Methods for determining vehicle contamination

Abstract

The invention relates to a method for determining the level of contamination (V) of a vehicle (1). According to the invention, it is provided that - image data of at least certain areas of the vehicle (1) are captured and stored by means of a number of vehicle capture units (E1 to E5), - based on signals received from other detection units, the vehicle's environmental conditions (1) are detected, - the image data are evaluated using computer vision methods and - if contamination (V) is detected in the captured image data of at least one of the specified areas of the vehicle (1), the detected contamination (V) is classified and a note about the contamination (V) in the vehicle (1) is issued.

Description

[0001] The invention relates to a method for determining the contamination of a vehicle.

[0002] From DE 10 2018 002 608 A1, a method for operating a motor vehicle is known. In this method, data from at least one external recording device is received by means of an electronic computing device of the motor vehicle, wherein the data characterizes the motor vehicle externally and the electronic computing device of the motor vehicle determines an external appearance of the motor vehicle depending on the data.

[0003] The invention is based on the objective of providing a method for determining the contamination of a vehicle.

[0004] The object is achieved according to the invention by a method which has the features specified in claim 1.

[0005] Advantageous embodiments of the invention are the subject of the subclaims.

[0006] According to the invention, a method for determining the contamination of a vehicle provides that - image data of at least certain areas of the vehicle are captured and stored by means of a number of the vehicle's detection units, - based on signals from other detection units, environmental conditions of the vehicle are detected, optionally creating a context for the image data using the detected signals, - the image data are evaluated using computer vision methods, and - if contamination is detected in the captured image data, at least in one of the specified areas

[0007] Areas of the vehicle are detected, the detected contamination is classified, and a notification regarding the contamination in the vehicle is issued.

[0008] By applying this procedure, it is possible to increase road safety, as the vehicle's occupant is informed about the contamination. In particular, the procedure can reduce the danger to other road users, especially by recognizing snow and / or ice deposits on a vehicle roof, which may be difficult for the occupant to see, as the vehicle occupant is alerted by the displayed warning. For example, this warning will continue to be displayed until the contamination, especially the snow and / or ice, is removed from the vehicle, particularly the roof. For example, the applied procedure can reduce wear and tear on vehicle parts, especially those caused by failure to remove the contamination.

[0009] The method increases user comfort in relation to vehicle operation, as the notification of soiling is automatically generated and displayed in the vehicle. For example, the method can also detect soiling on a vehicle license plate and alert the vehicle user.

[0010] In one implementation, the captured image data of at least the specified areas are compared with captured image data of at least the specified areas without soiling and with captured image data of at least the specified areas with various levels of soiling to determine whether soiling is present in at least one of the vehicle's areas. Specifically, the image data depicts the soiling states of the specified areas, with the specified areas being covered with snow, ice, and / or mud, in order to detect and classify soiling in the specified areas of the vehicle based on the currently captured image data of the specified areas. This makes it possible to generate a corresponding notification and display it in the vehicle as information for the vehicle user.

[0011] If the vehicle user complies with the warning issued by the vehicle relatively quickly, the risk of having to pay a fine can potentially be avoided.

[0012] In a further implementation, at least one machine learning model is used to classify the soiling based on the captured image data. This makes it possible to detect and classify any type of soiling in a specific area of ​​the vehicle using this model. Such a model can be pre-trained during vehicle development using images of exemplary soiling, and then further trained or refined using captured image data of soiling from the vehicle fleet. In particular, this retraining with image data of soiling from vehicles in a fleet, especially from a single vehicle manufacturer, takes place in a central computing unit, which can also be referred to as a cloud.

[0013] Furthermore, one implementation of the method involves evaluating captured image data from a specific area of ​​the vehicle using a model, and evaluating captured image data from at least one other specific area of ​​the vehicle using a further model. The respective image data is evaluated by the respective model at the start of a journey and / or at predetermined time intervals. This makes it possible to determine and classify the soiling in the respective area that occurs during vehicle operation. Specifically, the model is a machine learning model and / or an object detection model.

[0014] In a training course, the level of contamination in certain areas of the vehicle, determined using models, is used to infer contamination in another area, particularly an area outside the detection range of the number of detection units. Thus, based on the contamination detected in specific areas of the vehicle, it is concluded that the other area is also contaminated. Therefore, no additional detection unit is required to determine contamination in this other area.

[0015] In one possible implementation, the vehicle's current position and external appearance are described by characteristics that are sent to a central computing unit that is digitally linked to the vehicle. For example, this makes it possible to use the central computing unit to determine the vehicle's level of soiling. In other words, the central computing unit is a component for determining the vehicle's soiling.

[0016] One possible advanced development feature envisions using the central computer unit to identify other vehicles in the vehicle's fleet that are located in the vicinity of the vehicle. These other vehicles can also be used to determine the level of soiling on the vehicle itself or on other vehicles in the fleet, for example, if the vehicle's required number of detection units is missing or one of the detection units is defective.

[0017] In one implementation, the central computer unit determines the direction of travel and the trajectory of the vehicle and other vehicles in the fleet that are in its vicinity. This allows the system to determine if and when at least one of the other vehicles in the fleet is in the vicinity of the vehicle, in order to detect any contamination, at least of which may have occurred on the vehicle itself.

[0018] In one possible implementation, external sensors from at least one of the other vehicles detect signals related to the vehicle when it is within the detection range of the external sensors of that other vehicle. For example, the detection of signals by the external sensors of at least one of the other vehicles is activated when the central computer unit determines, based on the vehicle's characteristics, current position, and trajectory, that the vehicle is in the immediate vicinity of at least one of the other vehicles.

[0019] In particular, data anonymization methods are used, which, for example, remove personal information and images of people, license plates, etc. from the data. Location data is also altered, for example using a so-called hashing process, in such a way that a vehicle's position cannot be readily deduced from the data in the central computer unit.

[0020] If, in a further implementation, the system detects soiling of the vehicle based on signals from at least one other vehicle, the detected signals, particularly image data, along with the current position, orientation of the at least one vehicle, and the time of signal acquisition, are transmitted to the central computer unit. The central computer unit then sends a notification regarding the vehicle's soiling to the vehicle. For example, the vehicle user might be advised to clear snow and / or ice from the roof of their vehicle and / or to clean a dirty license plate.

[0021] Embodiments of the invention are explained in more detail below with reference to drawings.

[0022] This shows: Fig. 1. Schematic perspective view of a vehicle with a number of detection units and Fig. 2 schematically the vehicle and another vehicle.

[0023] Corresponding parts are provided with the same reference numerals in all figures.

[0024] Fig. Figure 1 shows a vehicle 1 with environmental sensors, for example for a parking assistant, wherein the environmental sensors have a number of detection units E1 to E5 arranged in and / or on the vehicle 1.

[0025] A first detection unit E1 is configured as a front camera, the detection area E of which is directed towards the front of the vehicle 1 with an opening angle of, for example, 70°. A second detection unit E2 is configured as a radar-based sensor, which is arranged in the area of ​​a radiator grille 1.1 of the vehicle 1, and a detection area of ​​the second detection unit E2, not shown in detail, is directed essentially in front of the vehicle 1. A third detection unit E3 is arranged in a side mirror 1.2 on the driver's side of the vehicle 1 and is configured, for example, as a 360° camera, whereas a fourth detection unit E4 is arranged in a lateral area of ​​a rear bumper unit 1.3. The fourth detection unit E4 is configured as an ultrasound-based sensor.A fifth detection unit E5 is designed as a rear camera, in whose detection range E there is a rear area of ​​the vehicle 1 and an area behind the vehicle 1, i.e. behind the rear area.

[0026] For example, the detection units E1 to E5 are arranged on and / or in the vehicle 1 in such a way that the detection areas E overlap section by section in order to obtain the most complete possible view of an exterior area of ​​the vehicle 1.

[0027] Vehicle 1 can be a motor vehicle, for example a passenger car or truck, a bus or minibus, but also an aircraft, a watercraft, an emergency vehicle, a construction machine, an agricultural or a military vehicle, for example an armored personnel carrier.

[0028] It is generally known that snow and / or ice on a vehicle 1 can pose a danger to other road users, or can pose a danger if the snow and / or ice is thrown towards other road users during the vehicle 1's operation, if these deposits have not been completely removed from the vehicle roof 1.4 before the journey begins. In particular, relatively tall SUVs and minibuses make it difficult to manually clean the vehicle roof 1.4 before a journey and may prevent or obstruct a vehicle user's view of the vehicle roof 1.4, especially if height is inversely correlated with vehicle height. Furthermore, contamination V of the vehicle 1, for example by deposits of de-icing salt, mud, and / or the like, can lead to increased wear and tear, especially on moving parts of the vehicle 1, such as ball bearings, etc.

[0029] According to the Fig. In the embodiment shown in Figure 1, a hood 1.5 and a trunk lid 1.6 have soiling V, for example in the form of snow and / or ice.

[0030] The following describes a method for automatically detecting and classifying contamination V of a vehicle 1.

[0031] Image data of the vehicle 1, particularly with regard to the hood 1.5 and the trunk lid 1.6 of the vehicle 1, are recorded by means of the recording units E1 to E5, in particular by means of the first recording unit E1 designed as a front camera and the fifth recording unit E5 designed as a rear camera.

[0032] Based on signals from other sensors of the vehicle 1 (not shown in detail), information such as the direction of light incidence, illuminance, weather conditions, especially rain, fog, etc., and in general information for contextualizing the captured image data are determined.

[0033] Alternatively or additionally, captured image data and / or signals from the detection units E1 to E5, as well as from other sensors of vehicle 1, such as a temperature sensor, an external humidity sensor, an external acoustic sensor, pressure sensors, and a light incidence sensor, can be used directly to detect water, snow, ice, and other objects on a vehicle surface or to assume the likely presence of water, snow, and / or ice on the vehicle surface. For this purpose, the current position of vehicle 1 and associated weather information can be taken into account.

[0034] The image data captured by the detection units E1 and E5 are evaluated using computer vision methods. Object detection is used to identify contamination V in specific areas of vehicle 1, classifying the detected contamination V and localizing its location. Alternatively or additionally, anomaly detection can identify deviations in the appearance of vehicle 1 from a predefined appearance and identify contamination V in specific areas of vehicle 1 as the likely cause.

[0035] Standard object detection and classification algorithms, particularly trained deep convoluted neural networks, can be used for the detection and classification of contaminants V. Pre-trained networks can be further trained with labeled, exemplary contaminants V, such as snow on the vehicle roof 1.4, mud splashes on a fender 1.7, etc., to optimize the detection and classification of contaminants V on the vehicle 1.

[0036] According to the Fig. In the embodiment shown in Figure 1, the first detection unit E1 detects the front area of ​​the vehicle 1, in particular the hood 1.5. The fifth detection unit E5 detects the rear area of ​​the vehicle 1, in particular the trunk lid 1.6.

[0037] A relatively large dataset of image data from the two acquisition units E1 and E5, covering a relatively large number of vehicles 1, is stored in a central computer unit 3. This dataset includes images of the hood 1.5 and trunk lid 1.6 both with and without soiling V. The image data includes, for example, snow, ice, dust, road salt residue, mud, etc. A pre-trained model was developed using this image data to evaluate and classify the image data acquired by the first acquisition unit E1 with respect to the hood 1.5, and another pre-trained model was developed to evaluate and classify the image data acquired by the fifth acquisition unit E5 with respect to the trunk lid 1.The models are retrained in the central computing unit 3 and distributed to the vehicles of a fleet, particularly those of a vehicle manufacturer. Thus, in vehicle 1, one model is used for the first data acquisition unit E1 and one model for the fifth data acquisition unit E5.

[0038] At the start of a journey of vehicle 1 and / or at regular intervals, the image data captured by the two detection units E1 and E5 are evaluated using the called models. If soiling V is detected on the hood 1.5 and / or the trunk lid 1.6, this soiling V, i.e., the captured image data containing the soiling V, is stored in vehicle 1 and / or transmitted to the central computer unit 3. If it is determined that the hood 1.5 and the trunk lid 1.6 are soiled, it is assumed that this soiling V is also present on the vehicle roof 1.4. This means that the condition of the hood 1.5 and the trunk lid 1.6 can be interpolated to the condition of the vehicle roof 1.4.

[0039] If snow and / or ice is detected as contamination V on the hood 1.5 and the trunk lid 1.6, a message is displayed in the vehicle 1 to the vehicle user, reminding them to clear the vehicle roof 1.4 of snow and / or ice. This procedure thus contributes to road safety.

[0040] For the aforementioned anomaly detection regarding vehicle 1, a basic model of the vehicle's external appearance can be trained using machine learning methods. For this purpose, image data of vehicle 1 is compressed into a low-dimensional space using an autoencoder, such as a convolutional autoencoder with neural networks, and then decompressed back to an initial size. Characteristic feature values ​​of vehicle 1 can thus be learned automatically. In particular, context can be taken into account during training; that is, the characteristic feature values ​​of vehicle 1 can, for example, depend on the lighting conditions.If current image data cannot be adequately reconstructed using the autoencoder, the current feature values ​​deviate from the characteristic feature values, so that contamination V and / or damage may be present as an anomaly of vehicle 1.

[0041] In another version, the anomaly detection model can be trained or at least pre-trained on fleet data from vehicles of the same type, make, color, special equipment, etc.

[0042] In an exemplary embodiment not shown in detail, a current position of the vehicle 1 as well as features describing its external appearance, in particular its color, shape, superstructure, etc., are linked to a data-technically connected system of the vehicle 1. Fig. 2. Central computing unit 3, shown as an example and greatly simplified representation, transmits data.

[0043] Using the central computing unit 3, other vehicles 2 belonging to vehicle 1's fleet that are located in the vicinity of vehicle 1 are identified. Furthermore, the direction of travel (i.e., orientation) and the trajectory of vehicle 1 and the other vehicles 2 in the vicinity of vehicle 1 are determined.

[0044] Using external sensors on at least one of the other vehicles 2 in the fleet, signals relating to vehicle 1 are detected when the latter is within the detection range of at least one sensor of the external sensors.

[0045] Vehicle 1 is identified within the fleet data of other vehicles 2 by evaluating captured signals, i.e., captured sensor information, for example, using computer vision methods, to determine whether vehicle 1 can be detected in the captured signals. For instance, camera images can be used to determine the type of captured object, such as road surface, road users, etc., with pixel-level precision, and to extract object properties, such as color, size, and direction of movement, from captured individual images or image sequences. Furthermore, the trajectory of vehicle 1 can be compared with the trajectories of other vehicles 2 contained in the sensor information.

[0046] If, based on the detected signals of at least one of the other vehicles 2, a contamination V of vehicle 1 is detected, the detected signals, particularly in the form of image data, are transmitted to the central computer unit 3 along with a current position, orientation, and time stamp. Subsequently, the central computer unit 3 transmits a notification to vehicle 1 regarding the detected contamination V.

[0047] In one embodiment of the method, a contamination detection model is specifically trained with training data, in which recordings of vehicle 1 are taken from an elevated position, for example from a bridge or an example in Fig. 2 higher vehicles shown, for example a bus, a truck, etc., can be included.

[0048] In a Fig.In the embodiment shown in Figure 2, a vehicle 1 is equipped with a front camera as the first detection unit E1, whose detection range E includes the front of the vehicle 1 and an area in front of it. A model for image classification is used in the vehicle 1 for the image data captured by the first detection unit E1. This model is called up when the vehicle 1 starts moving and / or at regular intervals. Whenever contamination V is detected on another vehicle 2, the type of detected contamination V, as well as, if applicable, the image data from the first detection unit E1, including the current position of the vehicle 1 and / or the other vehicle 2, its orientation, a time stamp, etc., are transmitted to the central computer unit 3.Each vehicle 1, 2 connected to the central computer unit 3 can now receive information from the central computer unit 3 about the types of soiling V detected by other vehicles 1, 2. If snow is detected and classified as a type of soiling V, the vehicle user can receive a notification. For example, the notification might read: "Don't forget to clear the snow off your vehicle's roof." Reference symbol list 1 vehicle 1.1 Radiator grille 1.2 Side mirrors 1.3 Bumper unit 1.4 Vehicle roof 1.5 Hood 1.6 Rear Lid 1.7 Fender 2 more vehicles 2.1 Vehicle roof 3 central computing unit E detection area E1 to E5 recording unit V Pollution QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2018 002 608 A1

[0002]

Claims

[1] Method for determining contamination (V) of a vehicle (1), characterized by , that - image data of at least certain areas of the vehicle (1) are recorded and stored by means of a number of recording units (E1 to E5) of the vehicle (1), - environmental conditions of the vehicle (1) are detected based on signals from other detection units, - the image data are evaluated using computer-assisted vision methods and - if contamination (V) is detected in the captured image data of at least one of the specific areas of the vehicle (1), the detected contamination (V) is classified and an indication of the contamination (V) in the vehicle (1) is output. [2] Method according to claim 1, characterized bythat the recorded image data of at least the specific areas are compared with recorded image data of at least the specific areas without contamination (V) and with recorded image data of at least the specific areas with different contamination states. [3] Method according to claim 1 or 2, characterized by that at least one pre-trained model for classifying the contamination (V) is retrained using the acquired image data. [4] Method according to claim 3, characterized by that recorded image data of a specific area of ​​the vehicle (1) are evaluated by means of a model and recorded image data of a further specific area of ​​the vehicle (1) are evaluated by means of a further model, wherein the respectively recorded image data are evaluated by means of the respective model at the start of a journey of the vehicle (1) and / or at predetermined time intervals. [5] Method according to claim 3 or 4, characterized bythat a contamination (V) of another area of ​​the vehicle (1) is inferred from a contamination (V) of the areas determined using the models. [6] Method according to one of the preceding claims, characterized by that a current position of the vehicle (1) and features describing an external appearance are sent to a central computer unit (3) which is data-technically linked to the vehicle (1). [7] Method according to claim 6, characterized by that by means of the central computer unit (3) further vehicles (2) of a vehicle fleet of the vehicle (1) are determined which are located in an environment of the vehicle (1). [8] Method according to claim 6 or 7, characterized by that a direction of travel and a trajectory of the vehicle (1) and the other vehicles (2) of the vehicle fleet that are in the vicinity of the vehicle (1) are determined by means of the central computer unit (3). [9] Method according to one of claims 6 to 8, characterized by that signals relating to the vehicle (1) are detected by means of an external sensor system of at least one of the further vehicles (2) when the vehicle is located in a detection range of the external sensor system of the at least one further vehicle (2). [10] Method according to one of claims 6 to 9, characterized by that if a contamination (V) of the vehicle (1) is determined on the basis of the detected signals of the at least one further vehicle (2), the detected signals are transmitted to the central computer unit (3) with a current position, an orientation and a time indication, and an indication relating to the contamination (V) of the vehicle (1) is transmitted to the vehicle (1) by means of the central computer unit (3).

Images