Method and system for monitoring road condition through machine learning system and method for training machine learning system

Abstract

The invention relates to a method and a system (10) for monitoring road conditions by means of a machine learning system (16) and a method for training a machine learning system (16). A method for training a machine learning system (16) for monitoring road conditions comprises the following steps: providing or capturing data by means of a sensor system (1) of a vehicle (2), the sensor system (1) detecting the surroundings of the vehicle (2) and using said data as training input data X, -providing or capturing data characterizing the road condition by means of a reference sensor (5) arranged in or on the vehicle (2) and using it as a training target value Y, and-training a machine learning system (16), in which training data comprising training input data X and training target values Y corresponding to the training input data X are provided, parameters of the machine learning system are adjusted by means of the training data such that the machine learning system generates output data Y'similar to the training target value Y when the training input data is input.

Description

Technical Field

[0001] The present invention relates to road condition monitoring (Road Condition Monitoring, Road Condition Observation) or the identification of lane conditions from a moving vehicle equipped with sensors for detecting the surrounding environment. This subject is extremely important for preventing accidents, because certain lane conditions or road surfaces can reduce the static friction or friction coefficient between the tires and the lane surface, thereby increasing the braking distance or braking can cause the vehicle to slip and slide to the side, thus causing serious accidents. Therefore, when automatic steering and / or braking interventions are carried out in assisted or autonomous driving (Advanced Driver Assistance Systems (ADAS) / Automated Driving (AD) Systems), the system must also assess the road conditions as realistically as possible before the vehicle passes. Since this cannot yet be achieved sufficiently reliably, driving tests of autonomous vehicles are usually carried out in sunny areas where ice and snow conditions are unlikely to occur.

[0002] In order to improve the recognition of road conditions, machine learning is applied in an innovative setting. The invention relates in particular to a method for training a machine learning system for road condition monitoring, a method for monitoring road conditions with the aid of a trained machine learning system, a road condition monitoring system and a vehicle with a road condition monitoring system. Background Art

[0003] It is known that YouTube videos sometimes show the segmentation of the same road based on a camera video data and the driving noise detected by a microphone at the same time. Dry and wet road conditions can be distinguished here: Road condition estimator using cameras and microphones (2018 edition) https: / / www.youtube.com / watch?v=H13jgv55o8w (Invoked on May 25, 2022). It is difficult or impossible to reliably distinguish more than two categories (wet / dry) using this method.

[0004] WO 2016 / 177372 A1 presents a method for identifying and evaluating environmental influences and lane condition information of a vehicle's surroundings. At least two digital images are generated successively by means of a camera device, and an identical image portion is selected in each image. Changes in image sharpness between image portions of at least two consecutive images are identified by means of a digital image processing algorithm, wherein the changes in image sharpness are weighted in a decreasing manner from the center of the image portion to the outside. Based on the identified changes in image sharpness between image portions of at least two consecutive images, surrounding state information is determined by means of a machine learning method, and lane condition information is determined based on the determined surrounding state information.

[0005] WO 2019 / 174682 A1 presents a method for classifying lane conditions based on image data of a vehicle camera system and a corresponding vehicle camera system. The method comprises the following steps:

[0006] - providing image data by a vehicle camera system, which is configured to image at least a portion of the surroundings outside the vehicle, wherein the portion at least partially includes a lane on which the vehicle is traveling,

[0007] - distinguishing diffuse and specular reflections of a roadway by evaluating the difference in appearance of at least one point of the roadway in at least two images from a camera system, wherein the images are taken from different viewing angles,

[0008] - determining whether there is any disturbance in at least one image of the camera system caused by dust being raised by at least one wheel of a passing vehicle,

[0009] - Taking into account the results regarding reflection type and interference intensity, the lane condition is classified into one of the following five lane condition categories:

[0010] a) Dry lane: The reflection type is diffuse reflection, no interference

[0011] b) Ordinary wet lane: reflection type is specular reflection with interference

[0012] c) Very wet lanes, with the risk of slipping: The reflection type is mirror reflection, which is very disturbing

[0013] d) Snowy lane: Reflection type is diffuse, with interference, or

[0014] e) Icy lane (thin ice layer): The reflection type is specular reflection, with no interference.

[0015] Known problems with camera systems are, for example, low signal-to-noise ratio in darkness. If the signal-to-noise ratio is increased by extending the exposure time, motion blur will occur when the camera system moves.

[0016] DE 102013002333 A1 shows a method for predictive / proactive determination of road conditions in a vehicle, in which a road surface is illuminated with a sensor beam, wherein the sensor beam is reflected and absorbed depending on the road condition of the road surface, wherein the road condition is determined based on the reflected sensor beam. The method is characterized in that the road surface in front of the vehicle in the direction of travel is illuminated. Since road surfaces of different road conditions have different optical properties and accordingly absorb certain wavelengths and reflect other wavelengths, the corresponding road condition of the illuminated road surface can be inferred from the reflected sensor beam. An example of this is the case of a wavelength of approximately 1550 nanometers, which is relatively strongly absorbed by ice.

[0017] DE 10 2014 214 243 A1 discloses a method for determining a road condition, wherein road condition data of a weather map and / or a road map are used to determine the road condition, wherein the road condition data of the weather map and / or the road map are re-digitized.

[0018] DE 102017223510 A1 discloses an optical sensor for surface evaluation, which comprises:

[0019] - at least one transmitting unit for transmitting a modulated light signal onto the surface to be evaluated;

[0020] - at least one detection unit for detecting an optical signal reflected by the modulated signal on a surface and for providing an electrical output signal dependent on the reflected optical signal; and

[0021] A control device for evaluating an output signal of at least one detection unit and determining the state of the surface to be evaluated, wherein the frequency spectrum of the modulated light signal transmittable by the at least one transmitting unit is substantially separated from the frequency spectrum of an interference spectrum.

[0022] All known methods have drawbacks and do not fully address the associated challenges. Of particular interest is the predictive and reliable identification of "black ice," i.e., frozen (icy) black road surfaces that are themselves difficult to see. Summary of the invention

[0023] It is therefore an object of the present invention to provide a method for predictive recognition of lane situations in an economical manner without compromising the robustness of the recognition of safety-relevant types of lane situations.

[0024] This object is achieved by the subject matter of the independent claims. Further embodiments of the invention are described in the dependent claims and in the following description.

[0025] The invention is based on the idea of ​​using surroundings sensors that are already installed on a vehicle for other tasks, in particular for detecting an area that is located in front of the vehicle in the direction of travel.

[0026] The shortcomings of existing surroundings detection sensors can be compensated by specialized and therefore mostly too expensive reference sensors for mass production, by including them in the generation of training data for a machine learning system for determining road conditions. To generate training data, surroundings detection sensors are often used in fleets of test vehicles for later series use. The detected data are then usually annotated manually in order to provide realistic road conditions for the training of the machine learning system. Manual annotation is done by humans, whose perception of road conditions is subjective. For example, when did the road become "wet"? Is there an old layer of ice underneath a visible layer of new snow? Is there a thin layer of ice on black road surface or is it dry black road surface? Moreover, manual annotation is time-consuming, costly, and expensive.

[0027] A second starting point of the invention is to use a reliable, highly accurate and well-defined road condition determination sensor as a reference sensor. Advantageously, the ("pure") road condition determination sensor can predictively determine whether there is water on the road surface and what the temperature of the road surface is. Transmitting and receiving devices operating at 1550 nanometers and in the wavelength range of approximately 2 micrometers to 10 micrometers can determine both of these.

[0028] One aspect of the present invention relates to a method for training a machine learning system for monitoring road conditions, comprising the following steps:

[0029] - data are provided or captured / recorded by a vehicle sensor system, wherein the sensor system detects the vehicle surroundings and uses the data as training input data,

[0030] - providing or acquiring data representing the road condition through a reference sensor disposed in or on the vehicle, and using the data representing the road condition as a training target value, and

[0031] - training a machine learning system, wherein training data comprising training input data and training target values ​​corresponding to the training input data are provided, and parameters of the machine learning system are adjusted / adapted with the aid of the training data such that the machine learning system generates output data similar / close / equivalent to the training target values ​​when the training input data is input.

[0032] The data of the sensor system (sensor data) may include, for example, image data, radar data and / or lidar data.

[0033] The machine learning system is trained with the aid of training data, i.e., the parameters of the machine learning system are adjusted in such a way that the output data generated by the machine learning system when the training input data are input are similar to the training target values. Taking a neural network as an example, the parameters include the weights between the individual input values ​​and neurons. Here, the machine learning system is trained with the aid of a supervised learning method, of which there are a large number of known learning methods. For example, a back propagation algorithm can be used to train a neural network. Here, the parameters of the machine learning system are adjusted during the training process in such a way that the error between the output data and the training target value is as small as possible. Here, the error between the output data and the training target value can be determined, for example, by the spacing between the output data and the training target value and the corresponding measure of the output data. It should be noted here that overfitting should be avoided, which can be achieved, for example, by analyzing the error between the output data generated by the test input data and the relevant test target value. Here, the test target value corresponds to the test input data, and the test input data and the test target value are not used to adjust the parameters of the machine learning system.

[0034] As an option, the parameters of the machine learning system may be output upon successful completion of the training, for example because the output data are sufficiently similar to the training target values, which may be predetermined, for example, by a threshold value of a similarity measure.

[0035] In one embodiment, the sensor system comprises a camera system of the vehicle, whereby the data provided or captured are image data, which are used as training input data. The reference sensor determines reference data as training target values ​​(labels) for the image data.

[0036] For example, the camera system may be a monocular camera device mounted on the vehicle, preferably behind the windshield, so that the area in front of the vehicle can be detected based on the visual perception of the vehicle driver. Alternatively, the camera system may be a stereo camera device that provides depth information about the vehicle's surroundings, or a satellite camera system, such as a panoramic surround camera system that includes multiple fisheye cameras that can observe different directions of the vehicle.

[0037] By subsequent processing of the image data and reference sensor data by a machine learning system (“artificial intelligence (AI) camera device”), the road condition monitoring function of the reference sensor can be transferred to the camera system using appropriate learning methods.

[0038] According to one embodiment, the reference sensor comprises a transmitting and receiving unit, which transmits electromagnetic radiation of at least one specified wavelength to the road and receives and measures the intensity reflected by the road, wherein the reference sensor is configured to indicate the probability of the presence of different road condition categories based on the measured values. By measuring the absorption rate at discrete wavelengths in the mid-infrared, it can be determined whether there is water on the road. By comparing the measurement results of two suitable wavelengths (ratio measurement method), it is possible to reliably identify whether there is water on the road.

[0039] A suitable reference sensor for representing the ground-truth data can identify all object learning categories of the camera system regardless of the light distribution (brightness) depending on the time of day.

[0040] This is a prerequisite to ensure that the learning results of the camera system (as a sensor system) can be optimally adapted to the light distribution (brightness) that varies according to the time of day through the exposure control of the camera device.

[0041] This ensures that the camera system can detect road conditions not only during the day, but also to a large extent at dusk and at night, provided that the camera device has sufficient sensitivity.

[0042] Advantageously, the observation wavelength ranges between the reference sensor and the camera system do not coincide. The camera observes in the visible light range, while the reference sensor observes, for example, in the infrared range.

[0043] In one embodiment, it can also be provided that the “motion blur” caused by different driving speeds and set exposure times is also learned in the images. Here, these two parameters can be included in the learning result either categorically or using a linear approximation method.

[0044] Advantageously, the speed over the ground can serve as a useful additional input variable.

[0045] According to one embodiment, it can be provided that weather data and driving dynamics data such as control interventions of the anti-lock braking system (ABS), traction control system (ASR) and / or electronic stability control system (ESC) of the vehicle can be added to the learning result in a beneficial manner. In this case, it can be mentioned that the label variables are early integrated.

[0046] In addition or as an alternative, according to one embodiment, for manually driven vehicles, in addition to the above-mentioned driving speed or, as an alternative, the safe distance to the vehicle driving in front, a driving style determined based on the longitudinal and lateral acceleration of the vehicle can also be added.

[0047] In one embodiment, the reference sensor assumes (at least) the probability of the following road condition categories:

[0048] “Dry”, “Wet”, “Snow”, “Ice” and “Unknown / Error”. For a camera sensor as a sensor system, an error means, for example, a predominantly black or white image or predominantly image noise due to underexposure.

[0049] According to one embodiment, the reference sensor comprises a pyrometer / thermometer for measuring the road temperature. The reference sensor may measure, for example, thermal (far) infrared radiation, such as in the range of 2 microns to 10 microns, in order to deduce the surface temperature from the thermal radiation. On the other hand, the thermal signature of the far infrared (IR), for example, is not visible in the visible spectrum, which is why it is difficult for people to identify black ice.

[0050] In one embodiment, the reference sensor itself uses:

[0051] wavelengths of 1550 nm and 980 nm in order to detect the presence of water by comparing the reflected intensity, and

[0052] Wavelengths in the 2- to 10-micron range are used to measure road temperature.

[0053] In one embodiment, the machine learning system is a neural network. Here, neural networks are particularly suitable for the method because they are easy to adjust.

[0054] The neural network is in particular a convolutional neural network. The nonlinearity of the convolutional neural network does not pose a problem for the usability of the method.

[0055] However, the method can also be used by other machine learning systems, such as decision tree learning, support vector machines, regression analysis or Bayesian networks. In addition, the method can also be used for multi-task classification systems. In this case, the multi-task classification system includes, for example, an encoder and multiple decoders. In addition, the machine learning system can also be divided into multiple subsystems. In this case, each subsystem has the functionality of the machine learning system described herein, but the subsystems differ from each other, for example in the type of machine learning system or the selection of training data. The output data generated by means of the subsystems can then be combined to obtain a better output.

[0056] According to one embodiment, the vehicle comprises a data transmission unit, which is configured to transmit training data to a server unit (backbone network). This enables the construction of a data driven ecosystem (DDE) for developing machine learning systems, for example to determine road conditions based on sensor data from conventional surrounding sensors.

[0057] In one embodiment, the server unit is configured to update a training data set including a (predetermined) amount of training data, wherein the size of the training data set remains unchanged, and wherein, based on one or more quality criteria (for example, the quality can be evaluated based on multiple key performance indicators (KPIs)), it is ensured that the relevance of the training data to the road condition determination is improved when the training data set is updated.

[0058] In order to achieve high reliability even in unknown scenarios, according to one embodiment, training is performed in a data-driven ecosystem (DDE) and data relevance is continuously improved. At the same time, at the back end of the data-driven ecosystem (DDE), automatic methods are used based on key performance indicators (DAgger) and manual judgment (active learning) is used with a significantly reduced workload.

[0059] The Data Driven Ecosystem (DDE) is designed so that once a certain limit is reached, the size of the dataset remains constant. As a result, the training time of the machine learning system does not increase. The backend computing power is used to improve the relevance of the training dataset.

[0060] According to one embodiment, the server unit is configured to achieve a balanced training data set during the updating process, so that the training data set has a high diversity, wherein relatively rare road conditions are fully represented by the training data. For example, rare conditions such as black ice, i.e. frozen black road surface, are particularly rare in probability, but also particularly dangerous, because the friction coefficient will drop significantly.

[0061] In one embodiment, the method for evaluating the relevance of training data is to assign high relevance to training data that is orthogonal to other training data already included in the training data set.

[0062] According to one embodiment, the sensor data of the detected area of ​​the road is divided into several sections, and the data provided or captured by the reference sensor represents the road conditions of the corresponding section.

[0063] The high spatial resolution preferably enables segmentation / semantic segmentation of the sensor data on the road ahead with the help of condensed recognition rules in the network, thus enabling predictive and spatially resolved measurements. This saves reaction time and reduces the application costs by up to two orders of magnitude compared to reference measurement technologies.

[0064] Another aspect of the present invention relates to a method for monitoring road conditions using a machine learning system, wherein the machine learning system has been trained as described above, wherein data detected by a vehicle sensor system that detects the vehicle's surrounding environment is provided to the machine learning system as input data, and the machine learning system generates output data representing the road conditions from the input data.

[0065] A third aspect of the present invention relates to a road condition monitoring system, which comprises: an input unit for receiving input data; a computer unit configured to execute the above-mentioned road condition monitoring method; and an output unit for outputting output data generated by the computer unit.

[0066] Another aspect of the present invention relates to a vehicle including a sensor system and the above-mentioned road condition monitoring system, wherein the sensor system is configured to detect the vehicle surroundings and provide the detected sensor data as input data to an input unit.

[0067] The following problems arise when building a training dataset (e.g. in a server unit):

[0068] a) Generate a large amount of similar data

[0069] b) There are rare categories

[0070] c) The categories are strongly correlated with the test location, e.g. more icy conditions are detected in Sweden, more wet conditions are detected in Germany, and more dry conditions are detected in southern Spain. Due to the correlation with the location, it is possible to mistakenly learn the location instead of the road conditions.

[0071] d) However, rare categories are also important. Rare categories are often neglected in practice, but these rare categories are particularly dangerous cases.

[0072] e) The wavelength ranges observed between the reference sensor and the camera are inconsistent, for example, the reference sensor is used to detect the absorption of water and ice at discrete wavelengths in the mid-infrared, and is used to derive the surface temperature from thermal radiation at thermal (far) infrared wavelengths. The camera observes light in the visible wavelength range. Thermal features in the far infrared (IR), for example, are not visible in the visible spectrum, which is why people have difficulty recognizing black ice. Instead, they infer the black ice road condition by observing the surrounding environment and other optical features. That is, in principle, the road condition can be distinguished into four or five categories using camera optical technology, but this is difficult and must be learned from a large number of examples, so pure camera optical road condition recognition has not necessarily been sufficiently robust so far.

[0073] Data-driven ecosystems provide the following solutions to these problems:

[0074] 1) Policy aggregation: If you learn a random decision forest, you don’t need a single decision tree, but a “bag full of decision trees”, i.e., the “bag of classifiers method”. This allows for majority decisions, and as learning progresses, variations of existing successful decision trees are randomly composed and added, while unsuccessful ones are discarded. Ultimately, the majority decision of many good classifiers will give excellent results. This approach works well if the computational effort of a single classifier is low. This is the case for decision trees with only a few “yes” and “no” criteria. This is an evolutionary approach that does not require the calculation of gradients or backpropagation algorithms.

[0075] 2) Back-propagation algorithm: Modern neural networks use the back-propagation algorithm to learn from the difference between the desired output and the random output after the initial initialization. Through the chain rule, the difference (gradient) is converted back to the corresponding layer of the network, and from this it is determined how the weights and biases in the network must be changed to obtain a better solution. In the back-propagation algorithm, the reuse of redundant data is disruptive because it introduces irrelevance. Specific, possibly very similar situations are over-learned and even meticulously memorized, but slightly different test data cannot be recognized as well as the training data. This is called overfitting.

[0076] 3) Data segmentation: Splitting the data into training and test data helps prevent overfitting in recognition, where the KPI (key performance indicator) to be measured, such as the precision or accuracy of the output corresponding to the test data, must be at least as good as the training data. It does not help if the training data is too close to the test data, for example because they are adjacent images in the same sequence. This situation must be avoided.

[0077] 4) Batch Normalization: Since the input data, output data, and data in intermediate layers are all unbiased and uniformly distributed, the neurons in the network function under the simplifying assumption that linear coupling to the next layer is sufficient. This can be achieved by renormalizing the layers layer by layer (Ioffe and Szegedy, Google)

[0078] 5) Dropout: During learning, clusters of neurons without functionality may form in the network. Our goal is to make each sub-region of the network contribute equally to the classification. A particular difficulty is the vanishing and exploding gradient problem. That is, when backtracking with the chain rule, the requirement for neurons to reduce their weights becomes exponentially smaller or larger layer by layer. To avoid this, approximately 50% of the neurons are randomly selected and disabled in each learning step. The signal strength of the remaining 50% of neurons is adjusted, and only these neurons are trained in the current step. In the next iterative training step, another 50% of the neurons are selected, forcing each subset of the network to carry the same function, so that the performance of the entire network depends only on its size, but almost no neurons remain unconnected because the relevant weights remain close to zero. 6) Regularization: This method is perhaps best compared to the measurement of crater depth. The deepest corresponds to the best solution, but the crater is not just a simple bowl, but also contains mountains and other craters. It is actually an entire crater field composed of overlapping craters. In simple methods of finding the deepest depth, for example by throwing a ball into the field, there is a risk that the ball may get stuck in a local minimum, i.e., not the best solution is determined, but only a local suboptimal solution. Regularization means that the crater landscape is filled with adhering powder. This allows smaller craters to be filled, at least after solidification, and thus reduces the chances of getting stuck in a local minimum. Regularization must be used with caution. Figuratively speaking, the crater landscape is not allowed to be filled and thus all differences are eliminated. However, using regularization in small doses, in the range of one thousandth, it is possible to eliminate special identification features and find more generally valid basic rules. However, this comes at the cost of a "clearly" reduced accuracy. As a result, simple key performance indicators (KPIs) will be worse, although in fact they are better classified.

[0079] 7) Activations: Activations are computed in parallel with the actual neural network via a visual back-propagation algorithm on a graphics processing unit (GPU). It indicates which neurons are involved in decision making for convolutional layers (not fully connected layers) and projects them back to the network input layer. That is, if the activations are superimposed with the corresponding input image, one can see which features are used for classification decisions. This helps understand the problem when an error is detected and provides valuable hints about “possible missing training examples”. In addition to directly observing the KPI, it also provides clear hints.

[0080] 8) Key Performance Indicator, or KPI for short: A simple binary classification detector can respond, not respond, report correctly, or report incorrectly in all combinations (i.e., the four cases here). If it detects, the recall is high. Imagine that the scene gets darker and darker until the image is black. If the output is an image, rather than just a class, then the pixels that depict the known features and are identified can be calculated. In addition, other pixels that do not belong to the known features will also be identified. These pixels are false positive pixels. Therefore, it is about two things: the identification and the correctness of the result. This can be plotted as a graph or summarized by the F2 score. In short, a key performance indicator (KPI) is a pre-defined metric that, averaged over a test dataset, indicates how well the network is trained.

[0081] 9) Data aggregation: It is very similar to policy aggregation. Here, one only needs to compute, for example, 500 neural networks and take a majority vote on their outputs, moving the problem from implementation time to training time and thus to the selection of relevant data. Thus, one only needs to compute one network, but start with a subset of the training data, for example 5% of it, and then look at the next 5% quota to see which images have been correctly classified by the first 5% of training. These are not needed directly. It is like preparing for an exam, where knowledge that has not yet been understood must be learned in the process. Data aggregation means that when the machine decides which data is important, it is those data that are orthogonal to all other data that have been learned.

[0082] 10) Active learning: Sometimes data is mislabeled. During the data aggregation process, the network will find that an image should not output both dry and wet conditions. This indicates that one of the images is wrong. But which image is wrong? In this case, manual judgment is required, but since the process is automated, only the controversial cases need to be judged.

[0083] 11) Dataset size does not change: The duration of learning does not increase continuously. The Data Driven Ecosystem (DDE) in turn independently decides whether the data is important or not. When the dataset eventually reaches its target size, data aggregation attempts to replace the old data classified as important with the new data. Here, the data quality is continuously measured based on multiple key performance indicators (KPIs). If the new solution in the form of a newly learned neural network is below average, it is discarded, and if it is above average, it is given priority.

[0084] 12) Data-driven ecosystem: In its backend, i.e. functions computed later in the cloud, the best networks computed so far are constantly challenged to see if they are better than before by replacing more important training data after training. This depends heavily on the test dataset, which must be tuned in exactly the same way as the training dataset. It should contain all the relevant cases in the surrounding context, as well as rare cases and edge / corner cases.

[0085] 13) Synthetic data (derived from real data): As the name suggests, rare situations are rare. To ensure that they are not underestimated, synthetic data can be derived from regular data with the help of generative adversarial networks (GAN) style transfer, i.e. synthetic data can answer the question of what a certain landscape looks like in another weather condition, or what a certain weather condition looks like in another location. Since a very large number of images of a location can be captured, but only a few rare situations can be captured, the conditions are transferred to another location. 14) Evaluation of speed-dependent time delay: The front camera can see the area of ​​the road ahead of the current measurement point. For safety reasons, a time shift must be performed so that the path of the measurement points on the road is consistent with the road image. Conversely, instead of classification, segmentation can also be calculated. If a network that can recognize the location of the road in the image is additionally used, the road can be learned in segments according to the road conditions and displayed accordingly.

[0086] 15) Averages are more informative than single values: In the context of reasoning, i.e. in network applications, outliers can also be improved and intercepted by averaging. For example, the weather conditions cannot be predicted to change moment by moment, but the road conditions are a phenomenon that can change over spatial boundaries, whether when exiting a garage, in a tunnel or crossing a bridge. Here, it can be expected that the changes are particularly significant. If this is recognized, then the results can be stabilized by averaging in all other cases.

[0087] 16) Test the network in synthetic scenarios and driving tests: To cover standard cases and known rare and edge cases, additional testing on normal training data is required to check the confidence of the behavior. It must survive the testing process.

[0088] 17) Securing the transmission: The simplest method is to update via a USB flash memory stick. The update function for the fleet after the product is delivered and during its service life is also useful. To avoid transmission errors and tampering throughout the data-driven ecosystem (DDE), checksums, cryptographic signatures and encryption are added to the actual data.

[0089] 18) Updates: For fleet applications, data is preferably transferred over the air (OTA). Here, it is assumed that future vehicles will be equipped with a SIM card and have a tunneled Internet connection to the update server. Of course, traditional methods such as USB flash memory sticks and file systems are also possible.

[0090] 19) Triggering interesting data: It is known from Tesla patents in particular that rare and extreme situations should be captured, in our case rare road weather conditions. In Tesla, the trigger itself is also a network trained to respond to specific scenarios. Here, the same technology will also be used for data-driven improvements of the product in a second continuous cycle. It is conceivable that only a few measurement vehicles equipped with reference sensors are equipped with trigger devices. However, the configuration of a trigger in each vehicle of the fleet is obviously more beneficial due to the larger number of application cases, because only in this way can the system be self-scaling.

[0091] 20) Feedback channels: Over-the-air (OTA) updates are preferred, but conventional updates via USB or other bus systems in the workshop are also possible. The benefit of an over-the-air (OTA) update channel for vehicles is that the vehicle manufacturer does not have to initiate a recall if a software update is required. Feedback channels also contribute to the continuous improvement of products. Here, the moments that are triggered and selected, for example, by a test driver or internal driver, as well as the measured values ​​assigned to them, are fed back. The amount of data from the various sensors of each manufacturer can reach several hundred megabytes (MB), which is a realistic order of magnitude. This means that there is much more data available than is subsequently required to train the corresponding sensor network. Therefore, good methods for selecting the training data and evaluating the network to be trained with the help of key performance indicators (KPIs) are particularly important.

[0092] 21) Virtualization of the ecosystem, connectivity and procurement: All of this takes place in the backend, preferably in the cloud. The entire backend can be virtualized, i.e. moved into a Docker container and run on a sufficiently efficient large cloud instance. The benefit is that post-processing can be scaled to the size of the fleet. Thus, there is a data connection between the fleet, suppliers and cloud providers. This in turn does not matter where the data is collated and new algorithms are integrated. There are no problems with cloud access worldwide. The same applies to the unfolding of the data.

[0093] 22) Anonymization and data protection processing without side effects: Based on the locally applicable data protection regulations, source anonymization should be considered. This is because if no personally specific data is stored, no laws will be violated in globally available cloud processing and data collation solutions. Here, it must be ensured that the anonymization does not have any unnecessary impact on the main functionality. However, in the case of road observation, anonymized faces and anonymized license plates should not cause problems unless the license plates are very large in the image and are, for example, re-annotated from a dark background to a light background. Extreme white tones in the image can be an indicator of snow accumulation, which would be incorrectly classified as likely snow. Again, these side effects must be tested and systematically avoided (e.g., making them unrecognizable using only the background color).

[0094] Adapting driving behavior to the road conditions: In the case of road condition measurement, all these efforts are aimed at making driving behavior more appropriate. The driver certainly does not want to be informed of road conditions at all times, because he / she is certainly an expert in this area. In the case of slippery roads, i.e. a significantly reduced coefficient of friction, a warning is given. The actual coefficient of friction can only be estimated, as it depends on the road and the tires. The main application of road condition measurement is highly automated driving, because the driving planning system must also be aware of road conditions. It may be necessary to drive slowly, understeer and extended braking distances may have to be taken into account. What we humans take for granted must be communicated to machines by technical means. Here, the fact that the technical means are cost-effective is an advantage.

[0095] Road condition monitoring is useful for warning the driver in slippery conditions (especially black ice), but more so in the area of ​​highly automated driving, where assumptions about braking distances must be constantly updated based on road weather, so that driving behavior can be adjusted to the weather conditions.

[0096] Average values ​​are more convincing than single values: In inference, i.e. in network applications, outliers can also be improved and intercepted by averaging. For example, the overall weather conditions cannot be expected to change from moment to moment; however, phenomena such as road conditions change at spatial boundaries, whether when leaving a garage or a tunnel or when crossing a bridge. Here, it can be expected that the changes are particularly pronounced. If this is recognized, the results can be stabilized by averaging in all other cases.

[0097] Evaluation of speed-dependent time delays: The front camera sees an area of ​​the road ahead of the current measuring point. For safety reasons, a time shift must be performed so that the path of the measuring points on the road matches the road image. Conversely, instead of classification, segmentation can also be calculated. If a network is additionally used that recognizes the road position in the image, the road can be learned in segments according to the road conditions and displayed accordingly. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] The embodiments of the present invention will be described in more detail below with reference to the accompanying drawings.

[0099] Figure 1 A vehicle is shown with a surroundings detection sensor system, a reference sensor and a processing unit;

[0100] Figure 2 A schematic diagram of a machine learning system is shown, wherein the system is trained based on data from surrounding environment detection sensors and reference sensors to generate output data representing road conditions;

[0101] Figure 3 A trained machine learning system is shown, wherein the system can generate output data representing road conditions based on data from surrounding environment detection sensors;

[0102] Figure 4 A road condition monitoring system is shown; and

[0103] Figure 5 A vehicle is shown which is equipped with a reference sensor having a transmitting and receiving unit. DETAILED DESCRIPTION

[0104] Figure 1 Shown is a schematic top view of a vehicle 2. The vehicle 2 has a surrounding environment detection sensor system 1, a reference sensor 5, and a processing unit 10. The surrounding environment detection sensor system 1 may be, for example, or may include, an image capture device. The image capture device may be a vehicle front camera. The front camera may be arranged inside the vehicle 2, for example, in the rearview mirror area, and detect the surrounding environment in front of the vehicle 2 through the windshield of the vehicle 2. Based on the signal or image data of the front camera, details such as objects of the surrounding environment of the vehicle 2 may be detected. Based on the surrounding environment detection, an advanced driving assistance system (ADAS) control unit or an autonomous driving (AD) control unit may provide ADAS functions or AD functions, such as lane recognition, lane keeping assistance, traffic sign recognition, speed limit assistance, traffic participant recognition, collision warning, emergency brake assistance, distance following control, construction site assistance, highway autonomous driving, cruise driver function and / or autonomous driving.

[0105] The image capture device typically includes an optical system or lens and an image capture sensor such as a complementary metal oxide semiconductor (CMOS) sensor.

[0106] Here, the proposed surroundings detection sensor system 1 takes into account the vehicle sensor setup in the context of assisted driving and autonomous driving. As an option, this can also be expanded to a multi-sensor setup. The advantage of a multi-sensor system is that the safety and reliability of the road traffic detection algorithm is improved by verifying the detection of multiple sensors. Here, the multi-sensor system can be any combination of the following sensors:

[0107] - one or more cameras,

[0108] - one to multiple radars,

[0109] -One to multiple ultrasound systems,

[0110] -One to multiple LiDARs,

[0111] - and / or one to multiple microphones.

[0112] A well-defined road condition determination sensor with high reliability and high accuracy can be used as reference sensor 5. It is advantageous here that the road condition determination sensor can predictively determine whether there is water on the road surface and, as an option, can also predictively determine the temperature of the road surface.

[0113] For this purpose, a transmitting and receiving unit is provided, which transmits electromagnetic radiation of at least one specified wavelength to the road and receives and measures the intensity reflected by the road. The reference sensor is provided to indicate the probability of the presence of different categories of road conditions based on the measured values. By measuring the absorption rate of discrete wavelengths in the mid-infrared, it can be determined whether there is water on the road. By comparing the measurement results of two suitable wavelengths (ratio measurement method), a robust identification of water on the road can be achieved. For example, the reference sensor can use the following wavelengths: 1550 nanometers and 980 nanometers to detect the presence of water by comparing the reflection intensity.

[0114] Transmitting and receiving devices operating in the wavelength range of approximately 2 to 10 micrometers can determine the road surface temperature. Alternatively, the temperature can also be measured using other pyrometers that can measure surface temperatures at a specified distance.

[0115] The vehicle 2 with the reference sensor 5 and the surroundings sensor system 1 is a test vehicle for generating training data. A corresponding vehicle with the surroundings sensor system 1 and the trained machine learning system 16 can be used as a production vehicle without using the expensive reference sensor 5 .

[0116] Figure 2 A schematic diagram of a machine learning system 16 is shown, wherein the system is trained based on data X from a surrounding environment detection sensor 1 and corresponding data Y from a reference sensor 5 to generate output data Y' representing a road condition.

[0117] Artificial neural networks can be used as machine learning system 16. In this case, neural networks are particularly suitable for the method because they are easy to adjust. In this case, neural networks are especially convolutional neural networks. As an alternative, other designs of machine learning system 16 can also be used, such as decision tree learning, support vector machines, regression analysis or Bayesian networks. In addition, multi-task classification systems can also be used. In this case, the multi-task classification system includes, for example, an encoder and multiple decoders. In addition, the machine learning system can also be divided into multiple subsystems. In this case, each subsystem has the functionality of the machine learning system 16 described in this article, but the subsystems differ from each other, for example in the type of machine learning system or the selection of training data. The output data generated by the subsystems can be combined to obtain improved output results.

[0118] The machine learning system is trained by supervised learning. The machine learning system 16 is trained based on the training data (the input data X_1, X_2, ..., X_n of the surrounding environment detection sensor 1 and the corresponding theoretical output data Y_1, Y_2, ..., Y_n determined by the reference sensor 5). By adjusting the weights (or parameters) of the machine learning system 16, the error function can be minimized, which describes the deviation between the output Y'_1, Y'_2, ..., Y'_n of the machine learning system for the input data X_1, X_2, ..., X_n and the corresponding theoretical output data Y_1, Y_2, ..., Y_n.

[0119] Figure 3 The trained machine learning system 16 is shown, which can generate output data Y' representing the road conditions from the (newly detected) data X of the surrounding environment detection sensor 1. The trained machine learning system 16 can be used in mass-produced vehicles, for example.

[0120] Figure 4 An embodiment of a road condition monitoring system 10 is schematically shown, which can determine road condition data and transmit them to a server unit 20. The road condition monitoring system 10 is electrically or wirelessly connected to at least one surroundings detection sensor 1 such as an image capture device in a vehicle 2.

[0121] The data or signal detected by the surrounding environment detection sensor 1 is transmitted to the input interface 12 of the road condition monitoring system 10. The processing unit (or data processor) 14 in the road condition monitoring system 10 processes the data. The processing unit 14 includes a machine learning system 16. The machine learning system 16 may include an artificial neural network, such as a convolutional neural network (CNN), which has been trained to classify road conditions. The classified road conditions can be transmitted to other vehicle control units (such as an automatic driving control unit (ADCU)) via the output interface 18. The data transmission unit 19 is used to wirelessly transmit data and / or classified road conditions to a server unit 20 (cloud, backbone network, infrastructure...). In order to enable the artificial neural network to process data in the vehicle in real time, the road condition monitoring system 10 or the processing unit 14 may include one or more hardware accelerators for the machine learning system 16 or the artificial neural network.

[0122] Figure 5 A vehicle 500 is shown with a reference sensor 505 with a transmitting and receiving unit. The vehicle has a sensor system 504, a road condition monitoring system 501 with a machine learning system 502 and a data transmission unit 503. The reference sensor 505 is arranged at the front of the vehicle, for example in front of the radiator, and has a beam direction 506 of the transmitting and receiving unit such that the angle α between the beam 506 and the road plane is about 70°. For a specific vehicle 500, the angle α can be set, for example, between 50° and 80°. The height of the exit of the beam 506 of the reference sensor 505 above the road can be in the range of 20 cm to 60 cm.

[0123] To generate a first dataset (seed phase), 2500 test drives were performed using a vehicle equipped with a camera, a global positioning system (GPS) and reference sensors. The global positioning system (GPS) is required, for example, for comparison with weather forecasts etc. in the network.

[0124] In addition to the individual images (e.g. one image per second), the evaluation results of the reference sensor are also stored. If additional information such as speed, acceleration, GPS, etc. is available, this is also stored together with the corresponding time stamp.

[0125] If a case of interest arises, additional manual classification by humans is also advantageous. This allows the corresponding camera image and reference measurement value to be found again in the data set later. An additional feature of the reference sensor enables marking on the feedback channel via a CAN output. Important are the cases in which the conclusions drawn by the human differ from those of the reference sensor.

[0126] Additional information ("meta information") can be stored, for example, in a separate classified file (txt, xml or json) with the same name. Time stamps should also not be missing, since some of the methods described later require a time shift between the image and the reference measurement data, because, for example, the front camera sees a road section earlier than the measurement point of the reference measurement technology.

[0127] In addition, it is preferred to name the camera images and classify the documents using a combination of best pre-classification, percentage, date, time, camera name, horizontal image angle, and vehicle:

[0128] D_99_20210507_2023CET_front_90_F-TZ-333.jpg

[0129] D_99_20210507_2023CET_front_90_F-TZ-333.xml

[0130] (D_99_20210507_2023CET_front_90_F-TZ-333.txt)

[0131] (D_99_20210507_2023CET_front_90_F-TZ-333.json)

[0132] D = Dry (W = Wet, I = Ice, S = Snow, E = Error)

[0133] 99 = 99% means "dry"

[0134] 20210507=YYYYMMDD

[0135] 2023CET=hour, minute, minute+time zone

[0136] front=front camera (left, right, rear)

[0137] 90 = 90° wide angle

[0138] F-TZ-333 = License plate number (optional)

[0139] Regarding the format: the txt format is easy to read but difficult to parse; the json format is the most machine readable but difficult to edit; the xml format is in the middle, it can still be edited by hand but is more machine readable.

[0140] The purpose of the preliminary pre-classification is to be able to sort by category by name and to change the labels in a very simple way. This can be done in the active learning phase described later. In the case of human rejection, the data can be further processed and the labels adjusted for this purpose. Only in this case can the contradictions in the data set be resolved.

[0141] The measurement data are preferably stored in subdirectories of the relevant date, since Linux systems have problems storing more than 65535 files per directory without special changes for typical variable sizes.

[0142] Another possible solution is to use a JSON database such as Mongo-DB to store image data and reference data instead of storing image documents and text documents which are easier to read.

[0143] As an option, the vehicle's GPS position can also be transmitted in the data. Later in fleet tests, license plate number and GPS were omitted.

[0144] GPS positions are helpful especially in the seeding phase, ie in the initial phase of network calculation, to select sufficiently different locations, or if the same location, just different time of day and weather conditions.

[0145] A specific example of evaluating reference sensor data in pseudocode form:

[0146] %lrm=(r980m+r1310m+r1550m+r1552m+1)>>2; / / Total brightness

[0147] %xrm=(((r1550m+r1552m)*1600) / r980m); / / smaller: higher water content

[0148] %yrm = ((r980m*192) / r1310m); / / Larger: higher proportion of ice *or* deeper water

[0149] %vrm=(r1550m*100 / (r1550m+r1552m)); / / The difference between the two wavelengths of 1550nm and 1552nm indicates that the laser is dirty

[0150] Here, R980 to R1552 are the reflected light intensities at the respective wavelengths measured in nanometers in the analog-to-digital conversion unit (ADU), i.e., the unit of the analog-to-digital (AD) converter, and XRM, YRM, and VRM are ratio measurements (with one denominator). The variables XRM and YRM refer to different wavelengths, VRM refers to different channels at the same wavelength of 1550 nanometers, and LRM is an absolute measurement, which is the surface brightness and is used to identify the difference between ice and snow.

[0151] % Icing probability for pyrometers for early fusion

[0152] pp_ice=-40*TC_can+100;%0%means Tc>2.5℃,100%means Tc<0℃

[0153] if (pp_ice>99) pp_ice=99; end

[0154] if (pp_ice<0) pp_ice=0; end

[0155] Measuring ice formation using absorbance at 1310 nm is unreliable and is better measured using a pyrometer instead because absorbance at 1550 nm affects both water and ice in the same way. 1310 nm affects only ice, but is significantly less effective than 1550 nm for identifying water.

[0156] Pyrometers detect thermal infrared light in the range of 2 microns to 10 microns, while 1550 nanometers is known as mid-infrared.

[0157] %If (yrm>yr2)% there is ice? Just use lasers!

[0158] If (pice>50)% early fusion: Identify ice based on pyrometer

[0159] psno = ((lrm-lr1)*100) / (lr2-lr1); % Consider snow vs. ice

[0160] pdry=0;% Simplify!

[0161] pwet=0;% simplified!

[0162] The probability of dryness, wetness, ice, snow is determined directly from xrm (absorption of water at 1550nm vs. 980nm), Tc (corrected road temperature from the pyrometer), and lrm (surface brightness). And this is done one after another in this order in the decision tree.

[0163] The basis for this is the vibration modes of water molecules and the radiation of black bodies, as well as the physics of direct and indirect semiconductors. Measurements can be made at these wavelengths in the infrared (IR), but this is not possible with silicon, which becomes "transparent" above 1000 nanometers. In other words, it is no longer suitable for use as a photodiode. Germanium can detect wavelengths up to 1550 nanometers, but this generates a lot of noise if it is not cooled. Indium gallium arsenide (InGaAs) works just as well as germanium at 1550 nanometers, and does not require cooling. The quantum efficiency can be increased by coating the surface, but coating is a very costly technology.

[0164] A complete data-driven ecosystem includes the following steps that run in a loop:

[0165] Test data is continuously input by means of a fleet of test vehicles with telematics devices that have OTA (over-the-air) update capabilities (e.g. according to the 5G mobile communication standard). The ADCU continuously (open-loop testing) determines AD or ADAS-related data from the input sensor data with the help of a pre-trained machine learning system to achieve automatic or assisted control of the vehicle. In the event of a trigger (e.g. an abnormal situation, such as the driver's behavior is different from what the ADCU predicted), the data is transmitted to a server unit / cloud. The data is checked in the cloud to determine whether it is better training data than the training data already there. For this purpose, the predicted abnormal situation (or extreme situation) can be verified, or more relevant data can be selected for other reasons. The machine learning system or neural network can be retrained using the optimized training data by means of data aggregation ("let the data decide") or by means of active learning (but with less manual annotation work). The actual improvement effect is verified by improving KPIs (key performance indicators) such as precision and / or recall. The neural network is then extensively tested in the simulated world, test scenarios, extreme situations or rare situations to improve safety. In the event of a successful test, the trained neural network is approved for use and can be transmitted to the test fleet's vehicles as an over-the-air update using a cryptographic signature and imported as an update to the ADCU.

[0166] Therefore, continuous improvement requires cyclic training in a data-driven ecosystem. Here, it is not about simply increasing the dataset, but about increasing the relevance of the training data. With style transfer, less representative categories are transferred from images of the same situation to other locations, so that the relevant situation is not learned based on the location, but on the appearance of the road. By time shifting and projecting the path of the measurement points into the image, the measurement result can be accurately assigned to a specific point on the road. With the bootstrapping method, that is, applying the same rules to all image points that contain the road (or other networks), the entire distribution on the road can be measured, which is forward-looking, thus saving time and facilitating the necessary adjustments to driving behavior.

Claims

1. A method for training a machine learning system (16) for monitoring road conditions, comprising the following steps: - data are provided or acquired by a sensor system (1) of the vehicle, wherein the sensor system detects the vehicle's surroundings and the data are used as training input data (X), - providing or acquiring data representing the road condition through a reference sensor (5) arranged in or on the vehicle, and using the data representing the road condition as a training target value (Y), and - training machine learning systems (16), Wherein, training data (X, Y) is provided, the training data includes training input data (X) and a training target value (Y) corresponding to the training input data (X), and The parameters of the machine learning system (16) are adjusted with the help of training data (X, Y) so that the machine learning system (16) generates output data (Y') similar to the training target value (Y) when the training input data (X) is input.

2. The method according to claim 1, wherein: The sensor system (1) comprises a vehicle camera system, so that the data provided or captured are image data, which are used as training input data (X).

3. The method according to claim 1 or 2, wherein: The reference sensor (5) comprises a transmitting and receiving unit, which transmits electromagnetic radiation (506) of at least one specified wavelength to a road (510) and receives and measures the intensity reflected by the road (510), wherein the reference sensor (5) is configured to provide the probability of existence of different categories of road conditions based on the measured values.

4. The method according to claim 3, wherein: The reference sensor (5) outputs the existence probability of the following categories of road conditions: "Dry", "Wet", "Snow", "Ice", and "Unknown / Error".

5. A method according to any one of the preceding claims, wherein: The reference sensor (5) comprises a pyrometer for measuring the road temperature.

6. The method according to claim 3, 4 or 5, wherein: Reference sensor (5) uses: Wavelengths of 1550 nanometers and 980 nanometers to detect the presence of water by comparing the intensity of reflections, and wavelengths in the range of 2 microns to 10 microns to measure road temperature.

7. A method according to any one of the preceding claims, wherein: Machine learning systems (16) are neural networks, especially convolutional neural networks.

8. A method according to any one of the preceding claims, wherein: The vehicle (2) comprises a data transmission unit (18) which is configured to transmit training data (X, Y) to a server unit (20).

9. The method according to claim 8, wherein: The server unit (20) is configured to update a training data set comprising a certain number of training data (X, Y), wherein the size of the training data set remains constant, wherein one or more quality criteria are used to ensure that the relevance of the training data (X, Y) for determining road conditions is increased when the training data set is updated.

10. The method according to claim 9, wherein: The server unit (20) is configured to achieve balance of the training data set during the updating process, so that the training data set has a higher diversity, wherein relatively rare road conditions can be fully reflected by the training data (X, Y).

11. The method according to claim 9 or 10, wherein: The correlation of the training data (X, Y) is evaluated so that the training data (X, Y) that is orthogonal to other training data (X, Y) already included in the training data set is given a high correlation.

12. A method according to any one of the preceding claims, wherein: The sensor data of the detected area of ​​the road (510) is divided into a plurality of sections, and the data provided or captured by the reference sensor (5) is used to characterize the road conditions of each section.

13. A method for monitoring road conditions using a machine learning system (16), wherein: A machine learning system (16) has been trained according to a method according to any one of claims 1 to 12, wherein providing data detected by a vehicle sensor system (1) for detecting the surroundings of a vehicle (2) as input data (X) to a machine learning system (16), The machine learning system (5) generates output data (Y') representing road conditions from input data (X).

14. A road condition monitoring system (10), comprising An input unit (12) for receiving input data (X); a computer unit (14) configured to execute the method according to claim 13; and An output unit (18) for outputting output data (Y') generated by the computer unit (14).

15. A vehicle (2) comprising a sensor system (1) and a road condition monitoring system (10) according to claim 14, wherein: The sensor system is provided for detecting the surroundings of a vehicle (2) and for providing the detected sensor data as input data (X) to an input unit (1).

Images