Method and system for monitoring the road condition by means of a machine learning system, and method for training the machine learning system

Abstract

The invention relates to a method and system (10) for monitoring the road condition by means of a machine learning system (16) and to a method for training the machine learning system (16). The method for training a machine learning system (16) for monitoring the road condition comprises the steps of: - providing or acquiring data by means of a sensor system (1) of a vehicle (2), wherein the sensor system (1) captures an environment of the vehicle (2), as training input data X, - providing or acquiring data characterizing the road condition by means of a reference sensor (5) fitted in or on the vehicle (2) as training target values Y, and - training the machine learning system (16), wherein training data comprising training input data X and training target values Y corresponding to these training input data X are provided, and the training data are used to adapt parameters of the machine learning system in such a manner that the machine learning system generates output data Y' similar to the training target values Y when the training input data are input.

Description

[0001] Method and system for road condition monitoring by a machine learning system and method for training the machine learning system

[0002] Field of the invention

[0003] The present invention relates to road condition monitoring (road condition observation).

[0004] Road condition detection from a moving vehicle equipped with sensors to detect its surroundings. This topic is of immense importance for accident prevention, as certain road conditions or surfaces reduce the static friction, or the coefficient of friction, between the tires and the road surface. This increases stopping distances or causes the vehicle to skid and swerve, which can lead to serious accidents. For automated steering and / or braking interventions in assisted or autonomous driving (ADAS / AD systems), the system must therefore also be able to assess the road condition as realistically as possible before the vehicle crosses it. Since this is not yet possible with sufficient reliability, driving tests for autonomous vehicles often take place in sunny regions where snow and ice are not expected.

[0005] To improve road condition detection, machine learning is to be used in an innovative setup. In particular, the present invention relates to a method for training a machine learning system for road condition monitoring, a method for road condition monitoring using a trained machine learning system, a road condition monitoring system, and a vehicle with a road condition monitoring system.

[0006] background

[0007] A well-known example is the YouTube video of a road condition segmentation based on video data from a camera and driving noise recorded by a microphone during the segmentation process. It distinguishes between dry and wet conditions: Road condition estimator using cameras and microphones (2018) https: / / www.youtube.com / watch?v=H13igv55o8w (accessed May 25, 2022). Differentiating between more than two classes (wet / dry) is difficult or not reliably possible with this approach.

[0008] WO 2016 / 177372 A1 describes a method for detecting and evaluating environmental influences and road condition information in the vicinity of a vehicle. A camera is used to successively generate at least two digital images, each of which has an identical image section selected. Digital image processing algorithms are used to detect changes in image sharpness between the image sections of the at least two consecutive images, with the image sharpness changes being weighted from the center of the image sections outward. Environmental condition information is determined using machine learning methods based on the detected changes in image sharpness between the image sections of the at least two consecutive images, and road condition information is determined based on the determined environmental condition information.

[0009] WO 2019 / 174682 A1 describes a method for classifying a road condition based on image data from a vehicle camera system and a corresponding vehicle camera system. The method comprises the following steps:

[0010] - Providing image data by a vehicle camera system which is configured to image at least a section of an environment outside the vehicle, wherein the section at least partially contains the roadway on which the vehicle is traveling

[0011] - Distinguishing between diffuse reflection and specular reflection of the road surface by evaluating differences in the appearance of at least one point of the road surface in at least two images of the camera system, the images being taken from different perspectives; ,

[0012] - Determine whether there are disturbances in at least one image of the camera system that have been caused by at least one wheel of a vehicle having disturbed a road surface when rolling over it,

[0013] - Classify the road surface condition taking into account the results regarding the reflection type and the disturbance intensity into one of the following five road surface condition classes: a) Dry road surface: diffuse reflection type without disturbance b) Normally wet road surface: specular reflection type with disturbance c) Very wet road surface with aquaplaning risk: specular reflection type with a lot of disturbance d) Snowy road surface: diffuse reflection type with disturbance or e) Icy road surface (black ice): specular reflection type without disturbance.

[0014] Known problems with camera systems include, for example, that the signal-to-noise ratio is low in the dark; if you improve the signal-to-noise ratio through longer exposure, you end up with motion blur if the camera system moves.

[0015] DE 102013002333 A1 discloses a method for predictive road condition determination in a vehicle, in which a road surface is illuminated with sensor beams, wherein the sensor beams are reflected and absorbed depending on the road surface condition, and wherein the road condition determination is performed based on the reflected sensor beams. The method is characterized in that the road surface is illuminated in front of the vehicle in the direction of travel. Since the different road surface conditions have different optical properties and are accordingly absorbent for certain wavelengths and reflective for others, the respective road condition of the illuminated road surface can be deduced from the reflected sensor beams. An example of this is the wavelength 1550 nm, which is comparatively strongly absorbed by ice.

[0016] DE 102014214243 A1 shows a method for determining road conditions, wherein

[0017] Road condition data from a weather map and / or road map for

[0018] Road condition determination can be used, whereby the road condition data from the weather map and / or road map are subjected to re-digitization.

[0019] DE 102017223510 A1 shows an optical sensor for assessing surfaces, comprising:

[0020] - at least one emission unit for emitting a modulated optical signal onto a surface to be assessed;

[0021] - at least one detection unit for detecting an optical signal of the modulated signal reflected at the surface and providing an electrical output signal dependent on the reflected optical signal; and

[0022] - a control device for evaluating the output signal of the at least one detection unit and determining a state of the surface to be assessed, wherein a frequency spectrum of the modulated optical signal emitted by the at least one emission unit is substantially delimited from a frequency spectrum of an interference spectrum.

[0023] All known methods have drawbacks and do not fully address the relevant problems. Of particular interest is the predictive and reliable detection of "black ice," i.e., a frozen (icy) black road that is difficult to see even for humans.

[0024] Summary

[0025] It is therefore an object of the invention to provide a method that enables the predictive detection of road conditions in an affordable manner without compromising the robustness of the detection of safety-relevant road condition types.

[0026] This problem is solved by the subject matter of the independent patent claims. Further developments of the invention emerge from the subclaims and the following description. One starting point is to use environmental detection sensors that are already installed in a vehicle for other tasks, and primarily detect areas that lie in front of the vehicle in the direction of travel.

[0027] The deficiencies of existing environmental detection sensors can be compensated for by specialized reference sensors, which are therefore usually too expensive for series production use, by also contributing to the generation of training data for a machine learning system for road condition determination. To generate training data, test vehicle fleets with the environmental detection sensors are often used for later series production use. The recorded data is then usually manually labeled to provide the actual road conditions for training the machine learning system. Manual labeling is performed by humans, although human perception of road conditions is subjective. At what point, for example,Is there a "wet roadway"? Is there an older layer of ice beneath a visible layer of fresh snow? Is there black ice on a black road or a dry black road? Furthermore, manual labeling is both time-consuming and costly.

[0028] A second starting point is to use an explicit road condition determination sensor with high reliability and precision as a reference sensor. It is advantageous if the ("pure") road condition determination sensor can predictively determine whether water is present on the road surface and what the road surface temperature is. A transmitting and receiving device operating in the wavelength ranges of 1550 nm and approximately 2 to 10 micrometers can determine both.

[0029] One aspect of the invention relates to a method for training a machine learning system for road condition monitoring, comprising the steps:

[0030] - Providing or recording data by a sensor system of a vehicle, wherein the sensor system detects an environment of the vehicle, as training input data, - Providing or recording data characterising the road condition by a reference sensor mounted in or on the vehicle as training target values, and

[0031] - Training the machine learning system, wherein training data comprising training input data and training target values ​​corresponding to this training input data are provided and parameters of the machine learning system are adapted by means of the training data such that the machine learning system generates output data that are similar to the training target values ​​when the training input data is input.

[0032] The data of the sensor system (sensor data) can include, for example, image data, radar data and / or lidar data.

[0033] The machine learning system is trained using the training data. This means that the machine learning system's parameters are adjusted so that, when the training input data is input, the machine learning system generates output data that is similar to the training target values. For example, in neural networks, the parameters include the weightings between individual input values ​​and neurons. The machine learning system is trained using a supervised learning method, although numerous learning methods are known. For example, the backpropagation method can be used to train a neural network. During training, the machine learning system's parameters are adjusted so that any error between the output data and the training target values ​​is as small as possible.The error between the output data and the training target values ​​is determined, for example, using the distance between the output data and the training target values ​​and a corresponding metric for the output data. It is important to avoid overfitting, which is achieved, for example, by considering the error between the output data generated from test input data and the corresponding test target values. The test target values ​​are assigned to the test input data, and the test input data and test target values ​​are not used to adjust the parameters of the machine learning system.

[0034] Optionally, the parameters of the machine system can be output if the training has been successfully completed, e.g. because the output data are sufficiently similar to the training target values, which can be specified, e.g., via a threshold value of a similarity metric.

[0035] In one embodiment, the sensor system comprises a camera system of the vehicle, so that the provided or recorded data is image data, and the image data serves as training input data. The reference sensor determines reference data as training target values ​​for image data (labels).

[0036] The camera system can, for example, be a monocular camera, preferably arranged behind the windshield in a vehicle, so that the area in front of the vehicle can be captured according to the visual perception of a driver. Alternatively, the camera system can be a stereo camera that provides depth information about the vehicle's surroundings or a satellite camera system, e.g., a surround-view camera system that includes several fisheye cameras facing in different directions of the vehicle.

[0037] Through the downstream processing of the image data and the reference sensor data by the machine learning system (“Al camera”), the capabilities of the reference sensor for road condition monitoring can be transferred to the camera system using appropriate learning procedures.

[0038] According to one embodiment, the reference sensor comprises a transmitting and receiving unit that emits electromagnetic radiation of at least one defined wavelength onto the road and receives and measures the intensity reflected by the road. The reference sensor is configured to indicate probabilities for the presence of various classes of road conditions based on the measured values. The presence of water on the road can be determined by measuring the absorption at discrete wavelengths in the mid-infrared. A comparison of the measurement results of two suitable wavelengths (ratiometric measurement) enables the robust detection of water on the road.

[0039] A suitable reference sensor for displaying ground truth data can detect all target learning classes of the camera system regardless of the time-of-day dependent light distribution (brightness).

[0040] This is the prerequisite for the learning result of a camera system (as a sensor system) to be optimally adapted to the very different light distribution (brightness) throughout the day by controlling the exposure of the cameras.

[0041] This ensures that the camera system can detect the road condition not only in daylight but also at dusk and at night, provided the camera is sufficiently sensitive.

[0042] Advantageously, the wavelength ranges of observation between the reference sensor and the camera system do not match. The camera observes in visible light, while the reference sensor observes in the infrared range, for example.

[0043] One embodiment provides for motion blur in the image caused by different driving speeds and the set exposure time to be learned. Both parameters can be incorporated into the learning result either in classes or using a linear approximation.

[0044] Advantageously, the ground speed can be a useful additional input variable.

[0045] According to one embodiment, weather and driving dynamics data such as ABS, ASR, and / or ESC control interventions of the ego vehicle can be advantageously incorporated into the learning result. In this case, one can speak of early fusion of the labeling variables. Furthermore, or alternatively, according to one embodiment, for manually driven vehicles, the driving style is also included based on vehicle accelerations in the longitudinal and transverse directions, in addition to the aforementioned driving speed, or alternatively, the safety distance to vehicles ahead.

[0046] In one embodiment, the reference sensor assumes probabilities for the presence of (at least) the following classes of road conditions: "dry," "wet," "snow," "ice," and "unknown / error." An error in the case of a camera sensor as a sensor system is, for example, a predominantly black or white image or predominantly image noise due to underexposure.

[0047] According to one embodiment, the reference sensor comprises a pyrometer that measures the temperature of the road. For example, the reference sensor can measure in the thermal (far) infrared, e.g., in the range of 2 to 10 pm, to derive the surface temperature from thermal radiation. Thermal features in the far IR, on the other hand, are invisible in the visible spectrum, which is why even humans have great difficulty detecting black ice.

[0048] In one embodiment, the reference sensor itself uses the following wavelengths: 1550 nm and 980 nm to detect the presence of water by comparing the reflected intensities and a wavelength in the range of 2 to 10 pm to measure the temperature of the road.

[0049] In one embodiment, the machine learning system is a neural network. Neural networks are particularly well suited for the described method because they can be easily adapted.

[0050] The neural network is, in particular, a convolutional neural network. The nonlinearity of the convolutional neural network does not pose a problem for the usability of the described method. However, the described method can also be used with other machine learning systems, for example, decision tree learning, support vector machines, regression analysis, or Bayesian networks. Furthermore, it is possible to use the described method in a multi-task classification system. The multi-task classification system, for example, comprises an encoder and a plurality of decoders. Furthermore, it is possible to divide the machine learning system into several subsystems.Each of the subsystems exhibits the functionality of the machine learning system described here, but the subsystems differ from each other, for example, in the type of machine learning system or the selection of training data. Output data generated by the subsystems can then be combined to obtain an improved output.

[0051] According to one embodiment, the vehicle comprises a data transmission unit, and the data transmission unit is configured to transmit training data to a server unit (backbone). This enables the creation of a data-driven ecosystem (DDE) for the development of machine learning systems, e.g., for determining road conditions based on sensor data from common environmental sensors.

[0052] In one embodiment, the server unit is configured to update a training data set comprising a (predetermined) amount of training data, wherein the size of the training data set is kept constant, and wherein one or more quality criteria (e.g., the quality can be assessed based on several KPIs) are used to ensure that the relevance of the training data with regard to road condition determination is increased when the training data set is updated.

[0053] To achieve high reliability even in unfamiliar scenarios, one exemplary embodiment uses a DDE for training with continuous improvement of data relevance. The DDE backend utilizes automated processes based on key performance indicators (DAgger) and, with significantly reduced effort, human judgment (active learning).

[0054] The DDE is designed to keep the dataset size constant once a limit is reached. This prevents training times for the machine learning system from increasing further. The computing power in the backend is used to increase the relevance of the training dataset.

[0055] According to one embodiment, the server unit is configured to achieve a balanced training data set during the update, so that the training data set exhibits a high degree of diversity, with less frequent road conditions being adequately represented by training data. Rare situations, such as black ice, i.e., a frozen, black road, are particularly rare, but also particularly dangerous, as the coefficient of friction is significantly reduced.

[0056] In one embodiment, the relevance of the training data is evaluated in such a way that a high relevance is attributed to the training data that is orthogonal to the other training data already contained in the training data set.

[0057] According to one embodiment, the sensor data for the detected area of ​​the road are divided into segments and the provided or recorded data of the reference sensor characterize the road condition for the segments.

[0058] The high spatial resolution, preferably with the help of the recognition rules condensed in the network, enables segmentation of the sensor data on the road ahead, thus enabling even predictive and spatially resolving measurements. This gains reaction time and, at the same time, reduces application costs by up to two orders of magnitude compared to the reference measurement technology. Another aspect relates to a method for road condition monitoring using a machine learning system, wherein the machine learning system has been trained as described above. Data acquired by a vehicle sensor system that detects the vehicle's surroundings is provided to the machine learning system as input data, and the machine learning system generates output data characterizing the road condition from the input data.

[0059] A third aspect relates to a road condition monitoring system comprising an input unit for receiving input data; a computer unit configured to carry out the above-described method for road condition monitoring; and an output unit for outputting the output data generated by the computer unit.

[0060] A further aspect relates to a vehicle comprising a sensor system, wherein the sensor system is configured to detect an environment of the vehicle and to provide the detected sensor data to the input unit as input data, as well as a road condition monitoring system as described above.

[0061] When setting up a training data set (e.g. in a server unit) the following problems can arise: a) A lot of similar data is generated b) There are rare classes c) The classes are very strongly correlated with the test locations, for example more ice is detected in Sweden, more wetness in Germany and more dryness in southern Spain. Due to the correlation with the location, the location could be incorrectly learned instead of the road condition. d) Rare classes are nevertheless relevant. The rare classes are practically neglected, but these are precisely the particularly dangerous cases. e) The wavelength ranges of the observation between the reference sensor and the camera do not match, for example the reference sensor observes at discrete wavelengths in the mid-infrared to detect absorption of water and ice and in the thermal (far) infrared to derive the surface temperature from the thermal radiation.The camera observes light in the visible wavelength range. Thermal features in the far-IR, for example, are invisible in the visible spectrum, which is why humans already have great difficulty detecting black ice. Instead, they infer the condition from observations of the surroundings and other optical features. This means that it is in principle possible to differentiate road conditions into four to five classes using a camera's optical system, but it is difficult; one would have to learn from an enormous number of examples, and therefore road condition detection based solely on camera's optical system is not necessarily sufficiently robust.

[0062] A data-driven ecosystem offers the following solutions to address such problems:

[0063] 1) Policy Aggregation: When you learn a Random Decision Forest, you don't use just one decision tree, but a whole "bag full of them", i.e. a "bag of classifiers method". This makes a majority decision possible and as the learning progresses, variants of existing, successful decision trees are randomly generated and added, while unsuccessful ones are discarded. In the end, you get a very good result through a majority decision of many good classifiers. This method is easy to use when the individual classifier is calculated sparingly. This is the case with decision trees with few yes / no criteria. It is an evolutionary process; no gradients are calculated and no backpropagation is used.

[0064] 2) Backpropagation: Modern neural networks learn using backpropagation from the difference between the desired output and the random output obtained after initial initialization. Using a chain rule, the differences (gradients) are translated back into the network's layers, thus determining how the weights and offsets in the network need to change to arrive at a better solution. In backpropagation, the repeated application of redundant data is disruptive because it introduces irrelevance. Certain, possibly very similar, cases are overlearned, almost meticulously memorized, but slightly different test data are not recognized as well as the training data—a process known as overfitting.

[0065] 3) Splitting the data: On the detection side, splitting the data into training and test data helps prevent overfitting. A key performance indicator (KPI) to be measured, such as the precision or accuracy of the output, must be at least as good for the test data as for the training data. It doesn't help if the training data is too close to the test data, for example, because they are adjacent images from the same sequence. This should be avoided.

[0066] 4) Batch Normalization: Neurons in the networks operate on the simplifying assumption that a linear coupling to the next layer is sufficient because the input data, the output data, and also the data in the intermediate layers are offset-free and evenly distributed. This can be achieved through layer-by-layer renormalization (Loffe and Szegedy, Google).

[0067] 5) Dropout: During the learning process, clumps of non-functional neurons can form in the network. The goal is for each sub-region of the network to contribute equally to the classification. A particularly difficult problem is the problem of vanishing and exploding gradients. This means that when propagating back using the chain rule, the requirement for the neurons to reduce their weights becomes exponentially smaller or exponentially larger in each layer. To prevent this, approximately 50% of the neurons are randomly selected and deactivated in each step. For the remaining 50%, the signal strength is adjusted, and only these are trained in the current step.In the next iterative training step, a different 50% of neurons are selected, ensuring that every subset of the network has the same functionality. The full network is then simply more powerful due to its size, but there are hardly any neurons that remain unconnected because the associated weights remain close to zero. 6) Regularization: This method is perhaps best compared to measuring the depth of a crater. The deepest point corresponds to the best solution, but craters are not simply shaped like a bowl; rather, they also contain mountains and other craters. In fact, it is an entire field of overlapping craters. With a simple search for the deepest point - in the example, a ball thrown into the field - there is a risk that the ball will get stuck in a local minimum, i.e., the best solution is not determined, but only the locally next best one.Regularization means injecting adhering powder into the crater landscape. This, at least after solidification, fills the smaller craters, thus reducing the chances of getting stuck in a local minimum. Regularization must be used very sparingly. Figuratively speaking, one must not allow the crater landscape to decay completely, thereby leveling out all differences. However, when used in small doses, in the per mille range, regularization has the potential to diminish the effectiveness of specific identifying features and instead find more general, fundamental rules. However, this comes at the cost of a "seemingly" reduced accuracy. A simple KPI thus performs worse, even though it actually classifies better.

[0068] 7) Activations: Activations can be calculated using a type of visual backpropagation on the GPU in parallel with the actual neural network. They indicate which neurons are involved in the decision for the convoluted layers (not the fully connected layers) and project this information back onto the network's input layer. This means that, if the activations are overlaid with the corresponding input image, a human can assess which features were used to make the classification decision. This helps understand what goes wrong when something is incorrectly detected and provides valuable clues to potentially missing training examples. It also provides a visual indicator beyond simply observing the KPIs.

[0069] 8) Key Performance Indicators, or KPIs for short: A simple two-class detector can either respond, not respond, report correctly, or report incorrectly – in all combinations, so here four cases. If detection occurs, then recall is high. It is conceivable to darken a scene further and further until the image is black. If the output were again an image, not just a class, one could count pixels that depict a known feature and that are recognized. In addition, other pixels not belonging to the known feature could also respond. These would then be false positives. So it's about two things: detection and the correctness of the result. This can either be plotted in a diagram or summarized in the F2 score. Simply put, a KPI is a predefined metric that, on average across the test dataset, indicates how well a network is trained.

[0070] 9) Data Aggregation: Very similar to Policy Aggregation. Instead of computing, say, 500 neural networks and coordinating their output based on the majority, the problem is shifted from execution time to training time, and thus to selecting the relevant data. This means you only have to compute one network, but you start with a subset of the training data, say 5% of it, and then look at the next 5% to see which images are already correctly classified by training the first 5%. These are simply not needed. It's like preparing for an exam, where you have to learn what you don't yet understand. Data aggregation is when the machine decides which data is relevant; it's the data that is orthogonal to all the other data it has already learned.

[0071] 10) Active Learning: Sometimes data is incorrectly labeled. The network notices this during data aggregation; it's impossible to output both dry and wet images at the same time. This suggests that an image is incorrect. But which one? In these cases, a human still has to make the decision, but because the process is largely automated, only the controversial cases need to be resolved.

[0072] 11) Constant dataset size: No steady increase in learning time. The DDE then independently decides whether the data is relevant. When the dataset finally reaches its target size, data aggregation can attempt to swap new data for old data that is deemed relevant. The quality is continuously measured using several KPIs. If the new solution, in the form of a newly learned neural network, is below average, it is discarded; if it is above average, it is favored.

[0073] 12) The Data-Driven Ecosystem: In its backend, i.e., the function that is later computed in the cloud, the best networks computed so far are constantly challenged to determine whether they can be improved after training by replacing them with more relevant training data. This depends largely on the test dataset, which, like the training dataset, must be adapted. It should contain the relevant cases in all environmental situations, as well as the rare cases and corner cases.

[0074] 13) Synthetic data (derived from real data): By definition, rare cases are rare. To avoid underrepresentation, synthetic data can be derived from regular data using GAN-style transfer. This means that the synthetic data answers the question of what a landscape looks like in a different weather condition, or what a weather condition looks like at a different location. Since you can take many images of locations, but only a few of rare conditions, you transfer the condition to another location.

[0075] 14) Speed-dependent, time-delayed analysis: What is visible on the front camera is further ahead on the road than the current measurement spot. To be sure, a time shift must be applied to align the trajectory of the measurement spot on the road with the image of the road. Conversely, segmentation can be calculated instead of categorical classification. If a network is also used that recognizes the position of the road in the image, the road can be learned in segmented form along with its road condition and then displayed accordingly.

[0076] 15) Averages speak louder than individual values: Even on the inference side, i.e., in the application of the network, there are opportunities for improvement and for compensating for outliers through averaging. For example, second-by-second changes in the overall weather situation are not to be expected; however, road conditions are a phenomenon that can very well change at spatial boundaries, be it when leaving a hall or a tunnel, or when crossing a bridge. Particularly significant changes are to be expected here. If this can be identified, the result can be stabilized in all other cases by averaging.

[0077] 16) Testing the networks in synthetic scenes and in road tests: To cover standard cases and known rare and corner cases, additional testing is performed on top of the regular training data to verify the plausibility of the behavior. The network must survive a test course.

[0078] 17) Transmission security: In the simplest case, an update is carried out via a USB stick. However, an update function for the vehicle fleet after delivery and throughout the product's lifetime is also useful. To prevent transmission errors and tampering in a complete DDE, checksums, a cryptographic signature, and encryption are added to the actual data.

[0079] 18) Update: For fleet applications, data should preferably be transferred over the air (OTA). This assumes that vehicles will be equipped with SIM cards in the future and can establish a tunneled internet connection to the update server. Of course, a conventional update, e.g., via a USB stick and file system, is always possible.

[0080] 19) Triggering interesting data: It is known from the Tesla patent in particular that rare cases and corner cases, in our case rare road weather situations, should be recorded. With Tesla, the trigger itself is a network that has been trained to respond to certain scenarios. This same technology would be used here to improve the product in a second, continuous cycle based on data. Triggering only in a few test vehicles equipped with reference sensors is also conceivable. However, due to the larger number of use cases, the greater benefit is clearly a trigger in every vehicle in the fleet, as this is the only way the system scales automatically. 20) Return channel: preferably Over The Air (OTA), but also conventionally in the workshop via USB or other bus systems.An OTA update channel to the vehicle offers the advantage that necessary software updates do not require a recall from the vehicle manufacturer. Furthermore, a feedback channel facilitates continuous product improvement. Triggered moments and other selected moments, such as those by test drivers, and associated measurements are reported back. Data volumes of up to several hundred MB for various sensors per manufacturer are a realistic order of magnitude. This means that far more data is available than subsequently needs to be trained into the corresponding sensor networks. This makes a good methodology for selecting the training data and evaluating the networks to be trained using KPIs all the more important.

[0081] 21) Ecosystem virtualization, connections, and sourcing: All of this happens in the backend, preferably in the cloud. The entire backend can be virtualized, i.e., moved to a Docker container and run on a large cloud instance with sufficient performance. This has the advantage that post-processing can be scaled along with the fleet size. Thus, there is a data connection between the fleet, provider, and cloud provider. This, in turn, is independent of the location where the data is curated and new algorithms are integrated. Access to the cloud should be possible worldwide without any problems. The same applies to the rollout of the data.

[0082] 22) Design anonymization and data protection without side effects: Due to locally applicable data protection regulations, source anonymization should be considered. Because if no personal data is stored, then no laws can be violated in the concept of globally available cloud processing and data curation. It must be ensured that anonymization does not have an unintended impact on the primary function. In the case of road condition monitoring, however, an anonymized face and license plate should not pose a problem, unless, perhaps, the license plate were very large in the image and were relabeled from a dark background to a light background. Extreme white tones in the image could be an indicator of snow, which would then be falsely classified as more likely. Such side effects should be tested and methodically avoided (e.g., obscuring only with background color).

[0083] Adapting driving behavior to road conditions: In the case of road condition measurement, the goal of all these efforts is simply appropriate driving behavior. The driver certainly doesn't want to be informed about the condition every second; they are certainly experts enough for that. Warnings would be issued in the case of slippery conditions, i.e., a significantly reduced coefficient of friction. The actual coefficient of friction can only be estimated, as it depends on the road and tires. The main application of road condition measurement is highly automated driving, as driving planning must also be informed of the road conditions. It may be necessary to drive more slowly, and understeering and longer braking distances may have to be expected. What is self-evident to us humans must be taught to machines using a technical method. It is advantageous if the technical method is cost-effective.

[0084] The purpose of road condition monitoring lies in the area of ​​driver warning in the event of slippery conditions (especially black ice), but even more so in the area of ​​highly automated driving. Here, braking distance assumptions must be constantly updated according to the road conditions, allowing driving behavior to adapt to the weather conditions.

[0085] Averages speak louder than individual values: Even on the inference side, i.e., in the application of the network, opportunities for improvement and for compensating for outliers arise through averaging. For example, second-by-second changes in the overall weather situation are not to be expected; road conditions are a phenomenon that can, however, very well change at spatial boundaries, be it when leaving a hall or a tunnel, or when crossing a bridge. Particularly significant changes are to be expected here. If this can be identified, the result can be stabilized in all other cases by averaging.

[0086] Speed-dependent, time-delayed analysis: What is visible on the front camera is further ahead on the road than the current measurement spot. To be sure, a time shift must be applied to align the trajectory of the measurement spot on the road with the image of the road. Conversely, segmentation can be calculated instead of categorical classification. If a network that recognizes the position of the road in the image is also used, the road can be learned in segmented form along with its road condition and then displayed accordingly.

[0087] Short description of the characters

[0088] In the following, exemplary embodiments of the invention are explained in more detail with reference to the drawings. Herein:

[0089] Fig. 1 shows a vehicle with an environment-sensing sensor system, a reference sensor and a processing unit;

[0090] Fig. 2 is a diagram of a machine learning system being trained using data from the environment sensing sensor and the reference sensor to generate output data characterizing the road condition;

[0091] Fig. 3 shows a trained machine learning system that can generate output data characterizing the road condition based on data from the environmental sensing sensor;

[0092] Fig. 4 a road condition monitoring system; and

[0093] Fig. 5 a vehicle with a reference sensor with a transmitting and receiving unit.

[0094] Detailed Description Fig. 1 schematically shows a top view of a vehicle 2. The vehicle 2 has an environment-detecting sensor system 1, a reference sensor 5, and a processing unit 10. The environment-detecting sensor system 1 can be or comprise, for example, an image recording device. The image recording device can be a front camera of a vehicle. The front camera can be arranged inside the vehicle 2 - for example, in the area of ​​the rearview mirror - and can capture the environment in front of the vehicle 2 through the windshield of the vehicle 2. Based on the signals or image data from the front camera, details of the environment of the vehicle 2 can be detected, e.g., objects. Based on the environment detection, ADAS or AD functions can be provided by an ADAS / AD control unit, e.g.lane recognition, lane keeping assistance, traffic sign recognition, speed limit assistance, road user recognition, collision warning, emergency braking assistance, distance following control, construction site assistance, a motorway pilot, a cruising chauffeur function and / or an autopilot.

[0095] The image recording device typically comprises an optic or lens and an image recording sensor, e.g. a CMOS sensor.

[0096] The proposed environmental detection sensor system 1 considers a sensor setup for vehicles in the context of assisted and autonomous driving. This can optionally be expanded to a multi-sensor setup. Multi-sensor systems have the advantage of increasing the reliability of detection algorithms for road traffic by verifying the detections of multiple sensors. Multi-sensor systems can be any combination of: one to several cameras, one to several radars, one to several ultrasound systems, one to several lidars, and / or one to several microphones. As reference sensor 5, it is advisable to use an explicit road condition determination sensor with high reliability and precision. It is advantageous if the road condition determination sensor can predictively determine whether water is present on the road surface and, optionally, also the temperature of the road surface.

[0097] A transmitting and receiving unit that emits electromagnetic radiation of at least one defined wavelength onto the road and receives and measures the intensity reflected by the road is ideal for this purpose. The reference sensor is configured to indicate probabilities for the presence of various classes of road conditions based on the measured values. The presence of water on the road can be determined by measuring the absorption at discrete wavelengths in the mid-infrared. A comparison of the measurement results of two suitable wavelengths (ratiometric measurement) enables the robust detection of water on the road. For example, the reference sensor can use the following wavelengths:

[0098] 1550 nm and 980 nm to detect the presence of water by comparing the reflected intensities.

[0099] A transmitting and receiving device operating in the wavelength range of approximately 2 to 10 micrometers can determine the temperature on the road surface. Alternatively, another pyrometer can be used to measure the temperature of a surface at a defined distance.

[0100] Vehicle 2 with reference sensor 5 and environment-sensing sensor system 1 is a test vehicle for generating training data. A corresponding vehicle with environment-sensing sensor system 1 and a trained machine learning system 16 can be used as a production vehicle without the cost-intensive reference sensor 5.

[0101] Fig. 2 shows a representation of a machine learning system 16 which uses

[0102] Data X of the environment detection sensor system 1 and corresponding data Y of the reference sensor 5 are trained to generate output data Y' that characterizes the road condition.

[0103] An artificial neural network can serve as the machine learning system 16. Neural networks are particularly well suited for the described method because they can be easily adapted.

[0104] The neural network is, in particular, a convolutional neural network. Alternatively, other configurations of a machine learning system 16 can be used, for example, decision tree learning, support vector machines, regression analysis, or Bayesian networks. Furthermore, it is possible to use a multi-task classification system. The multi-task classification system comprises, for example, an encoder and a plurality of decoders. Furthermore, it is possible to divide the machine learning system into several subsystems. Each of the subsystems has the functionality of the machine learning system 16 described here; however, the subsystems differ from one another, for example, in the type of machine learning system or in the selection of training data. Output data generated by the subsystems can then be combined to obtain an improved output.

[0105] The machine learning system is trained using supervised learning. The machine learning system 16 is trained using training data (input data X_1, X_2, ..., X_n from the environment-sensing sensor 1 and corresponding target output data Y_1, Y_2, ..., Y_n determined by the reference sensor 5). By adjusting weights (or parameters) of the machine learning system 16, an error function is minimized that specifies deviations between outputs Y'_1, Y'_2, ..., Y'_n of the machine learning system for input data X_1, X_2, ..., X_n and corresponding target output data Y_1, Y_2, ..., Y_n.

[0106] Fig. 3 shows a trained machine learning system 16 that can generate output data Y' characterizing the road condition based on (newly acquired) data X from the environment detection sensor 1. The trained machine learning system 16 is used, for example, in a production vehicle.

[0107] Fig. 4 schematically shows an embodiment of a road condition monitoring system 10 that can determine road condition data and transmit it to a server unit 20. The road condition monitoring system 10 is electrically or wirelessly connected to at least one environmental detection sensor 1, e.g., an image recording device, in a vehicle 2.

[0108] The data or signals acquired by the environmental detection sensor 1 are transmitted to an input interface 12 of the road condition monitoring system 10. The data is processed in the road condition monitoring system 10 by a processing unit (or data processor) 14. The processing unit 14 includes a machine learning system 16. The machine learning system 16 may include an artificial neural network, for example, a CNN, which has been trained to classify the road condition. The classified road condition can be transmitted to other vehicle control units (e.g., an ADCU, automated driving control unit) via an output interface 18. A data transmission unit 19 serves to wirelessly transmit data and / or the classified road condition to a server unit 20 (cloud, backbone, infrastructure, etc.).In order for the artificial neural networks to process the data in the vehicle in real time, the road condition monitoring system 10 or the processing unit 14 may include one or more hardware accelerators for machine learning systems 16 or artificial neural networks.

[0109] Fig. 5 shows a vehicle 500 with a reference sensor 505 having a transmitting and receiving unit. The vehicle has a sensor system 504, a road condition monitoring system 501 with a machine learning system 502, and a data transmission unit 503. The reference sensor 505 is arranged in the front of the vehicle, for example, in front of the radiator, and has a beam direction 506 of the transmitting and receiving unit such that the angle α between beam 506 and road level is approximately 70°. The angle α can be set for specific vehicles 500, for example, between 50° and 80°. The height at which the beam 506 emerges from the reference sensor 505 above the road can be in the range of 20 to 60 centimeters.

[0110] To generate an initial data set (seed phase), 2,500 test drives can be undertaken with a vehicle equipped with cameras, GPS, and a reference sensor. GPS is needed, for example, to compare with online weather forecasts.

[0111] In addition to the individual images (e.g., one image every second), the evaluation of the reference sensor is saved. If additional information is available—speed, acceleration, GPS, etc.—this is also saved with the appropriate timestamp.

[0112] Additional human classification can also be advantageous when an interesting situation arises. This allows the associated camera images and reference measurements to be subsequently found in the data sets. An additional feature of a reference sensor enables labeling via a CAN output on a return channel. Situations in which the human reaches a different conclusion than the reference sensor are important.

[0113] For example, the additional information ("metainformation") can be stored in a separate classification file (txt, xml, or json) with the same name. A time reference should also be included, as some of the methods described later require a time offset between images and reference measurement data, for example, because the front camera sees the section of road earlier than the measurement spot of the reference measurement technology.

[0114] It is further preferred to label the camera images and classification files with a combination of best pre-classification, percentage, date, time, camera name, etc. Image angle and vehicle to name: D_99_20210507_2023CET_front_90_F-TZ-333.jpg D_99_20210507_2023CET_front_90_F-TZ-333.xml (D_99_20210507_2023CET_front_90_F-TZ-333.txt) (D_99_20210507_2023CET_front_90_F-TZ-333.json) D = dry (W = wet, I = ice, S = snow, E = Error) 99 = 99% for “dry” 20210507 = YYYYMMDD 2023CET = HHMM+time zone front = front camera (left, right, back) 90 = 90° wide angle

[0115] F-TZ-333 = License plate (optional)

[0116] Regarding the formats: txt is easy to read but difficult to parse, json is the easiest to machine-read but difficult to edit, and xml is a middle ground, still editable by hand but more machine-readable.

[0117] The intention of the pre-classification is to allow sorting by class by name and to provide a very easy way to change the label. This can occur in the active learning phase described later. If the human overrules the decision there, the data can be reused preferentially and the label can be adjusted accordingly. Only then is it possible to resolve inconsistencies in the data set.

[0118] Preferably, the measurement data should be stored in subdirectories of the day in question, since Linux systems run into problems with more than 65,535 files per directory without special modification of the typical variable sizes.

[0119] Another possible solution is a JSON database, e.g. Mongo DB, for image and reference data instead of the more easily readable image and text files.

[0120] Optionally, the vehicle's GPS position should also be included in the data. License plate and GPS will be omitted later in the fleet test.

[0121] The GPS position can be particularly helpful in the seed phase, i.e. the initial phase of the network calculation, to select sufficiently different locations, or if the same locations, then only with different times of day and weather conditions.

[0122] A concrete example of evaluating data from the reference sensor in the form of pseudocode:

[0123] % lrm=(r980m+r1310m+r1550m+r1552m+1 )»2; / / Total brightness

[0124] % xrm=(((r1550m+r1552m)*1600) / r980m); / / smaller: higher water content

[0125] % yrm=((r980m*192) / r1310m); / / larger: higher proportion of ice *or* deeper water

[0126] % vrm=(r1550m*100 / (r1550m+r1552m)); / / Inequality between the two wavelengths 1550nm and 1552nm indicates contamination of the laser

[0127] Here, r980..r1552 are the intensities of the reflected light at the respective wavelength in nm, measured in ADU, i.e., the units of the AD converter. Xrm, Yrm, and Vrm are ratiometric measurements (there is a denominator). The quantities Xrm and Yrm refer to different wavelengths, Vrm to different channels of the same wavelength (1550 nm), and Irm is an absolute measurement, a surface brightness that allows the distinction between ice and snow to be distinguished.

[0128] % Ice probability from the pyrometer for early fusion pp_ice=-40*TC_can+100; % 0% for Tc>2.5°C, 100% for Tc<0°C if (pp_ice>99) pp_ice=99; end if (pp_ice<0) pp_ice=0; end

[0129] The ice measurement with 1310nm absorption is unreliable and is advantageously replaced by pyrometer measurement, the 1550nm absorption occurs equally for water and ice, 1310nm effects are only seen for ice, but are significantly less pronounced than water detection with 1550nm.

[0130] The pyrometer detects in the 2..10 pm range, thermal infrared, whereas 1550 nm is considered mid-IR. % if(yrm>yr2) % Is ice present? Only lasers! if(pice>50) % Early fusion: ice based on the pyrometer psno=((lrm-lr1 )*100) / (lr2-lr1 ); % Consider snow vs. ice pdry=0; % simplified! pwet=0; % simplified!

[0131] Probabilities for dry, wet, ice, and snow are determined directly from xrm (water absorption at 1550nm vs. 980nm), Tc (corrected road temperature from the pyrometer), and Irm (surface brightness). These are determined sequentially in a decision tree and in that order.

[0132] The fundamentals are the vibrational modes of the water molecule and blackbody radiation, as well as the physics of direct and indirect semiconductors. Measurement at these wavelengths in the IR is suitable, but not possible with silicon, because silicon becomes "transparent" above 1000 nm, meaning it is no longer suitable as a photodiode. Germanium detects 1550 nm, but produces very high noise levels if not cooled. InGaAs works just as well as germanium at 1550 nm, but no cooling is required. Surface coating improves quantum efficiency, but this coating is a very cost-intensive technique.

[0133] A complete data-driven ecosystem, for example, includes the following steps, which are carried out cyclically:

[0134] Test data is continuously collected using a fleet of test vehicles equipped with an OTA-updatable telematics unit (e.g., in accordance with the 5G mobile communications standard). An ADCU uses a pre-trained machine learning system to continuously determine AD or ADAS-relevant data for automated or assisted vehicle control from the collected sensor data (open loop testing). In the event of a trigger (e.g., anomalies; for example, the driver behaves differently than predicted by the ADCII), data is transmitted to the server unit / cloud. In the cloud, the data is checked to see whether it is better training data than that already available. For this purpose, the predicted anomalies (or corner cases) can be verified, or more relevant data can be selected for other reasons. The optimized training data can be used to retrain the machine learning system orNeural networks are developed using data aggregation (“you let the data decide”) or active learning (although the human labeling effort is reduced). Actual improvements are verified by increasing KPIs (such as precision and / or recall). The neural network is then extensively tested in simulation worlds, test scenarios, corner cases or rare cases to increase safety. If successful, the trained neural network is released and can be transmitted with a cryptographic signature as an over-the-air update to the vehicles of the test fleet and imported as an update for the ADCU.

[0135] For continuous improvement, closed-loop training in a data-driven ecosystem is necessary. This not only increases the data set, but also the relevance of the training data. Severely underrepresented classes are transferred from images of the same situation to other locations using style transfer, so that the situation is learned not from the location, but from the appearance of the road. By time-shifting and projecting the trajectory of the measuring point into the image, the measurement can be assigned to a specific location on the road. Bootstrapping, i.e. applying the same rules to all pixels containing the road (another network), allows the entire distribution on the road to be measured, proactively and thus saving time, which benefits necessary adjustments to driving behavior.

Claims

Patent claims 1. A method for training a machine learning system (16) for road condition monitoring, comprising the steps: - providing or recording data by a sensor system (1) of a vehicle, wherein the sensor system detects an environment of the vehicle, as training input data (X), - providing or recording data characterising the road condition by a reference sensor (5) mounted in or on the vehicle as training target values ​​(Y), and - Training the machine learning system (16), wherein training data (X, Y) comprising training input data (X) and training target values ​​(Y) corresponding to said training input data (X) are provided, and parameters of the machine learning system (16) are adapted by means of the training data (X, Y) such that the machine learning system (16) generates output data (Y') similar to the training target values ​​(Y) upon input of the training input data (X).

2. The method according to claim 1, wherein the sensor system (1) comprises a camera system of the vehicle, so that the provided or recorded data are image data and the image data serve as training input data (X).

3. The method according to claim 1 or 2, wherein the reference sensor (5) comprises a transmitting and receiving unit which emits electromagnetic radiation (506) of at least one defined wavelength onto the road (510) and receives and measures the intensity reflected by the road (510), and wherein the reference sensor (5) is configured to indicate probabilities for the presence of different classes of road conditions on the basis of the measured values.

4. The method according to claim 3, wherein the reference sensor (5) outputs probabilities for the presence of the following classes of road conditions: , dry", "wet", "snow", "ice" and "unknown / error".

5. Method according to one of the preceding claims, wherein the reference sensor (5) comprises a pyrometer which measures the temperature of the road.

6. Method according to one of claims 3, 4 or 5, wherein the reference sensor (5) uses the following wavelengths: 1550 nm and 980 nm to detect the presence of water by comparing the reflected intensities and a wavelength in the range of 2 to 10 pm to measure the temperature of the road.

7. Method according to one of the preceding claims, wherein the machine learning system (16) is a neural network, in particular a convolutional neural network.

8. Method according to one of the preceding claims, wherein the vehicle (2) comprises a data transmission unit (18), and the data transmission unit (18) is configured to transmit training data (X, Y) to a server unit (20).

9. The method according to claim 8, wherein the server unit (20) is configured to update a training data set comprising a quantity of training data (X, Y), wherein the size of the training data set is kept constant, and wherein it is ensured on the basis of one or more quality criteria that the relevance of the training data (X, Y) with regard to the road condition determination is increased when the training data set is updated.

10. The method according to claim 9, wherein the server unit (20) is configured to achieve a balance of the training data set during the update, so that the training data set has a high diversity, wherein rarer road conditions are sufficiently represented by training data (X, Y).

11. Method according to claim 9 or 10, wherein the relevance of the training data (X, Y) is assessed such that a high relevance is attributed to the training data (X, Y) which are orthogonal to the other training data (X, Y) already contained in the training data set.

12. Method according to one of the preceding claims, wherein the sensor data for the detected area of ​​the road (510) are divided into segments and the provided or recorded data of the reference sensor (5) characterize the road condition for the segments 13. A method for road condition monitoring using a machine learning system (16), wherein the machine learning system (16) has been trained according to a method according to one of claims 1 to 12, wherein data acquired by a vehicle sensor system (1) which detects an environment of the vehicle (2) is provided to the machine learning system (16) as input data (X), and the machine learning system (5) generates output data (Y') characterizing the road condition from the input data (X).

14. Road condition monitoring system (10) comprising an input unit (12) for receiving input data (X); a computer unit (14) configured to carry out a method according to claim 13; and an output unit (18) for outputting the output data (Y') generated by the computer unit (14).

15. A vehicle (2) comprising a sensor system (1), wherein the sensor system is configured to detect an environment of the vehicle (2) and to provide the detected sensor data to the input unit (1) as input data (X), and a road condition monitoring system (10) according to claim 14.

Images