Method and system for monitoring road surface conditions by a machine learning system, and method and system for training a machine learning system

Abstract

The present invention relates to a method and a system for monitoring a road surface condition by a machine learning system (16), and a method and a system (10) for training the machine learning system (16). A method for training a machine learning system (16) for monitoring the road condition includes the following steps: - providing or capturing data by a sensor system (1) of a vehicle (2), wherein the sensor system (1) captures the surroundings of the vehicle (2) as training input data X; - providing or capturing data symbolizing the road condition as training target values Y by a reference sensor (5) provided in or on the vehicle, i.e., in the vehicle (2); and - training the machine learning system (16). Provided is training data including the training input data X and the training target values Y corresponding to the training input data X. Subsequently, parameters of the machine learning system are adjusted such that when the training input data is input, the machine learning system creates output data Y' similar to the training target values Y.

Description

Technical Field

[0001] The present invention relates to road condition monitoring (Road Condition Monitoring, Road Condition Observation) or road condition recognition from a moving vehicle equipped with sensors for capturing the surroundings. This theme is very meaningful from the perspective of accident avoidance because on certain road conditions or road surfaces, the friction or coefficient of friction between the tire and the road surface is reduced. That is, the braking distance may be extended, or the vehicle may slip or spin during braking, which can lead to serious accidents. In automated intervention and / or braking intervention (ADAS / AD system) in assisted or autonomous driving, the system must be able to estimate the road condition as realistically as possible before the vehicle reaches that point. However, since this has not yet been reliably achieved, experiments on autonomous driving are often carried out in clear areas without the worry of snow or ice.

[0002] To better recognize road conditions, machine learning should be adopted in an innovative setup. The present invention particularly relates to a method for training a machine learning system for monitoring road conditions, a method for monitoring road conditions using the trained machine learning system, a system for monitoring road conditions, and a vehicle equipped with the system for monitoring road conditions.

Background Art

[0003] Among others, a YouTube video of a method for segmenting road conditions based on the driving sound of the same road collected by a microphone simultaneously with the video data of a camera is known. Here, a dry road surface and a wet road surface can be distinguished: Road condition estimator using cameras and microphones(2018) https: / / www.youtube.com / watch?v=H13jgv55o8w (as of May 25, 2022). Distinguishing between two or more (wet / dry) is difficult or unreliable with this approach.

[0004] WO 2016 / 177372 A1 discloses a method for recognizing and evaluating the influence from the environment of the vehicle periphery and road surface condition information. At least two temporally consecutive digital images in which the same image area is selected respectively are created using a camera. Using an algorithm of digital image processing, a change in image sharpness between the image areas of at least two temporally consecutive images is detected, and at this time, the weighting of the image sharpness change is implemented so as to decrease from the center of the image area toward the outside. According to the recognized change in image sharpness between the image areas of at least two temporally consecutive images, environmental state information is deduced using a machine learning method, and road surface condition information is identified according to the deduced environmental state information.

[0005] WO 2019 / 174682 A1 discloses a method for classifying a road surface condition based on image data of an in-vehicle camera system and an in-vehicle camera system corresponding thereto. The method has the following steps: - A step of providing image data by an in-vehicle camera system configured to be able to depict at least one area outside the vehicle, provided that the area at least partially includes the lane on which the vehicle is traveling; - A step of distinguishing between diffuse reflection and specular reflection of the road surface by evaluating the visual differences of at least one point on the road in at least two images of the camera system, provided that the images are taken from different shooting viewpoints; - A step of determining whether there is an obstacle caused by a road surface covering being rolled up when at least one tire of the vehicle rolls thereon in at least one image of the camera system; - A step of classifying the road surface condition into the following five road surface condition classes in consideration of the results of the reflection type and the obstacle level; a) Dry road surface: The reflection type is diffuse and there is no obstacle b) Normally wet road surface: Reflection type is specular, with obstacles c) Very wet road surface, with risk of hydroplaning: Reflection type is specular, with significant obstacles d) Snow-covered road surface: Reflection type is diffuse, with obstacles, or e) Frozen road surface (ice burn): Reflection type is specular, without obstacles.

[0006] Known problems of camera systems are, for example, when improving the signal-to-noise ratio by long exposure, the interference distance in the dark becomes small, while when the camera system is moving, motion blur occurs.

[0007] DE 102013002333 A1 discloses a method for evaluating a predictive road condition in a vehicle in which a road surface is irradiated by sensor light rays. In this method, the sensor light rays are reflected and absorbed according to the condition of the road surface, and subsequently, the evaluation of the road condition is performed based on the reflected sensor light rays. This method is characterized in that the road surface in front of the traveling direction of the vehicle is irradiated. Various road conditions of the road surface have different optical characteristics, and accordingly, specific wavelengths are absorbed and other wavelengths act so as to be reflected, so that the road condition of the irradiated road surface can be determined from the reflected sensor light rays. As an example, the wavelength 1550 nm that is relatively strongly absorbed by ice can be mentioned.

[0008] DE 102014214243 A1 discloses a method for identifying a road condition. Here, road condition data of a weather map and / or a road map is / are referred to for identifying the road condition, and the road condition data obtained from the weather map and / or the road map is redigitized.

[0009] DE 102017223510 A1 discloses an optical sensor for evaluating a surface including: - At least one irradiation unit for irradiating a modulated optical signal to the surface to be evaluated; - At least one detection unit for capturing an optical signal reflected from the surface of a modulated optical signal and providing an electrical output signal that depends on the reflected optical signal; and, - Control means for evaluating the output signal of at least one detection unit and determining the state of the surface to be evaluated, provided that here, the frequency spectrum of the modulated optical signal irradiated from the at least one irradiation unit is substantially distinct from the frequency spectrum of the interference spectrum.

[0010] However, all known methods have drawbacks and do not fully solve the related problems. Of particular interest is the predictive and reliable detection of "black ice", which is difficult for humans to see, i.e., a black road covered with (frozen) ice.

Prior Art Documents

Patent Documents

[0011]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Summary of the Invention

Problems to be Solved by the Invention

[0012] Therefore, an object of the present invention is to provide a method that enables a predictive recognition of the road surface condition at a reasonable price without impairing the robustness of the detection of road surface condition types related to safety.

Means for Solving the Problems

[0013] The above problem is achieved by the subject matter recited in the independent claims. Developments of the invention are indicated by the dependent claims as well as the following description.

[0014] A first starting point is to use the surrounding capture sensors already installed in the vehicle for other purposes, in particular to capture the area located in front of the vehicle's direction of travel.

[0015] The inadequacies of existing surrounding capture sensors can be compensated for by using specialized reference sensors that are, for that reason, too expensive for mass production use in many cases and also contributing to the generation of training data for a machine learning system for road condition identification. To generate the training data, in many cases, a fleet of test vehicles equipped with surrounding capture sensors is introduced for later mass production use. The captured data is usually manually labeled to provide the actual road conditions for training the machine learning system. Manual labeling is performed by humans, but human perception of road conditions is subjective. For example, at what level is the road "wet"? Is there an old ice layer under the visible fresh snow cover? Is that black road covered with ice or just a dry black road? In addition, manual labeling is time-consuming and costly.

[0016] A second starting point is to use an explicit road condition identification sensor with high reliability and accuracy as a reference sensor. Here, it is advantageous if a (pure) road condition identification sensor can predictively identify whether there is water on the road surface and the temperature of the road surface. Transmission and reception means operating in the wavelength range of 1550 nm and approximately 2 to 10 micrometers can identify both.

[0017] One aspect of the invention relates to a method for training a machine learning system for monitoring road conditions, which includes the following steps: - Providing or capturing data by a sensor system of a vehicle, provided that the sensor system captures a peripheral part of the vehicle as training input data. - Providing or capturing data symbolizing a road condition as a training target value by a reference sensor provided in or on the vehicle, and - Training a machine learning system. Provided that training data including the training input data and the training target values corresponding to these training input data is provided, and subsequently, The parameters of the machine learning system are adjusted by the training data such that when the training input data is input, the machine learning system creates output data similar to the training target values.

[0018] The data (sensor data) of the sensor system can include, for example, image data, radar data, and / or lidar data.

[0019] Using this training data, a machine learning system is trained, i.e., the parameters of the machine learning system are adjusted such that when the training input data is input, the machine learning system creates output data similar to the training target value. The parameters include, for example, in the case of a neural network, the weights between individual input values and neurons. The training of the machine learning system is then carried out by the learning method being monitored, and many learning methods are known. And, for example, the backpropagation method can be used for training a neural network. During training, the parameters of the machine learning system are adjusted such that the error between the output data and the training target value is minimized as much as possible. The error between the output data and the training target value is then determined, for example, via the distance between the output data and the training target value and a metric for the corresponding output data. At that time, care must be taken not to overfit, which can be obtained, for example, by examining the error between the output data created from the test input values and the test target values belonging thereto. The test target values are then assigned to the test input values, and the test input values and the test target values are not used for adjusting the parameters of the machine learning system.

[0020] Optionally, when the training ends successfully, for example, if the output data sufficiently satisfies the similarity to the training target value that can be set by the threshold of the similarity metric, the parameters of the machine system can be output.

[0021] In one embodiment, the sensor system includes a camera system of a vehicle, the provided or captured data is image data, and these image data function as training input data. The reference sensor defines reference data (labels) as training target values for the image data.

[0022] The camera system can be, for example, a monocular camera disposed inside the vehicle, preferably behind the windshield, whereby a front area of the vehicle corresponding to the visual recognition of the vehicle driver can be captured. Alternatively, the camera system can be a stereo camera capable of providing depth information of the vehicle surroundings, or a satellite camera system including fisheye cameras directed in different directions of a plurality of vehicles, for example, a surround view camera system.

[0023] By subsequent processing of the image data by the machine learning system and the reference sensor data (the "AI camera"), the capabilities of the reference sensors regarding road condition monitoring are imparted to the camera system by a suitable learning method.

[0024] According to an embodiment, the reference sensor includes a transceiver that irradiates the road with electromagnetic rays of at least one defined wavelength, receives and measures the intensity reflected from the road, and the reference sensor is configured to provide the probability of the presence of each road condition class based on the measurement values. The presence of water on the road can be determined by measuring the absorption of the mid-infrared wavelength of the discontinuous infrared. Regarding the water on the road, robust detection is possible by comparing (ratio measurement) the measurement results of two suitable wavelengths.

[0025] A reference sensor suitable for representing ground-truth data is one that can detect all target learning classes of the camera system regardless of the time-dependent light distribution (brightness).

[0026] This is a prerequisite for the learning result of the camera system (as a sensor system) to be ideally adjustable to very different light distributions (brightness) depending on the time of day by the exposure control of the camera.

[0027] Thus, although a highly sensitive camera is required as a premise, it can be ensured that the road surface condition can be recognized not only during the day but also in the early morning, evening, and at night.

[0028] Furthermore, in the reference sensor and the camera system, it is advantageous that the observation wavelength ranges do not match. That is, the camera performs observation in the visible light region, and the reference sensor performs observation, for example, in the infrared range.

[0029] In one embodiment, motion blur (blurring due to movement) in the image caused by various driving speeds and the set exposure time is also a learning target. At this time, both parameters can be reflected not only within the class but also in the learning result by linear approximation.

[0030] At this time, the ground speed is advantageous as a useful additional input value.

[0031] In one example, weather data and driving dynamics data such as ABS, ASR, and / or ESC control intervention of the ego vehicle are also assumed to be advantageously reflected in the learning result. This case can be said to be an early fusion of labeling variables.

[0032] Additionally, or alternatively, according to one embodiment, in a vehicle driven by a person, in addition to the above speed, vehicle acceleration in the longitudinal and lateral directions, or alternatively, the driving style is also assumed to be considered based on the safe distance to the vehicle traveling ahead.

[0033] In one embodiment, the reference sensor considers the probabilities for (at least) the following classes of road conditions: "Dry", "Wet", "Snow", "Ice", and "Unknown / Error". Examples of errors when using the camera sensor as the sensor system include mainly black or white images, or overall image noise due to insufficient exposure.

[0034] According to one embodiment, the reference sensor includes a pyrometer for measuring the temperature of the road. For example, the reference sensor can measure thermal (far) infrared in the range of, for example, 2 to 10 μm in order to derive the surface temperature from the thermal radiation. The thermal characteristics in the far infrared (IR) are invisible in the visible spectrum, in contrast, making it very difficult for people to recognize black ice.

[0035] In one embodiment, the reference sensor uses the following wavelengths: 1550 nm and 980 nm are added to detect the presence of water by comparing the reflected intensities, and To measure the temperature of the road, wavelengths in the range of 2 to 10 μm are measured.

[0036] In one example, the machine learning system is a neural net (work). Neural nets are particularly suitable for the above method because they are easily adjustable.

[0037] Here, the neural net is particularly preferably a convolutional neural network. The non-linearity of the convolutional neural network does not impair the usefulness of the above method.

[0038] The above method can also be used in combination with other machine learning systems such as, for example, decision trees, support vector machines, regression analysis, or Bayesian networks. Furthermore, the above method can also be used in a multi-task classification system. The multi-task classification system includes, for example, an encoder and a plurality of decoders. Furthermore, it is also possible to divide the machine learning system into a plurality of subsystems. In that case, each subsystem has the functions of the machine learning system described here, but each subsystem is different from each other, for example, in terms of the type of machine learning system or the selection of training data. Therefore, the output data created using the subsystems can be combined to obtain an improved output.

[0039] According to an embodiment, the vehicle includes a data transmission unit configured to transmit training data to a server unit (backbone). This enables the development of a data driven ecosystem (DDE) for developing a machine learning system, such as identifying road conditions based on sensor data from common peripheral sensors.

[0040] In an example, the server unit can update a training dataset that includes a (pre - given) set of training data, provided that the size of the training dataset is kept constant and that the relevance of the training data for road condition identification is guaranteed to be enhanced when updating the training dataset according to one or more quality criteria (e.g., the quality is evaluated by multiple KPIs).

[0041] In this case, in order to achieve high reliability even in unknown scenarios, in an example, the training is performed in a DDE where the relevance of data is continuously improved. In the backend of the DDE, automatic methods (DAgger) based on Key Performance Indicators (KPIs) and human judgment with significantly reduced effort (active learning) are also utilized.

[0042] The DDE is designed such that the size of the dataset is kept constant after reaching the limit. This ensures that the training time for the machine learning system does not increase further. The computing power of the backend is used to enhance the relevance of the training dataset.

[0043] According to an embodiment, the server unit is configured to ensure the balance of the training dataset during update so that the training dataset has a high diversity, so that rare road conditions can also be sufficiently described by the training data. For example, black ice, that is, a rare situation such as a black road with a frozen surface, is very rare but also very dangerous because the friction coefficient is extremely reduced.

[0044] In one example, the relevance of the training data is evaluated based on a way that gives high relevance to the training data that is orthogonal to the training data already included in other training datasets.

[0045] In one embodiment, the sensor data of the captured area of the road is segmented, and the road condition of each segment is characterized by the data provided by or captured by the reference sensor.

[0046] Due to the high regional resolution, the segmentation of the sensor data of the road ahead is enabled by the recognition rules condensed in the network, and as a result, predictive and high-resolution measurements are realized. Thereby, the response time is shortened, and at the same time, the cost of the application can be reduced by up to two orders of magnitude compared with the reference measurement technology.

[0047] A further aspect is a machine learning system is trained as described above, data captured by a vehicle sensor system that captures the periphery of the vehicle is provided as input data to the machine learning system, and the machine learning system creates output data characterizing the road condition from the input data relates to a method for monitoring the road condition using a machine learning system.

[0048] A third aspect is an input unit for receiving input data; A computing unit configured to implement the above method for monitoring road conditions; and, An output unit for outputting output data created by the computing unit relates to a road condition monitoring system including the same.

[0049] A further aspect is a sensor system, wherein the sensor system captures the periphery of the vehicle and provides the captured sensor data as input data to the input unit; and relates to a vehicle including the above road condition monitoring system as described above.

[0050] When constructing a training dataset (e.g., within a server unit), the following problems may occur: a) A large number of similar data are generated b) There are rare classes c) The classes are highly correlated with the test area, so ice is detected more frequently in Sweden, water is detected more frequently in Germany, and dry conditions are detected more frequently in southern Spain. Due to such correlations with the area, it is possible that the state of the area rather than the road conditions is erroneously learned. d) However, rare classes are also important. Rare classes are substantially ignored, but these are precisely the very dangerous cases. The wavelength ranges for observation of the reference sensor and the camera do not match. The reference sensor observes discrete wavelengths of mid-infrared light, for example, to detect absorption by moisture or ice, and thermal (far) infrared light to derive the surface temperature from thermal radiation. The camera observes light in the visible wavelength range. Since the thermal features in the far-infrared (IR) are invisible in the visible spectrum, it is very difficult for humans to recognize black ice. Instead, the state is derived by observing the environment and other optical features. That is, although it is in principle possible to distinguish the road condition into four to five classes optically with a camera, road condition recognition relying only on the camera optical method is not sufficiently robust because a very large number of examples must be learned.

[0051] To solve such problems, the data-driven ecosystem provides the following measures:

[0052] 1) Policy aggregation: When training a random decision forest, instead of using only one decision tree, the so-called "bag of classifiers" method that uses a "bag full" of decision trees is adopted. This enables majority voting, and as the learning progresses, random variants of multiple existing successful decision trees are generated and added, while those that are not successful are discarded. Finally, very good results can be obtained by majority voting with a large number of good classifiers. This method can be well applied by calculating individual classifiers conservatively. This applies to the case of decision trees with a small number of YES / NO decisions. This is an evolutionary method where gradients are not calculated and backpropagation is not performed.

[0053] 2) Backpropagation: The latest neural networks learn using backpropagation from the difference between the desired output and the random output obtained after the initial initialization. Using the chain rule, the difference (gradient) is backpropagated through the layers of the network, which determines how the weights and offsets within the network should be changed to reach a better solution. In backpropagation, repeatedly using redundant data is obstructive because irrelevant information is input. That is, in such so-called "Overfitting," specific, and in some cases very similar cases are over-learned as if they were memorized exactly, and slightly different test data is not recognized as well as the training data.

[0054] 3) Data splitting: To address overfitting, it is effective to split the data on the recognition side into training data and test data, and the KPIs (Key Performance Indicators) to be measured, such as the precision and accuracy of the output for the test data, must be equal to or better than those for the training data. However, if the training data and the test data are very close, for example, adjacent images of the same sequence, the problem cannot be solved. This should be avoided.

[0055] 4) Batch Normalization: Since it is assumed that not only the input data and output data but also the data in the intermediate layers are evenly distributed without offsets, the neurons in the network operate based on the simple assumption that a linear combination with the next layer is sufficient. This can be achieved by per-layer renormalization (Ioffe and Szegedy, Google).

[0056] 5) Dropout: In the learning process, there is a possibility that a block of non-functional neurons may form within the network. The goal is for each sub-region within the network to contribute equally to classification. Particularly difficult are the problems of vanishing gradients and exploding gradients. That is, when backpropagating using the chain rule, the problem where the requirement for a neuron to decrease its weights becomes exponentially smaller or exponentially larger for each layer. To avoid this, in each learning step, approximately 50% of the neurons are randomly selected and deactivated (dropout). For the remaining 50%, the signal intensity is adjusted, and only these are trained in this step. In the next iterative training step, another 50% of the neurons are selected so as to force each subset of the network to have the same function, but as a whole network, only the performance improves according to its size, and since the associated weights remain approximately zero, almost no unconnected neurons remain.

[0057] 6) Regularization: This method would be most suitable to be likened to depth measurement in a crater. The deepest point is the best solution, but the crater does not have a simple saucer shape and contains mountains and further craters within it. It is actually a crater field consisting of overlapping craters. For example, if one simply looks for the deepest point, there is a risk that, like a sphere thrown into the field, the sphere will remain at a local minimum, i.e., not the optimal solution will be found, but only the locally closest solution will be determined. Regularization means filling the crater topography with adhering powder. This, at least after solidification, fills small craters and reduces the possibility of remaining at local minima. However, regularization must be used very sparingly. Figuratively speaking, filling the crater topography completely and flattening all differences would be meaningless. However, when used in a very small promille range, regularization has the effect of suppressing the influence from specific recognition features and finding basic rules that are more generally applicable. However, as a result, there is a cost of a seemingly decreased accuracy. Therefore, simple KPIs show inferior results even though they are actually better classified.

[0058] 7) Activations: Activations can be calculated by a kind of visual backpropagation on the GPU in parallel with the actual neural network. They indicate which neurons are involved in the decision in the convoluted layers (rather than the fully connected layers) and project this onto the input layer of the network. That is, by overlaying the activation with the input image to which it is related, one can evaluate which features were used in the classification decision. This helps to understand what was the problem when something was misdetected and provides valuable clues regarding potentially missing training examples. This provides visual clues in addition to the simple observation of KPIs.

[0059] 8) Key Performance Indicators (KPIs): In a simple two-class detector, there are four cases as all combinations - respond, not respond, report correctly, or report incorrectly. When detected, the recall (probability) is high. For example, a scene can be gradually darkened until the image finally becomes black. If the output is an image again and not just a class, pixels representing recognized known features can be counted. Additionally, other pixels not belonging to the known features may also respond. In this case, it is a false positive. In short, it relates to two other issues, namely, recognition and the accuracy of the results. These can be plotted in a graph or summarized in an F2 score. Briefly speaking, a KPI is a pre-defined metric that indicates how well the network is trained across the entire test dataset.

[0060] 9) Data aggregation: This is very similar to policy aggregation. The difference is, for example, instead of calculating 500 neural networks and adjusting their outputs by majority vote, it moves the problem at runtime to the training time and focuses on the selection of relevant data. This way, only one network needs to be calculated. Here, start with a subset of the training data, say 5% of it, and examine in the next 5% of the dataset which images have already been correctly classified by the first 5% of the training. These images are not needed at all. It can be said to be similar to the method of exam preparation where one only learns what is not yet understood. Data aggregation is for the machine to determine which data is relevant, and these are data orthogonal to all other data that has already been learned.

[0061] 10) Active Learning: Sometimes, data may be mislabeled. Since an image cannot be output as both "dry" and "wet" at the same time, the network will notice this error during data aggregation. That is, the image is incorrect. But which one? In such cases, humans ultimately need to make a judgment, but since the process is highly automated, only the contentious cases need to be judged.

[0062] 11) The learning time for a certain size of the dataset does not constantly increase. DDE independently rejudges whether the data is relevant. When the dataset finally reaches the target size, the data aggregation can also attempt to exchange new data with old data based on the relevance evaluation. At that time, the quality is constantly measured using multiple KPIs. If the new answer as a newly learned neural network is below the average, it is discarded, and if it is above the average, it is prioritized.

[0063] 12) Data Driven Ecosystem: This is a continuous challenge to obtain the best-ever computed network by improving, after training, through the exchange with more relevant training data in the backend, i.e., in the functions that are later calculated in the cloud. This highly depends on the test dataset, which needs to be properly adjusted like the training dataset. It should include not only relevant cases in all surrounding situations but also rare cases and corner cases.

[0064] 13) Synthetic data (derived from real data): By definition, rare cases are rare. To ensure that these are not underestimated, synthetic data can be derived from normal data using GAN style transfer. In short, the synthetic data answers the question of what the scenery looks like in different weather conditions, or what the weather conditions look like in different locations. While images of many locations can be taken, images of rare conditions are limited, so the conditions are transferred to other locations.

[0065] 14) Time-shifted evaluation depending on speed: What can be seen by the front camera is located on the road ahead of the current measurement point. To ensure accuracy, it is necessary to shift the time to match the trajectory on the road at the measurement point with the image of the road. Conversely, it is also possible to calculate segmentation instead of category classification. Additionally, if a network that recognizes the position of the road in the image is also used, the road can be segmented, learned, and displayed according to the road conditions.

[0066] 15) The average value tells more than individual values: On the inference side, i.e., in the application of the network, the calculation of the average value creates possibilities for improvement and correction of outliers. For example, while it is not considered that the overall weather conditions change every second, regarding the road conditions, it is a phenomenon that can change at spatial boundaries, such as when coming out of a garage, passing through a tunnel, or crossing a bridge. In the above examples, particularly large changes can occur. If these can be recognized, the results can be stabilized by averaging in all other events.

[0067] 16) Testing of the network in synthetic scenes and driving tests: To cover standard cases, as well as known rare cases and corner cases, additional tests are performed on the normal training data to confirm the validity of the behavior. Passing in one test case is the condition.

[0068] 17) Ensuring transmission reliability: In the simplest case, updates are performed via a USB stick. However, an update function from vehicle fleet delivery to end-of-life is also suitable for the purpose. To avoid transmission errors and tampering in a complete DDE, checksums, cryptographic signatures, and encryption are added to the actual data.

[0069] 18) Updates: In fleet applications, the data is preferably transmitted wirelessly (OTA, Over The Air). In the future, it is assumed that vehicles will be equipped with SIM cards and can establish a tunneled Internet connection to the update server. Needless to say, conventional updates via USB sticks, file systems, etc. are always possible.

[0070] 19) Triggering of related data: In particular, from Tesla patents, it is known that rare cases and corner cases, i.e., rare road weather conditions in this case, should be recorded. In Tesla, the trigger itself is also a network trained to respond to specific scenarios. This technology could also be adopted here to improve the product in a data-driven manner in a second continuous cycle. It is also conceivable to perform triggering only in a small number of measurement vehicles equipped with reference sensors. However, since the system can only be automatically scaled by a large number of application cases, there are clear advantages to triggering in all vehicles within the fleet.

[0071] 20) Return channel: Wireless (OTA, Over The Air) is preferred, but it is also possible via USB or other bus systems in a maintenance factory as before. The OTA update channel for vehicles provides the advantage that vehicle manufacturers do not need to recall when software updates are required. Furthermore, a return channel for continuous product improvement is also beneficial. At this time, the triggered moment, and other moments selected by, for example, male and female test drivers or male and female drivers, and the corresponding measured values are fed back. As for the data volume, up to several hundred megabytes per manufacturer for various sensors is a realistic scale. That is, much more data is available than the data required to input and train into the corresponding sensor network. Therefore, the method for selecting training data and the method for evaluating the network trained using KPIs are very important.

[0072] 21) Virtualization, connection and sourcing of the ecosystem: All of these are implemented in the back-end, preferably in the cloud. It is possible to virtualize the entire back-end, that is, migrate to Docker-Containers and operate on a large-scale cloud instance with sufficient performance. This has the advantage that post-processing can be scaled according to the size of the fleet. In short, a data connection is established between the fleet, the provider and the cloud provider. This is independent of the place where data is curated and new algorithms are integrated. It is preferable that access to the cloud is easily possible from anywhere in the world. The same applies to data deployment.

[0073] 22) Design of anonymization and data protection without side effects: Based on local data protection regulations, it is necessary to consider source anonymization. If no data that can identify individuals is stored, there is no violation of the law in the concept of cloud processing and data curation that can be used worldwide. However, it must be confirmed that anonymization does not have unintended effects on the main function. However, when observing road conditions, anonymized faces and anonymized license plates should not be a problem, except for cases where, for example, the license plate is very large in the image and the label is switched from a dark background to a bright background. Extreme white tones in the image can be an indicator of snow, but it may be misjudged as having a higher probability, for example. Similar side effects should be tested and avoided methodologically (for example, for anonymization, only use the background color).

[0074] Adjustment of driving behavior according to road conditions: The purpose of all these efforts is simply to achieve appropriate driving behavior in measuring road conditions. Male / female drivers should not expect themselves to report on road conditions every second, believing that they have sufficient expertise. Warnings should be issued in cases of freezing, i.e., when the friction coefficient is significantly reduced. Since the actual friction coefficient depends on the road and the tires, it can only be estimated. The main use of road condition measurement is highly automated driving, because the driving plan must also take into account the road conditions. In some cases, driving at a low speed is necessary, and furthermore, understeering may be expected and the braking distance may also become longer. Even what people take for granted must be taught to machines in a technical way. At that time, it is advantageous that the cost-effectiveness of the technical method is high.

[0075] The significance of monitoring road conditions lies in the field of use for warning drivers of slippery conditions (especially black ice), but the field of use for highly automated driving is even more significant. Here, assumptions regarding braking distances must be constantly updated according to the weather conditions of the road, whereby the driving behavior is adapted to the weather conditions.

[0076] The average value tells more than individual values: Even on the inference side, i.e., in the application of the network, calculating the average value creates possibilities for improvement and correction of outliers. For example, although it is not considered that the overall weather conditions change every second, regarding road conditions, it is a phenomenon that can change at spatial boundaries, such as when exiting a garage, when passing through a tunnel, or while crossing a bridge. In the above examples, particularly large changes can occur. If these can be recognized, in all other events, the results can be stabilized by averaging.

[0077] Time-shifted evaluation depending on speed: What can be seen with the front camera is located on the road ahead of the current measurement point. To ensure accuracy, it is necessary to shift the time to match the trajectory on the road at the measurement point with the image of the road. Conversely, it is also possible to calculate segmentation instead of category classification. Additionally, if a network that recognizes the position of the road in the image is also used, the road can be segmented, learned, and displayed according to the road conditions.

[0078] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Description of the drawings:

Brief Description of the Drawings

[0079]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

DETAILED DESCRIPTION OF THE INVENTION

[0080] FIG. 1 schematically shows a top view of a vehicle 2. The vehicle 2 is equipped with a sensor system 1 for capturing the surroundings, a reference sensor 5, and a unit 10 for performing processing. The sensor system 1 for capturing the surroundings can be, for example, an image capturing means, or can include these. The image capturing means can be a front camera of the vehicle. The front camera is disposed inside the vehicle 2, for example, near the rearview mirror, and can capture the peripheral part in front of the traveling direction of the vehicle 2 through the front glass of the vehicle 2. Based on the signal or image data of the front camera, details of the peripheral part of the vehicle 2, for example, an object can be detected. Based on these peripheral detections, for example, ADAS functions or AD functions such as lane recognition, lane keeping support, road sign recognition, speed limit assistant, traffic participant recognition, collision warning, emergency brake assistant, inter-vehicle distance maintenance following control, construction site assistant, autobahn pilot (highway autopilot), cruising driver function, and / or autopilot are provided by an ADAS / AD control unit. The image capturing means typically includes an optical system or a lens and an image capturing sensor, for example, a CMOS sensor.

[0081] The proposed peripheral capture sensor system 1 here assumes a sensor setup for vehicles in the context of assisted and autonomous driving. This can optionally be extended to a multi-sensor setup. The multi-sensor system has the advantage that it can enhance the safety of detection algorithms for road traffic by verifying detections by multiple sensors. The multi-sensor system can be, for example, any combination of the following candidates: - One or more cameras, - One or more radars - One or more ultrasonic systems - One or more lidars, and / or - One or more microphones.

[0082] As the reference sensor 5, a dedicated road condition identification sensor with high reliability and accuracy is suitable. Here, it is advantageous if the road condition identification sensor can predictively identify whether there is water on the road surface and, optionally, the temperature of the road surface.

[0083] A transceiver unit that irradiates the road with electromagnetic rays of at least one defined wavelength, receives the intensity reflected from the road, and measures it is suitable for this application. The reference sensor is set to indicate the probability of the presence of each road condition class based on the measurement values. The presence of water on the road can be determined by measuring the absorption of the mid-infrared wavelength of discontinuous infrared. Regarding water on the road, robust detection is possible by comparing (ratio measurement) the measurement results of two suitable wavelengths. For example, the reference sensor can use the following wavelengths: Wavelengths 1550 nm and 980 nm for comparing each reflection intensity and detecting the presence of water.

[0084] Transmitting and receiving means operating in the wavelength range of 2 to 10 micrometers can determine the road surface temperature. Alternatively, it is also possible to use an additional pyrometer that can measure the temperature of the surface at a defined distance for temperature measurement.

[0085] A vehicle 2 equipped with a reference sensor 5 and a sensor system 1 for capturing the surroundings is a test vehicle for generating training data. The corresponding vehicle equipped with the sensor system 1 for capturing the surroundings and the trained machine learning system 16 can be used as a mass-produced vehicle without using the costly reference sensor 5.

[0086] FIG. 2 shows a depiction of a machine learning system 16 that is trained based on the data X of the surrounding capture sensor system 1 and the corresponding data Y of the reference sensor 5 to generate output data Y' that characterizes the road conditions.

[0087] As the machine learning system 16, an artificial neural network (network) can be adopted. Since the neural network is easily adjustable, it is particularly suitable for the above method.

[0088] Here, the neural network is particularly preferably a convolutional neural network.

[0089] Alternatively, further forms of the machine learning system 16, for example, decision tree learning, support vector machine, regression analysis, or Bayesian network, etc. can be adopted. Furthermore, it is also possible to use a multi-task classification system. The multi-task classification system includes, for example, an encoder and a plurality of decoders. Furthermore, it is also possible to divide the machine learning system into a plurality of subsystems. In that case, each subsystem has the functions of the machine learning system 16 described here, but each subsystem is different from each other in terms of, for example, the type of machine learning system or the selection of training data. Therefore, the output data created using the subsystems can be combined to obtain an improved output.

[0090] The training of the machine learning system is performed using supervised learning. The machine learning system 16 is trained based on training data (input data X_1, X_2, ..., X_n captured by the surrounding sensors 1 and corresponding target output data Y_1, Y_2, …, Y_n determined by the reference sensor 5). By adjusting the weights (or parameters) of the machine learning system 16, an error function indicating the deviation between the output Y‘_1, Y‘_2, …, Y‘_n of the machine learning system for the input data X_1, X_2, …, X_n and the corresponding target output data Y_1, Y_2, …, Y_n is minimized.

[0091] Figure 3 shows a trained machine learning system 16 that can generate output data Y‘ characterizing the road condition from the (newly obtained) data X of the surrounding capture sensor 1. The trained machine learning system 16 is used, for example, in mass-produced vehicles.

[0092] Figure 4 schematically shows an example of a road condition monitoring system 10 that can determine road condition data and transmit it to the server unit 20. The road condition monitoring system 10 is electrically or wirelessly connected to at least one surrounding capture sensor 1 inside the vehicle 2, for example, an image capturing means.

[0093] The data or signals captured by the peripheral capture sensor 1 are transmitted to the input interface 12 of the road condition monitoring system 10. The data is processed by a processing unit (or data processor) 14 within the road condition monitoring system 10. The processing unit 14 includes a machine learning system 16. The machine learning system 16 can include an artificial neural network such as a CNN trained to classify road conditions, for example. The classified road conditions can be transmitted via the output interface 18 to a further in-vehicle sensor unit (e.g., ADCU, automated driving control unit). The data transmission unit 19 is used for wireless transmission of the data and / or the classified road conditions to the server unit 20 (such as cloud, backbone, infrastructure, etc.).

[0094] So that the neural network can process the data in the vehicle in real time, the road condition monitoring system 10 or the processing unit 14 can include one or more hardware acceleration means for the machine learning system 16 or the neural network.

[0095] Figure 5 shows a vehicle 500 equipped with a reference sensor 505 having a transceiver unit. The general vehicle has a road condition monitoring system 501 equipped with a sensor system 504, a machine learning system 502, and a data transmission 503. The reference sensor 505 is arranged in front of the vehicle, for example, in front of the radiator, and its transceiver unit is set in a beam direction 506 where the angle a between the beam 506 and the road surface is about 70°. The angle a can be set, for example, between 50° and 80° in the case of a specific vehicle 500. The height of the exit of the beam 506 from the reference sensor 505 can be within the range of 20 to 60 centimeters above the road.

[0096] To generate the initial dataset (seed phase), 500 test drives can be performed using vehicle 2 equipped with a camera, GPS, and reference sensors. The GPS is necessary, for example, to compare with weather forecasts on the network.

[0097] In addition to a single image (e.g., one image per second), the evaluation of the reference sensors is also saved. If additional information - speed, acceleration, GPS, etc. - is available, these are also saved with appropriate timestamps.

[0098] If an interesting situation occurs, additionally, manual classification by a person may also be advantageous. This allows the relevant camera images and reference measurements to be rediscovered later within the dataset. With the additional functionality of the reference sensors, it is possible to label the return channel via the CAN output. What is important here is the situation where a person arrives at a different conclusion from the reference sensor.

[0099] For example, additional information ("meta-information") can be saved in a separate classification file with the same name except for the extension (txt, xml, json). In some of the techniques described later, for example, the front camera captures the target section of the road earlier than the measurement points of the reference measurement technique, and a time offset is required between the image and the reference measurement data, so a time reference cannot be omitted either.

[0100] Furthermore, it is desirable to name the camera images and classification files using the best preliminary classification, percentage, date, time, camera name, horizontal field of view, and vehicle combination: D_99_20210507_2023CET_front_90_F-TZ-333.jpg D_99_20210507_2023CET_front_90_F-TZ-333.xml (D_99_20210507_2023CET_front_90_F-TZ-333.txt) (D_99_20210507_2023CET_front_90_F-TZ-333.json) D = dry (dry), (W = wet (moist), I = ice (frozen), S = snow (snow accumulation), E = Error (error)) 99 = 99% means "dry (dry)" 20210507 = YYYYMMDD 2023CET = HHMM + time zone front = front camera (left, right, back) 90 = angle 90° F-TZ-333 = vehicle number (optional) Regarding the format: The txt format is easy to read but difficult to analyze. The json format has the highest machine readability but is difficult to edit. The xml format is in between. It is still possible to edit manually, and the machine readability is also good.

[0101] The purpose of the preliminary classification carried out in advance is to enable sorting by class by name and to provide a very simple means for changing labels. This can occur in the active learning phase described later. If a person overrides it, the data will be preferentially continued to be used, and the label can be adjusted accordingly. This is the only way to resolve conflicts in the dataset.

[0102] Since measurement data will cause problems if there are more than 65535 files per directory unless the typical variable sizes in the Linux system are specially modified, it is desirable to store them in subdirectories for each day.

[0103] Another possible solution is to use a JSON database such as MongoDB instead of using only image files or text files that are easy to read for images and reference data.

[0104] Optionally, it is better to be able to transmit the GPS position of the vehicle as data. The vehicle number and GPS will be omitted in later fleet tests.

[0105] The GPS position helps to select sufficiently different locations, or, if at the same location, different times and weather conditions, especially during the seed phase, i.e., the initial stage of the net calculation.

[0106] Specific example of evaluating data from a reference sensor in the form of pseudocode: %% lrm=(r980m+r1310m+r1550m+r1552m+1)>>2; / / Total luminance % xrm=(((r1550m+r1552m)*1600) / r980m; / / If this is smaller: the proportion of water is large % yrm=((r980m*192) / r1310m); / / If this is larger: the proportion of ice * or * deep water is large % vrm=(r1550m*100 / (r1550m+r1552m)); / / If both wavelengths 1550nm and 1552nm are different, laser contamination may be considered

[0107] Here, r980..r1552 are the intensities of the reflected light at wavelengths in nanometers (nm), measured in units of ADU, i.e., the units of an analog (A)-to-digital (D) converter. The values xrm, yrm, and vrm are ratio measurements (i.e., there is a denominator). The values xrm and yrm are for different wavelengths, the value vrm is for different channels at the same wavelength of 1550nm, and the value lrm is an absolute measurement, indicating the surface brightness and capable of distinguishing between ice and snow.

[0108] Freezing probability % by pyrometer used for initial fusion pp_ice=-40*TC_can+100; % 0% when Tc>2.5°C, 100% when Tc<0°C if(pp_ice>99) pp_ice=99; end if(pp_ice<0) pp_ice=0; end

[0109] Regarding ice measurement, since the measurement by absorption at 1310 nm has low reliability, it is advantageous to replace it with measurement by a pyrometer where absorption at 1550 nm occurs equally for water and ice, the effect at 1310 nm is only seen in ice, and it is clearly weaker than water detection at 1550 nm.

[0110] The pyrometer detects thermal infrared in the range of 2..10 μm, while 1550 nm is called mid-infrared (IR). % if(yrm>yr2)% Frozen? Laser only! if(pice>50)% Initial fusion: Freezing measurement by pyrometer psno=((lrm-lr1)*100) / (lr2-lr1);% Consider snow vs. ice pdry=0;% Simplification! pwet=0;% Simplification!

[0111] The probabilities of dry, wet, ice, and snow are directly determined from xrm (water absorption in the comparison of 1550 nm and 980 nm), Tc (corrected road surface temperature from the pyrometer), and lrm (surface brightness). Specifically, using a decision tree, they are determined sequentially in this order.

[0112] The basic principles are the vibration modes of water molecules, blackbody radiation, and the physics of direct and indirect semiconductors. Measurements at these wavelengths in the infrared (IR) are useful, but above 1000 nm, silicon is "transparent" and thus no longer suitable as a photodiode, making it impossible with silicon. Germanium detects 1550 nm but becomes noisy without cooling, while InGaAs functions similarly to germanium for 1550 nm and does not require cooling. Surface coatings improve quantum efficiency, but coating is a very costly technology.

[0113] For example, a complete data-driven ecosystem includes the following steps, which are executed periodically: Using a fleet of test vehicles equipped with a telematics unit with an OTA update function (e.g., compliant with the mobile communication standard 5G), test data is continuously collected. Using a pre-trained machine learning system, the ADCU continuously calculates data related to autonomous driving (AD) or advanced driver assistance systems (ADAS) from the collected sensor data (in open loop testing). When a trigger occurs (e.g., an anomaly, e.g., when the driver's behavior is different from the ADCU's prediction), the data is sent to the server unit / cloud. In the cloud, it is checked whether the data is better training data than the data already there. For this purpose, predicted anomalies (or corner cases) can be verified or more relevant data can be selected for other reasons.

[0114] Using the optimized training data, the retraining of the machine learning system or neural network can be performed by data aggregation ("let the data decide") or active learning (while reducing the burden of manual labeling work). The actual improvement is verified by increasing KPIs (such as precision and / or recall). Subsequently, the neural network is extensively tested in the simulation world, test scenarios, corner cases, or rare cases to enhance safety. If successful, the trained neural network is approved and transferred as an over-the-air update with a cryptographic signature to the vehicles in the test fleet and installed as an update for the ADCU.

[0115] To succeed in continuous improvement, cycle training in a data-driven ecosystem is necessary. In this case, not only is the dataset increased, but the relevance of the training data is also enhanced. Significantly undervalued classes are transferred to other locations using style transfer from images of the same situation, so that the situation is learned based on the appearance of the road rather than being location-based. By projecting the time shift and the trajectory of the measurement points onto the image, the measurement results can be accurately assigned to specific points on the road. By bootstrapping, that is, applying the same law to all pixels including the road (a different network), the overall distribution on the road can be measured predictively, thus with a time margin that is advantageous for adjusting the required driving behavior.

Claims

1. A method for training a machine learning system (16) for monitoring road conditions, characterized by including the following steps: - Providing or capturing data by a sensor system (1) of a vehicle, provided that the sensor system captures the periphery of the vehicle as training input data (X), - Providing or capturing data symbolizing road conditions as training target values (Y) by a reference sensor (5) provided in or on the vehicle, and - Training the machine learning system (16), provided that training data (X, Y) including the training input data (X) and the training target values (Y) corresponding to these training input data (X) is provided, and subsequently, the parameters of the machine learning system (16) are adjusted by the training data (X, Y) such that when the training input data (X) is input, the machine learning system (16) creates output data (Y') similar to the training target values (Y).

2. The method according to claim 1, characterized in that the sensor system includes a camera system (1) of the vehicle, the provided or captured data is image data, and these image data function as training input data (X).

3. The reference sensor (5) includes a transceiver unit that irradiates the road (510) with electromagnetic rays (506) of at least one defined wavelength, receives and measures the intensity reflected from the road (510), and is configured such that based on the measured values, it can give the probability of the existence of each road condition class. The method according to claim 1 or 2, characterized by this.

4. The method according to claim 3, characterized in that the reference sensor (5) outputs probabilities for the following road condition classes: "Dry", "Wet", "Snow", "Ice", and "Unknown / Error".

5. The method according to any one of the preceding claims, characterized in that the reference sensor (5) includes a pyrometer for measuring the temperature of the road.

6. The method according to any one of claims 3, 4, or 5, characterized in that the reference sensor (5) uses the following wavelengths: To compare the reflected intensities and detect the presence of water, 1550 nm and 980 nm are added, and To measure the temperature of the road, wavelengths in the range of 2 to 10 μm are measured. **Claim 7** The method according to any one of the preceding claims, characterized in that the machine learning system (16) is a neural network, particularly preferably a convolutional neural network. **Claim 8** The method according to any one of the preceding claims, characterized in that the vehicle (2) includes a data transmission unit (18), and the data transmission unit (18) is configured to transmit training data (X, Y) to the server unit (20). **Claim 9** The method according to claim 8, characterized in that the server unit (20) can update a training dataset including a set of training data (X, Y), the size of the training dataset is kept constant, and when updating the training dataset according to one or more quality criteria, it is ensured that the relevance of the training data (X, Y) to the road condition identification is enhanced. **Claim 10** The method according to claim 9, characterized in that the server unit (20) is configured to ensure the balance of the training dataset during update so that the training dataset has a high diversity, and rare road conditions can also be sufficiently described by the training data (X, Y). **Claim 11** The method according to claim 9 or 10, characterized in that the relevance of the training data (X, Y) is evaluated based on the method of giving high relevance to the training data (X, Y) that is orthogonal to the training data (X, Y) already included in other training datasets. **Claim 12** The sensor data of the captured area of the road (510) is segmented into segments, and The road condition of each segment is characterized by the data provided by or captured by the reference sensor (5). The method according to any one of the preceding claims, characterized in that. **Claim 13** The machine learning system (16) is trained by the method according to any one of claims 1 to 12, Data captured by a vehicle sensor system (1) that captures the surrounding area of a vehicle (2) is provided as input data (X) to a machine learning system (16), and the machine learning system (5) creates output data (Y') that characterizes the road condition from the input data (X). A method for monitoring a road condition using a machine learning system (16), characterized in that.

14. An input unit (12) for receiving input data (X); A calculation unit (14) configured to be able to implement the method according to 13 for monitoring the road condition; and An output unit (18) for outputting output data (Y') created by the calculation unit (14). A road condition monitoring system (10) including.

15. A sensor system (1), wherein the sensor system captures the surrounding area of the vehicle (2) and provides the captured sensor data as input data (X) to the input unit (1); and the road condition monitoring system (14) according to Claim 14. A vehicle (2) including.

Images