Computer-implemented method for assisting a driving situation relating to a vehicle driving a roadway with at least one occupant

Abstract

A computer-implemented method for assisting a driving situation (10) relating to a vehicle (12) driving a roadway (14) with at least one occupant (16), the roadway including a roadway section (18) ahead of the vehicle (12), the method comprising: collecting and / or receiving roadway section data (34) as monitoring data (36) of the driving situation (10), the roadway section data (34) being obtained by monitoring, by the vehicle (12), the roadway section (18) during driving with respect to one or more damaged roadway portions (20) suitable for affecting a drivability relating to the occupant (16) of the vehicle (12).

Description

[0001] DESCRIPTION

[0002] Computer-implemented method for assisting a driving situation relating to a vehicle driving a roadway with at least one occupant

[0003] TECHNICAL FIELD

[0004] The invention relates to a computer-implemented method for assisting a driving situation relating to a vehicle driving a roadway with at least one occupant. The invention further relates to a driving assistance device, a vehicle, a computer program, and a computer-readable data carrier.

[0005] BACKGROUND

[0006] Existing systems and methods for drivable road detection perform road detection using classical image processing techniques on specific datasets. They may predict road-boundaries for unstructured settings (road conditions) and provide boundaries of drivable road areas in unstructured settings by taking dashboardmounted camera images and performing a set of rigid image processing operations. Some methods make use of laser radar and lidar information to predict road boundaries as an auxiliary input alongside images. Pothole detection may also be performed in prior art, for example by using accelerometer sensors and image information.

[0007] It may still be difficult to use a varied set of unstructured road image data and hence to generalize the drivable road detections to a greater set of unstructured roads. In particular, robustness and reliability in detecting drivable areas of the road may be challenging as existing methods may overlook or may have difficulties to integrate pothole detection in unstructured settings. Existing methods usually do not combine the drivable road information from other vehicles that have previously passed through the same road.

[0008] Furthermore, it may still be difficult to provide the most optimized way to traverse the road. Also, it would be desirable to consider the vehicle type, size, weight, vehicle tire pressure, suspension quality, etc. This would enable personalized suggestions for a particular vehicle. For example, the optimized traversal path suggestion might be different for a hatchback, sedan, or SUV.

[0009] It would be desirable to analyze the driver capabilities and / or style capabilities. This would enable suggesting alternate routes. For example, if the road is very unstructured and has a lot of potholes, a novice driver might not be able to comfortably traverse through it, and they would find it easier to use an alternate route. Furthermore, existing systems and methods may miss accounting for occupant health conditions in a vehicle, so that alternate routes tailored to the occupant cannot be suggested. For example, an orthopedic patient with back or spine issues might feel very uncomfortable or incur injuries while passing through degraded roads.

[0010] A method for detecting unstructured road boundary is disclosed in US 2016 / 1 71 314 A1. A real-time robust unstructured road detection method is disclosed in CN 107 122 757 A.

[0011] SUMMARY OF THE INVENTION

[0012] The object of the invention is to improve a comfort and / or safety level of a driving situation relating to a vehicle driving a roadway.

[0013] The object of the invention is achieved by the subject-matter of the independent claims. Advantageous embodiments of the invention are subject-matter of the dependent claims.

[0014] In one aspect, the invention provides a computer-implemented method for assisting a driving situation relating to a vehicle driving a roadway with at least one occupant, the roadway including a roadway section ahead of the vehicle, the method comprising: a) Collecting and / or receiving roadway section data as monitoring data of the driving situation, the roadway section data being obtained by monitoring, by the vehicle, the roadway section during driving with respect to one or more damaged roadway portions suitable for affecting a drivability relating to the occupant of the vehicle; b) Feeding the monitoring data into a first-level driving situation module for determining at least one first-level driving situation feature of the driving situation, the first-level driving situation module including at least one first data analysis model adapted for determining the first-level driving situation feature; c) Feeding the first-level driving situation feature and, preferably, the monitoring data into at least one second-level driving situation module for determining at least one second-level driving situation feature of the driving situation, the second-level driving situation module including at least one second data analysis model adapted for determining the second-level driving situation feature; d) Determining an optimal route of the driving situation with respect to the drivability by means of a third data analysis model using as input the monitoring data, the first-level driving situation feature, and / or the second-level driving situation feature; and e) Assisting the driving situation based on the determined optimal route.

[0015] An advantage of the method may be that roadway section data as monitoring data of the driving situation is collected and / or received. In the driving situation, a vehicle is driving a roadway that has damaged roadway portions in a roadway section ahead. As basic input for determining an optimal route of the driving situation the roadway section data is used. The optimal route may be a traversal path through the roadway section or an alternate route avoiding the roadway section. This avoids or reduces negative impact of potential damaged roadway portions on the driving situation. Thus, an overall comfort and / or safety level may be improved.

[0016] Preferably, the method further comprises one or both of the following: a1) Collecting and / or receiving vehicle section data as monitoring data of the driving situation, the vehicle data being obtained by monitoring, by the vehicle, characteristics of the vehicle during driving; and / or a2) Collecting and / or receiving occupant data as monitoring data of the driving situation, the occupant data being obtained by monitoring, by the vehicle, the occupant of the vehicle during driving. As advanced input for determining an optimal route of the driving situation vehicle data and / or occupant data may be used. The optimal route can thus additionally be determined based on characteristics of the vehicle and / or the occupant. For example, for large vehicles an impact of damaged roadway portions on drivability may be minor, while the impact of damaged roadway portions on drivability may be major for small vehicles. For example, for ambulance vehicles transporting injured occupants, an impact of damaged roadway portions may also play an important role. Furthermore, older occupants may also prefer a high level of comfort and / or safety. By including additional input, an optimal, individualized, or personalized route can be determined.

[0017] Preferably, the first data analysis model is adapted to perform one, several or all of the following: f1) Detecting a roadway section and / or an undamaged roadway portion of a roadway section not affecting the drivability; and / or f2) Detecting a damaged roadway portion of a roadway section affecting the drivability, particularly potholes or cracks, of a roadway section; and / or f3) Estimating a body volume and / or body weight of an occupant of a vehicle; and / or f4) Estimating an age of an occupant of a vehicle; and / or f5) Profiling an occupant of a vehicle based on identification.

[0018] Preferably, the first data analysis model is adapted to analyze the roadway section data, the occupant data, and / or the vehicle data. Additionally, a plurality of first data analysis models may be adapted to perform one, several or all of the steps f 1 ) to f5). For example, for each of the steps f1) to f5), there may exist a corresponding first data analysis model.

[0019] Preferably, the second data analysis model is adapted to perform one or both of the following: g 1 ) Detecting an undamaged roadway portion of a roadway section not affecting the drivability; and / or g2) Categorizing a driving style of a driving occupant of a vehicle. Preferably, the second data analysis model is adapted to analyze the roadway section data, the occupant data, the vehicle data, and an output from the one or more first data analysis models. Additionally, a plurality of second data analysis models may be adapted to perform one or both of the steps g1) and g2). For example, for each of the steps g1) and g2), there may exist a corresponding second data analysis model.

[0020] Preferably, the third data analysis model is adapted to perform one or both of the following: hi) Predicting a traversal path through the roadway section; and / or h2) Determining an alternate route of the driving situation avoiding the roadway section.

[0021] An advantage of the method may be that it can predict a traversal path through the roadway section and / or determine an alternate route of the driving situation avoiding the roadway section. The method determines that if meandering through the roadway section would result in an uncomfortable and / or unsafe ride, it may select and / or suggest the alternate route.

[0022] Preferably, the data analysis models respectively include a machine learning model, a neural network model, a regression model, an analytical model, a search algorithm, a linear model, a non-linear model, and / or the like.

[0023] The data analysis models may be based on and / or derived from algorithms described in the documents and / or websites as referenced herein, see [1] to

[0024] .

[0024] Preferably, the drivability corresponds to a comfort and / or safety level of the occupant during the driving situation.

[0025] Preferably, the method further comprises: i1) Generating, by a data processing device, community-based data by collecting and / or aggregating the monitoring data, the first-level driving situation feature, the second-level driving situation feature, and / or the determined optimal route of a plurality of driving situations each relating to a vehicle driving said roadway; and i2) Determining the optimal route using as further input the community-based data.

[0026] Another advantage of the method may be that community-based data may be generated. A vehicle in a driving situation can refer to the community-based data, if the same or another vehicle has already passed through the roadway section in the past. Thus, the optimal route can be determined in an improved and faster way, while the community-based data can grow with each following the optimal route.

[0027] In another aspect, the invention provides a driving assistance device, comprising means for carrying out the method of any of the preceding embodiments.

[0028] In another aspect, the invention provides a vehicle, comprising the driving assistance device.

[0029] In another aspect, the invention provides a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of the preceding embodiments.

[0030] In another aspect, the invention provides a computer-readable data carrier having stored thereon the computer program.

[0031] Aspects of preferred embodiments of the invention may be summarized as follows:

[0032] In order provide an optimal traversal path and / or alternate routes for a vehicle under unstructured road conditions, the proposed system may detect the drivable portions of a road. Preferred embodiments provide a system which uses cameras, physiological sensors (heart and respiration rate), and vehicle data (type of vehicle, suspension quality, tire pressure, etc.).

[0033] Artificial intelligence based neural network models such as support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks, or other types of models including linear models and / or non-linear models may be used. Neural networks may also include feed-forward neural networks, convolutional neural networks (CNN), recurrent neural networks (RNN).

[0034] Algorithms used in preferred embodiments of the invention are disclosed and described in the following documents or on the following websites:

[0035] [1] Olaf Ronneberger et al.: “U-Net:ConvolutionalNetworksforBiomedical

[0036] I mageSegmentation”; htts : / / do i .org / 10.48550 / arXi v .1505.04597;

[0037] [2] Changqian Yu et al.: “BiSeNet: Bilateral Segmentation Network for Real-time

[0038] Semantic Segmentation”; https : / / do i .org / 10.48550 / arXi v .1808.00897;

[0039] [3] Shaoqing Ren et al.: “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”; https: : / / do i .org / 10.48550 / arXi v .1506.01497;

[0040] [4] Joseph Redmon, Ali Farhadi: “YOLOv3: An Incremental Improvement”; https: : / / do i .org / 10.48550 / arXi v .1804.02767;

[0041] [5] Tsung-Yi Lin et al.: “Focal Loss for Dense Object Detection”;

[0042] [6] Kaiming He et al.: “Deep Residual Learning for Image Recognition”;

[0043] [7] Christian Szegedy et al.: “Rethinking the Inception Architecture for Computer Vision”; https: : / / do i .org / 10.48550 / arXi v .1506.01497;

[0044] [8] Ralf C. Staudemeyer, Eric Rothstein Morris: “Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networks”; htps: / / doi.org / 10.48550 / arXiv.1909.09586;

[0045] [9] Gated Recurrent Unit: https: / / paperawithcode.com / method / gm

[0046]

[0010] Leo Breiman: “Random Forests”; https: : / / do i .org / 10.1023 / A.1010933404324;

[0047]

[0011] Leo Breiman: “ Random Forests”;

[0048]

[0012] Gradient Boosting Classifier: https: / / sgikit- jeam.Qrg / stabje / ntgduses / generated / skleam,ensemble .GradientBoostingClasslfier. html?highlighf-gradient%20boosting#sklearn.ensemble.GradientBoostingClassifie

[0049]

[0013] Gradient Boosting Regressor: httpsd / seikit-

[0050]

[0014] Alexander Toshev, Christian Szegedy: “DeepPose: Human Pose Estimation via Deep Neural Networks”; https: : / / do i .org / 10.48550 / arXi v .1312.4659; and

[0051]

[0015] Zhe Cao et al.: “OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”; https: : / / do i .org / 10.48550 / arXi v .1812.08008:

[0052] An idea is to provide governing bodies with information on roadway conditions for better maintenance of degraded roads. A further idea is to provide safe and comfortable rides for vehicle occupants by detecting drivable portions of roads.

[0053] Road segmentation from a single camera is an important problem for Advanced Driver Assistance Systems (ADAS) as well as autonomous driving systems. In general, camera- based road segmentation solutions work well when

[0054] - the roads are well-maintained;

[0055] - the roads have clear lane markings; and / or

[0056] - the weather is clear.

[0057] However, it is still a challenging problem to segment the drivable path in unstructured scenarios, where the roads are neither well-maintained nor well- marked. Such roads may not be well-made or broken with cracks and pot-holes that make road segmentation even more challenging. It may be difficult to drive on such roads and the presence of potholes not only increases driving difficulty, but the lack of information about their presence can make driving unsafe.

[0058] Automated inspection and assessment of functional and physical conditions of the road are particularly important for the purpose of maintaining the roads and vehicle safety. With the inclusion of precise location along with pictures of the degraded roads, the road maintenance bodies can work efficiently.

[0059] Also, if the driver has prior information about potholes and / or degraded roads, they can take an informed decision. By factoring in the occupant profile(s), their health conditions, driving patterns, and vehicular information, the system preferably proposes: - optimized path(s) to traverse through the degraded route; and / or

[0060] - alternate routes, if the road conditions are undrivable

[0061] Embodiments of the invention preferably have the following advantages and effects:

[0062] - preferred embodiments use a varied set of unstructured road image data, and hence can generalize the drivable road detections to a greater set of unstructured roads; and / or

[0063] - preferred embodiments improve robustness in detecting drivable road areas as they integrate pothole detection and utilize community-based drivable road portion aggregation; and / or

[0064] - by using community-based drivable road portion aggregation, embodiments preferably combine the drivable road information from other vehicles that have previously passed through the same road, thereby providing the optimum way to traverse through the road; and / or

[0065] - embodiments preferably consider the vehicle type, size, weight, vehicle tire pressure, suspension quality, etc. to provide personalized optimum route suggestions for vehicles; and / or

[0066] - embodiments preferably analyze driver capabilities to gauge driver / passenger comfort in traversing the path ahead, and accordingly may also suggest alternate route(s); and / or

[0067] - embodiments preferably analyze the driver capabilities and / or style and therefore can suggest alternate routes. For example, if the road is very unstructured and has a lot of potholes, a novice driver might not be able to comfortably traverse through it, and would find it easier to use an alternate route; and / or

[0068] - preferably, embodiments evaluate the health (or health concerns) of the occupants, and utilize this information to further personalize optimum route suggestions / proposals; and / or

[0069] - preferred embodiments of the invention provide optimum route traversal path(s) / alternative routes based on the driver’s capabilities as well as taking occupant health into consideration.

[0070] BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the invention are now explained in more detail with reference to the accompanying drawings of which

[0071] Fig. 1 shows a driving situation relating to a vehicle driving a roadway;

[0072] Fig. 2 shows a first embodiment of a computer-implemented method for assisting the driving situation; and

[0073] Fig. 3 shows an embodiment of a system in which a second embodiment of the method is implemented.

[0074] DETAILED DESCRIPTION OF THE DRAWINGS

[0075] Fig. 1 shows a driving situation 10 relating to a vehicle 12 driving a roadway 14. The vehicle 12 has at least one occupant 16. The vehicle 12 may be autonomous, semi-autonomous or a non-autonomous.

[0076] The roadway 14 includes a roadway section 18 ahead of the vehicle 12. The roadway section 18 includes a plurality of damaged roadway portions 20 that are suitable for affecting a drivability relating to the occupant 16 of the vehicle 12. The damaged roadway portions 20 may be, for example, potholes 22 or cracks 24 of the roadway section 18.

[0077] The drivability may be, for example, a comfort and / or safety level of the occupant 16 during the driving situation 10.

[0078] An idea is to determine an optimal route 26 of the driving situation 10 with respect to the drivability. The optimal route 26 may be a traversal path 28 through the roadway section 18 or an alternate route 30 of the driving situation 10 avoiding the roadway section 18. The optimal route 26 may be used for assisting the driving situation 10. In this context, the term “assisting” means, for example, an active navigation of the vehicle 12 by means of navigation signals generated by a driving assistance device 32 for following the optimal route 26 and / or a passive assistance of the driving assistance device 32 by outputting a navigation recommendation to a driving occupant 16 of the vehicle 12 enabling to follow the optimal route 26 and / or outputting a warning message to the driving occupant 16 that the optimal route 26 has been left.

[0079] Fig. 2 shows a first embodiment of a computer-implemented method for assisting the driving situation 10.

[0080] In a step S11 , the method includes:

[0081] - Collecting and / or receiving roadway section data 34 as monitoring data 36, the roadway section data 34 being obtained by monitoring, by the vehicle 12, the roadway section 18 during driving with respect to the one or more damaged roadway portions 20 .

[0082] The monitoring may be performed by at least a first monitoring device 38 of the vehicle 12 suitable for monitoring the roadway section 18. For example, the first monitoring device 38 may be an exterior camera device, a radar device, a LIDAR device, or a combination thereof.

[0083] In a step S12, the method includes:

[0084] - Feeding the monitoring data 36 into a first-level driving situation module 40 for determining at least one first-level driving situation feature 42 of the driving situation 10, the first-level driving situation module 40 including at least one first data analysis model 44 adapted for determining the first-level driving situation feature 42.

[0085] In a step S13, the method includes:

[0086] - Feeding the first-level driving situation feature 42 and, preferably, the monitoring data 36 into at least one second-level driving situation module 46 for determining at least one second-level driving situation feature 48, the second-level driving situation module 46 including at least one second data analysis model 50 adapted for determining the second-level driving situation feature 48.

[0087] In a step S14, the method includes:

[0088] - Determining the optimal route 26 of the driving situation 10 with respect to the drivability by means of a third data analysis model 52 using as input the monitoring data 36, the first-level driving situation feature 42, and / or the second-level driving situation feature 46.

[0089] In a step S15, the method includes:

[0090] - Assisting the driving situation 10 based on the determined optimal route 26.

[0091] The data analysis models 44, 50, 52 used in steps S12, S13, and S14 may be based on one or more approaches, respectively, such as a machine learning model, a neural network model, a regression model, an analytical model, a search algorithm, a linear model and / or a non-linear model. For example, the third data analysis models 52 could be based on a neural network algorithm, an A* search algorithm with a personalized cost map and / or other algorithms.

[0092] The neural network models may respectively be based on support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks. The neural network models may further respectively include feedforward neural networks, convolutional neural networks (CNN), and / or recurrent neural networks (RNN).

[0093] Fig. 3 shows an embodiment of a system 54 in which a second embodiment of the method is implemented.

[0094] The system 54 includes the vehicle 12 with the driving assistance device 32.

[0095] In a step S21 , the method includes:

[0096] - Classifying a quality of the roadway section 18. The step S21 is used as a pre-assessment of the roadway section 18, where the driving assistance device 32 determines whether the roadway section 18 is unstructured or not, based on the roadway section data 34 of the first monitoring device 38.

[0097] In a step S22, the roadway section data 34, vehicle data 56, and occupant data 58 is inputted as monitoring data 36 to the driving assistance device 32.

[0098] The vehicle data 56 may be generated by monitoring characteristics of the vehicle 12 during driving. For this, the vehicle 12 may include at least a second monitoring device 60 for monitoring characteristics of the vehicle 12. The characteristics may be a type, a suspension quality, a tire pressure, etc. of the vehicle 12.

[0099] The occupant data 58 may be generated by monitoring the occupant of the vehicle 12. For this, the vehicle 12 may include at least a third monitoring device 61 for monitoring the occupant 16. For example, the third monitoring device 61 may be an in-cabin camera device 62 for monitoring an inner region of the vehicle 12 and / or a physiological sensor device 64 for monitoring a physiological parameter, e.g., a heart rate and / or a respiration rate, of the occupant 16.

[0100] In a step S23, the step S12 is performed as described with reference to Fig. 2. This means, the monitoring data 36 is fed into the first-level driving situation module 40 including the first data analysis model 44.

[0101] The first data analysis model 44 may be adapted or trained for detecting the roadway section 18 and / or undamaged roadway portions 20 thereof. Thus, the first data analysis model 44 fed with the monitoring data 36, for example the roadway section data 34, can determine a general road feature 66 as first-level driving situation feature 42. The first data analysis model 44 may use semantic (binary) segmentation based on a neural network model. For this, the first data analysis model 44 may be based on a CNN, e.g., ll-Nets, BiSeNet, or a combination thereof. Furthermore, the first data analysis model 44 may be trained on a dataset having images containing an exterior environment in front of the vehicle. The images may be primarily taken from unstructured roadway settings. For example, the dataset may contain images obtained from a front facing camera attached to a car, wherein each object on a road is labelled. Probable safe driving areas beside the road may also be labelled. An example of such a dataset is “India Driving Dataset” which is described and downloadable under:

[0102]

[0016] https: : / / id d .insa an.jlit.ac.in / .

[0103] Another example is described under:

[0104]

[0017] https: : / / ww w .a2d2.audi / a2d2 / en / d atas et.html

[0105] Alternatively, or additionally, the first data analysis model 44 may be adapted or trained for detecting damaged roadway portions 20, particularly potholes 22 or cracks 24, of a roadway section 18. Thus, the first data analysis model 44 fed with the monitoring data 36, for example the road section data 34, can determine a roadway damage feature 68 as first-level driving situation feature 42. The first data analysis model 44 may use object detection based on a neural network model. For this, the first data analysis model 44 may be based on a CNN, e.g., FasterRCNN, YOLO, RetinaNet, or a combination thereof. Furthermore, the first data analysis model 44 may be trained on a dataset having images containing an exterior environment in front of the vehicle. The images may be primarily taken from unstructured roadway settings. For example, the dataset may contain images of roadways 14 with labelled damages roadway portions 20. An example of such a Retina Net dataset is the pothole dataset compiled at Electrical and Electronic Department, Stellenbosch University, 2015, described in:

[0106]

[0018] S. Nienaber et al.: “Detecting potholes using simple image processing techniques and real-world footage”; SATC, July 2015, Pretoria, South Africa; and

[0107]

[0019] S. Nienaber et al.: “A Comparison of Low-Cost Monocular Vision Techniques for Pothole Distance Estimation”; IEEE CIVTS, December 2015, Cape Town, South Africa.

[0108] Alternatively, or additionally, the first data analysis model 44 may be adapted or trained for estimating a body volume and / or body weight of an occupant 16 of a vehicle 12. Thus, the first data analysis model 44 fed with the monitoring data 36, for example the occupant data 58 from the in-cabin device 62, can determine an occupant body feature 70 as first-level driving situation feature 42. The first data analysis model 44 may use neural network models to estimate a body-pose of the occupant 16. Mathematical calculations may be done using the body-part coordinates in order to estimate the occupant body-volume / weight. For this, the first data analysis model 44 may be based on a CNN, e.g., DeepPose, OpenPose, or a combination thereof. Furthermore, the first data analysis model 44 may be trained on a dataset having images containing clearly visible occupant bodies and their respective volumes. For example, the dataset may contain images of people wearing different cloths, wherein each clothing item is labelled with different body poses. An example of such a dataset is “DeepFashion2” which is described and downloadable under:

[0109]

[0020] https: : / / ai t hub.c om / switchable n o rms / DeepFas

[0110] Alternatively, or additionally, the first data analysis model 44 may be adapted or trained for estimating an age of an occupant 16 of a vehicle 12. Thus, the first data analysis model 44 fed with the monitoring data 36, for example the occupant data 58 from the in-cabin device 62, can determine an occupant age feature 72 as first- level driving situation feature 42. The first data analysis model 44 may use age prediction (regression) based on a neural network model. For this, the first data analysis model 44 may be based on a CNN, e.g., ResNet, lnceptionV2, or a combination thereof. Furthermore, the first data analysis model 44 may be trained on a dataset having images containing clearly visible faces and people’s ages. For example, the dataset may contain face images of people with annotations of age, gender, and ethnicity. An example of such a dataset is “UTKFace” which is described and downloadable under:

[0111]

[0021] https: : / / su s anqq. github.io / UTK F a ce / .

[0112] Alternatively, or additionally, the first data analysis model 44 may be adapted or trained for profiling an occupant of a vehicle based on identification. Thus, the first data analysis model 44 fed with the monitoring data 36, for example the occupant data 58 from the in-cabin device 62, can determine an occupant profile feature 74 as first-level driving situation feature 42. The first data analysis model 44 may use face recognition based on a neural network model. For this, the first data analysis model 44 may be based on a CNN, .e.g., ResNet, lnceptionV2, or a combination thereof. Furthermore, the first data analysis model 44 may be trained on a dataset having images containing clearly visible faces. For example, the dataset may contain face photographs of people collected from the web, wherein each face is labelled with the name of the person pictured. An example for such a dataset is “Labeled faces in the Wild” which is described and downloadable under:

[0022]

[0113] In a step S24, the step S13 is performed as described with reference to Fig. 2. This means, the first-level driving situation features 42, and preferably, the monitoring data 36 is fed into the second-level driving situation module 46 including the second data analysis model 50.

[0114] The second data analysis model 50 may be adapted for detecting the undamaged roadway portions 20 of the roadway section 18. Thus, the second data analysis model 50 fed with the first-level driving situation features 42, for example the general road feature 66 and the roadway damage feature 68, can determine a drivable-road feature 76 as second-level driving situation feature 48. The second data analysis model 50 may use a superimposition of the general road feature 66 and the roadway damage feature 68.

[0115] Alternatively, or additionally, the second data analysis model 50 may be adapted or trained for categorizing a driving style of a driving occupant 16 of a vehicle 12. Thus, the second data analysis model 50 fed with the first-level driving situation features 42, for example the occupant profile feature 74, and the monitoring data 36, for example the vehicle data 56 and the occupant data 58 from the physiological sensor device 64, can determine a driving-style feature 78. For example, the driving-style feature 78 may be aggressive or conservative. The second data analysis model 50 may be based on a classification using a neural network model or decision trees. For this, the second data analysis model 50 may be based on decision trees that use either bagging (e.g., random forests) or boosting methods (XGBoost, Gradient Boosting Trees). If a neural network model is used, RNN-based (LSTMs, GRUs, etc.) neural networks may be used to learn from time-series data. The second data analysis model 50 may be trained on a dataset having tabular time-series data from vehicle occupant’s vital signs during a ride and / or data from vehicle components and / or mappings of the ride to the drivers (driver identity data). A dataset for training parameters acquired by a vehicle 12 via on-board diagnostics and through micro-devices embedded in an occupant smartphone is downloadable under:

[0116] Another dataset concerning vital signs of a vehicle driver is described under:

[0117] In a step S25, the step S14 is performed as described with reference to Fig. 2. This means, the optimal route 26 of the driving situation 10 is determined with respect to the drivability by means of the third data analysis model 52.

[0118] The monitoring data 36, the first-level driving situation features 42, and / or the second-level driving situation features 48 are inputted to the third data analysis model 52. The third data analysis model 52 may be adapted or trained for predicting the traversal path 28 through the roadway section 18 where drivable roadway portions 20 have been identified. Alternatively, or additionally, the third data analysis model 52 may be adapted or trained for determining / suggesting the alternate route 30 of the driving situation 10 avoiding the roadway section 18, in particular, if the system 54 finds that meandering through the drivable roadway portions 20 of the roadway section 18 would result in an uncomfortable and / or unsafe ride.

[0119] The third data analysis model 52 may determine the optimal route 26 based on the monitoring data 36, for example the vehicle data 56 and the occupant data 58 from the physiological sensor device 64, the first-level driving situation features 42, for example the occupant body feature 70 and the occupant age feature 72, and the second-level driving situation features 48, for example the drivable-road feature 76 and the driving-style feature 78.

[0120] The system 54 as shown in Fig. 3 also includes a data processing device 80.

[0121] In a step S26, the method includes:

[0122] - Generating, by the data processing device 80, community-based data 82 by collecting and / or aggregating the monitoring data 36, the first-level driving situation features 42, and / or the second-level driving situation features 48 of a plurality of driving situations 10 each relating to a vehicle 12 driving said roadway 14; and - Determining the optimal route 26 using as further input the community-based data 82.

[0123] The determining of the optimal route 26 may, for example, be improved by community-aggregation of the drivable-road feature 76. The data processing device 80 may be configured as a cloud database which gets populated with the drivable-road feature 76 from vehicles 12 that are driving the same roadway 14. Community-aggregation may ensure that undamaged roadway portions 20 for a particular roadway section 18 are ensembled from different camera angles, thereby making the detections more robust.

[0124] In a step S27, the step S15 is performed as described with reference to Fig. 2.

[0125] The invention also provides the driving assistance device 32 comprising means for carrying out the described method, the vehicle 12 with the driving assistance device 32, a computer program (not shown) comprising instructions which, when the program is executed by a computer, cause the computer to carry out the described method, and a computer-readable data carrier (not shown) having stored thereon the computer program.

[0126] REFERENCE SIGNS

[0127] 10 driving situation

[0128] 12 vehicle

[0129] 14 roadway

[0130] 16 occupant

[0131] 18 roadway section

[0132] 20 roadway portion

[0133] 22 pothole

[0134] 24 crack

[0135] 26 optimal route

[0136] 28 traversal path

[0137] 30 alternate route

[0138] 32 driving assistance device

[0139] 34 roadway section data

[0140] 36 monitoring data

[0141] 38 first monitoring device

[0142] 40 first-level driving situation module

[0143] 42 first-level driving situation feature

[0144] 44 first data analysis model

[0145] 46 second-level driving situation module

[0146] 48 second-level driving situation feature

[0147] 50 second data analysis model

[0148] 52 third data analysis model

[0149] 54 system

[0150] 56 vehicle data

[0151] 58 occupant data

[0152] 60 second monitoring device

[0153] 61 third monitoring device

[0154] 62 in-cabin camera device

[0155] 64 physiological sensor device

[0156] 66 general road feature

[0157] 68 roadway damage feature

[0158] 70 occupant body feature

[0159] 72 occupant age feature occupant profile feature drivable-road feature driving-style feature data processing device community-based data

Claims

CLAIMS1. A computer-implemented method for assisting a driving situation (10) relating to a vehicle (12) driving a roadway (14) with at least one occupant (16), the roadway including a roadway section (18) ahead of the vehicle (12), the method comprising: a) Collecting and / or receiving roadway section data (34) as monitoring data (36) of the driving situation (10), the roadway section data (34) being obtained by monitoring, by the vehicle (12), the roadway section (18) during driving with respect to one or more damaged roadway portions (20) suitable for affecting a drivability relating to the occupant (16) of the vehicle (12); b) Feeding the monitoring data (36) into a first-level driving situation module (40) for determining at least one first-level driving situation feature (42) of the driving situation (10), the first-level driving situation module (40) including at least one first data analysis model (44) adapted for determining the first-level driving situation feature (42); c) Feeding the first-level driving situation feature (42) and, preferably, the monitoring data (36) into at least one second-level driving situation module (46) for determining at least one second-level driving situation feature (48) of the driving situation (10), the second-level driving situation module (46) including at least one second data analysis model (50) adapted for determining the second-level driving situation feature (48); d) Determining an optimal route (26) of the driving situation (10) with respect to the drivability by means of a third data analysis model (52) using as input the monitoring data (36), the first-level driving situation feature (42), and / or the second-level driving situation feature (48); and e) Assisting the driving situation (10) based on the determined optimal route (26).

2. The method according to claim 1 , further comprising one or both of the following: a1) Collecting and / or receiving vehicle data (56) as monitoring data (36) of the driving situation (10), the vehicle data (56) being obtained by monitoring, by the vehicle (12), characteristics of the vehicle (12) during driving; and / ora2) Collecting and / or receiving occupant data (58) as monitoring data (36) of the driving situation (10), the occupant data (58) being obtained by monitoring, by the vehicle (12), the occupant (16) of the vehicle (12) during driving.

3. The method according to any of the preceding claims, characterized in that the first data analysis model (44) is adapted to perform one, several or all of the following: f1) Detecting a roadway section (18) and / or an undamaged roadway portion (20) of a roadway section (18) not affecting the drivability; and / or f2) Detecting a damaged roadway portion (20) of a roadway section (18) affecting the drivability, particularly potholes (22) or cracks (24), of a roadway section (18); and / or f3) Estimating a body volume and / or body weight of an occupant (16) of a vehicle (12); and / or f4) Estimating an age of an occupant (16) of a vehicle (12); and / or f5) Profiling an occupant (16) of a vehicle (12) based on identification.

4. The method according to any of the preceding claims, characterized in that the second data analysis model is adapted to perform one or both of the following: g1) Detecting an undamaged roadway portion (20) of a roadway section (18) not affecting the drivability; and / or g2) Categorizing a driving style of a driving occupant (16) of a vehicle (12).

5. The method according to any of the preceding claims, characterized in that the third data analysis model (52) is adapted to perform one or both of the following: hi) Predicting a traversal path (28) through the roadway section (18); and / or h2) Determining an alternate route (30) of the driving situation (10) avoiding the roadway section (18).

6. The method according to any of the preceding claims, characterized in that the data analysis models (44, 50, 52) respectively include a machine learning model, a neural network model, a regression model, an analytical model, a search algorithm, a linear model, a non-linear model, and / or the like.

7. The method according to any of the preceding claims, characterized in that the drivability corresponds to a comfort and / or safety level of the occupant (16) during the driving situation (10).

8. The method according to any of the preceding claims, further comprising:11) Generating, by a data processing device (80), community-based data (82) by collecting and / or aggregating the monitoring data (36), the first-level driving situation feature (42), the second-level driving situation feature (48), and / or the determined optimal route (26) of a plurality of driving situations (10) each relating to a vehicle (12) driving said roadway (14); and12) Determining the optimal route (26) using as further input the community-based data (82).

9. A driving assistance device (32), comprising means for carrying out the method of any of the preceding claims.

10. A vehicle (12), comprising a driving assistance device (32) according to claim 9.

11. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of the claims 1 to 8.

12. A computer-readable data carrier having stored thereon the computer program of claim 11.

Images