ASSISTANCE SYSTEM ON THE ROAD SURFACE

Abstract

A driving assistance device (700) includes a processor configured to generate a risk map using road surface data and vehicle data; and to generate a driving instruction based on the risk map; wherein the risk map represents risk areas with respect to a road surface near a vehicle; and wherein the road surface data represent road surface irregularities near the vehicle.

Description

Technical area

[0001] Various aspects of this disclosure generally concern the detection of road surface irregularities using sensor data, the assessment of the risk associated with those detected irregularities, and the generation of instructions based on the assessed risk. background

[0002] The quality of a road surface or a drivable unpaved surface (e.g., off-road) varies considerably, e.g., from country to country, between rural and urban areas, and between different road types (e.g., asphalt, gravel, off-road unpaved roads, etc.). Although most major roads in Europe have very smooth and regular surfaces, free of potholes or bumps, there are still many secondary roads, e.g., gravel roads, that have poor or uneven surfaces. Looking at road quality worldwide, poor surface quality is the rule rather than the exception.

[0003] Although many advanced driver assistance systems (ADAS) can improve comfort and safety on the road (e.g., emergency braking or adaptive cruise control), these systems do not analyze the entire road surface in front of the vehicle (e.g., the road surface the vehicle will travel over) to provide instructions to the driver, such as regarding path, ride, speed, or chassis adjustments.

[0004] Various systems can extract the condition of the road surface (driveable area) from sensor data. Other systems can detect road bumps (e.g., speed bumps) based on sensor data. Such results can be shared, e.g., via wireless communication (e.g., vehicle-to-vehicle (V2V) communication or via a cloud-based system). However, such systems only deal with individual (isolated) findings (e.g., a single bump, a single uneven area). Thus, such systems cannot provide meaningful driving instructions for operating on a gravel road consisting primarily of uneven or bumpy areas, or for operating on a road with numerous bumps and / or potholes. Such roads require additional system components that can convert detected bumps and potholes into meaningful driving instructions.Such an analysis could significantly increase road safety by providing drivers or automatic assistance systems with instructions to avoid potholes or reduce speed, thereby reducing the risk of tire blowouts, mechanical failures or loss of control of the vehicle. Brief description of the drawings

[0005] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed generally upon illustrating the exemplary principles of the disclosure. In the following description, various exemplary embodiments of the disclosure are described with reference to the following drawings, in which: Fig. 1 shows elements of the driver assistance device; Fig. 2 shows an alternative implementation of the driver assistance device; Fig. 3 shows detailed versions of elements 112 to 122 as they appear in Fig. 1 are shown; Fig. 4 shows an apparatus for detecting road surface irregularities based on tracking a vehicle's behavior; Fig. Figure 5 shows the detection of a pothole using a vehicle’s headlights; Fig. 6 shows, by means of the headlights of a vehicle, that there is no pothole; Fig. 7 shows a driving assistance device; and Fig. 8 shows a method for generating driving aids. Description

[0006] The following detailed description refers to the accompanying drawings, which, by way of illustration, show exemplary details and embodiments in which aspects of the present disclosure may be practiced.

[0007] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0008] In the drawings, unless otherwise indicated, the same or similar elements, features and structures are designated by the same reference numerals.

[0009] The terms "at least one" and "one or more" may be understood to have a numerical quantity greater than or equal to one (e.g., one, two, three, four, [...], etc.). The phrase "at least one of" with respect to a group of elements may be used herein to mean at least one element from the group of elements. For example, the phrase "at least one of" with respect to a group of elements may be used herein to mean a selection of: one of the listed elements, a plurality of one of the listed elements, a plurality of individual listed elements, or a plurality of a multiple of individual listed elements.

[0010] The words "plural" and "multiple" in the description and claims expressly refer to a set greater than one. Accordingly, all expressions that expressly refer to the above words (e.g., "several [elements]," "multiple [elements]") refer to a set of elements, expressly referring to more than one of the mentioned elements. For example, the phrase "a plurality" can be understood to refer to a numerical quantity greater than or equal to two (e.g., two, three, four, five, [...], etc.).

[0011] The terms "group of," "set of," "collection of," "series of," "sequence of," "grouping of," etc., in the description and claims, if any, refer to a set equal to or greater than one, i.e., one or more. The terms "proper subset," "reduced subset," and "smaller subset" refer to a subset of a set that is not equal to the set, i.e., a subset of a set that contains fewer elements than the set.

[0012] The term "data" as used herein may be understood to include information in any suitable analog or digital form, e.g., in the form of a file, a portion of a file, a set of files, a signal or stream, a portion of a signal or stream, a set of signals or streams, and the like. Furthermore, the term "data" may also be used to refer to information, e.g., in the form of a pointer. However, the term "data" is not limited to the aforementioned examples and may take various forms and represent any information as understood in the art.

[0013] The terms "processor" or "controller" used here, for example, can be understood as any type of technological unit that enables the processing of data. The data can be processed according to one or more specific functions performed by the processor or control unit. Furthermore, a processor or a control unit, as used here, can be understood as any type of circuit, e.g., any type of analog or digital circuit. A processor or a control unit can therefore be or comprise an analog circuit, a digital circuit, a mixed-signal circuit, a logic circuit, a processor, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), an integrated circuit, an application-specific integrated circuit (ASIC), etc., or any combination thereof.Any other implementation of the respective functions described in more detail below may also be understood as a processor, controller, or logic circuit. It is understood that two (or more) of the processors, controllers, or logic circuits described herein may be implemented as a single unit with equivalent functionality or the like, and conversely, that a single processor, controller, or logic circuit described herein may be implemented as two (or more) separate units with equivalent functionality or the like.

[0014] The term "memory" refers to a computer-readable medium (e.g., a non-transitory computer-readable medium) in which data or information can be stored for retrieval. References to "memory" may include volatile or non-volatile memory, including random-access memory (RAM), read-only memory (ROM), flash memory, solid-state memory, magnetic tape, a hard disk drive, an optical drive, a 3D XPoint™, or any combination thereof. Registers, shift registers, processor registers, data buffers, etc., are also included herein under the term memory. The term "software" refers to all types of executable instructions, including firmware.

[0015] Unless explicitly stated, the term "transmission" encompasses both direct (point-to-point) and indirect transmission (via one or more intermediate points). Similarly, the term "receive" encompasses both direct and indirect reception. Furthermore, the terms "send," "receive," "communicate," and similar terms encompass both physical transmission (e.g., the transmission of radio signals) and logical transmission (e.g., the transmission of digital data over a logical software-level connection).For example, a processor or an ECU can send or receive data in the form of radio signals over a software-level connection with another processor or ECU, with the physical transmission and reception being handled by radio-layer components such as the RF transceiver and antennas, and the logical transmission and reception being performed over the software-level connection by the processors or ECUs. The term "communicate" includes both sending and receiving, i.e., unidirectional or bidirectional communication in one or both directions, i.e., inbound and outbound. The term "compute" includes both "direct" computations using a mathematical expression / formula / relationship and "indirect" computations using lookup or hash tables and other array indexing or search operations.

[0016] In the following, the term "image sensor" is used to refer to both a camera sensor and a LIDAR (Light Detection and Ranging) sensor. The image sensor may be configured to generate image data in a conventional image format (e.g., .jpg, .gif, .tiff, .bmp, .dng, or other) or in a point cloud format. Where a "sensor" is specified to collect sensor data about a vehicle's surroundings, the sensor may be or include a camera sensor or a LIDAR sensor. Should a camera sensor be used, those skilled in the art will understand that a variety of camera sensors may be used, and then one or more photogrammetry techniques may be used to generate 3D or depth information for use in object / obstacle detection and mapping.

[0017] Various aspects of this disclosure describe the detection and navigation of a vehicle in the face of a bumpy (e.g., uneven) road surface and / or a road surface with multiple potholes and unpaved, drivable terrain. For the sake of simplicity of this description and to avoid unnecessary repetition, the detected bumps, potholes, and / or uneven road or terrain surfaces are referred to herein simply as "road surface irregularities."

[0018] As described above, conventional driver assistance systems are focused on detecting individual objects or obstacles, and such systems may be inadequately able to meet the requirements of poorly maintained roads, such as gravel or unpaved roads, or otherwise bumpy roads or roads with significant potholes. Detection and warning of a single pothole or bump may be insufficient to provide meaningful instructions for driving a vehicle over a generally bumpy road or one with numerous potholes. Instead, meaningful driving instructions are required, informing the driver, for example, that a lane change or a different route is better or even optimal, that a particular pothole should not be driven over at more than 10 km / h, or that a specific suspension setting should be selected.To achieve this goal, the following describes an approach in which the system creates a risk map from which driving instructions are derived. The risks depicted in the risk map should not be static, but can vary depending on the vehicle, load, weather conditions, time of day, etc.

[0019] For clarity, the term "driving instruction" in the context of a vehicle operated by a human driver can be understood as a recommendation for a driving action that the human driver can take. That is, the driver-assistance device can recommend a specific course of action (e.g., slow down, change lanes, etc.), and the human driver can choose to implement that course of action or not. In the context of autonomous or semi-autonomous vehicles (e.g., L3+), the instruction is an instruction for action sent from the driver-assistance device to the autonomous driving system, which can then decide whether or how to implement the instruction.

[0020] A rough road assistance system can provide a human driver with sensible driving instructions to reduce the risk of vehicle damage due to bumps and / or potholes. Such risks may include, but are not limited to, tire blowouts, mechanical failure, damage to the vehicle, or loss of control. Alternatively or additionally, the driving instructions can improve driving comfort by reducing the forces on the human driver and / or passengers due to driving over bumpy roads and / or potholes, for example, by reducing speed when negotiating these bumps, selecting road surface irregularities to negotiate with the least impact on the vehicle and / or driver or passengers, or by taking an appropriate course or avoiding road surface irregularities altogether.

[0021] The driver assistance device may use one or more cameras, LIDAR sensors, or radar sensors to collect road surface data by detecting potholes or bumps and creating a list of such detected potholes or bumps for further processing. The driver assistance device may then use this detected road surface data to generate instructions (e.g., instructions for a driver and / or driving instructions), such as for speed and a driving path. Such instructions may be determined from a risk analysis or a simulation. These instructions may optionally be displayed, e.g., on the vehicle's dashboard or in a head-up display (HUD). If the vehicle is operated with an L3(+) system (e.g.,a system that meets or exceeds an L3 level of autonomous functions according to the international SAE classification), the instruction can also be used as input to the driving strategy for autonomous driving.

[0022] The driving assistance device disclosed herein can provide a human driver or an autonomous driving system with sensible driving instructions to reduce the risk of tire blowouts, mechanical failures, loss of control, or the like, and / or improve driving comfort on bumpy roads full of bumps and potholes. The system can use the results of existing bump and pothole detection methods to create a risk map of the driving area. The system can create the risk map using road surface data corresponding to the detected road surface irregularities and what is referred to herein as vehicle data, which may include, but is not limited to, vehicle speed, vehicle load, type or distribution of vehicle load, ground clearance, chassis, number and location of axles / wheels, center of gravity, or the condition or nature of the road surface or terrain.Based on this risk map, and optionally with other information from ADAS systems or sensors, the driver assistance device can calculate a trajectory and / or an optimal speed, and / or make chassis adjustments. This can then be made available to the driver. In some cases, the information can be displayed to the driver as a camera image overlay (or similar), in a HUD, or in or on the dashboard. Alternatively or additionally, this driver assistance device can also be used for L3+ vehicles to change their planned trajectory in self-driving mode.

[0023] As described in more detail below, the driver assistance device may also include several extensions to adjust the suspension and / or tire pressure and to integrate further sensor data (or ADAS data), for example, including the material or material properties of a surface (e.g., asphalt, gravel, sand, grass, bushes, puddles, etc.; strength, flexibility, sinking behavior, etc.), e.g., to cope with situations involving soft sand or to observe how a preceding or otherwise observed vehicle reacts to a bump in the road. In addition, data from systems that monitor the driver and passengers and provide personal data (e.g., discomfort, sleep, frailty, etc.) may be incorporated into the determination of a driving instruction.

[0024] In Fig.Figure 1 shows elements of the driver assistance device. First, the driver assistance device must detect bumps and / or potholes in the road surface, and the corresponding data from this detection is generally referred to here as road surface data. To do this, the vehicle must either have access to information about the road surface bumps (e.g., in a database), or the vehicle can detect the road surface bumps. Regarding access to information about road surface bumps, e.g., in a database, the vehicle can access a database or a list of road surface bumps (e.g., bumps / potholes), e.g., with dimensions and locations. Although such a list can be configured in different ways depending on the implementation, an example format might look like this: type Location (e.g. in vehicle coordinates) Dimensions bumper x = 5 m, y = 0 m, z = 0 m (i.e. the bump is 5 m in front of the vehicle) x = 0.2 m, y = 6 m, z = 0.4 m (i.e. the bump is 40 cm high and 6 m wide) pothole x = 10m, y = -1m, z = 0m (i.e. the bump is 105m in front of the vehicle and 1 meter to the right) x = 0.2 m, y = 0.2 m, z = -0.05 m (i.e. the pothole is 5 cm deep) pothole x = 8m, y = 3m, z = 0m (i.e. bumper 8m in front of the vehicle and 3m to the left, e.g. on the opposite lane) x = 0.4 m, y = 0.4 m, z = -0.4 m (i.e. a large pothole with a depth of 40 cm)

[0025] Such information may, for example, be available in one or more cloud-based mapping services, so that not every vehicle needs to be equipped with sensors or systems for detecting bumps / potholes. In this way, the vehicle may include a transceiver configured to receive a wireless signal representing the road surface data mentioned above. The vehicle (one or more processors in the vehicle) may be configured to control the transceiver to establish a wireless connection with a service (e.g., a database service and a mapping service) and retrieve road surface data, such as information related to the vehicle's location or its expected location. This list of road surface irregularities can then be used to generate meaningful driving advice, as described in more detail below.

[0026] Fig.Figure 1 shows a configuration in which the vehicle detects information about road surface irregularities based on sensor data (e.g., image sensor data and / or LIDAR data). In this figure, the vehicle generates sensor data 102 about a road surface near the vehicle through its own sensors (e.g., cameras, LIDAR, etc.) or otherwise has access to raw sensor data (e.g., via a transceiver or stored information). The following describes the steps for detecting road surface irregularities from this sensor data. First, the driver assistance device may be configured to perform road surface detection 104, analyzing the sensor data to distinguish a road surface from a non-road surface.The result of such an analysis may include the sensor data in which the road surface is highlighted, marked, or otherwise delineated, as shown in 106.

[0027] Following the detection of the road surface 104, the driver assistance device may perform road surface irregularity detection (pothole detection) 108, in which the defined road surface is examined for road surface irregularities (see resulting image 110). This may be done using any conventional sensor data analysis method capable of isolating a feature (in this case, the road) within the sensor data. Alternatively or additionally, the driver assistance device may include an artificial neural network that may be configured to receive image sensor data indicative of a detected road surface and detect the road surface irregularities (e.g., bumps and / or potholes) in this sensor data. Regardless of the method, the result of the detection 108 may be data (e.g.,The pothole detection system 108 may display a list of the identified road surface irregularities (e.g., the sensor data or data corresponding to the sensor data) in which road surface irregularities are marked, delineated, highlighted, or otherwise identified 110. Alternatively or additionally, the pothole detection system 108 may display a list of the identified road surface irregularities. This list may include any type of road surface irregularity, location of road surface irregularity, or dimensions of road surface irregularity.

[0028] The driver assistance device can then proceed with the creation of a risk map 112. For this step, the driver assistance device can receive the output of the bump / pothole detector 108 (e.g., the list of road surface irregularities). However, this input alone may not be sufficient to estimate the risk of loss of comfort or control, a collision, tire failure, mechanical failure, or the like. Therefore, the risk map generator 112 can be configured to combine the road surface irregularity information with vehicle data, which, as mentioned above, can include information from one or more additional sources. The vehicle data can include, for example, tire pressure, suspension information (e.g.,from accelerometers), information about vehicle loading and load distribution, or sensors in the vehicle cabin that observe / monitor passenger comfort. Alternatively or additionally, the vehicle data may also include passenger comfort data, which may represent the comfort of a passenger in the vehicle. Alternatively or additionally, the vehicle data may also include the current speed of the vehicle. In addition, information about road friction, e.g. from anti-lock braking system (ABS) / electronic stability program (ESP) sensors, may be taken into account to determine whether the vehicle is currently on a loose or soft surface, which may influence risk assessments.

[0029] Using this data, the risk map generator 112 can determine the risks for each bump, pothole, or other irregularity in the road surface. If the road surface is discretized as a grid, the risks can be estimated for each grid cell. In this context, it should be noted that various approaches can be used for risk estimation at runtime (i.e., while the vehicle is traveling on the bumpy road). It is possible to use learning-based solutions such as artificial intelligence (e.g., an artificial neural network), which can, for example, use a colored bird's-eye view image as input to create a risk map for each aspect, which can then be accumulated into a single risk representation. Alternatively, or additionally, one or more predefined look-up tables can be used. An example look-up table is shown below: hump / punch hole Vehicle weight Speed ​​[km / h] Probability of penetration comfort Probability of loss of control hump +20cm 2 tons 5 Low High No hump +20cm 2 tons 10 Low OK No hump +20cm 2 tons 15 Low Low Low hump +20cm 2 tons 20 Medium Very low Medium hump +20cm 3.5 tons 20 High Very low High Pothole - 40cm 2 tons 5 Low Low Low Pothole - 40cm 2 tons 10 Medium Very low Medium Pothole - 40cm 2 tons 15 Medium Very low Medium Pothole - 40cm 2 tons 20 High Very low High Pothole - 40cm 3.5 tons 20 Very high Very low Very high

[0030] This table indicates the irregularity of the road surface in the "Bumps / Potholes" category. The vehicle data is specified as "Vehicle Weight" and "Speed." Of course, other or additional vehicle data can also be taken into account, as specified here. Based on this, the risk of a flat tire ("Flat Probability"), the estimated comfort level for a human driver or passenger ("Comfort"), and the risk of loss of control ("Loss of Control Probability") are calculated. Based on these findings, the system can generate one or more driving instructions.

[0031] In an alternative configuration, this online system can be combined with an offline system that can be configured to consider both recorded and simulation data to relate comfort, the probability of mechanical failure, etc., to different speeds and road surface irregularities. Such an offline system can be used to train the learning-based approach or to populate / update the look-up table.

[0032] A look-up table or a learning-based approach can provide a baseline for each risk level (online), which can then be adjusted, e.g., with an offset using the current sensor data in the cabin. For example, human comfort (e.g., the comfort of a driver or passenger) varies from person to person, as each person reacts differently to road surface irregularities. Therefore, the risk assessment results can be adjusted by observing the current comfort level. For example, if the in-cabin systems detect severe head shaking (e.g., due to road bumps), but the risk assessment system reports a supposedly high comfort level, the high comfort level can be adjusted accordingly to a lower value. This offset can be dynamically adjusted based on the incoming / available sensor data.

[0033] The system can optionally be configured to not only evaluate bumps / potholes in isolation, but also to detect patterns or repetitions (e.g., a wave pattern that may develop spontaneously, such as on an unpaved road). On roads with such a wave pattern, or on roads with many closely spaced road surface irregularities, an isolated analysis may lead to suboptimal results. Therefore, a combined analysis is provided to determine the best response, e.g., the best speed given the current tire pressure, vehicle weight, and wave profile.

[0034] Furthermore, it should be noted that human drivers or passengers may have interests other than those that can promote or prolong mechanical stability. For example, it may feel better for a human occupant to drive faster rather than slower on such "bumpy" roads; however, a lower speed would be better for the mechanical components and tires. Taking this fact into account, the driver assistance device may attempt to determine the best compromise between these competing aspects, or the driver assistance device may weight these competing interests based on one or more pre-determined preferences (e.g., always prioritize mechanical integrity over comfort, etc.).In this way, the risk map generator 112 can create one or more risk maps that can link risks to each bump / pothole, or alternatively, on a grid basis, as shown in 114. That is, the risk map can be a data visualization tool in which individual road surface irregularities are identified relative to the vehicle or relative to one or more reference points (e.g., relative to the vehicle, in Cartesian space, etc.), and each of the road surface irregularities can be assigned a risk. The risk map can alternatively be divided into sectors, such that a sector corresponding to one or more road surface irregularities is associated with a specific risk (e.g., corresponding to a single road surface irregularity or corresponding to multiple road surface irregularities). The risk map can include a discretized grid, which in turn has a plurality of cells.In this way, each cell of the plurality of cells may correspond to a discrete area of ​​the road surface and a risk associated with the vehicle traveling over that discrete area of ​​the road surface.

[0035] Using the risk data from the previous module, the driver assistance device can then generate driving instructions 116 based on the risk map created by the risk map generator 112. In this way, the driver assistance device can calculate a feasible driving path with any speed profile, suspension setting (e.g., harder, softer, higher, or lower ride height), or the recommended mode (e.g., 2-wheel driver or 4-wheel drive, if available).

[0036] For this purpose, the driver assistance device can use a variety of planners (e.g., trajectory planners). For example, the driver assistance device can include or use a particle filter with positive weighting for low-risk regions and negative weighting for high-risk regions. Each particle can then represent a next step along a path, so that the result of the filtering process is a near-optimal driving path, including speed instructions.

[0037] In this context, the planner can be allowed to use the entire driving area, including the lanes for oncoming traffic, without having to object. To prevent the planner from issuing an instruction that could lead the driver into a collision or near-collision with other road users, the driver assistance device can receive data from other ADAS systems 118. In this way, the driving instruction generator 116 can take into account perceived environmental information from other road users when planning the route. For example, in the absence of oncoming traffic, the driving instruction generator 116 can generate an instruction instructing the driver or vehicle to drive in an adjacent lane (e.g., for oncoming traffic). Alternatively, the driving instruction generator 116 can instruct the vehicle or driver to stay on the roadway where there is oncoming traffic.In this case, the driving instruction generator 116 may issue an alternative instruction, e.g., to reduce speed to maintain an acceptable risk level. An instruction from the driving instruction generator 116 is illustrated in 120 for illustrative purposes, depicting a driving path around identified risk areas (corresponding to the road surface irregularities).

[0038] Although the risk map generator 112 and the driving instruction generator 116 are depicted as separate units, it is expressly understood that they may be combined into a single unit and / or a single processor. In an alternative configuration, each of these units may be combined into a single artificial intelligence (e.g., an artificial neural network) that can be trained to perform both risk assessment and the generation of driving instructions. Alternatively or additionally, the driving instruction generator 116 may be combined with one or more decision modules, for example, to evaluate whether tire pressure needs to be adjusted (decreased or increased).

[0039] Once the driving instruction has been created, the display instruction module 122 transmits this information to the driver, if present. In this way, the display instruction module 122 can use the created path, speed, or other instructions from the previous module(s) to interact with the driver. This can be done in several ways.

[0040] The simplest approach is to simply display a speed instruction on the vehicle's dashboard / HUD 124. This allows a visual representation of the speed instruction to be displayed. This could be a number (e.g., "recommended speed: 40 km / h"), a speedometer display, a deceleration display, or something else.

[0041] Alternatively or additionally, the display instruction module 122 may overlay the determined path over a camera image to provide a better visual context for the best route. This may be communicated to the driver in a road visualization or map format depicting a preferred travel path. Alternatively or additionally, this may be communicated to the driver in the form of an instruction, e.g., "turn left," "change lanes," "move two meters to the left," etc.

[0042] Alternatively or additionally, the display instruction module 122 may provide instructions to the driver regarding tire pressure. In many vehicles, tire pressure can only be changed manually, so the driver can be instructed to increase or decrease the tire pressure to reduce the load on the vehicle and / or improve driver comfort. However, in some vehicles, tire pressure may be automatically or semi-automatically adjustable, so that the tire pressure can be adjusted based on an instruction, such as an instruction in an autonomous driving system, or based on a driver's authorization or instruction.

[0043] In addition to normal driving instructions, the driver assistance system can optionally include a rescue guidance mode. Such a rescue guidance can be activated if the driver has already steered into an area corresponding to a road surface irregularity. Instead of a recommended lane, the system can then provide rescue guidance such as "slow down," "keep right," or similar, allowing the driver to negotiate the high-risk area. Depending on the driver's profile and the capabilities of the vehicle's assistance systems, active intervention (e.g., adding torque to the steering wheel in a specific direction) may also be possible. If the vehicle comes to a stop in an area corresponding to a road surface irregularity, more detailed rescue recommendations can be calculated (e.g., "activate all-wheel drive and drive slowly straight ahead").

[0044] Fig.2 shows an alternative implementation of modules 104 and 108 from Fig.1, in which the road surface irregularities are identified for the subsequent creation of a risk map and driving instructions. In this figure, the road surface detection module 202 detects the road surface as described in 104, and the bump and pothole detection module 204 detects the road surface irregularities as described in 108. However, as shown here, the output of these modules is a list of road surface irregularities (e.g., bumps / potholes) 206, which serves as the basis for subsequent driving instructions. In a first option, the bump and pothole detection module 204 can be configured to output the list of road surface irregularities 206. In an alternative configuration, the list of road surface irregularities 206 can be obtained from one or more cloud-based services 208.In this way, the vehicle can send a request to the cloud-based service 208 with a current position of the vehicle (e.g., a GPS position, a position relative to another reference point, etc.). In response, the cloud-based service 208 can provide a map of the surrounding areas along with information about road surface irregularities, from which the bump and pothole detection module 204 can derive the list of road surface irregularities 206. Alternatively, the cloud-based service 208 can transmit the list of road surface irregularities 206 directly to the vehicle. In this way, the list of road surface irregularities 206 can include coordinates relative to a reference point, e.g., the reported location of the vehicle or others.

[0045] Fig. 3 shows the Fig.1 as 112 to 122 in greater detail. In this way, the driver assistance device receives the list of road surface irregularities 302 and creates a risk map 304 from the list of road surface irregularities 302 along with additional information, as generally shown at 306, where the additional information may include tire pressure sensors, suspension sensors (e.g., acceleration sensors), information about vehicle load and / or load distribution, interior sensors for observing or monitoring passenger comfort, the current vehicle speed, information about road friction, or information about pavement quality. Such information may also be available elsewhere in the vehicle, and the driver assistance device may be configured to receive these other measurements or data and incorporate them into the calculation of the risk map.

[0046] As illustrated at least in 308, the risk map may represent one or more risks associated with a road surface irregularity. In this context, it is useful to distinguish the present disclosure from a conventional device configured to identify a single obstacle or damaged area on the road surface. Such conventional devices are ill-suited to detecting risks within a poor road surface, where the entire road surface may be uneven or damaged, or where the damaged areas are close together. In such areas, a large portion or even all of the road surface may be considered to pose a risk, for example, an above-average risk.In such cases, the driver assistance device can identify relative risk levels associated with the road surface irregularities, allowing areas to be identified that correspond to several different risk levels (e.g., at least 2 risk levels, at least 5 risk levels, or at least 10 risk levels, etc.). In this way, it is possible to determine a path along which the lowest risk occurs. Such risk maps can be divided into sectors, so that the vehicle's surroundings are divided into a plurality of sectors, and each sector is assigned a risk corresponding to the quality of the road surface when viewed in light of the additional information described above.

[0047] Based on the risk map, the driving instruction generator 310 generates one or more driving instructions that pose the lowest risk. This may primarily involve specifying a driving path (see 314) and / or a driving speed. For example, a driver may be instructed to continue on the current driving path, change the driving path according to an instruction, increase speed, or decrease speed.

[0048] Note that the driving instruction generator 310 can optionally generate its driving instructions based on the risk map and one or more ADAS systems 312. In this way, the output of other ADAS systems, such as a perception system, can be considered when selecting a driving path to avoid journeys that might otherwise expose the vehicle to increased risk, e.g., by entering the path of another vehicle.

[0049] The display instruction module 316 can then translate the instruction into a notification for the driver. Such a notification can be a visual notification, for example, displayed on a HUD, a dashboard, or the like. Alternatively or additionally, the driver instruction can be or include an audio instruction and / or a haptic instruction.

[0050] In addition to detecting road surface irregularities based on sensor data, the driving assistance device can optionally be configured to detect road surface irregularities by analyzing the behavior of another vehicle ahead, as described in Fig.4. In this way, the driver assistance device can detect relevant attributes of the behavior of another vehicle (a leading vehicle) to model and predict a road surface irregularity and / or the vehicle's future driving behavior based on the other vehicle's actions. In an optional configuration, speed and direction of travel can be used to determine the road surface. In other optional configurations, one or more behavior models can be used for specific vehicle classes (e.g., motorcycle, truck, car) or specific vehicle types (e.g., pickup truck, SUV).

[0051] By tracking the vehicle and comparing it with predicted behavior, anomalies in driving behavior can be detected. Since these anomalies can correspond to road surface irregularities, information about road surface irregularities can be obtained from these observations. Examples of such anomalies include a vehicle leaning to one side (e.g., due to a pothole), exhibiting bumpy behavior (e.g., due to a bump), or sliding sideways (e.g., due to sand). The location of these anomalies is calculated, and a risk map is created for further processing.

[0052] Fig.Figure 4 shows a device for detecting road surface anomalies based on tracking vehicle behavior. This device can be configured to detect a vehicle 402, e.g., using sensor data (e.g., camera data or LIDAR data). This can be done using known techniques, e.g., vehicle or object detection techniques using image sensor data, or by modifying the road detection and / or road surface irregularity detection modules, as described above, to identify vehicles.

[0053] The device can be configured to predict the behavior of an identified vehicle 404. For example, a vehicle exhibiting smooth operation is predicted to maintain smooth operation; a vehicle traveling straight is predicted to continue traveling straight; a vehicle traveling in a level configuration (e.g., without leaning to one side) is predicted to continue traveling in a level configuration. The device can be configured to predict any type of future behavior and compare it to the observed behavior, without limitation and notwithstanding the above examples.

[0054] The vehicle behavior is tracked 406 or observed, and this observed behavior 406 is compared with the predicted behavior in 404. If the driving behavior matches the predicted behavior, no anomaly is detected. However, if the road surface behavior deviates from the predicted behavior, the resulting deviation can be attributed to a road surface anomaly. In this way, the anomaly detector 408 can determine a road surface anomaly from the difference between the road surface behavior 406 and the predicted behavior 404. For example, a vehicle that is predicted to travel smoothly but then suddenly exhibits a bumpy movement may indicate a bump or speed hump, and such an irregularity in the road surface can be associated with the location where the distracting behavior occurred.Likewise, a vehicle that is predicted to be traveling smoothly but suddenly leans to the left may indicate a pothole or other unevenness in the road surface, particularly in the area of ​​the left tires.

[0055] Based on the determinations made by the anomaly detector 408, the device may create the risk map 410 for use as described above. Additionally or alternatively, the device may update a previously created risk map 410 to include road surface irregularities that might not otherwise have been detected based on the visualization of the road surface itself.

[0056] By predicting and comparing the vehicle's behavior, road surface irregularities can be detected and stored in a risk map. A locally stored risk map can be created along with the location. Whenever the vehicle travels the same road again, the risk map can be reused to improve the risk calculation. This risk map can then also be used in route creation (e.g., choosing the route with the fewest potholes).

[0057] Fig. 5 and Fig.6 show the detection of unevenness in the road surface using light. Potholes can cause unnecessary mechanical stress on a vehicle, which can lead to or contribute to vehicle damage. Detecting a pothole can be particularly difficult at night because it is particularly hard for the human eye to see. Therefore, the vehicle can optionally be configured to calculate the risk posed by a pothole based on the shadow the pothole creates due to the vehicle's headlights. The underlying principle is that a vehicle's headlights shining over a pothole create a shadow inside the pothole that is obscured by the edge of the pothole. This is shown in Fig.5, where the vehicle's headlights shine on the edge of the pothole, creating a shadow area within the pothole. The image sensor captures an image of the pothole in the shadowed area, and the device detects a pothole based on the shadow. Typically, a larger shadow indicates a deeper pothole, and a smaller shadow indicates a shallower pothole. In comparison between Fig. 5 and Fig. 6 shows Fig. 6 A comparatively small bump where the light falling on the edge of the bump does not create a shadow (e.g., due to the angle of incidence of the headlight beam on the edge of the bump). The absence of a shadow indicates either that there is no pothole or that the pothole or irregularity is comparatively small.

[0058] Fig.7 shows a driving assistance device 700 with a processor 702 configured to generate a risk map using road surface data and vehicle data and to generate an instruction based on the risk map, wherein the risk map represents risk areas related to a road surface near a vehicle, and wherein the road surface data represents irregularities of a road surface near the vehicle. The driving assistance device 700 may further include a sensor 704 configured to receive data representing the surroundings of the vehicle; wherein the processor 702 is further configured to identify the road surface irregularities based on the data. The driving assistance device 700 may further include a transceiver 706 configured to receive a wireless signal representing the road surface data.

[0059] Fig. 8 shows a method for generating driving assistance 800, including generating a risk map using road surface data and vehicle data 802; and generating an instruction based on the risk map 804; wherein the risk map represents risk areas related to a road surface near a vehicle 806; and wherein the road surface data represents irregularities of a road surface near the vehicle 808.

[0060] Reference has been made at various points to the observation of a driver or one or more passengers in the cabin, e.g., using image sensors. This can be done using any existing image sensor configured to capture image data of the cabin interior, or using one or more additional image sensors specifically dedicated to this task. One or more processors can be configured to receive the appropriate image sensor data and perform one or more evaluations of the data to identify a driver and / or passenger and make one or more determinations regarding a comfort level of the driver and / or passenger.Such determinations regarding comfort levels may be made by observing body movements, so that jerky, bouncing, or rapidly rocking movements may be associated with discomfort, while immobile bodies or bodies exhibiting only minimal movement may be associated with comfort. Alternatively, or additionally, the processor may be configured to isolate and determine one or more facial expressions in the image sensor data, so that comfort or discomfort can be inferred from these facial expressions (e.g., smiles or neutral facial expressions may be associated with comfort, while frowns, expressions associated with disgust, expressions associated with nausea, etc., may be associated with discomfort). In this way, the processor may be configured to dynamically adjust the risk levels in the risk map based on the passenger comfort data.

[0061] As mentioned herein, the processor may be configured to perform each of the determination steps described herein using direct computations or using one or more artificial neural networks. Such artificial neural networks may be configured to receive the road surface data and the vehicle data and output the risk map.

[0062] The conversion of an identified road surface irregularity along with vehicle data into a risk can be achieved according to another configuration using a look-up table. This way, the look-up table can contain risk quantities corresponding to the road surface irregularities and the vehicle data. Thus, the processor can be configured to generate the risk map by looking up a road surface irregularity and corresponding vehicle data and determining the resulting risk using the look-up table.

[0063] Further aspects of the disclosure are described using examples.

[0064] In Example 1, a driving assistance device is configured with a processor to generate a risk map using road surface data and vehicle data; and generate an instruction based on the risk map; wherein the risk map represents risk areas related to a road surface near a vehicle; and wherein the road surface data represents irregularities of a road surface near the vehicle.

[0065] In Example 2, the driving assistance device according to Example 1 includes an image sensor configured to obtain data representing the surroundings of the vehicle; wherein the processor is further configured to generate the road surface data by identifying road surface irregularities based on the data.

[0066] In Example 3, the driving assistance device according to Example 1 further includes a transceiver configured to receive a wireless signal representing the road surface data.

[0067] In Example 4, the driving assistance device according to any one of Examples 1 to 3, wherein the risk map includes a discretized grid having a plurality of cells, and wherein each cell of the plurality of cells corresponds to a discrete region of the road surface and a risk associated with the vehicle traveling over the discrete region of the road surface.

[0068] Example 5: The driving assistance device according to any one of examples 1 to 4, wherein the vehicle data includes tire pressure, suspension sensor data, accelerometer data, vehicle load data, or vehicle load distribution data.

[0069] Example 6: The driving assistance device according to any one of examples 1 to 5, wherein the vehicle data includes passenger comfort data representing the comfort of a passenger in the vehicle.

[0070] In Example 7, the driving assistance device according to Example 6 further comprises an image sensor in the cabin configured to generate a signal representing an image of the passenger in the vehicle; wherein the processor is further configured to determine the comfort of the passenger in the vehicle from the signal.

[0071] In Example 8, the driving assistance device is as in Example 6 or 7, wherein the processor is configured to dynamically adjust the risk levels in the risk map based on the passenger comfort data.

[0072] In Example 9, the driving assistance device according to any one of Examples 1 to 8 further includes an artificial neural network configured to receive the road surface data and the vehicle data and output the risk map; wherein the processor that generates the risk map includes the processor that generates the risk map using the artificial neural network.

[0073] In Example 10, the driving assistance device according to any one of Examples 1 to 8 further includes a look-up table having risk quantities corresponding to irregularities of a road surface and vehicle data; wherein the processor generating the risk map includes the processor generating the risk map using the look-up table.

[0074] Example 11: The driving assistance device according to any one of examples 1 to 10, wherein the instruction includes a driving instruction, and wherein the driving instruction includes a driving path or a driving speed.

[0075] Example 12: The driving assistance device according to any one of examples 1 to 11, wherein the instruction includes an instruction to change a suspension setting.

[0076] Example 13: The driving assistance device according to any one of examples 1 to 12, wherein the instruction includes an instruction to switch between a two-wheel drive mode and a four-wheel drive mode.

[0077] Example 14: The driver assistance device of any one of examples 1 to 13, wherein the processor is further configured to output the instruction on a vehicle display.

[0078] Example 15: The driving assistance device of any one of examples 1 to 14, wherein the processor is further configured to cause an autonomous driving system to operate the vehicle according to the instruction.

[0079] In Example 16, the driving assistance device is according to any one of Examples 1 to 15, wherein the processor is further configured to determine one or more patterns within the road surface data; wherein generating the driving instruction based on the risk map comprises generating the driving instruction based on the determined one or more patterns.

[0080] In Example 17, the driving assistance device is according to any one of Examples 2 to 16, wherein the vehicle is a first vehicle; and wherein the processor is further configured to: detect a second vehicle in the data of the surroundings of the first vehicle; and generate road surface data based on a movement of the detected second vehicle.

[0081] In Example 18, the driving assistance device according to any one of Examples 2 to 17, wherein the processor is further configured to: detect, in the data of the surroundings of the vehicle, a shadow cast by a headlight of the vehicle shining on an irregularity of the road surface; generate road surface data based on the detected shadow.

[0082] In Example 19, a method for generating driving assistance, comprising: generating a risk map using road surface data and vehicle data; and generating an instruction based on the risk map; wherein the risk map represents risk areas related to a road surface near a vehicle; and wherein the road surface data represents irregularities of a road surface near the vehicle.

[0083] In Example 20, the method for generating the driving assistance according to Example 19 comprises: comprising an image sensor configured to obtain data representing the surroundings of the vehicle; further comprising generating the road surface data by identifying road surface irregularities based on the data.

[0084] In Example 21, the method for generating driving assistance according to Example 19 further comprises a transceiver configured to receive a wireless signal representing the road surface data.

[0085] In Example 22, the method for generating driving assistance according to any one of Examples 19 to 21, wherein the risk map includes a discretized grid having a plurality of cells, and wherein each cell of the plurality of cells corresponds to a discrete region of the road surface and a risk associated with travel of the vehicle over the discrete region of the road surface.

[0086] Example 23: The method for generating driving assistance according to any one of examples 19 to 22, wherein the vehicle data comprises tire pressure, suspension sensor data, accelerometer data, vehicle load data, or vehicle load distribution data.

[0087] Example 24: The method for generating driving assistance according to any one of examples 19 to 23, wherein the vehicle data includes passenger comfort data representing the convenience of a passenger in the vehicle.

[0088] In Example 25, the method for generating driving assistance according to Example 24 further comprises an image sensor in the cabin configured to generate a signal representing an image of the passenger in the vehicle; further comprising determining the comfort of the passenger in the vehicle from the signal.

[0089] In Example 26, the method for generating the driving assistance according to Example 24 or 25 further comprises dynamically adjusting the risk levels in the risk map based on the passenger comfort data.

[0090] In Example 27, the driving assistance generation method according to any one of Examples 19 to 26 further comprises an artificial neural network configured to receive the road surface data and the vehicle data and output the risk map; further comprising generating the risk map using the artificial neural network.

[0091] In Example 28, the method for generating driving assistance according to any one of Examples 19 to 26 further comprises a look-up table having risk quantities corresponding to irregularities of a road surface and vehicle data; further comprising generating the risk map by the processor using the look-up table.

[0092] Example 29: The method for generating driving assistance according to any one of examples 19 to 28, wherein the instruction includes a driving instruction and wherein the driving instruction includes a driving path or a driving speed.

[0093] Example 30: The method for generating driving assistance according to any one of examples 19 to 29, wherein the instruction includes an instruction to change a chassis setting.

[0094] Example 31: The method for generating driving assistance according to any one of examples 19 to 30, wherein the instruction includes an instruction to switch between a two-wheel drive mode and a four-wheel drive mode.

[0095] In Example 32, the method for generating the driving assistance according to any one of Examples 19 to 31 further comprises causing the instruction to be scrolled on a vehicle display.

[0096] In Example 33, the method for generating the driving assistance according to any one of Examples 19 to 32 further comprises causing an autonomous driving system to operate the vehicle according to the instruction.

[0097] In Example 34, the method for generating driving assistance according to any one of Examples 19 to 33 further comprises determining one or more patterns within the road surface data; wherein generating the driving instruction based on the risk map comprises generating the driving instruction based on the determined one or more patterns.

[0098] In Example 35, the method for generating driving assistance according to any one of Examples 20 to 34, wherein the vehicle is a first vehicle; and further detecting a second vehicle in the data of the surroundings of the first vehicle; and generating road surface data based on a movement of the detected second vehicle.

[0099] In Example 36, the method for generating driving assistance according to any one of Examples 20 to 35 further comprises detecting, in the data of the surroundings of the vehicle, a shadow cast by a headlight of the vehicle shining on an irregularity of the road surface, and generating road surface data based on the detected shadow.

[0100] In Example 37, a non-transitory computer-readable medium containing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of any of Examples 19-36.

[0101] Although the above descriptions and accompanying illustrations depict components as separate elements, those skilled in the art will appreciate the various ways to combine or integrate discrete elements into a single element. These include combining two or more circuits into a single circuit, mounting two or more circuits on a common chip or chassis to form an integrated element, executing discrete software components on a common processor core, etc. Conversely, those skilled in the art will recognize the possibility of dividing a single element into two or more discrete elements, such asthe division of a single circuit into two or more separate circuits, the division of a chip or chassis into discrete elements originally intended on it, the division of a software component into two or more sections and their execution on a separate processor core, etc.

[0102] It is believed that the implementations of the methods described herein are demonstrative in nature and can therefore be implemented in a corresponding device. It is also believed that implementations of the devices described herein can be implemented as a corresponding method. It is therefore understood that a device corresponding to a method described herein can include one or more components configured to perform each aspect of the corresponding method.

[0103] All acronyms defined in the above description also apply to all claims contained herein.

Claims

[1] . Driving assistance device (700) comprising: a processor (702) configured to: a risk map using road surface data and vehicle data created; and a driving instruction is drawn up based on the risk map; wherein the risk map represents risk areas in relation to a road surface near a vehicle; and where the road surface data represents road surface irregularities near the vehicle. [2] . Driving assistance device (700) according to claim 1, comprising: a sensor (704) configured to collect data representing the environment or environmental characteristics of the vehicle; wherein the processor (702) is further configured to generate the road surface data by identifying road surface irregularities based on the data. [3] The driving assistance device (700) according to claim 1, further comprising a transceiver (706) configured to receive a wireless signal representing the road surface data. [4] The driving assistance device (700) according to any one of claims 1 to 3, wherein the risk map comprises a discretized grid having a plurality of cells, and wherein each cell of the plurality of cells corresponds to a discrete region of the road surface and a risk associated with the vehicle traveling over the discrete region of the road surface. [5] The driving assistance device (700) according to any one of claims 1 to 4, wherein the vehicle data comprises tire pressure, suspension sensor data, accelerometer data, vehicle load data, vehicle load distribution data, chassis data, or occupant data. [6] The driving assistance device (700) according to any one of claims 1 to 5, wherein the vehicle data comprises occupant comfort data representing the comfort of the driver or a passenger in the vehicle; further comprising a sensor in the cabin configured to generate a signal representing the state of the driver or the passenger in the vehicle; wherein the processor (702) is further configured to detect the comfort of the driver or the passenger in the vehicle from the signal. [7] The driving assistance device (700) according to any one of claims 1 to 6, further comprising an artificial neural network configured to receive the road surface data and the vehicle data and output the risk map; wherein the processor (702) that generates the risk map comprises the processor (702) that generates the risk map using the artificial neural network. [8] The driving assistance device (700) according to any one of claims 1 to 6, further comprising a look-up table including risk quantities corresponding to road surface irregularities and vehicle data; wherein generating the risk map comprises generating the risk map using the look-up table. [9] The driving assistance device (700) according to any one of claims 1 to 8, wherein the driving instruction comprises a driving path, a driving speed, an instruction to change a chassis setting, or an instruction to switch between a single axle drive mode and a multiple axle drive mode. [10] The driving assistance device (700) according to any one of claims 1 to 9, wherein the processor (702) is further configured to cause the driving instruction to be shown on a vehicle display. [11] The driving assistance device (700) of any one of claims 1 to 10, wherein the processor (702) is further configured to identify one or more patterns within the road surface data; wherein generating the driving instruction based on the risk map comprises generating the driving instruction based on the one or more identified patterns. [12] The driving assistance device (700) according to any one of claims 2 to 11, wherein the vehicle is a first vehicle; and wherein the processor (702) is further configured: to detect a second vehicle in the data surrounding the first vehicle; and To generate road surface data based on movement of the detected second vehicle. [13] The driving assistance device (700) according to any one of claims 2 to 12, wherein the processor (702) is further configured to: detects in the data of the vehicle's surroundings a shadow cast by a headlight of the vehicle shining on a road surface irregularity; and Road surface data is generated based on the detected shadow. [14] A method for generating driving assistance, comprising: Creating a risk map using road surface data and vehicle data; and Generation of an instruction based on the risk map; wherein the risk map represents risk areas in relation to a road surface near a vehicle; and where the road surface data represents road surface irregularities near the vehicle. [15] A non-transitory computer-readable medium containing instructions that, when executed by one or more processors, cause the one or more processors to: to create a risk map using road surface data and vehicle data; and to prepare an instruction based on the risk map; wherein the risk map represents risk areas in relation to a road surface near a vehicle; and where the road surface data represents road surface irregularities near the vehicle. [16] Vehicle having a driver assistance device according to one of claims 1 to 13.

Images