Control device, method, and computer program product

Abstract

The subject according to the present invention relates to a control device, method, and computer program product for controlling a driver assistance system of a vehicle.

Description

Control device, method, and computer program product

[0001] The present subject matter relates to control devices, methods, and computer program products for controlling driver assistance systems in a vehicle.

[0002] Current advanced driver assistance systems (ADAS) use onboard sensors installed in vehicles to detect obstacles in the area around the vehicle and intervene if there is a risk of collision. To avoid false triggers, an obstacle is continuously observed / detected several times before activating intervention. This increases the reliability of calculations of the position and velocity of the detected obstacle, for example. However, repeated detections require a certain observation time, which may then force the ADAS to brake heavily, for example, which may cause discomfort to the vehicle driver. This situation may occur especially when an obstacle suddenly appears from a blind spot in front of or behind the vehicle. One potential way to mitigate this situation is to use V2X (vehicle-to-everything) communications to obtain information beyond the detection range of the vehicle's onboard sensors.

[0003] Patent Literature 1 describes a system and apparatus for detecting moving objects that enter the field of view of a vehicle's camera device. The system uses a positioning device installed in the vehicle itself and position information periodically received from a mobile terminal carried by the moving object. When a moving object enters the field of view of the camera, a detection area for the mobile unit is set in the image captured by the camera based on the position information of the mobile terminal of the mobile unit and the vehicle's own traveling direction, and the mobile unit is detected.

[0004] WO2015 / 098510A1JP2008-293009A

[0005] However, direct control of ADAS by external devices is difficult because it is essential to avoid erroneous activation of external devices due to delays, unstable transmission, and malicious transmission of V2X information. Therefore, it is important to evaluate the V2X information (e.g., in terms of the specifications and application scope of the external device) and initiate ADAS functions using only suitable parameters of the V2X information.

[0006] While the above issues have been addressed previously, the use of a mix of information originating from on-board sensors and V2X information (external sensor information) poses further challenges that are addressed by the present disclosure.

[0007] More specifically, for example, one of the typical application scenarios of V2X relates to blind spot warning, which means that the blind spot warning alerts the driver when an obstacle is detected in the blind spot area of ​​the on-board sensor(s).

[0008] Patent Literature 2 describes such blind spot warning to the driver of the own vehicle by using V2X information in addition to the own vehicle sensors. The system uses the following logic: if an obstacle is detected by both, i.e., by on-board sensor(s) and V2X, the obstacle is not considered to be an obstacle inside the blind spot (i.e., blind spot area); on the other hand, if the obstacle is detected only by V2X, the obstacle is considered to be an obstacle inside the blind spot area.

[0009] However, some situations that may occur during driving are not resolved / addressed by this approach. For example, a) this approach cannot handle situations where the onboard sensors cannot detect an obstacle due to detection failure, etc. If an obstacle is detected only by V2X but the detected obstacle is within the detection area of ​​the onboard sensors (not in a blind spot area), the ADAS or automated driving system (AD) should provide a warning to the driver, such as a sensor failure condition warning and / or an unreliable status of the ADAS. This warning supports the driver in understanding that ADAS or AD driving is unreliable and that the driver should drive on their own. b) Furthermore, the above-described prior art approach has difficulties with prioritizing warnings. For example, when the host vehicle is traveling in a forward direction, it is a very dangerous situation when the first obstacle is located in front of the host vehicle and within a short distance to the host vehicle. In this situation, a first obstacle ahead of the host vehicle blocks the sensor's detection area, so that other obstacles behind the first obstacle, such as a second, third, etc., can be detected only by V2X. In other words, this situation may occur when a first, close obstacle blocks the sensor's view, thereby creating a huge blind spot, in which additional obstacles (more distant) may be located, and the additional obstacles can be detected only by V2X due to the blind spot created by the first obstacle. In such a situation, the driver would naturally not accept a (strong) blind spot warning for the second, third, etc., obstacles (more distant) because the host vehicle is already in a dangerous situation facing the very close first obstacle, and in this case, the driver should not be distracted by a blind spot warning that is not very urgent at this moment.

[0010] The subject matter described herein addresses the technical object of improving driving safety in vehicles equipped with driver assistance systems and / or automated driving systems, in particular in situations where obstacles are detected by V2X rather than by on-board sensor(s). This object is achieved by the subject matter of the independent claims. Further preferred developments are described by the dependent claims.

[0011] According to the subject matter specified in the appended claims, a control device, a method, and a computer program product are proposed. Furthermore, an ego-vehicle equipped with a control device of such a computer program product is also part of the present disclosure. In particular, the subject matter disclosed herein improves the performance of a driver assistance or automated driving system in the event of detection of an obstacle by V2X rather than by on-board sensor(s) of the ego-vehicle.

[0012] The driver assistance system controlled by the disclosed subject matter may be, for example, automatic emergency braking (AEB), adaptive cruise control (ACC), or lane change assist (LCA). Other types of driver assistance systems may also be combined with the proposed subject matter. The system may also include or further have automated driving capabilities.

[0013] One aspect of the present disclosure relates to a control device for controlling a driver assistance system of a vehicle, i.e., an ego-vehicle, the control device including an on-board sensor obstacle information acquisition unit configured to acquire obstacle information detected by one or more on-board sensors covering a sensor detection area and a blind spot area, which is an area outside the sensor detection area. In other words, the term sensor detection area is used for an area covered by the field of view of an on-board sensor(s) of the ego-vehicle, and a blind spot or blind spot area is an area not covered by the on-board sensor(s) of the vehicle. However, an external sensor, i.e., V2X information provided to the ego-vehicle, may cover the blind spot area.

[0014] Furthermore, the control device has an on-board sensor sensing functionality status acquisition unit configured to acquire functionality status (functionality status information) of one or more on-board sensors, which will be described below in connection with various examples. The functionality status may cover reliability information of the on-board sensor(s), degradation of the on-board sensor's sensing area due to various factors, size, shape, dimensions of the sensor's sensing area of ​​the host vehicle, etc.

[0015] Furthermore, the control device comprises an external sensor obstacle information acquisition unit configured to acquire obstacle information from one or more external sensors, where obstacle information may in particular mean wirelessly received V2X information from other external sensors, such as sensors mounted on other vehicles, to roadside sensor units, etc.

[0016] The control device further includes an assistance determination unit configured to determine whether an external sensor-detected obstacle is located within the blind spot area. The term external sensor-detected obstacle specifically refers to an object or obstacle detected only by the external sensor. Additionally or alternatively, the control device may also cover a situation where both the onboard sensor and the external sensor detect an obstacle, but the reliability of the external sensor is higher than that of the onboard sensor. The situation where the reliability of the external sensor is higher than that of the onboard sensor may be determined by any unit or instance of the control device, for example, when a self-diagnosis detects a failure of the onboard sensor or when bad weather may reduce the reliability of the onboard sensor(s). This may preferably be included in the functionality status information.

[0017] Preferably, a check whether an external sensor detected obstacle is inside the blind spot area is performed by / against functionality status information, which may include the sense in which the actual shape of the sensor detection area is taken into account, its reliability, etc. Such information preferably originates from an input to the assistance determining unit, in particular from a detection functionality status obtaining unit of the on-board sensor.

[0018] Based on the information determined by the assistance determination unit, the control device is able to provide more useful outputs to the driver of the host vehicle, thereby increasing safety.

[0019] Preferably, the control device may further include an obstacle identification unit configured to identify obstacles detected by the sensors, where the identification may include identifying the type of obstacle, the class of obstacle, whether two obstacles detected by different sensors are the same, etc. This may further increase the accuracy of the control device and therefore improve reliability and safety.

[0020] Further, the control device includes: a first driver assistance execution unit configured to perform driver assistance based on on-board sensor obstacle information regarding obstacles detected by the on-board sensor(s), the on-board sensor obstacle information including various data such as type, size, pose, speed, etc.; a second driver assistance execution unit configured to perform driver assistance, preferably for the blind spot area, when the assistance determination unit determines that there is an external sensor-detected obstacle in the blind spot area; and a third driver assistance execution unit configured to perform driver assistance when the assistance determination unit determines that there is an external sensor-detected obstacle in the sensor detection area but not in the blind spot area.

[0021] Driver assistance may also include issuing various types of guidance and / or warnings, such as visual guidance displayed on the host vehicle, audio guidance, etc.

[0022] Preferably, the assistance determination unit may be configured to determine whether an external sensor-detected obstacle is in a blind spot area based on the functionality status information, the reliability determined for the onboard(s), and / or the risk. In other words, this determination is preferably made based on an input to the assistance determination unit, which may include various data, as will be described in connection with the examples in the description of the figures.

[0023] Preferably, the control device may include a driving behavior acquisition unit configured to acquire driving behavior of the host vehicle, which may include speed data, direction data, and other data that may be provided by an odometry, satellite navigation system, etc. When the driving behavior acquisition unit is part of the control device, the first driver assistance execution unit may be configured to execute driver assistance based on the on-board sensor obstacle information and the driving behavior information.

[0024] The provision of various driver assistance execution units has the advantage that the information of the assistance decision unit can be put into practice and in particular used to communicate with the driver, resulting in increased driving safety.

[0025] Additionally, the following modifications may be provided:

[0026] The onboard sensor detection functionality status acquisition unit is configured to acquire the sensor detection area of ​​one or more onboard sensors; the assistance judgment unit is configured to determine whether the external sensor-detected obstacle is in a blind spot area based on the position of the external sensor-detected obstacle and the sensor detection area; the second driver assistance execution unit is configured to send a blind spot warning to the driver of the vehicle when the external sensor-detected obstacle exists / is outside the detection area, preferably based on reliability and risk; and the third driver assistance execution unit is configured to send an obstacle caution warning to the driver when the external sensor-detected obstacle exists / is inside the detection area (sensor detection area), preferably based on reliability and risk.

[0027] In this preferred embodiment, the input from the on-board sensor detection functionality status acquisition unit to the assistance determination unit is particularly related to the determination of the sensor detection area, i.e., the size, shape, dimensions, and location of the sensor detection area. In this case, driving safety is enhanced, and the second driver assistance execution unit can issue a blind spot warning, particularly when an external sensor-detected obstacle is detected outside the determined sensor detection area. Since the sensor detection area may change from time to time due to the movement of the vehicle, changes in the surroundings of the vehicle, etc., the sensor detection area is refreshed for each new information or at a certain interval. Furthermore, the third driver assistance execution unit can issue an obstacle warning to the driver when an external sensor-detected obstacle is inside the sensor detection area. In other words, various scenarios are covered to provide the driver with more detailed information.

[0028] Preferably, the third driver assistance execution unit is further configured to send a sensor failure warning if an external sensor-detected obstacle is detected within the detection area more times than a threshold value. The threshold value may be preset. Alternatively, the threshold value may be replaced by a sensor failure counter or a certain period of time during which two or more "sensor failures" occur, where a sensor failure means that the on-board sensor(s) failed to detect an obstacle within the sensor detection area of ​​the on-board sensor and that this obstacle was detected only by an external sensor. This increases the reliability and therefore the safety of the control device.

[0029] Furthermore, the following modifications may be provided to any of the modifications and aspects of the control device provided and described in this disclosure: The on-board sensor sensing functionality status acquisition unit is configured to determine the sensor sensing area from the mounting position, sensing range, and / or sensing angle of the on-board sensor(s).

[0030] Furthermore, the following modifications may be provided to any of the modifications and aspects of the control device provided and described in this disclosure: a surrounding map information acquisition unit configured to acquire information on the position(s) and shape of surrounding buildings from map information is provided, and the sensing functionality status acquisition unit of the on-board sensor is additionally configured to calculate the sensor detection area using the building information. The building information may also cover other information other than information about buildings, such as any static (or dynamic) objects around the host vehicle, such as trees, walls, natural structures, etc. Using this additional information, the sensor detection area can be determined more accurately, for example because buildings may block the sensor detection area of ​​the host vehicle, thereby limiting / reducing the sensor detection area of ​​the host vehicle.

[0031] Further preferred variations of the above aspect(s) may include the following.

[0032] The on-board sensor detection functionality status acquisition unit is configured to acquire (self-)diagnostic information of the on-board sensor(s), which may be provided by the on-board sensor(s), from the respective unit that performs the diagnosis, or from an obstacle detection software program, etc.; the assistance judgment unit is configured to determine whether an external sensor-detected obstacle is in the blind spot area based on the self-diagnostic information; the second driver assistance execution unit is configured to send a blind spot warning to the driver, preferably based on reliability and risk, when an external sensor-detected obstacle is present and the self-diagnostic information indicates a normal status; and the third driver assistance execution unit is configured to send an obstacle caution warning and a sensor failure warning to the driver, preferably based on reliability and risk, when an external sensor-detected obstacle is present and the self-diagnostic information indicates an abnormal status.

[0033] This modification can cover situations where the sensor detection area is reduced due to on-board sensor failure etc., and can be used to adapt the behavior of the driver assistance execution unit.

[0034] Preferably, the on-board sensor detection functionality status acquisition unit is configured to acquire self-diagnosis information including one or more invalid sensor detection areas; the assistance judgment unit is configured to determine whether an external sensor-detected obstacle is in a blind spot area based on the position of the detected obstacle and the one or more invalid detection areas; the second driver assistance execution unit is configured to send a blind spot warning to the driver, preferably based on reliability and risk, when the external sensor-detected obstacle is outside the invalid sensor detection area; and the third driver assistance execution unit is configured to send a sensor failure warning and an obstacle caution warning to the driver, preferably based on reliability and risk, when the external sensor-detected obstacle is inside one or more of the invalid sensor detection areas automatically extracted by known mud or raindrop detection algorithms known in the art and not further described in this specification.

[0035] This modification can cover situations where the sensor detection area is reduced due to on-board sensor failure etc., and can be used to adapt the behavior of the driver assistance execution unit.

[0036] Further preferred variations of the above aspect(s) may include the following.

[0037] The on-board sensor detection functionality status acquisition unit is configured to acquire surrounding environment information; the assistance judgment unit is configured to determine whether an external sensor-detected obstacle is in a blind spot area based on the sensor specifications (which may include the type of on-board sensor, field of view, mounting pose, etc.) and the environment information; the second driver assistance execution unit is configured to send a blind spot warning to the driver when an external sensor-detected obstacle is present and the environment is not classified as a severe condition (for the on-board sensor), preferably based on reliability and risk (preferably meaning that a warning can be issued only when the reliability of the detection result is sufficiently confirmed in a known manner and / or when the situation is identified as having a predetermined or higher risk, which can also be determined based on a known procedure); and the third driver assistance execution unit is configured to send an obstacle caution warning and a harsh environment warning to the driver when an external sensor-detected obstacle is present and the environment is classified as a severe condition (for the on-board sensor), preferably based on reliability and risk.

[0038] Classification of the extreme conditions into different categories can be performed in real time, for example based on a predetermined formula, table, etc.

[0039] This modification can cover situations where the sensor detection area is reduced or the detection of the on-board sensor(s) is less reliable due to external conditions such as heavy rain, snow, dark areas, etc., and can be used to adapt the behavior of the driver assistance execution unit.

[0040] Preferably, the sensing functionality status obtaining unit of the on-board sensor is configured to obtain weather information as the ambient environment information.

[0041] Preferably, the sensing functionality status acquisition unit of the on-board sensor is configured to acquire time information as the ambient environment information.

[0042] Further preferred variations of the above aspect(s) may include the following.

[0043] The on-board sensor detection functionality status acquisition unit is configured to acquire intermediate detection information, for example from an obstacle detection program / algorithm; the assistance judgment unit is configured to determine whether an external sensor-detected obstacle is in a blind spot area based on the position and the position of the intermediate detection information; the second driver assistance execution unit is configured to send a blind spot warning to the driver when the intermediate detection information is not present in proximity to an external sensor-detected obstacle, preferably based on reliability and risk; and the third driver assistance execution unit is configured to send an obstacle caution warning to the driver when the intermediate detection information is present in proximity to an external sensor-detected obstacle, preferably based on reliability and risk.

[0044] Such variations use intermediate (detection) information that may result from possible intermediate steps in the object detection procedure (as known in the art) and may be raw or semi-processed raw data about the detected object. In other words, cases are covered where possible detected artifacts may be included in the assistance decision process, which may increase the time between detection and warning output, so that the driver may be warned early (even if the detection on which the warning is based may later be considered a false positive).

[0045] Further preferred variations of the above aspect(s) may include the following.

[0046] The on-board sensor detection functionality status acquisition unit is configured to acquire the position of an on-board sensor obstacle detected by the on-board sensor(s) and preferably evaluate the risk posed thereby; the assistance judgment unit is configured to determine whether the external sensor detected obstacle is in a blind spot area based on the distance to the detected on-board sensor obstacle; the second driver assistance execution unit is configured to send a blind spot warning to the driver, preferably based on reliability and risk, when the closest on-board sensor obstacle is not close, preferably checked by a predetermined threshold distance; and the third driver assistance execution unit is configured to send a weak warning to the driver, preferably based on reliability and risk, when the distance of the closest on-board sensor obstacle is close, for example, close by less than a predetermined distance.

[0047] This variant can particularly handle situations where two obstacles are detected, one by an on-board sensor and another by an external sensor only, for example, also due to the fact that the sensor detection area may be reduced due to the presence of the first detected obstacle. In such a situation, if the obstacle detected by the first on-board sensor is very close and therefore poses a (high) risk to the driver, the control device may not provide a strong warning about the second detected obstacle in the blind spot area, as this may distract the driver from his / her attempts to deal with the first (very close) detected obstacle.

[0048] Furthermore, preferably, a surrounding map acquisition unit configured to acquire road structure information around the host vehicle may be provided, and the assistance determination unit is configured to determine that an external sensor-detected obstacle is within a blind spot area when the host vehicle is in an intersection.

[0049] Furthermore, preferably, the second driving aid is a warning system configured to indicate to the driver the type and direction of an obstacle.

[0050] Furthermore, preferably, the second driving assistance and the third driving assistance are configured to use object tracking time information for warning determination.

[0051] Furthermore, preferably, the external sensor obstacle information acquisition unit is configured to acquire identity information including a license plate, a blinker on condition, a brake on condition, a vehicle overall length, a vehicle width, and / or a vehicle color, and the obstacle identification unit is configured to use the identity information to identify whether the external sensor detected obstacle and the onboard sensor obstacle are the same obstacle, thereby reducing false detections.

[0052] Furthermore, an aspect may relate to a method for controlling a driver assistance system of a vehicle, including the above-mentioned steps preferably performed by a control device.

[0053] Furthermore, an aspect may relate to a computer program product storable in a memory, comprising instructions that, when executed by a computer, cause the computer to perform the method(s) described above.

[0054] As noted above, the present disclosure and claimed subject matter provides additional benefits to vehicle driver safety by improving ADAS / AD-related control.

[0055] FIG. 1 schematically illustrates units of a control device according to the disclosed subject matter. FIG. 2 schematically illustrates a general example of a control device according to the disclosed subject matter. FIG. 3 schematically illustrates units of a first example control device according to the disclosed subject matter. FIG. 4 illustrates a flowchart of steps performed by a first driver execution unit. FIG. 5 illustrates a flowchart of steps performed for obstacle identification. FIG. 6a schematically illustrates an exemplary sensor detection area. FIG. 6b schematically illustrates an exemplary sensor detection area. FIG. 6c schematically illustrates an exemplary sensor detection area. FIG. 7a schematically illustrates an example of determining a sensor detection area using additional surrounding map information. FIG. 7b schematically illustrates an example of determining a sensor detection area using additional surrounding map information. FIG. 8a schematically illustrates an exemplary front sensor-only scenario for determining a sensor detection area between buildings, etc. FIG. 8b schematically illustrates an exemplary front sensor and corner radar sensor scenario for determining a sensor detection area between buildings, etc. FIG. 8c schematically illustrates an exemplary front sensor and spinning lidar radar sensor scenario for determining a sensor detection area between buildings, etc. FIG. 9 illustrates a flowchart of steps for determining whether an obstacle detected by an external sensor is inside or outside the sensor detection area. FIG. 10a illustrates a flowchart of steps for determining whether to output a blind spot warning. FIG. 10b illustrates an example scenario related to the flowchart of FIG. 10a. FIG. 10c schematically illustrates an example of a blind spot warning related to the flowchart of FIG. 10a. FIG. 11a illustrates a portion of a flowchart of steps for determining whether to output an obstacle caution warning and / or a sensor failure warning. FIG. 11b illustrates another portion of a flowchart of steps for determining whether to output an obstacle caution warning and / or a sensor failure warning. FIG. 11c schematically illustrates an example of a warning related to the flowcharts of FIGS. 11a and 11b. FIG. 12a schematically illustrates a sensor detection area defined by the field of view of an onboard sensor and occluding a building, etc.FIG. 12b illustrates a use case scenario and related technical benefits of the present disclosure related to a specific example of a sensor detection area according to FIG. 12a. FIG. 12c illustrates a use case scenario and related technical benefits of the present disclosure related to a specific example of a sensor detection area according to FIG. 12a. FIG. 12d illustrates a use case scenario and related technical benefits of the present disclosure related to a specific example of a sensor detection area according to FIG. 12a. FIG. 13 schematically illustrates a unit of a second example of a control device according to the disclosed subject matter. FIG. 14a illustrates a schematic diagram of a scenario in which an on-board sensor has completely failed. FIG. 14b illustrates a schematic diagram of a scenario in which an on-board sensor has partially failed. FIG. 15a illustrates steps in a flowchart for processing an assistance decision in the case of a full-area failure of one or more on-board sensors. FIG. 15b illustrates steps in a flowchart for processing an assistance decision in the case of a partial failure of one or more on-board sensors. FIG. 16a illustrates a portion of a flowchart for determining whether to output an obstacle caution warning and / or a sensor failure warning. FIG. 16b shows another portion of a flowchart of steps for determining whether to output an obstacle attention warning and / or a sensor failure warning. FIG. 16c schematically shows an example of a warning associated with the flowcharts of FIGS. 16a and 16b. FIG. 17 schematically shows a unit of a third example of a control device according to the disclosed subject matter. FIG. 18a schematically shows areas of a sensor detection area affected by various environmental conditions. FIG. 18b shows a flowchart of steps for assistance determination associated with environmental conditions as shown by FIG. 18a. FIG. 19a shows a portion of a flowchart of steps for determining whether to output an obstacle attention warning and / or a sensor failure warning. FIG. 19b shows another portion of a flowchart of steps for determining whether to output an obstacle attention warning and / or a sensor failure warning. FIG. 19c schematically shows an example of a warning associated with the flowcharts of FIGS. 19a and 19b. FIG. 20 schematically shows a unit of a fourth example of a control device according to the disclosed subject matter. FIG. 21a schematically shows use of intermediate detection information. FIG. 21b shows a flow chart of the steps of aided decision making related to the use of intermediate detection information.FIG. 22a shows a portion of a flowchart of steps for determining whether to output an obstacle caution warning and / or a sensor fault warning. FIG. 22b shows another portion of a flowchart of steps for determining whether to output an obstacle caution warning and / or a sensor fault warning. FIG. 22c schematically shows an example of a warning associated with the flowcharts of FIGS. 22a and 22b. FIG. 23 schematically shows a unit of a fifth example control device according to the disclosed subject matter. FIG. 24a shows a flowchart of steps associated with a fifth example of the disclosed subject matter for issuing a weak blind spot warning or blind spot warning. FIG. 25a shows a flowchart of steps associated with a fifth example of the disclosed subject matter for issuing a weak blind spot warning. FIG. 25b shows a schematic of an exemplary blind spot warning. FIG. 25c shows a schematic of an exemplary weak blind spot warning.

[0056] 1 illustrates a control device 1 of the present disclosure, including some of the units of the control device 1 described in more detail below. It should be noted that, in general, the control device 1 may be mounted on a vehicle V (not shown in FIG. 1 , which may instead be identified below as the ego-vehicle). The vehicle V includes one or more on-board sensors (or simply sensors) that detect one or more obstacles in the vicinity of the vehicle V. The on-board sensor(s) (not shown) may include, for example, radar sensors, camera sensors, lidar sensors, sonar sensors, and / or any other sensors suitable for detecting obstacles in the vicinity of the vehicle V.

[0057] Additionally, the control device 1 is communicatively connected to one or more external sensor(s) (not shown), which may be connected to the control device 1 via vehicle-to-location (V2X) communications. The external sensor(s) may also be radar sensors, camera sensors, lidar sensors, sonar sensors, GNSS sensors, and / or any other sensors suitable for detecting obstacles in the surroundings of the vehicle V, and the external sensor(s) may, for example, be included in another vehicle (other than the ego vehicle), in a roadside unit, and / or in a mobile device. The communication between the on-board sensor(s) and the control device 1 may be performed in real time, while the communication between the external sensor(s) and the control device 1 may, for example, be performed via a cellular network.

[0058] Both the on-board sensor(s) and the external sensor(s) may detect obstacles in the area surrounding the vehicle V and may determine multiple obstacle parameters including, for example, obstacle type, position, direction, speed, yaw rate, and acceleration of the detected obstacles.

[0059] 1 shows in particular some of the units of the control device 1 according to the present disclosure, the units being indicated by rectangular boxes containing the name / technical function of the unit. The units depicted in FIG. 1 include units that are common to the following embodiments or possible variations of the embodiments not explicitly mentioned, while variations of the unit-specific technical functions are possible. The units and the entire control device 1 may be configured by software, hardware, or a combination thereof.

[0060] The unit of Figure 1 includes an on-board sensor obstacle information acquisition unit 10 that receives (or acquires) data from one or more on-board sensors of the vehicle V via a communication line (not shown) between the unit and the sensor(s) as known, which may be a wired line and / or a wireless line.

[0061] Further, an external sensor obstacle information acquisition unit 11 is provided which receives (or acquires) data from one or more external sensors via a communication line (not shown) between said unit and the external sensor(s) as known, the communication line being wireless.

[0062] Both of the above-mentioned units provide / transmit the acquired sensor information / data to an obstacle identification unit 12, as shown in FIG. 1, which is provided to identify detected obstacles. Identification may involve determining that an object (obstacle) detected by the on-board sensor is the same as an object detected by an external sensor, which determination may also be made by comparing predetermined characteristics of the detected object, including, for example, the location of the detected object. Identification may also involve identifying what type of obstacle has been detected, for example, by a trained Machine Learning / Artificial Intelligence (ML / AI) subunit, which has been trained to identify, for example, humans, other vehicles, traffic signs, etc. For such technical functions, the ML / AI subunit is pre-trained with training data including images of each possible detected object / obstacle.

[0063] The assistance determination unit 13 performs a judgment or discrimination action, as will be described in connection with specific examples below, and based on this result, can select or activate further actions by the control device 1 or a unit of the control device 1. In particular, the assistance determination includes determining whether an "external sensor-detected obstacle" is within a sensor detection area SDA, which specifies the area / field of view covered by one or more on-board sensors of the host vehicle V. An external sensor-detected obstacle is preferably understood as an obstacle / object detected only by an external sensor (and acquired by the respective unit). Alternatively, an external sensor-detected obstacle may also be detected by the external sensor(s) and the on-board sensor(s), but in that case this term is used only if the on-board sensor(s) have low reliability or lower reliability compared to the external sensor(s).

[0064] The assistance determining unit 13 further receives information for making a determination decision / judgment from the on-board sensor sensing functionality status obtaining unit 14. In the following examples, modifications of the on-board sensor sensing functionality status obtaining unit 14 and the information transmitted to the assistance determining unit 13 will be described in detail. It is possible to combine different modifications, that is, to combine and use different information from the following different examples.

[0065] Figure 2 shows further units of the control device 1, with the names of their technical configurations / functions as in Figure 1. In particular, Figure 2 shows functions / units downstream of the assistance decision unit of the control device 1, including, inter alia, preferably three driver assistance execution units 15-17, which may be separate units or a combined unit with at least three different driver assistance execution functions. The driver assistance execution units receive information from a further unit, which is a driving behavior unit 18.

[0066] 2 further shows that the output of the on-board sensor obstacle information acquisition unit 10 flows to an obstacle identification unit 12 and to a first driver assistance execution unit 15 which executes standard / usual / known driving assistance as known from ADAS driving or even (semi-)AD driving. The output of the assistance decision unit 13 is provided to a second driver assistance execution unit 16 and / or a third driver assistance execution unit 17. The functions of these driver assistance execution units are explained in more detail below.

[0067] 3 and the following figures show a first embodiment (embodiment 1) of a preferred specific modification of the control device 1 as generally described above. In addition to the above-mentioned units, a further surroundings map information acquisition unit 14aa is provided, which inputs information to the on-board sensor detection functionality status acquisition unit 14a. In FIG. 3, it is further specified by way of example that the assistance determination unit 13a particularly performs the internal or external determination of external sensor-detected obstacles as described above, the second driver assistance execution unit 16a particularly performs the blind spot warning process, and the third driver assistance execution unit 17a particularly performs the obstacle attention warning and / or sensor fault warning process.

[0068] FIG. 4 shows the steps of a process related to a first driver assistance performed by a first driver assistance execution unit 15, which may, for example, provide a forward collision warning (FCW) function to the driver, or any other type of ADAS or function as currently known in the art, but will be briefly described here. As shown in FIG. 3 , on-board sensor information is input to the first driver assistance execution unit 15, which receives this information, for example, regarding objects / obstacles detected by on-board sensors. The workflow of FIG. 4 then begins by setting a parameter or flag (FLG) “activate_FCW_FLG” to the value 0 (step S1_15) and using an index m having possible values ​​of 1, 2, ..., M (step S2_15). Thereafter, a TTC value in meters, which is the time to collision from the actual position of the host vehicle V to the detected obstacle, is determined and this TTC value is compared with a respective predetermined threshold "TH_TTC" (steps S3, S4_15). If the TTC value is less than the threshold, i.e. if the obstacle detected by the on-board sensors is found to be close to the host vehicle V, activate_FCW_FLG is set to the value 1 (step S5_15), whereas if this is not the case, activate_FCW_FLG remains at the value 0. Subsequently (step S6_15), the first driver assistance execution unit 15 determines whether activate_FCW_FLG is at the value 1. If yes, the FCW is activated (step S7_15), otherwise the FCW is not activated and the flow returns to START.

[0069] 5 shows the steps of the process for obstacle identification performed by the obstacle identification unit 12, which receives obstacle information / parameters OP1[t][M] from on-board sensor(s) as an example, where t denotes time (timestamp / time information) and M denotes a variable M for a matrix of obstacle parameters, and the obstacle identification unit 12 receives obstacle information / parameters OP2[t][N] from external sensor(s), where N is a matrix of obstacle parameters (steps S1, 2_12). Subsequently, an index p is set to zero, n goes from 1, 2,...N, m goes from 1, 2,...M, and a flag "identified_flg[n]" is set to the value zero (steps S3-4a_12). Then, the distance in meters between the obstacle "described" by the detected OP1[t][M] and the detected obstacle OP2[t][N] is determined and a comparison is performed with a threshold "TH_distance" (steps S6, 7_12), which sends either a "YES" if the determined distance is smaller than the threshold, or a "NO" in the other case. If the output is YES, it may further be checked whether attributes are identified. For the latter step, the attribute information of OP2 may be used to identify the object class (car, pedestrian, ...), size (length, width, etc.), V2X information (sending car only), subclass (car, truck, bus, ...), color (white, black, ...), brake lights, blinker lights, license plate number, etc. (steps S8, 9_12). The logic behind the above steps is that a close distance between an onboard detected obstacle and an externally detected obstacle may mean that it is in fact the same obstacle, which is further confirmed by attribute checking, i.e., if the distance is very close and one or more attributes are identical, the respective identified flags are set to 1, which prevents the externally detected obstacle OP2[t][n] from being registered as OPU[t][p] in steps S10_12 and S11_12, and further warning process flows being executed based on it.In other words, this part excludes the misidentified obstacles detected only by the external sensors and is further processed by the assistance judgment unit 13 .

[0070] If identification is possible, i.e., YES, identified_flg[n] is set to 1. In the next step, the value of identified_flg[n] is compared with the value 0, and if the comparison returns YES, an OP2[t][n] entry is registered in OPU[t][p] and p is incremented. The process returns to START as if identified_flg[n] is not equal to the value 0.

[0071] 6a to 6c non-exhaustively illustrate examples having different sensor detection areas (different shapes, sizes, etc.) around the host vehicle V. In this Example 1, the on-board sensor sensing functionality status acquisition unit 14a can determine the accurate and actual sensor detection area SDA of the host vehicle V by using information provided in a database or the like, including sensor specifications such as detection range, detection angle, etc., information from the on-board sensor(s) such as dynamic performance information from the sensors, history of detection results such as maximum distance, maximum angle, etc. Using the information, the sensor field of view (sensor detection area) of the host vehicle V can be determined, and for example, different fields of view as in FIGS. 6a to 6c can be determined.

[0072] Furthermore, in the first embodiment, the surrounding map information acquisition unit 14aa provides map information as shown in Fig. 7a and Fig. 7b. For example, Fig. 7a shows a case where "all" map information is used, but "all" preferably means that information from a certain radius around the actual position of the host vehicle is used. Alternatively, Fig. 7b shows that only surrounding feature amounts of the actual position of the host vehicle V, such as information about the positions and shapes of buildings and the like in the immediate surroundings, are used.

[0073] Using the information received from the surrounding map information acquisition unit 14aa, the onboard sensor sensing functionality status acquisition unit 14a can modify the initially determined sensor detection area(s) SDA by, for example, reducing portions of the sensor detection area where buildings block the field of view of one or more onboard sensors. Non-exhaustive and schematic examples of the results of such modified sensor detection areas are shown in FIGS. 8a to 8c. The upper section (a1, b1, c1) shows the initial sensor detection area SDA, and the grayed-out areas are modified, reduced areas resulting from four exemplary buildings in the vicinity of the ego-vehicle's position. The final sensor detection areas SDA are then shown by (a2, b2, c2) for three different exemplary cases. It is clear that the sensor detection area SDA changes due to the movement of the ego-vehicle V and, in some cases, due to the movement of non-static objects that may block the sensor detection area SDA.

[0074] 9 shows the assistance determination process performed by the assistance determination unit 13a. The assistance determination process starts with index initialization of p=1, 2,... P and setting the flags blindspot_flg[p] and mis_flag[p] to the value of 0 (steps 1-3_13). Further steps 4-8_13a are as follows:

[0075] The OPU[p] received from the previous unit is located in the sensor detection area SDA. This can be done, for example, by projecting the contour of the sensor detection area SDA and the position of the OPU[p] onto a 2D XY map, which allows the relationship between position and contour to be examined. Other known calculation methods can be used alternatively or additionally.

[0076] In the following comparison, it is checked whether OPU[n] is located inside the sensor detection area SDA, if the result is YES, mis_flag[p] is set to the value 1, if the result is NO, blindspot_flg[p] is set to the value 1. Then, the blind spot warning process and / or the obstacle attention warning process are executed as shown in Figures 10a-c and 11a-c, where OPU[n] relates to the obstacle identified as the external sensor detected obstacle.

[0077] 10a shows a flowchart in which the step of possibly activating the blind spot warning is executed in the second driver assistance execution unit 16a. Specifically, a minimum TTC (MinimumTTC) value of 1000 is initially set, p is initialized as 1, 2, ... P, and the value of Age[p] is calculated (steps S1-3_16). The age value indicates the absolute time length / duration from the first direction of the obstacle up to the current / actual time. This calculation can be performed based on known techniques. Subsequently, if the calculated Age[p] is found to be greater than a predetermined threshold TH_AGE_BS, i.e., if YES, a further check is performed as to whether blindspot_flg[p] is 1, and if YES, TTC[p] is calculated by known methods (steps S4-6_16), while if either of the two previous comparison steps is NO, the process jumps to comparing TTC[p] with MinimumTTC (step S7_16). If YES (TTC[p] is smaller than the MinimumTTC value), the MinimumTTC value is set to the TTC[p] value (step S8_16), while if NO, the MinimumTTC value is not changed.

[0078] Subsequently, a further comparison is made between MinimumTTC and TTC_BS (TTC threshold for activating blind spot warning). If MinimumTTC is smaller (YES), the blind spot warning is activated (steps S9, 10_16). If NO, the process returns to START. Fig. 10b shows a potential exemplary scenario in which the ego-vehicle V is driving along a route and there is a person walking towards the ego-vehicle's future route behind a building (a detected obstacle that can only be detected by an external sensor due to the building blocking the SDA). Fig. 10c then shows a possible display image displayed to the driver, such as on the ego-vehicle's screen. In this image, the driver is warned about the person in the blind spot by such an image, whereas the appearance / design of the image may be completely different. Furthermore, a warning sound may be emitted.

[0079] Furthermore, Fig. 11a shows a process executed by the third driver assistance execution unit 17a for executing an obstacle attention warning process and / or a sensor failure warning process. Fig. 11a shows the first step of this process up to "A" (steps S1-8_17), after which the process continues as shown in Fig. 11b (steps S9-13_17). The steps in Fig. 11a are as described for Fig. 10a, except that a predetermined threshold TH_AGE_SM (SM = sensor failure) is used and the miss_flg value is used instead of blindspot_flg. The threshold TH_AGE_SM is configured to check whether the sensor failure condition persists, because the obstacle detection algorithm may cause momentary detection failure and the third driver assistance system should not be activated by a momentary detection failure. In Fig. 11b, the MinimumTTC value is compared with a predetermined TTC_SM value, which is the TTC threshold of obstacles caused by sensor detection failures, and if the MinimumTTC value is lower (YES), an obstacle caution warning is activated (an exemplary and different design / image or warning may be displayed), and the SensorMissCnt value, which counts sensor failures, is increased and compared with a predetermined TH_MissCNT sensor failure count threshold, as shown in the top of Fig. 11c. On the other hand, if the MinimumTTC value is not lower than TTC_SM (NO), the process jumps directly to comparing the SensorMissCnt value with the predetermined TH_MissCNT sensor failure count threshold. If the SensorMissCnt value is higher than the predetermined TH_MissCNT sensor (YES), then additionally or alternatively, a sensor fault alert is activated; if NO, then the sensor fault alert is not activated and the process returns. Each sensor fault alert may appear as shown in the bottom of FIG. 11c.

[0080] FIG. 12 is represented by four schematic diagrams 12a-12d illustrating possible benefits of the subject matter disclosed herein and as specified above, which are particularly beneficial to the driver of the ego-vehicle V, since they can properly and more accurately warn / warn the driver when ADAS driving, AD driving, or any other type of driving support system encounters a fault, problem, or simply in the case of a situation that is difficult for an automated driving / driving support system to handle. FIG. 12a shows that the sensor detection area is reduced due to an obstructing element, such as a building, in the vicinity of the ego-vehicle V. The sensor detection area SDA can be determined by the above process so that the shape / size / dimension / covered area of ​​the onboard sensor detection area is known to the ego-vehicle V / the ego-vehicle's control device at any time. FIG. 12b shows a situation in which an object / obstacle is detected within the sensor detection area by the onboard sensor. This can result in normal driver assistance, which in the present disclosure is preferably processed by the first driver assistance execution unit. FIG. 12b shows multiple detected obstacles, two of which are inside the sensor detection area (i.e., the on-board sensor detection area) and one of which is outside the sensor detection area and can only be detected by the external sensor(s) (V2X information). This situation can also be handled appropriately with a warning of the possible blind spot obstacle (the external one) and guidance by the first driver assistance execution unit regarding the internal obstacle. FIG. 12d then shows a situation that the present subject matter can beneficially handle compared to the prior art, because one of the obstacles inside the sensor detection area is considered to be detected only by the external sensor(s), even though the on-board sensor(s) should be able to detect this obstacle because it is inside the sensor detection area of ​​the on-board sensor.Such a situation then results in two warnings: an obstacle caution warning and a sensor fault warning, which are possible due to the above-described configuration of the control device 1 disclosed in this specification, in particular the work of the assistance judgment unit and the second and third driver assistance execution units.

[0081] In summary, Example 1 enables a blind spot warning to be reliably issued when an obstacle is present in the blind spot of an onboard sensor(s) and is detected by an external sensor(s). Furthermore, for an obstacle detected within the (onboard) sensor detection area (i.e., not in the blind spot area), an obstacle attention warning is issued. If such an obstacle is detected only by the external sensor(s) and not by the onboard sensor(s), a sensor failure warning (sensor failure) may be issued, preferably if non-detection occurs two or more times within a certain period of time. In other words, the present subject matter can actually handle situations when the onboard sensor cannot detect an obstacle due to a detection failure or the like. If an obstacle is detected only by V2X and the detected obstacle is within the detection area of ​​the onboard sensor (not in the blind spot area), the ADAS or automated driving system (AD) provides a warning to the driver, such as a sensor failure condition warning and / or an unreliable ADAS status. In that case, the warning supports the driver to understand that ADAS driving or AD driving is not reliable and that the driver should drive on his / her own, etc.

[0082] 13 shows a second embodiment (embodiment 2) which may be an alternative configuration to embodiment 1. The functionality described in relation to embodiment 2 may be combined with embodiment 1.

[0083] The control device 1 having the units shown in FIG. 13 differs from Example 1 in the following ways: The assistance determination unit 13b receives various input information from the on-board sensor detection functionality status acquisition unit 14b, and in particular, the assistance determination unit 13b receives (self-)diagnosis results / information from the on-board sensor(s) or from the respective control programs of the sensor(s) or other units. Using this information, the assistance determination unit 13b is able to detect a fault in the on-board sensor(s) of the host vehicle V and can notify appropriately. Further details not described in relation to Example 2 shall be considered equivalent or similar to those in Example 1 and described above.

[0084] Indeed, Example 2 is comparable to Example 1 in that Example 2 can take into account situations where the sensor detection area SDA is not fully reliable or has deteriorated, but instead of the sensor detection area SDA being modified due to external objects such as buildings, etc., Example 2 takes into account internal sensor failure or deterioration due to mud, rain, etc.

[0085] Figure 14a shows a situation in which one of the on-board sensor(s) covering the left rear portion of the ego-vehicle V has completely failed. Such information is provided by a sensor, or a sensor control program, or a self-diagnosis program / unit (not shown) provided in the ego-vehicle V or in the control device 1. Figure 14a shows one non-limiting example of self-diagnosis from a radar sensor via the CAN bus. Therefore, the sensor detection area of ​​the ego-vehicle V in Figure 14a does not cover the left rear triangular portion marked in gray.

[0086] Another example is shown in Figure 14b, where the sensor detection area SDA of a forward looking sensor is partially reduced / degraded due to mud, rain, etc. on the sensor, such as a camera, lidar, etc. Thus, the degraded and newly determined sensor detection area SDA results as shown in the bottom part of Figure 14. The determination of the (partially) degraded sensor detection area SDA can be performed, for example, by image analysis as known in the art.

[0087] Issuing a warning is illustrated in conjunction with the flowcharts of FIGS. 15 and 16, which are similar to the flowchart of Example 1. FIG. 15a illustrates a process illustrating a situation in which the assistance judgment unit 13b is notified that a whole-area fault has been detected, as illustrated by the example of FIG. 14a. In such a case, steps S1_13b to S8_13b of the workflow in FIG. 15a are executed, which are identical to the steps described in conjunction with FIG. 9, and therefore their description is omitted. It should be noted that the sensor detection area SDA in the steps in FIG. 15a will naturally be the sensor detection area SDA resulting from the fault information, as illustrated by FIG. 14a. In the case where Example 1 and Example 2 are combined, the sensor detection area may be a combination of FIG. 14a and map-based information, for example, as described in FIG. 8. In the combined case, the assistance judgment unit 13 will receive both types of information from the onboard sensor detection functionality status acquisition unit 14. Then, as in Fig. 10a, a blind spot warning process is executed, and further warning functions are executed (steps S1_17b to S13_17b) as shown by Fig. 16a and Fig. 16b. The steps in Fig. 16a and Fig. 16b are similar to Fig. 11a and Fig. 11b, except that instead of first initializing the failure_flg value to have a value of 0 and proceeding to compare TTC[p] with MinimumTTC as in Fig. 11a, in Fig. 16a, the failure_flg value is first set to 1 before comparing TTC[p] with MinimumTTC. In Fig. 16b, compared to Fig. 11b, an obstacle caution warning is activated after a YES in the comparison between MinimumTTC and TTC_SM without setting sensormisscnt higher as in Fig. 11b. Furthermore, the next step in FIG. 16 is then to check if the failure_flg value is set to 1, if yes, a sensor failure warning is displayed as shown in FIG.

[0088] 15b is a case comparable to FIG. 15a, but further illustrates a situation in which the assistance determination unit 13b is notified that a partial failure of one or more on-board sensors has occurred, for example, due to raindrops, snow, mud, etc., with consequences such as those described in FIG. 14b as one possible example. Thus, the workflow uses the same method / steps as in FIG. 15a, but the blindspot_flg and mis_flag values ​​are set differently compared to FIG. 15a and FIG. 9, i.e., YES and NO are exchanged. In that case, the process flow of FIG. 10a is executed with respect to the second assistance execution unit 16b, and the steps of FIG. 16a and FIG. 16b are executed by the third assistance execution unit 17b as described above.

[0089] It should be noted that, again, the embodiments may be combined with each other, i.e., the input from the on-board sensor sensing functionality status acquisition unit to the assistance determination unit may be combined by providing input information from embodiment 1 and embodiment 2, or by having two on-board sensor sensing functionality status acquisition units providing specific information. Furthermore, the second and third driver assistance execution units may need to be adapted to be able to perform two different processes or combined adaptation processes as described above. The same considerations regarding the possible combinations of embodiment 1 and embodiment 2 also apply to the following embodiments. Also, two or more embodiments of the present disclosure may be combined accordingly.

[0090] FIG. 17 shows a third embodiment (embodiment 3) which again takes into account various information from the on-board sensor sensing functionality status acquisition unit 14c, in particular ambient environment information, for example regarding weather, time of day, etc.

[0091] FIG. 18 a illustrates how the sensor detection area portions marked in gray may be impaired or unreliable in terms of their sensor detection function due to adverse weather conditions, such as heavy rain, that block the respective sensors, e.g., LIDAR, whose information may be wirelessly received by an external information source. Furthermore, the illumination level around the host vehicle V may be detected based on the time of day, the particular sensor, etc., and when the illumination level is low, the onboard sensor(s), e.g., a camera, may be less reliable. Again, as in the previous example, factors such as those shown non-limitingly in FIG. 18 a may result in a reduction of the sensor detection area SDA, or at least in information that part or all of the sensor detection area is more unreliable. When such information is provided to the assistance determination unit 13 c, the adapted workflow of FIG. 18 b may be executed in the assistance determination unit 13 c using the respective sensor detection areas (or reliable parts of the sensor detection areas) and the process flow (steps S1-8_13 c) of FIG. 18 b, otherwise similar to that described before FIG. 15.

[0092] Furthermore, the process flow executed by the third driver assistance execution unit 17c is shown in Figures 19a and 19b (steps S1-13_17c), which is similar to the process flow of Figures 16a and 16b, except that the sensor fault warning is replaced by a harsh environment warning as exemplarily depicted in Figure 19c.

[0093] FIG. 20 illustrates a fourth embodiment (Example 4) as a further modification, in which the assistance determination unit 17d receives intermediate results / information regarding obstacle detection within the sensor detection area SDA. Such information is shown in a non-exhaustive list of examples in FIG. 21 . For example, this may include raw data object detection results as shown in the upper left of FIG. 21a. Such detection results of the onboard sensor(s) are typically first filtered with respect to a confidence check (upper right of FIG. 21a), as is known in the art. Then, the detection results are further filtered by tracking the detected object over a certain period of time (lower left of FIG. 21a), as is known in the art. Then, typically, a final result (lower right of FIG. 21a) is produced. However, in Example 4, detected objects at any stage before the final stage as shown in the lower right of FIG. 21a may be used, such as from the upper left, upper right, or lower left of FIG. 21a, and input to the assistance determination unit as "intermediate detection results / information." Therefore, the assistance judgment workflow as shown by FIG. 21b is executed by the assistance judgment unit, and is modified as follows compared with FIG.

[0094] The assistance determination unit 13d receives all or part of the intermediate detection results / information from the onboard sensor(s) denoted as OPI[Q], then sets indexes p and q, and sets their respective flag values ​​to the value 0. Subsequently, the distance between OPU[p] and OPI[q] is calculated, and then the distance is compared with a predetermined threshold TH_Candidate. If the distance is less than the threshold (YES), the value of mis_flag[p] is set to 1, and if the check returns NO, the value of blindspot_flg[p] is set to 1. Then, the blind spot warning procedure continues as described in connection with FIG. 10a, and the obstacle caution warning procedure continues as shown in FIG. 22. Here, if OPI[q] is present in close proximity to OPU[p], this means that the presence of OPU[p] was initially captured by its own vehicle sensors but was not ultimately detected due to the use of sensors or algorithms, for example, because it was difficult to distinguish an obstacle from background noise. On the other hand, if OPI[q] is not present around OPU[p], this means that OPU[p] was only captured by external sensors, and there is a higher probability that there is an obstacle in the blind spot of its own vehicle sensors.

[0095] In particular, Figures 22a and 22b (steps S1-13_17d) show the workflow as described in relation to Figures 11a and 11b, and the example warning display in Figure 22c is the same as in Figure 11c.

[0096] 23 shows a further fifth example (Example 5), which again has an alternative arrangement for the input to the assistance determination unit 13e, resulting in modifications to the second driver assistance execution unit 16e and the third driver assistance execution unit 17e. This case particularly covers a situation where one detected obstacle blocks part of the sensor detection area SDA, such that further objects / obstacles can only be detected by the external sensor(s). This situation would normally result in a blind spot warning, for example, in the above example, due to the presence of the detected obstacle in the blind spot area. However, if the first detected obstacle is, for example, very close to the host vehicle V, the driver may want to first pay attention to the closer first detected obstacle, and therefore driver safety would be increased if the second (further away) obstacle was not immediately warned of by the blind spot warning, or if the warning were less intrusive. While the prior art cannot handle such a situation, the present disclosure can safely handle such a situation, particularly with the configuration of Example 5.

[0097] The assistance determination unit 13e receives directly or indirectly (from unit 14ee) surrounding map information as in Example 1. The first option is illustrated by Fig. 23. In addition, the on-board sensor sensing functionality status acquisition unit 14e provides input regarding the position and distance of obstacle(s) detected by the on-board sensors.

[0098] The process of obtaining the information by the on-board sensor detection functionality status acquisition unit 14e is shown in Figure 24a, which starts with initializing an index m, and Spress_BSW_FLG, a predetermined threshold for distinguishing whether there is a very close obstacle immediately in front of or next to the vehicle and activating the weak BSW mode, is set to a value of 0, and then the TTC [m] to the obstacle detected by the on-board sensor is calculated and compared with a predetermined threshold TH_TTC (steps S1-4_14e).

[0099] If TTC[m] is lower than the threshold (YES), it means that the obstacle detected by the onboard sensor is very close (closer than the threshold value), so Spress_BSW_FLG is set to a value of 1, while if NO, the process jumps to "RETURN" (step S5_14e).

[0100] Further, the modified assistance judgment unit workflow is shown by Fig. 24b, including steps S1 to S7_13e. It starts by initializing an index p and setting the values ​​of blindspot_flg[p] and weak_blindspot_flg[p] to 0. Then, it checks whether the value of Spress_BSW_FLG is 1. If the value of Spress_BSW_FLG is 1 (YES), weak_blindspot_flg[p] is set to 1; otherwise (NO), blindspot_flg[p] is set to 1. Subsequently, the blind spot warning and weak blind spot warning processes are executed, and Fig. 10a is the case of blind spot warning and weak blind spot warning shown in Fig. 25b (the same as in Fig. 10c), and Fig. 25a is the case of weak blind spot warning.

[0101] Figure 25a shows a modified third driver assistance execution warning process (steps S1-10_17e) that issues a weak blind spot warning comparable to the blind spot warning of Figure 10a, but checks whether the weak blind spot flag is 1, and therefore may result in the issuance of a weak blind spot warning, as exemplarily shown for Figure 25c. In this non-limiting example, the blind spot warning is shown without an audio warning, as in Figure 25c, while the standard blind spot warning may issue an audio warning, for example, as shown in Figure 25b.

[0102] Example 5 particularly covers a situation where the prior art has difficulty in prioritizing warnings. For example, when the ego-vehicle V is traveling in a forward direction, it is a very dangerous situation when there is a first obstacle in front of the ego-vehicle V at a short distance from the ego-vehicle V. In this situation, the first obstacle in front of the ego-vehicle V blocks the detection area of ​​the sensor, so that other obstacles located behind the first obstacle, such as a second obstacle, a third obstacle, etc., can be detected only by V2X. In other words, this situation may occur when a first nearby obstacle blocks the sensor's view, thereby creating a blind spot in which further obstacles (more distant) may be located, and the further obstacles can be detected only by V2X due to the blind spot created by the first obstacle. In such a situation, the driver would naturally not accept a blind spot warning for a second, third, etc. obstacle (more distant) because the vehicle V is already in a dangerous situation facing a first obstacle that is very close, and in that case the driver would not be distracted by a blind spot warning that is not as urgent at this moment.

[0103] In summary, a method, device and / or computer program may be provided that improves driving safety, especially for drivers of vehicles.

[0104] It is further noted that embodiments of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.), or an embodiment combining software and hardware aspects. Further, embodiments of the present disclosure may take the form of a computer program product on a computer-readable medium having computer-executable program code embodied in the medium.

[0105] It should be noted that arrows in the figures may be used to represent communications, transfers, or other activities involving two or more entities. A bidirectional arrow generally indicates that activity can occur in both directions (e.g., a command / request in one direction and a corresponding reply in the other, or a peer-to-peer communication initiated by either entity), although in some situations activity may not necessarily occur in both directions.

[0106] It should be noted that while a unidirectional arrow may generally indicate activity that is exclusive or dominant in one direction, in certain circumstances such directional activity may actually involve activity in both directions (e.g., a message from a sender to a receiver and an acknowledgment from the receiver back to the sender, or the establishment of a connection before a transfer and the termination of the connection after a transfer). Therefore, arrows of this type used in particular figures to represent particular activities are exemplary and should not be considered limiting.

[0107] Aspects, embodiments are described above with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, etc. It will be understood that each block of the flowchart illustrations and / or block diagrams, and / or combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-executable program code.

[0108] Computer-executable program code may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a particular machine such that the program code, executing via the processor of the computer or other programmable data processing apparatus, creates means for implementing the functions / acts / output specified in one or more blocks of the flowcharts, block diagrams, figures, and / or written descriptions.

[0109] These computer-executable program codes may also be stored in computer-readable memory, and the computer-executable program code may instruct a computer or other programmable data processing apparatus to function in a specific manner, such as to produce an article of manufacture including instruction means that implement the functions / acts / output specified in the flowcharts, block diagram block(s), figures, and / or written descriptions.

[0110] The computer-executable program code may also be loaded onto a computer or other programmable data processing apparatus and run on the computer or other programmable data processing apparatus to produce a computer-implemented process such that the program code running on the computer or other programmable data processing apparatus provides steps that implement the functions / acts / output specified in the flowcharts, block diagram block(s), figures, and / or written descriptions. Alternatively, the steps or acts implemented by the computer program may be combined with steps or acts implemented by an operator or human to perform the embodiments.

[0111] Communications networks generally may include public and / or private networks, may include local area networks, wide area networks, metropolitan area networks, storage, and / or other types of networks, and may employ communications technologies including, but in no way limited to, analog, digital, optical, wireless (e.g., Bluetooth), networking, and internetworking technologies.

[0112] It should also be noted that devices may use communication protocols and messages (e.g., messages created, transmitted, received, stored, and / or processed by devices), and such messages may be conveyed over a communication network or medium.

[0113] Unless the context otherwise requires, this disclosure should not be considered limited to any particular communication message type, communication message format, or communication protocol. Thus, communication messages may generally include, but are not limited to, frames, packets, datagrams, user datagrams, cells, or other types of communication messages.

[0114] Unless the context requires otherwise, it will be understood that references to particular communication protocols are exemplary and that alternative embodiments may employ variations of such communication protocols (e.g., modifications or extensions of protocols that may be made from time to time) or other protocols, whether known or that may be developed in the future, as appropriate.

[0115] It should also be noted that logic flows may be described herein to illustrate various aspects, and that the logic flows should not be construed to limit the present disclosure to any particular logic flow or implementation aspect of the logic. The described logic may be divided into different logic blocks (e.g., programs, modules, functions, or subroutines) without changing the overall result.

[0116] In many cases, logic elements may be added, modified, omitted, executed in a different order, or implemented using various logic constructs (e.g., logic gates, looping primitives, conditional logic, and other logic constructs) without changing the overall result.

[0117] The present disclosure may be embodied in many different forms, including, but not limited to, computer program logic for use with a processor (e.g., a microprocessor, microcontroller, digital signal processor, or general-purpose computer), programmable logic for use with a programmable logic device (e.g., a Field Programmable Gate Array (FPGA) or other PLD), discrete components, an integrated circuit (e.g., an Application Specific Integrated Circuit (ASIC)), or any other means including any combination thereof. Computer program logic implementing some or all of the described functionality is typically implemented as a set of computer program instructions that are converted into a computer-executable form, stored intact on a computer-readable medium, and executed by a microprocessor under the control of an operating system. Hardware-based logic implementing some or all of the described functionality may be implemented using one or more appropriately configured FPGAs.

[0118] Computer program logic implementing all or part of the functionality previously described herein may be embodied in various forms, including, but in no way limited to, source code form, computer executable form, and various intermediate forms (e.g., forms produced by an assembler, compiler, linker, or locator).

[0119] Source code may include a series of computer program instructions implemented in any of a variety of programming languages ​​(e.g., object code, assembly language, or high-level languages ​​such as Fortran, C, C++, JAVA, or HTML) for use with various operating systems or operating environments. Source code may define and use various data structures and communication messages. Source code may be in a computer-executable form (e.g., via an interpreter), or source code may be converted into a computer-executable form (e.g., via a translator, assembler, or compiler).

[0120] Computer-executable program code for carrying out operations of embodiments of the present disclosure may be written in an object-oriented, scripting, or non-scripting programming language, such as Java, Perl, Smalltalk, C++, etc. However, computer program code for carrying out operations of embodiments may also be written in conventional procedural programming languages, such as the "C" programming language or similar programming languages.

[0121] Computer program logic implementing all or part of the functionality previously described herein may execute at different times (e.g., simultaneously) on a single processor, or may execute at the same or different times on multiple processors, and may execute under a single operating system process / thread or under different operating system processes / threads.

[0122] Thus, the term "computer process" may generally refer to the execution of a set of computer program instructions, regardless of whether different computer processes are executed on the same processor or on different processors, and regardless of whether different computer processes are executed under the same operating system process / thread or under different operating system processes / threads.

[0123] A computer program may be fixed, either permanently or temporarily, in any form (e.g., source code form, computer executable form, or intermediate form), on a tangible storage medium such as a semiconductor memory device (e.g., RAM, ROM, PROM, EEPROM, or flash programmable RAM), a magnetic memory device (e.g., a diskette or fixed disk), an optical memory device (e.g., a CD-ROM), a PC card (e.g., a PCMCIA card), or other memory device.

[0124] A computer program may be embodied in any form in a signal that is transmittable to a computer using any of a variety of communication technologies, including, but in no way limited to, analog, digital, optical, wireless (e.g., Bluetooth), networking, and internetworking technologies.

[0125] The computer program may be distributed in any form, such as on a removable storage medium accompanied by printed or electronic documentation (e.g., commercially available software), may be preloaded with the computer system (e.g., on a system ROM or fixed disk), or may be distributed from a server or electronic bulletin board via a communications system (e.g., the Internet or the World Wide Web).

[0126] Hardware logic (including programmable logic used in conjunction with a programmable logic device) implementing all or a portion of the functionality previously described herein may be designed using traditional manual methods, or may be designed, captured, simulated, or documented electronically using a variety of tools, such as Computer Aided Design (CAD), hardware description languages ​​(e.g., VHDL or AHDL), or PLD programming languages ​​(e.g., PALASM, ABEL, or CUPL).

[0127] Any suitable computer readable medium may be utilized, including, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or medium.

[0128] More specific examples of computer-readable media include, but are not limited to, an electrical connection having one or more wires, or other tangible storage media, such as a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM, or flash memory), a compact disc read-only memory (CD-ROM), or other optical or magnetic storage device.

[0129] The programmable logic may be fixed, either permanently or temporarily, in a tangible storage medium, such as a semiconductor memory device (e.g., RAM, ROM, PROM, EEPROM, or flash programmable RAM), a magnetic memory device (e.g., a diskette or fixed disk), an optical memory device (e.g., a CD-ROM), or other memory device.

[0130] The programmable logic may be fixed with signals that are transmittable to a computer using any of a variety of communication technologies, including, but in no way limited to, analog, digital, optical, wireless (e.g., Bluetooth), networking, and internetworking technologies.

[0131] The programmable logic may be distributed on a removable storage medium accompanied by printed or electronic documentation (e.g., commercially available software), may be preloaded with a computer system (e.g., on a system ROM or fixed disk), or may be distributed from a server or bulletin board via a communications system (e.g., the Internet or the World Wide Web). Of course, some aspects may be implemented as a combination of both software (e.g., a computer program product) and hardware. Still other embodiments thereof may be implemented entirely in hardware or entirely in software.

[0132] While certain exemplary aspects have been described and illustrated in the accompanying drawings, it is to be understood that such aspects are illustrative and that the embodiments are not limited to the specific configurations and arrangements shown and described, as various other modifications, combinations, omissions, alterations, and substitutions are possible in addition to those set forth in the preceding paragraphs.

[0133] Those skilled in the art will appreciate that various adaptations, modifications, and / or combinations of the described embodiments themselves may be constructed. Accordingly, it is to be understood that, within the scope of the appended claims, the present disclosure may be practiced other than as specifically described herein. For example, unless expressly stated otherwise, the steps of processes described herein may be performed in a different order than described herein, and one or more steps may be combined, separated, or performed simultaneously. Those skilled in the art will also appreciate, in light of the present disclosure, that different embodiments or aspects described herein may be combined to form other embodiments.

[0134] 1 Control device 10 On-board sensor obstacle information acquisition unit 11 External sensor obstacle information acquisition unit 12 Obstacle identification unit 13 Assistance judgment unit 13a Assistance judgment unit 13b Assistance judgment unit 13c Assistance judgment unit 13d Assistance judgment unit 13e Assistance judgment unit 14 On-board sensor detection functionality status acquisition unit 14a Detection functionality status acquisition unit 14aa Surrounding map information acquisition unit 14b Detection functionality status acquisition unit 14c Detection functionality status acquisition unit 14e Detection functionality status acquisition unit 14ee Surrounding map information acquisition unit 15 First driver assistance execution unit 16 Second driver assistance execution unit 16a Second driver assistance execution unit 16b Second driver assistance execution unit 16c Second driver assistance execution unit 16d Second driver assistance execution unit 16e Second driver assistance execution unit 17 Third driver assistance execution unit 17a Third driver assistance execution unit 17b Third driver assistance execution unit 17c Third driver assistance execution unit 17d Third driver assistance execution unit 17e Third driver assistance execution unit 18 Driving behavior unit SDA Sensor detection area V Vehicle

Claims

1. A control device for controlling a driving assistance system of a vehicle, comprising: an on-board sensor obstacle information acquisition unit configured to acquire on-board sensor obstacle information detected by one or more on-board sensors covering a sensor detection area and not covering a blind spot area; an on-board sensor detection functionality status acquisition unit configured to acquire functionality status information of the one or more on-board sensors; an external sensor obstacle information acquisition unit configured to acquire obstacle information from one or more external sensors; and an assistance judgment unit configured to determine whether an external sensor detected obstacle is within the blind spot area based on the functionality status information.

2. The control device of claim 1, further comprising: a first driver assistance execution unit configured to execute driver assistance based on the on-board sensor obstacle information; a second driver assistance execution unit configured to execute driver assistance when it is determined that there is an external sensor-detected obstacle in the blind spot area; and a third driver assistance execution unit configured to execute driver assistance when it is determined that there is an external sensor-detected obstacle in the sensor detection area but not in the blind spot area.

3. The control device of claim 2, wherein the second driver assistance execution unit is configured to send a blind spot warning to a driver of the vehicle notifying the driver that the external sensor-detected obstacle is outside the sensor detection area, and the third driver assistance execution unit is configured to send an obstacle caution warning to the driver notifying the driver that the external sensor-detected obstacle is inside the sensor detection area.

4. The control device according to claim 3, wherein the third driver assistance execution unit is configured to additionally send a sensor failure warning when the assistance determination unit determines that there is an external sensor-detected obstacle within the sensor detection area and that the one or more on-board sensors detect the obstacle a threshold number of times or less.

5. The control device of claim 1, wherein the on-board sensor detection functionality status acquisition unit is configured to determine the sensor detection area from the mounting position, detection range, and / or detection angle of the on-board sensor.

6. The control device of claim 1, further comprising a surrounding map information acquisition unit configured to acquire information on the positions and shapes of surrounding buildings from map information, and the on-board sensor detection functionality status acquisition unit is configured to calculate the sensor detection area by additionally using the information on the positions and shapes of the buildings.

7. The control device of claim 2, wherein the on-board sensor detection functionality status acquisition unit is configured to acquire diagnostic information of the on-board sensor; the assistance judgment unit is configured to determine whether the external sensor-detected obstacle is in the blind spot area based on the self-diagnosis information; the second driver assistance execution unit is configured to send a blind spot warning to the driver when the external sensor-detected obstacle is present and the self-diagnosis information indicates a normal status; and the third driver assistance execution unit is configured to send an obstacle caution warning and a sensor failure warning to the driver when the external sensor-detected obstacle is present and the self-diagnosis information indicates an abnormal status.

8. The control device of claim 7, wherein the on-board sensor detection functionality status acquisition unit is configured to acquire the self-diagnosis information including one or more invalid sensor detection areas; the assistance judgment unit is configured to determine whether the external sensor-detected obstacle is in the blind spot area based on the position of the detected obstacle and the one or more invalid sensor detection areas; the second driver assistance execution unit is configured to send a blind spot warning to the driver when the external sensor-detected obstacle is outside the invalid sensor detection area; and the third driver assistance execution unit is configured to send a sensor failure warning and an obstacle caution warning to the driver when the external sensor-detected obstacle is inside one or more of the invalid sensor detection areas.

9. The control device of claim 2, wherein the on-board sensor detection functionality status acquisition unit is configured to acquire surrounding environment information; the assistance judgment unit is configured to determine whether the external sensor-detected obstacle is in the blind spot area based on the specifications of the on-board sensor and the surrounding environment information; the second driver assistance execution unit is configured to send a blind spot warning to the driver when the external sensor-detected obstacle exists and the surrounding environment is not classified as a harsh condition; and the third driver assistance execution unit is configured to send an obstacle caution warning and a harsh environment warning to the driver when the external sensor-detected obstacle exists and the surrounding environment is classified as a harsh condition.

10. The control device according to claim 9, wherein the on-board sensor sensing functionality status acquisition unit is configured to acquire weather information as the surrounding environment information.

11. The control device according to claim 9, wherein the sensing functionality status acquisition unit of the on-board sensor is configured to acquire time information as the surrounding environment information.

12. The control device of claim 2, wherein the on-board sensor detection functionality status acquisition unit is configured to acquire intermediate detection information, which is information different from the on-board sensor obstacle information used by a first driver assistance execution unit, and the intermediate detection information is information after an obstacle is detected by the one or more on-board sensors but before determining that the detected obstacle is present; the assistance judgment unit is configured to determine whether the external sensor-detected obstacle is in the blind spot area based on a position of the detected obstacle and a position of the intermediate detection information; the second driver assistance execution unit is configured to send a blind spot warning to the driver based on reliability and risk when the intermediate detection information is not present in proximity to the external sensor-detected obstacle; and the third driver assistance execution unit is configured to send an obstacle attention warning to the driver based on the reliability and the risk when the intermediate detection information is present in proximity to the external sensor-detected obstacle.

13. The control device according to claim 2, wherein the on-board sensor detection functionality status acquisition unit is configured to acquire the position of an on-board sensor obstacle acquired by an on-board sensor; the assistance judgment unit is configured to determine whether a detected external sensor detected obstacle is in the blind spot area based on a distance to the detected on-board sensor obstacle; the second driver assistance execution unit is configured to send a blind spot warning to the driver when the distance of a closest on-board sensor obstacle is not within a predetermined distance and is not close; and the third driver assistance execution unit is configured to send a weak warning to the driver when the distance of the closest on-board sensor obstacle is within a predetermined distance and is close.

14. The control device according to claim 13, further comprising a surrounding map information acquisition unit configured to acquire road structure information around the host vehicle, and the assistance determination unit configured to determine that an external sensor-detected obstacle is in the blind spot area when the host vehicle is in an intersection.

15. The control device of claim 2, wherein the second driver assistance execution unit is a warning system configured to indicate to the driver a type and a direction of an obstacle.

16. The control device of claim 2, wherein the second driver assistance execution unit and the third driver assistance execution unit are configured to use object tracking time information for warning decisions.

17. The control device of claim 1, wherein the external sensor obstacle information acquisition unit is configured to acquire identity information including a license plate, a blinker on condition, a brake on condition, a vehicle overall length, a vehicle width, and / or a vehicle color, and the obstacle identification unit is configured to use the identity information to identify whether an external sensor detected obstacle and an on-board sensor obstacle are the same obstacle.

18. A method of controlling a driver assistance system of a vehicle, comprising steps performed by a control device as recited in claim 1.

19. A computer program product storable in a memory comprising instructions which, when executed by a computer, cause the computer to carry out the method of claim 18.

Images