Method and system for determining the presence of a hidden danger

Abstract

Method (1000) for determining the presence of a hidden danger, via a control system, comprising: Identifying an operating scene (200, 400, 600, 700, 800, 900) for a host vehicle (101, 201, 401, 601, 701, 801, 901) based on information that includes information corresponding to the current geographic location of the host vehicle (101, 201, 401, 601, 701, 801, 901); Identifying an operating situation for the host vehicle (101, 201, 401, 601, 701, 801, 901) based on information that includes information corresponding to dynamic conditions within the operating scene (200, 400, 600, 700, 800, 900); Collecting and classifying information from a variety of proximity sensors (119), wherein the proximity sensors (119) are connected to the host vehicle (101, 201, 401, 601, 701, 801, 901) and include an ambient light sensor, a vision system, an acoustic sensor and include a seismic sensor; Estimating a multitude of hidden hazard presence probabilities according to the information from each of the multitude of proximity sensors (119), the operating scene (200, 400, 600, 700, 800, 900), the operating situation and a comparison process (1013); and performing a fusion process with the multitude of hidden hazard presence probabilities to determine the presence of a hidden hazard; where the comparison process (1013): a) retrieving rules corresponding to the operating scene (200, 400, 600, 700, 800, 900) and the operating situation, defining overlap and non-overlap zones within a headlight pattern of the host vehicle (101, 201, 401, 601, 701, 801, 901), creating brightness comparisons for the zones, and comparing the brightness comparisons with collected and classified areas of overlapping light within the headlight pattern of the host vehicle (101, 201, 401, 601, 701, 801, 901); or b) retrieving acoustic signatures corresponding to the operating scene (200, 400, 600, 700, 800, 900) and the operating situation, and comparing the signatures with collected and classified acoustic waveforms; or c) includes retrieving seismic signatures that correspond to the operating scene (200, 400, 600, 700, 800, 900) and the operating situation, and comparing the signatures with collected and classified seismic waveforms.

Description

[0001] The description refers to situational awareness in road vehicles.

[0002] Vehicle systems are known to monitor the vehicle's surroundings to enhance the driver's situational awareness; examples include forward and reverse distance, distance rate, and vision systems. Such systems can be used to provide the driver with warnings and control inputs regarding objects, including other vehicles. These systems can be employed in autonomous and semi-autonomous vehicle control systems, such as adaptive cruise control, parking assistance, lane keeping assist, and blind spot warnings for adjacent lanes. However, known system capabilities and implementations primarily focus on line-of-sight detection.

[0003] US 2007 / 0286475A1 describes an object detection unit for detecting an object present around the vehicle using a plurality of sensors, comprising: an existence probability calculation unit configured to calculate each existence probability of the object for each sensor based on a normal distribution centered on an output value of each sensor; an existence probability correction unit configured to calculate each corrected existence probability by correcting each of the aforementioned existence probabilities with a detection rate of each sensor; and a fusion existence probability calculation unit configured to calculate a fusion existence probability of the object by fusing each of the aforementioned corrected existence probabilities.

[0004] DE 10 2009 006 113 A1 describes a device and a method for providing an environment representation of a vehicle with at least one first sensor device and at least one second sensor device as well as an evaluation device, wherein the sensor devices provide information about objects detected in an environment of the vehicle in the form of sensor objects, wherein a sensor object represents an object detected by the respective sensor device, and the sensor objects include as an attribute at least a probability of existence of the represented object, and the sensor objects detected by the at least one first sensor device and by the at least one second sensor device are subjected to an object fusion in which fusion objects are generated to which at least a probability of existence is assigned as an attribute.wherein the existence probabilities of the fusion objects are fused based on the existence probabilities of the sensor objects, wherein the fusion of the existence probability of one of the sensor objects is carried out depending on the respective sensor device from which the corresponding sensor object is provided.

[0005] German patent DE 10 2019 205 607 A1 describes a predictive lane-change system for supporting a host vehicle currently in a lane and adjacent to a neighboring lane. The predictive lane-change system may include an identification module that identifies a potential lane-change location and receives data on nearby vehicles associated with it. The predictive lane-change system may also include a forecasting module that predicts future kinematic data for a number of nearby vehicles at a future time. Finally, the predictive lane-change system may include a determination module that determines whether a gap will be available at the potential lane-change location at that future time, based on the future kinematic data.The predictive lane-change system may include a lane-change module that initiates a lane-change maneuver for the host vehicle in response to a determination that the gap will be available at a future time at the potential lane-change location.

[0006] It can be considered a task to specify an improved procedure and an improved system for determining the presence of a hidden danger.

[0007] An inventive method for determining the presence of a hidden hazard, via a controller, comprises identifying an operating scene for a host vehicle based on information corresponding to the current geographic location of the host vehicle; identifying an operating situation for the host vehicle based on information corresponding to dynamic conditions within the operating scene; collecting and classifying information from a plurality of proximity sensors, wherein the proximity sensors are connected to the host vehicle and include an ambient light sensor, a vision system, an acoustic sensor, and a seismic sensor; and estimating a plurality of hidden hazard presence probabilities based on the information from each of the plurality of proximity sensors, the operating scene, and thethe operational situation and a comparison process, and include performing a fusion process with the multitude of hidden hazard presence probabilities to determine the presence of a hidden hazard. The comparison process includes: a) retrieving rules that correspond to the operational scene and the operational situation and define overlap and non-overlap zones within a headlight pattern of the host vehicle, including creating brightness comparisons for the zones and comparing the brightness comparisons with collected and classified areas of overlapping light within the headlight pattern of the host vehicle; or b) retrieving acoustic signatures that correspond to the operational scene and the operational situation, including comparing the signatures with collected and classified acoustic waveforms; or c) retrieving seismic signatures,which correspond to the operational scene and situation, and includes comparing the signatures with collected and classified seismic waveforms.

[0008] A system according to the invention for determining the presence of a hidden danger comprises a host vehicle and a plurality of proximity sensors connected to the host vehicle, including an ambient light sensor, a vision system, an acoustic sensor and a seismic sensor.The system may further include a controller configured to identify an operating scene for the host vehicle based on information corresponding to the host vehicle's current geographic location, to identify an operating situation for the host vehicle based on information corresponding to dynamic conditions within the operating scene, to collect and classify information from the multitude of proximity sensors, to estimate a multitude of hidden hazard presence probabilities corresponding to the information from each of the multitude of proximity sensors, the operating scene, the operating situation, and a comparison process, and to perform a fusion process with the multitude of hidden hazard presence probabilities to determine the presence of a hidden hazard.The comparison process includes: a) retrieving rules that correspond to the operating scene and situation and define overlap and non-overlap zones within a headlight pattern of the host vehicle, creating brightness comparisons for the zones, and comparing the brightness comparisons with collected and classified areas of overlapping light within the headlight pattern of the host vehicle; or b) retrieving acoustic signatures that correspond to the operating scene and situation and comparing the signatures with collected and classified acoustic waveforms; or c) retrieving seismic signatures that correspond to the operating scene and situation and comparing the signatures with collected and classified seismic waveforms.

[0009] The description refers to the drawings in which: Fig. 1 shows an exemplary system for situation recognition in the case of hidden dangers; Fig. 2 shows an exemplary vehicle operation scene that does not present any hidden danger; Fig. Figure 3 shows an exemplary vehicle operation scene illustrating an exemplary hidden danger; Fig. Figure 4 shows an exemplary vehicle operation scene illustrating an exemplary hidden danger; Fig. Figure 5 shows an exemplary vehicle operation scene, illustrating hidden dangers; Fig. Figure 6 shows an exemplary vehicle operation scene illustrating an exemplary hidden danger; Fig. Figure 7 shows an exemplary vehicle operation scene illustrating an exemplary hidden danger; Fig. Figure 8 shows an exemplary vehicle operation scene illustrating an exemplary hidden danger; Fig. Figure 9 shows an exemplary vehicle operation scene illustrating exemplary hidden dangers; and Fig. Figure 10 shows an exemplary procedure for assessing hidden dangers.

[0010] The following description is merely exemplary and is not intended to limit the present description, its application, or use. In the drawings, corresponding reference numerals denote identical or corresponding parts and features. As used herein, control module, module, controller, control unit, electronic control unit, processor, and similar terms mean one or more combinations of one or more application-specific integrated circuits (ASICs), electronic circuits, central processing units (preferably microprocessors), and associated memory (read-only memory (ROM), random-access memory (RAM), electrically programmable read-only memory (EPROM), hard disk, etc.).A control module may consist of a microcontroller or other device executing one or more software or firmware programs or routines, combinational logic circuitry, input / output (I / O) circuitry and devices, suitable signal conditioning and buffering circuitry, high-speed clocks, analog-to-digital (A / D) and digital-to-analog (D / A) circuitry, and other components to provide the described functionality. A control module may include a variety of communication interfaces, including point-to-point or discrete lines and wired or wireless interfaces to networks, including wide area networks (WANs) and local area networks (LANs), in-vehicle control networks, and factory and service networks. Control module functions as described herein may be implemented in a distributed control architecture across multiple networked control modules.Software, firmware, programs, instructions, routines, code, algorithms, and similar terms refer to any control-executable instruction sets, including calibrations, data structures, and lookup tables. A control module has a set of control routines that are executed to provide described functions. Routines are executed, for example, by a central processing unit (CPU) and can monitor inputs from sensor devices and other networked control modules, and execute control and diagnostic routines to manage the operation of actuators. Routines can be executed at regular intervals during normal engine and vehicle operation. Alternatively, routines can be executed in response to the occurrence of an event, software calls, or on demand via user interface inputs or requests.

[0011] During semi- or fully autonomous operation of a vehicle on the road by a driver, the vehicle can be an observer in an operating scene and situation. An operating scene (scene) is generally understood to be the essentially static driving environment, e.g., the roadway and surrounding infrastructure, while an operating situation (situation) is generally understood to be the essentially kinetic, dynamic, and temporal conditions within the scene, such as other vehicles on the roadway, objects, and hazards. An observing vehicle can be referred to here as the host vehicle. Other vehicles sharing the roadway can be referred to here as target vehicles.

[0012] A host vehicle can be equipped with various sensors and communication hardware and systems. An example of a host vehicle, 101, is in Fig. Figure 1 illustrates an exemplary system 100 for situation detection in the case of hidden hazards, as described herein. The host vehicle 101 may contain a control system 102 comprising a plurality of networked electronic control units (ECUs) that may be communicatively coupled via a bus structure 111 to perform control functions and information exchange, including the execution of control routines locally or in a distributed manner. The bus structure 111 may be part of a Controller Area Network (CAN) or other similar network known to those skilled in the art. An exemplary control unit may include an engine control module (ECM) 115 that primarily performs functions related to the monitoring, control, and diagnosis of internal combustion engines based on a plurality of inputs 121.While the inputs 121 are depicted as being directly coupled to the ECM 115, the inputs to the ECM 115 can be provided by a variety of known sensors, calculations, derivations, syntheses, other control units, and sensors via the bus structure 111, or determined within the ECM 115, as is understandable to a person skilled in the art. A person skilled in the art recognizes that a variety of other control units 117 can be part of the network of control units on board the host vehicle 101 and can perform other functions relating to various other vehicle systems (e.g., chassis, steering, brakes, transmission, communication, infotainment, etc.). A variety of vehicle-related information can be generally available and accessible to all networked control units, e.g., vehicle dynamics information such as speed, heading, steering angle, multi-axis accelerations, yaw, pitch, roll, etc.

[0013] Another exemplary control unit may include an external object computation module (EOCM) 113, which primarily performs functions related to sensing the environment outside the vehicle 101, and in particular related to roadway, surface, and object detection. EOCM 113 receives information from a variety of sensors 119 and other sources. By way of example only, and without limitation, EOCM 113 may receive information from one or more radar systems, lidar systems, ultrasonic systems, vision systems (e.g., cameras), global positioning systems (GPS), vehicle-to-vehicle communication systems, and vehicle-to-infrastructure communication systems, as well as from on-board or off-board databases, e.g., map, road segment, navigation, and infrastructure information, and crowdsourced information.GPS and database information can provide much of the driving scene information, while the multitude of sensors can provide much of the driving situation information. The EOCM can have access to position and speed data of the host vehicle, line-of-sight, distance, and speed data of the target vehicle, and image-based data that can be useful in determining or validating road and target vehicle information, such as road features and geometric, distance, and speed information of the target vehicle, to name a few. However, many such acquisition systems are limited in their usefulness or have been restricted in their application to objects, including other vehicles, within an incomplete line of sight.According to the present description, certain other sensor technologies can be used to improve situational awareness of hidden hazards, such as ambient light sensors, acoustic sensors, and seismic sensors. Similarly, image processing systems can be adapted to improve situational awareness of hidden hazards, as described above. It is therefore understood that certain sensors can be considered line-of-sight (LOS) sensors in the sense that they rely on the direct detection of objects within the scene and depend primarily on an unobstructed line of sight between the sensor and the detected object. Examples of such LOS sensors include radar, lidar, ultrasonic, and vision sensors.In contrast, certain sensors can also be considered proximity sensors in the sense that they rely on the indirect detection of objects that are not within the line of sight between the sensor and the detected object. Such proximity sensors detect influences or excitations within the scene, which can be processed to infer the presence of the object from which the influences or excitations originate. Examples of such proximity sensors include ambient light sensors, acoustic sensors, seismic sensors, and vision sensors. The sensors 119 can be positioned at various circumferential points around the vehicle, e.g., front, rear, corners, sides, etc., as represented in vehicle 101 by large dots at these positions. Other sensor positioning is conceivable and may include forward-facing sensors through the vehicle's windshield, which, for example,The sensors are mounted in front of a rearview mirror or integrated into such a mirror assembly. The positioning of the sensor 119 can be selected to provide the desired coverage for specific applications. For example, positioning the sensors 119 at the front and front corners, and otherwise at the front, may be preferred to improve the situational awareness of hidden hazards while driving forward, in accordance with the present description. However, it is acknowledged that an analogous arrangement of the sensors 119 at the rear or facing backward may be preferred to improve the situational awareness of hidden hazards while driving in reverse, as described in the present description. Certain sensors, such as seismic sensors, may not be primarily sensitive to directional information, and therefore their placement may not be position-critical.Seismic sensors can be mounted on the sprung or unsprung mass of the host vehicle. In one embodiment, seismic sensors can be integrated into the unsprung mass of the chassis on one or more tire pressure monitors (TPMs) connected to each wheel. Known TPMs can be mounted on one end of a tire valve stem on the inside of the wheel or on the opposite end of the valve stem, where the valve stem cap is typically located. Known TPMs advantageously use low-power, high-frequency communication to transmit information to the associated vehicle. It is known that TPM systems use remote keyless entry (RKE) control units to receive TPM signals.A person skilled in the art can therefore easily adapt known TPMs to transmit seismic information to the associated vehicle to improve situational awareness of hidden hazards, in accordance with the present description. While the sensors 119 are shown to be directly coupled to the EOCM 113, the inputs can be provided to the EOCM 113 via the bus structure 111, as is familiar to those skilled in the art. The host vehicle 101 can be equipped with radio communication capabilities, which are generally described in 123 and relate in particular to GPS satellite communication 107, vehicle-to-vehicle (V2V) communication, and vehicle-to-infrastructure (V2I) communication, e.g., with terrestrial radio towers 105. The present description of the exemplary system 100 for situational awareness of hidden hazards does not claim to be exhaustive.Furthermore, the description of the various exemplary systems should not be interpreted as exhaustive. Someone with ordinary technical knowledge will understand that some, all, and additional technologies from the described exemplary system 100 can be used in various implementations of hidden hazard situational awareness in accordance with the present description.

[0014] The Fig. Figures 2-9 show a variety of exemplary vehicle operation scenes according to the present description. In scene 200 of Fig. 2. A host vehicle 201 and a first target vehicle 203 can travel in the same direction on different lanes of a roadway. Both vehicles can have their headlights on. Both vehicles have a respective headlight illumination pattern that is essentially directed forward in their respective direction of travel. In addition, both vehicles can travel so close to each other that their respective headlight patterns overlap in an area 209 that is brighter than the non-overlapping areas 205, 207. In Fig. 2. Area 205 is primarily illuminated by the headlights of the host vehicle 201, while area 207 is primarily illuminated by the headlights of the first target vehicle 203. According to the present description, illumination areas 205 and 209 can be detected by the forward-looking sensor(s) of the host vehicle, such as one or more vision systems and ambient light sensors. The host vehicle 201 can distinguish area 209 from area 205 and recognize that area 209 is brighter due to a contributing, additive light source within the host vehicle's headlight illumination pattern area. The host vehicle 201 can also detect the presence and position of the first target vehicle 203 and attribute the additional light source to the headlight system of the first target vehicle.

[0015] The Fig. Figure 3 shows the same scene 200 and situation in relation to the host vehicle 201 and the first target vehicle 203. Fig. Figure 3 additionally shows a second target vehicle 202 traveling in the opposite direction in a left-turn lane, approaching at an angle to cross the lanes of the lead vehicle 201 and the first target vehicle 203. The second target vehicle 202 may be hidden from the line of sight of the driver of the host vehicle 201, as the second target vehicle is obscured by the intervening first target vehicle 203. The second target vehicle 202 may also operate with its headlights on. All vehicles have a corresponding headlight beam pattern, essentially directed forward in their respective directions of travel. Furthermore, all vehicles may travel so close to each other that their respective headlight patterns overlap in certain areas. In the scenario of Fig. Area 215 can overlap with the headlight patterns of host vehicle 201 and the second target vehicle 202; area 209 can overlap with the headlight patterns of host vehicle 201 and the first target vehicle 203; area 211 can overlap with the headlight patterns of the first target vehicle 203 and the second target vehicle 202; area 213 can overlap with the headlight patterns of host vehicle 201, the first target vehicle 203, and the second target vehicle 202. Areas 205 and 207 can be primarily illuminated by the headlight pattern of host vehicle 201. According to the present description, the lighting areas 205, 207, 215, 213 and 209 can be distinguished from the forward-facing sensor(s) of the host vehicle, for example, one or more vision systems and ambient light sensors.The host vehicle 201 can differentiate these areas based on brightness and recognize that areas 209, 213, and 215 are brighter than areas 205 and 207 due to a contributing, additive light source within the host vehicle 201's headlight illumination pattern area. Similarly, the host vehicle 201 can recognize that area 213 is brighter than areas 205, 215, and 209 due to a contributing, additional light source within the host vehicle's headlight illumination pattern area. The host vehicle 201 can detect the presence and position of the first target vehicle 203 and attribute the additional light source in area 209 to the headlight system of the first target vehicle 201. Based on scene and situation information, the host vehicle 201 can deduce that the additive light sources in areas 213 and 215 originate from an obscured target vehicle (second target vehicle 202).

[0016] Similarly, it shows Fig. 4 an alternative scene 400 and situation in relation to the host vehicle 401 and the first target vehicle 403. Fig. Figure 4 shows the first target vehicle 403, which is traveling in the right-hand lane of the respective roadway. The first target vehicle 403 may be obscured from the view of the host vehicle 401 by the building 411. In one embodiment, both vehicles may be traveling with their headlights switched on. In an alternative embodiment, only the obscured target vehicle 403 may be operating with its headlights switched on. Both vehicles are shown with their respective headlight beam patterns, which are directed substantially forward in the respective direction of travel. Furthermore, both vehicles may be traveling so close to each other that their respective headlight beam patterns overlap in an area 409 that is brighter than the non-overlapping areas 405, 407, and 413. In the embodiment in which only the target vehicle 403 is traveling with its headlights switched on, only the lighting pattern of the target vehicle 403 is projected. Fig. 4. Areas 405 and 413 are primarily illuminated by the headlights of the host vehicle 401, while area 407 is primarily illuminated by the headlights of the first target vehicle 403. According to the present description, the illumination areas 405, 407, 413, and 409 can be distinguished by the forward-facing sensor(s) of the host vehicle 401, for example, by one or more vision systems and ambient light sensors. The host vehicle 401 can distinguish area 409 from areas 405, 407, and 413 and recognize that area 409 is brighter due to a contributing, additive light source within the headlight illumination pattern area of ​​the host vehicle 401. Based on the scene and situation information, the host vehicle 401 can conclude that the additive light source of area 409 and the light source of area 407 are due to a concealed target vehicle (first target vehicle 403).In the alternative embodiment, where only the target vehicle 403 is driving with its headlights switched on, the area illuminated by the headlights of the target vehicle 403 can be detected by the forward-looking sensor(s) of the host vehicle 401, for example, by one or more vision systems and ambient light sensors. The host vehicle 401 can distinguish the headlight-illuminated area in front of the target vehicle 403 from other areas and recognize that the headlight-illuminated area is brighter due to the headlight illumination pattern of the target vehicle. Based on the scene and situation information, the host vehicle 401 can infer that the additive light source of the headlight illumination area is due to an obscured target vehicle (the first target vehicle 403). In both embodiments, the host vehicle 401 can detect areas with additive light or areas with different light brightness.

[0017] The Fig. Figure 5 shows the same scene 400 and situation in relation to the host vehicle 401 and the first target vehicle 403. Fig. Figure 5 additionally shows a second target vehicle 402 traveling in the same direction as the first target vehicle 403 in the left lane of the respective roadway. The second target vehicle 402 may be hidden from the line of sight of the driver of the host vehicle 401, with the second target vehicle being obscured by the intervening first target vehicle 403, or by both the first target vehicle 403 and building 411. The second target vehicle 402 may also operate with its headlights on. All vehicles have a distinct headlight illumination pattern, essentially directed forward in their respective direction of travel. Furthermore, all vehicles may travel so close to each other that their respective headlight patterns overlap in certain areas. In the scenario of Fig. Area 410 can be overlapped by the headlight patterns of the host vehicle 401 and the second target vehicle 402; area 406 can be overlapped by the headlight patterns of the host vehicle 401 and the first target vehicle 403; area 408 can be overlapped by the headlight patterns of the first target vehicle 403 and the second target vehicle 402; area 414 can be overlapped by the headlight patterns of the host vehicle 401, the first target vehicle 403, and the second target vehicle 402. Areas 405 and 413 can be primarily illuminated by the headlight patterns of the host vehicle 401; area 407 can be primarily illuminated by the headlight patterns of the first target vehicle 403; and area 412 can be primarily illuminated by the headlight patterns of the second target vehicle 402.According to the present description, the illumination areas 405, 406, 407, 408, 410, 412, 413, and 414 can be detected by the forward-facing sensor(s) of the host vehicle 401, for example, by one or more vision systems and ambient light sensors. The host vehicle 401 can differentiate these areas based on their brightness and recognize that areas 406, 408, and 410 are brighter than areas 405, 407, 412, and 413 due to a contributing, additive light source within the host vehicle 401's headlight illumination pattern area. Likewise, the host vehicle 401 can recognize that area 414 is brighter than areas 406, 408, and 410 due to contributing, additional light sources within the host vehicle's headlight illumination pattern area. The host vehicle 401 can also be deployed at a specific time as the situation progresses (e.g.,If one or both host vehicles 401 and the first target vehicle 403 are moving forward, the presence and position of the first target vehicle 403 can be detected, and the additive light source of area 406 can be assigned to the headlight system of the first target vehicle. Based on the scene and situation information, the host vehicle 401 can deduce that the additive light sources in areas 414 and 408 are due to an obscured target vehicle (second target vehicle 402).

[0018] The Fig. Figure 6 shows a scene 600 and the situation with respect to the host vehicle 601, the first target vehicle 603, and the second target vehicle 602. The host vehicle 601 and the first target vehicle 603 may be traveling in the same direction in different lanes of a roadway. The second target vehicle 602 may be traveling in the opposite direction in a left-turn lane and approach at an angle to cross the lanes of the host vehicle 601 and the first target vehicle 603. The second target vehicle 602 may be hidden from the line of sight of the driver of the host vehicle 601 because the second target vehicle is obscured by the intervening first target vehicle 603. The second target vehicle 602 may be operating with its headlights on. The headlights of the second target vehicle 602 can produce relatively compact beams 605 that precede the second target vehicle and project into an area ahead of the host vehicle 601.According to the present description, one or both beams 605 can be detected by the forward-looking sensor(s) of the host vehicle, e.g., by a vision system. The host vehicle 601 can distinguish the beams 605 from other light sources that may occur within the scene 600. The host vehicle 601 can detect the presence and position of the first target vehicle 603 and assign one or more beams 605 to a hidden target vehicle (second target vehicle 602), inferring the location of the second target vehicle 602 based on the scene and situation information.

[0019] Similarly, it shows Fig. 7 an alternative scene 700 and situation in relation to the host vehicle 701 and the first target vehicle 703. Fig. Figure 7 shows the first target vehicle 703, which is traveling in the right-hand lane of the respective roadway. The first target vehicle 703 may be obscured from the view of the host vehicle 701 by building 711. The first target vehicle 703 may have its headlights on. The headlights of the first target vehicle 703 may produce relatively compact beams 705 that precede the first target vehicle and project into an area ahead of the host vehicle 701. According to the present description, one or both beams 705 may be detected by the forward-looking sensor(s) of the host vehicle, e.g., by a vision system. The host vehicle 701 can distinguish the beams 705 from other light sources that may occur within the scene 700.The host vehicle 701 can detect the presence and position of the first target vehicle 703 and assign one or more beams 705 to a hidden target vehicle (first target vehicle 703) and infer the first target vehicle 703 based on the scene and situation information.

[0020] The Fig. Figure 8 shows a scene 800 and a situation involving the host vehicle 801, the first target vehicle 803, and the second target vehicle 802. The host vehicle 801 and the first target vehicle 803 may be traveling in the same direction in different lanes of a roadway. The second target vehicle 802 may be traveling in the opposite direction in a left-turn lane and approach at an angle to cross the lanes of the host vehicle and the first target vehicle 801 and 803, respectively. The second target vehicle 802 may be hidden from the line of sight of the driver of the host vehicle 801 because the second target vehicle is obscured by the intervening first target vehicle 803. The second target vehicle 802 may generate an audible waveform 808, for example, through its drivetrain and road noise. Additionally, the second target vehicle 802 can excite a seismic waveform 806 through its kinetic contact with the road surface.The first target vehicle 803 can similarly generate an acoustic waveform 807 and excite a seismic waveform 805. According to the present description, the acoustic and seismic waveforms can be detected by the acoustic and seismic sensors of the host vehicle, respectively. The host vehicle 801 can distinguish the acoustic waveform 808 of the second target vehicle 802 from other audible sounds (including the acoustic waveform 807 of the first target vehicle 803) that may occur within the scene 800. The host vehicle 801 can distinguish the seismic waveforms from other seismic sounds (including the seismic waveform 805 of the first target vehicle 803) that may occur within the scene 800.The host vehicle 801 can detect the presence and position of the first target vehicle 803 and assign the sound waveform 808 and / or the seismic waveform 806 to a hidden target vehicle (second target vehicle 802) and infer the location of the second target vehicle 802 based on the scene and situation information.

[0021] Similarly, it shows Fig. 9 an alternative scene 900 and situation in relation to the host vehicle 901, the first target vehicle 903 and the second target vehicle 902. Fig. Figure 9 shows the first target vehicle 903, traveling in the right lane of the respective roadway, and the second target vehicle, traveling in the same direction in the left lane of the respective roadway. The second target vehicle 902 may be hidden from the line of sight of the operator of the host vehicle 901, with the second target vehicle being obscured by the intervening first target vehicle 903, or by both the first target vehicle 903 and the building 911. The second target vehicle 902 may generate an audible waveform 908, for example, through its drivetrain and road noise. Additionally, the second target vehicle 902 may excite a seismic waveform 906 through its kinetic contact with the road surface. The first target vehicle 903 may similarly generate an acoustic waveform 907 and excite a seismic waveform 905. According to the present description, the acoustic and seismic waveforms can be detected by the acoustic sensors and the seismic sensors, respectively.The host vehicle 901 can detect the second target vehicle 902 using its seismic sensors. It can distinguish the acoustic waveform 908 of the second target vehicle 902 from other audible sounds (including the acoustic waveform 907 of the first target vehicle 903) that may occur within scene 900. The host vehicle 901 can also distinguish the seismic waveforms from other seismic sounds (including the seismic waveform 905 of the first target vehicle 903) that may occur within scene 900. Finally, the host vehicle 901 can detect the presence and position of the first target vehicle 903 and assign the acoustic waveform 908 and / or the seismic waveform 906 to a hidden target vehicle (second target vehicle 902), inferring its location based on scene and situation information.

[0022] The Fig.Figure 10 shows an exemplary process 1000 for estimating hidden hazards according to the present description. The process begins at 1001, for example, after the fulfillment of access conditions such as the ignition of the vehicle. At 1003, the scene is identified and may include references to GPS, map, road segment, navigation, infrastructure, and crowd-sourced information 1002 corresponding to the current geographic location of the host vehicle. At 1005, the situation is identified and may include references to target vehicles on the roadway, objects and hazards from vehicle sensor inputs 1006, including line-of-sight sensors, and references to time of day, weather, and other relevant information from local and / or remote resources and databases 1004, including crowd-sourced resources.Scene and situation recognition (1002-1006) can provide a common process basis to support the various hidden hazard detection methods described here, which are based, for example, on ambient light sensors, acoustic sensors, seismic sensors and vision systems.

[0023] At 1007, a variety of proximity sensor inputs 1008 are received for estimating the hidden hazard. These inputs can include, for example, ambient light sensors, acoustic sensors, seismic sensors, and vision systems 1008. The process at 1007 collects and classifies the information from the various proximity sensors 1008. According to one embodiment, at least two of the ambient light sensors, acoustic sensors, seismic sensors, and vision systems provide information for collection and classification. According to another embodiment, at least one of the ambient light sensors and vision systems, and at least one of the acoustic sensors and seismic sensors, provide information for collection and classification.Forward-facing ambient light sensors and / or cameras of the image processing system can classify areas of overlapping light within the illumination pattern of the host vehicle's headlights based on brightness / intensity and location. Cameras of the image processing system can capture images predefined to correspond to light beam concentrations and classify them, for example, according to a variety of predetermined light beam characteristics, including radiance / extent, beam intensity, beam direction, etc. Acoustic and seismic sensors can detect corresponding acoustic and seismic waveforms and classify them, for example, according to a variety of predetermined acoustic or seismic characteristics, including energy level, continuity, frequency content, etc.

[0024] At 1009, the presence of hidden hazards can be estimated based on the information 1007 collected and classified by the sensors 1008 and a comparison process 1013 and / or a trained dynamic neural network (DNN) process 1015. Regarding the detection of ambient light and the overlap of headlight patterns, the comparison process 1013 can include retrieving rules from local and / or remote databases 1004 that define overlap and non-overlap zones within the host vehicle's headlight pattern in relation to scene and situation information. Brightness comparisons are made for these zones, and the classified areas of overlapping light within the host vehicle's headlight pattern from 1007 are compared with them. Thus, the presence of a concealed target vehicle can be inferred from the comparison data.With regard to the camera of the vision system, which images light rays, the comparison process 1013 can include retrieving typical light ray images of concealed target vehicles from local and / or remote databases 1004 in relation to the scene and situation information. These typical images can be used as comparison images for the collected and classified images of the vision system at 1007. Thus, a concealed target vehicle can be inferred from the comparison data. With regard to acoustic and seismic waveforms, the comparison process 1013 can include retrieving acoustic and seismic signatures of interest from local and / or remote databases 1004 in relation to the scene and situation information. These signatures can be used as comparison data for the collected and classified acoustic and seismic waveforms at 1007.A hidden target vehicle can therefore be deduced from the comparison data. Each inference regarding a hidden target vehicle can be assigned a corresponding confidence level.

[0025] Regarding the DNN process 1015, offline training is performed. The training process can include, for each of the ambient light sensors, acoustic sensors, seismic sensors, and image processing systems, corresponding data collection across a matrix of road conditions and situations involving one or more vehicles of different types. The collected data can be subjected to manual annotation of facts and ground truth through observation, preferably via scene capture by the vision system. The annotated data is then used to train specific DNN models for each of the ambient light sensors, acoustic sensors, seismic sensors, and image processing systems. These trained DNN models are used in the DNN process, where the collected and classified information 1007 from the sensors 1008 is fed into the respective DNN model to provide hidden hazard inferences.

[0026] Each of the comparison processes 1013 and the DNN process 1015 can be used alone or in combination to provide the respective inferences about hidden hazards for each of the proximity sensors 1008. The presence of hidden hazards can be estimated as probabilities of presence at 1011. Process 1000 can then be repeated, with the scene and situation information being continuously updated.

[0027] The process at 1011 can pass the multitude of probabilities corresponding to the multitude of proximity sensors 1008 to the fusion process 1021, where the multiple probabilities can be fused and a final determination regarding the presence of a hidden hazard can be made. The data fusion at 1021 can be based on rule-based priorities and weights from 1019. 1019 can receive scene- and situation-based inputs from 1017, which in turn receives scene- and situation-related information determined in 1003 and 1005. For example, 1017 can determine road characteristics such as surface type and quality, highway or rural road, urban or rural, etc. 1017 can also determine environmental characteristics such as current weather conditions, noise levels, proximity to public transportation, time of day, light pollution, etc. Based on such characteristics, the process at 1019 can apply dependent rules.For example, at night in urban locations with sparse lighting, ambient light sensors can be prioritized, and a higher weighting can be assigned to their detections. Similarly, at night in urban locations with strong lighting, acoustic and seismic sensors can be prioritized over ambient light sensors, and a higher weighting can be assigned to their detections. These weightings can be used to fuse the multitude of probabilities corresponding to the multitude of proximity sensor inputs at 1021, thereby fusing the multitude of probabilities to make a final decision regarding the presence of a hidden hazard. At 1023, the presence of a hidden hazard is reported to the vehicle operator or used as a control input in the vehicle's control system.

Claims

[1] Method (1000) for determining the presence of a hidden danger, via a control, comprising: Identifying an operating scene (200, 400, 600, 700, 800, 900) for a host vehicle (101, 201, 401, 601, 701, 801, 901) based on information that includes information corresponding to the current geographic location of the host vehicle (101, 201, 401, 601, 701, 801, 901); Identifying an operating situation for the host vehicle (101, 201, 401, 601, 701, 801, 901) based on information that includes information corresponding to dynamic conditions within the operating scene (200, 400, 600, 700, 800, 900); Collecting and classifying information from a variety of proximity sensors (119), wherein the proximity sensors (119) are connected to the host vehicle (101, 201, 401, 601, 701, 801, 901) and include an ambient light sensor, a vision system, an acoustic sensor and include a seismic sensor; Estimating a multitude of hidden hazard presence probabilities according to the information from each of the multitude of proximity sensors (119), the operating scene (200, 400, 600, 700, 800, 900), the operating situation and a comparison process (1013); and performing a fusion process with the multitude of hidden hazard presence probabilities to determine the presence of a hidden hazard; where the comparison process (1013): a) retrieving rules corresponding to the operating scene (200, 400, 600, 700, 800, 900) and the operating situation, defining overlap and non-overlap zones within a headlight pattern of the host vehicle (101, 201, 401, 601, 701, 801, 901), creating brightness comparisons for the zones, and comparing the brightness comparisons with collected and classified areas of overlapping light within the headlight pattern of the host vehicle (101, 201, 401, 601, 701, 801, 901); or b) retrieving acoustic signatures corresponding to the operating scene (200, 400, 600, 700, 800, 900) and the operating situation, and comparing the signatures with collected and classified acoustic waveforms; or c) includes retrieving seismic signatures that correspond to the operating scene (200, 400, 600, 700, 800, 900) and the operating situation, and comparing the signatures with collected and classified seismic waveforms. [2] System (100) for determining the presence of a hidden danger, comprising: a host vehicle (101, 201, 401, 601, 701, 801, 901); a variety of proximity sensors (119) connected to the host vehicle (101, 201, 401, 601, 701, 801, 901) and comprising an ambient light sensor, a vision system, an acoustic sensor and a seismic sensor; a controller configured to: Identify an operating scene (200, 400, 600, 700, 800, 900) for the host vehicle (101, 201, 401, 601, 701, 801, 901) based on information that includes information corresponding to the current geographic location of the host vehicle (101, 201, 401, 601, 701, 801, 901); an operating situation for the host vehicle (101, 201, 401, 601, 701, 801, 901) is identified based on information that includes information corresponding to dynamic conditions within the operating scene (200, 400, 600, 700, 800, 900); Information is collected and classified from the multitude of proximity sensors (119); a multitude of hidden hazard presence probabilities according to the information from each of the multitude of proximity sensors (119), the operating scene (200, 400, 600, 700, 800, 900), the operating situation and a comparison process (1013); and performs a fusion process with the multitude of hidden danger presence probabilities to determine the presence of a hidden danger; where the comparison process (1013): a) retrieving rules corresponding to the operating scene (200, 400, 600, 700, 800, 900) and the operating situation, defining overlap and non-overlap zones within a headlight pattern of the host vehicle (101, 201, 401, 601, 701, 801, 901), creating brightness comparisons for the zones, and comparing the brightness comparisons with collected and classified areas of overlapping light within the headlight pattern of the host vehicle (101, 201, 401, 601, 701, 801, 901); or b) retrieving acoustic signatures corresponding to the operating scene (200, 400, 600, 700, 800, 900) and the operating situation, and comparing the signatures with collected and classified acoustic waveforms; or c) includes retrieving seismic signatures that correspond to the operating scene (200, 400, 600, 700, 800, 900) and the operating situation, and comparing the signatures with collected and classified seismic waveforms.

Images