SOUND VISUALIZATION IN A HEAD-UP DISPLAY FOR A VEHICLE

Abstract

A method and vehicle with sound visualization are provided. The vehicle includes a graphic projection display; external microphones mounted on the vehicle; and a processing device programmed to receive audio signals from the external microphones; retrieve map data in an area around the vehicle; identify a sound of interest from the audio signals; identify the location of the source of the sound of interest; determine the stationary or moving state of the source; if the source is in a moving state, determine the direction and speed of the source's movement; determine a preferred driving maneuver from the map data and the location, stationary state, moving state, direction, and / or speed; determine a graphic illustrating the preferred driving maneuver; and display the graphic on the graphic projection display.

Description

INTRODUCTION

[0001] The technical field generally concerns systems for warning a driver of a vehicle, such as by providing visual warnings in a head-up display in response to the detection of sound.

[0002] As the active noise cancellation (ANC) function of vehicles is enhanced, external noise may not be transmitted effectively to the driver. Additionally, if the driver turns up the volume or is distracted, there is a greater risk of overlooking external noise.

[0003] Accordingly, it is desirable to detect external noises from outside a vehicle, to identify the identity and location of the sound source, and to provide the driver with information regarding the identity and location of the sound source. Furthermore, other desirable features and characteristics of the present invention will become apparent from the following detailed description and the accompanying claims in conjunction with the accompanying drawings and the preceding technical field and background. DESCRIPTION

[0004] In one embodiment, a vehicle is provided and includes a graphic projection display; external microphones mounted on the vehicle; and a processing device programmed to receive audio signals from the external microphones; retrieve map data in an area around the vehicle; identify a sound of interest from the audio signals; identify the location of a source of the sound of interest; determine a stationary or moving state of the source; if the source is in a moving state, determine the direction and speed of the source's movement; determine a preferred driving maneuver from the map data and the location, stationary state, moving state, direction, and / or speed; determine a graphic illustrating the preferred driving maneuver; and display the graphic on the graphic projection display.

[0005] In certain embodiments of the vehicle, the graphic projection display includes an essentially transparent windshield head-up display, which incorporates one of light-emitting particles and microstructures over a predefined area of ​​the windshield, enabling a luminescent display while allowing a view through it.

[0006] In certain embodiments, the vehicle further includes a haptic device, and the processing device is programmed to activate the haptic device in order to provide a haptic warning to a user of the source of the noise of interest.

[0007] In certain embodiments, the vehicle further includes an audio device, and the processing device is programmed to receive a current audio output from the audio device; and to activate the audio device to provide an audio warning to a user of the source of the noise of interest.

[0008] In certain vehicle configurations, the audio warning includes a vocalization of instructions for the preferred driving maneuver.

[0009] In certain embodiments of the vehicle, the external microphones include front external microphones mounted at the front of the vehicle and rear external microphones mounted at the rear of the vehicle, wherein the processing device is programmed to identify the location of a source of the noise of interest based on a decibel difference between audio signals received by the front external microphones and the rear external microphones.

[0010] In certain embodiments of the vehicle, the interior microphones include a front left interior microphone and a front right interior microphone mounted at the front of the vehicle, and a rear left interior microphone and a rear right interior microphone mounted at the rear of the vehicle, wherein the processing device is programmed to identify the location of a source of the noise of interest based on a decibel difference between audio signals received by the front interior microphones and the rear interior microphones.

[0011] In certain embodiments, the vehicle further includes interior microphones located inside the vehicle, and the processing device is programmed to receive interior audio signals from the interior microphones.

[0012] In another embodiment, a method is provided that includes operating a vehicle; retrieving, with a processor, map data in an area around the vehicle; receiving, with the processor, an audio signal from an external microphone mounted at an external location on the vehicle; processing, with a processor, the audio signal to determine the location of an audio signal source, optionally to determine whether the source is moving, and if so, to determine the direction and speed of the source's movement; determining, from the map data and from the location, direction, and / or speed, a preferred driving maneuver; and activating a warning device, with the processor, to transmit a warning to the driver of the vehicle regarding the preferred driving maneuver.

[0013] In certain embodiments of the method, determining from the map data and from the location, direction and / or speed of the preferred driving maneuver involves identifying that the location is in an oncoming lane that is separated from the vehicle by a hard median.

[0014] In certain embodiments, the method also includes determining, via the processor, whether the audio signal contains an emergency siren.

[0015] In certain embodiments, the method further includes receiving a current audio output, and transmitting the warning to the driver includes transmitting an audio warning.

[0016] In certain embodiments, the method also involves using a large language model to generate the audio warning in the form of vocalized speech.

[0017] In certain embodiments of the method, transmitting the warning to the driver involves transmitting a haptic warning.

[0018] In certain embodiments of the method, transmitting the warning to the driver involves displaying a visual warning in the form of a graphic illustrating the preferred driving maneuver.

[0019] In another embodiment, a method is provided that includes operating a vehicle along a path; determining, with a processor, that the vehicle is moving on a hill; determining, with the processor, that the visual perception of the path is blocked by a crest of the hill; detecting sounds from the path blocked by the crest of the hill with a microphone mounted on the vehicle; receiving an audio signal from the microphone with the processor; processing the audio signal with the processor to determine the location of the audio signal source, to determine that the source is moving, and to determine the direction and speed of the source's movement; and determining, from the map data and from the location, direction, and / or speed, a preferred driving maneuver.and the activation of a warning device, with the processor, to transmit a warning to a driver of the vehicle regarding the preferred driving maneuver.

[0020] In certain embodiments, the method further includes receiving a current audio output, and transmitting the warning to the driver includes transmitting an audio warning.

[0021] In certain embodiments, the method also involves using a large language model to generate the audio warning in the form of vocalized speech.

[0022] In certain embodiments of the method, transmitting the warning to the driver involves transmitting a haptic warning.

[0023] In certain embodiments of the method, transmitting the warning to the driver involves displaying a visual warning in the form of a graphic illustrating the preferred driving maneuver. DESCRIPTION OF THE FIGURES

[0024] The present disclosure is described below in conjunction with the following drawing figures, where the same reference numerals denote the same elements and where: Fig. 1 a functional block diagram of a vehicle with a system for determining and transmitting a warning in response to auditory stimuli from an environment of the vehicle according to an embodiment; Fig. 2 is a schematic diagram that shows a system architecture of the vehicle's system. Fig. 1 according to one embodiment; Fig. 3. A flowchart is a procedure for operating the vehicle from Fig. 1 illustrated according to one embodiment; Fig. 4. A flowchart is a procedure for operating the vehicle from Fig. 1 illustrated according to one embodiment; and Fig. 5. A flowchart is a procedure for operating the vehicle from Fig. 1 illustrated according to one embodiment. DETAILED DESCRIPTION

[0025] The following detailed description is merely exemplary and is not intended to limit the disclosure or the application and uses thereof. Furthermore, there is no intention of being bound to any theory presented in the preceding introduction or summary, or in the following detailed description.

[0026] As used herein, the term module refers to any hardware, software, firmware, electronic control component, processing logic and / or processor device, individually or in any combination, including without limitation: application-specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated or group) and memory executing one or more software or firmware programs, a combinational logic circuit and / or other suitable components providing the described functionality.

[0027] Embodiments of the present disclosure can be described herein with respect to functional and / or logical block components and various processing steps. It is understood that such block components can be implemented by any number of hardware, software, and / or firmware components configured to perform the specified functions. For example, an embodiment of the present disclosure may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, or the like, which can perform a variety of functions under the control of one or more microprocessors or other control devices.Furthermore, the person skilled in the art will recognize that embodiments of the present disclosure can be practiced in connection with any number of systems and that the systems described herein are merely exemplary embodiments of the present disclosure.

[0028] For the sake of brevity, conventional techniques relating to signal processing, data transmission, signaling, control, and other functional aspects of the systems (and the individual operating components of the systems) may not be described in detail herein. Furthermore, the connecting lines shown in the various figures contained herein are intended to represent exemplary functional relationships and / or physical couplings between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in an embodiment of the present disclosure.

[0029] According to one or more exemplary embodiments, methods and systems are provided for monitoring an environment around a vehicle (or other machine, device, or system for which threat or object detection is desirable), for detecting potential threats, and for presenting context-related notifications to a user (e.g., driver or passenger) of the vehicle. One embodiment of a system is configured to acquire detection data from one or more vehicle sensors and data relating to vehicle dynamics (e.g., speed, direction) and to identify one or more potential threats represented by detected objects.The system captures or determines a predicted trajectory of a detected object and generates a notification to the user. This notification takes into account the user's attention level and the threat level to provide information about the predictive dynamics of a threat (or a combined threat), provide relevant context, and capture the user's attention. As discussed below, the notification is adapted based on the threat level and attention to provide the user with a level of detail and sufficient stimulus to ensure that the user is alerted to a threat and has enough context to respond.

[0030] In one embodiment, the system collects environmental data indicating the vehicle's surroundings and driving context (e.g., road layout, weather, traffic, etc.). Based on this information, the system determines a mitigation strategy or route instruction and generates a notification using one or more available modalities, contextualized based on the threat level.

[0031] The notification uses one or more of different modalities, including a visual modality (graphics, text, etc.) such as on a head-up display, an auditory modality (e.g., a beep, a tone, a series of beeps, or vocalization of instructions), and a haptic modality (e.g., steering wheel and / or seat vibration). The haptic and auditory modalities can be configured as directional signals to prompt the user to focus their attention on a location of a threat. The combination and / or features of each modality are used to create a notification that enhances the user's awareness of a given context without unduly distracting the user.

[0032] The embodiments described herein offer a number of advantages. The system provides benefits, including improved situational awareness, both in providing relevant information to the user intuitively and in effectively and immediately communicating the seriousness of a detected threat and providing avoidance instructions. The system thus improves the user's reaction time and enhances accident prevention compared to conventional systems.

[0033] Embodiments described herein recognize that, as the active noise cancellation (ANC) function of vehicles is enhanced, external noise may not be effectively transmitted to the driver. Additionally, if the driver is distracted by loud interior noise or other distractions, there is a risk of an accident. Embodiments described herein provide a system that uses externally mounted microphones or other acoustic sensors to detect noise around the front and rear of the vehicle. Furthermore, embodiments can confirm the left or right positioning of the noise source based on left and right internally mounted microphones located inside the vehicle. The microphone arrangement can enable the detection of important sounds, such as human conversation or car horns, and the detection of people or vehicles nearby.Furthermore, embodiments thereof can visually display information on the head-up display (HUD) to inform the driver about the nature of the sound and the precise location of its source. Haptic warnings can be used to notify the driver from which direction the sound is coming.

[0034] Furthermore, embodiments thereof can identify a sound as an emergency warning, such as a siren from an ambulance, fire engine, or police vehicle. Using map coordinates of the area around the vehicle, the system processor can identify a most desirable route to bypass the sound source or a safe route that passes the sound source. Instructions regarding the most desirable or safest route can be displayed on the head-up display and / or communicated via a voice announcement from the vehicle's loudspeakers.

[0035] Embodiments herein can prevent accidents in advance by determining the type and location of noises occurring around the vehicle and visually warning the device via a head-up display that is easily visible to the driver.

[0036] Embodiments described herein can detect noise around the front and rear of the vehicle using a microphone mounted outside the vehicle, determine the decibel difference between the front and rear microphones, and pinpoint the exact location of the noise. Embodiments described herein can detect the frequency band of the noise detected by the external microphone, detect the same frequency band using microphones mounted on the left and right sides of the vehicle interior, and determine the left and right direction of the noise by comparing the decibel difference detected by the left and right microphones. The type and location of the noise can then be determined, and a warning can be displayed via the head-up display with an alarm sound. Further notification can be provided using haptic directional warnings (towards the seat and / or steering wheel) and through acoustic outputs.Text-to-speech-based feedback using a major language model can be used to generate a vocalization request through the loudspeakers, such as providing navigation instructions to safely bypass or pass the sound source.

[0037] With reference to Fig. Figure 1 shows a vehicle 10 according to various embodiments. The vehicle 10 generally comprises a chassis 12, a body 14 enclosing a vehicle cabin 15, front wheels 16, and rear wheels 18. The body 14 is arranged on the chassis 12 and essentially encloses components of the vehicle 10. The body 14 and the chassis 12 can together form a frame. The wheels 16 and 18 are each rotatably coupled to the chassis 12 near respective corners of the body 14.

[0038] In various embodiments, the vehicle 10 is operated by a driver, i.e., not by an autonomous vehicle that is automatically controlled to transport passengers from one place to another. The vehicle 10 is depicted as a passenger car in the illustrated embodiment; however, it is understood that any other vehicle, including motorcycles, trucks, all-terrain vehicles (SUVs), recreational vehicles (RVs), watercraft, aircraft, etc., can also be used.

[0039] As in Fig. As shown in Figure 1, the vehicle 10 generally includes a drive system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one control unit 34, and a communication system 36. The drive system 20 may, in various embodiments, include an internal combustion engine, an electric machine such as a traction motor, and / or a fuel cell drive system. The transmission system 22 is configured to transmit power from the drive system 20 to the vehicle wheels 16 and 18 at selectable speed ratios. According to various embodiments, the transmission system 22 may include a gear-ratio automatic transmission, a continuously variable transmission, or another suitable transmission. The braking system 26 is configured to provide braking torque to the vehicle wheels 16 and 18.The braking system 26 can, in various embodiments, include friction brakes, brake-by-wire, a regenerative braking system such as an electric motor, and / or other suitable braking systems. The steering system 24 influences the position of the vehicle wheels 16 and 18. Although shown for illustrative purposes as including a steering wheel, the steering system 24 may not include a steering wheel in some embodiments considered within the scope of this disclosure. As shown, the steering wheel 24 is located within reach of a driver's seat 25 in the cabin 15.

[0040] As in Fig. As shown in Figure 1, the vehicle 10 includes a front windshield 50 that encloses the cabin 15. Furthermore, the vehicle includes a head-up display 60 configured to project light onto the windshield 50, converting the light into a visible display. The head-up display 60 is configured to effectively present information to the operator of the vehicle 10 by reducing operator strain and enabling the operator to minimize unnecessary eye scans and gaze behaviors, allowing them to remain focused on driving and visual monitoring.

[0041] As in Fig. As shown in Figure 1, the vehicle 10 incorporates a haptic feedback system 70. The haptic feedback system 70 is configured to provide a warning or other communication through the sensation of touch via vibrations, movement, or other forces. As illustrated, the haptic feedback device 70 can provide a haptic signal through the driver's seat 25. Additionally or alternatively, the haptic feedback device 70 can provide a haptic signal through the steering wheel 24, through pedals (not illustrated), through the floor of the cabin 15, or in some other way.

[0042] As in Fig. As shown in Figure 1, the vehicle 10 includes an audio / visual input / output system 80. For example, the audio / visual input / output system 80 may include an input / output touchscreen for receiving input from a user and for visually displaying information and / or graphics. Furthermore, the audio / visual input / output system 80 may include speakers for transmitting information and / or for emitting warnings, alerts, or other audio signals and for playing music or entertainment programs.

[0043] As in Fig. As shown in Figure 1, the sensor system 28 includes one or more detection devices 40a-40n that detect observable conditions of the external environment and / or the internal environment of the vehicle 10. The detection devices 40a-40n may include, among others, radar, lidar (light detection and distance measurement), acoustic sensors, global positioning systems, optical cameras, thermal cameras, ultrasonic sensors, and / or other sensors. For example, the detection devices 40 may include an acoustic sensor, such as a microphone, etc. In the embodiment of Fig. 1 The vehicle includes a front left external acoustic sensor or microphone 40a, a front right external acoustic sensor or microphone 40b, a rear left external acoustic sensor or microphone 40c, a rear right external acoustic sensor or microphone 40d, a front left internal acoustic sensor or microphone 40e, a front right internal acoustic sensor or microphone 40f, a rear left internal acoustic sensor or microphone 40g, and a rear right internal acoustic sensor or microphone 40h. Other arrangements with more or fewer microphones are considered. In certain embodiments, the external microphones are mounted on an external surface of the vehicle 10, while the internal microphones are mounted inside the cabin 15 of the vehicle 10 or at other internal locations.

[0044] The actuator system 30 includes one or more actuator devices 42a-42n that can control one or more vehicle features, such as the drive system 20, the transmission system 22, the steering system 24, the braking system 26, the head-up display 60, the haptic feedback system 70, or the audio / visual system 80.

[0045] In various embodiments, the vehicle features may also include interior and / or exterior vehicle features, such as doors, a trunk, and cabin features such as air, lighting, etc. (not numbered).

[0046] In the embodiment of Fig. 1. The communication system 36 is configured to wirelessly transmit information to and from other entities 48, such as other vehicles (“V2V” communication), infrastructure (“V2I” communication), remote systems, and / or personal devices. In one exemplary embodiment, the communication system 36 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using IEEE 802.11 standards or using cellular data communication. However, additional or alternative communication methods, such as a dedicated short-range communication (DSRC) channel, are also considered within the scope of this disclosure.DSRC channels refer to one-way or two-way short-range to medium-range wireless communication channels specifically designed for automotive use and a corresponding set of protocols and standards.

[0047] In the embodiment of Fig. 1. The data storage device 32 stores data for use in facilitating the operation of the vehicle 10 and / or for automatically controlling certain aspects of the operation of the vehicle 10. In various embodiments, the data storage device 32 stores defined maps of the navigable environment. In various embodiments, the defined maps can be predefined by and received from a remote system. For example, the defined maps can be compiled by the remote system and transmitted to the vehicle 10 (wirelessly and / or via a wired connection) and stored in the data storage device 32. It is understood that the data storage device 32 can be part of the controller 34, separate from the controller 34, or part of the controller 34 and part of a separate system.

[0048] As in Fig. As shown in Figure 1, the controller 34 includes at least one processor 44 and a computer-readable storage device or a computer-readable storage medium 46. The computer-readable storage medium 46 and / or the storage device 32 can store a pre-programmed driving maneuver of the vehicle 10, wherein the pre-programmed driving maneuver can indicate an upcoming driving path of the vehicle 10. The processor 44 can be any custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among several processors assigned to the controller 34, a microprocessor-based semiconductor (in the form of a microchip or chipset), a macroprocessor, any combination thereof, or generally any device for executing instructions.The computer-readable storage device or computer-readable storage medium 46 can, for example, include volatile and non-volatile storage in read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables while the processor 44 is turned off.The computer-readable storage device or computer-readable storage medium 46 can be implemented using any of a number of known storage devices, such as PROMs (programmable read-only memory), EPROMs (electrical PROM), EEPROMs (electrically erasable PROM), flash memory, or any other electrical, magnetic, optical, or combined storage devices capable of storing data, some of which represent executable instructions used by the controller 34 in controlling the operation of the vehicle 10.

[0049] In the embodiment of Fig. 1. The instructions can include one or more separate programs, each containing an ordered list of executable instructions for implementing logical functions. When executed by the processor 44, the instructions receive and process signals from the sensor system 28, perform logic, calculations, procedures, and / or algorithms to automatically control the components of the vehicle 10, and generate control signals to the actuator system 30 to automatically control specific components of the vehicle 10 based on the logic, calculations, procedures, and / or algorithms. Although in Fig. 1 where only one controller 34 is shown, embodiments of the vehicle 10 may include any number of controllers 34 which communicate via any suitable communication medium or combination of communication media and which work together to process the sensor signals, perform logic, calculations, procedures and / or algorithms and generate control signals to automatically control certain features of the vehicle 10.

[0050] In various embodiments of Fig. 1 embody one or more instructions of the controller 34. The controller includes the non-volatile, computer-readable medium 46, which stores a pre-programmed driving maneuver of the vehicle 10, indicates a driving path of the vehicle 10, and, in particular, indicates a driving path along which the vehicle 10 will travel. The controller also includes the processor 44, which is configured to receive audio data from at least one acoustic source 41 in the vicinity of the vehicle 10. The acoustic source 41 can be any source that emits an acoustic wave or sound wave, which travels, for example, through the air from the acoustic source 41 to the vehicle 10. The controller 34, in particular the functionality of the processor 44, is described with reference to Fig. 2 described in more detail.

[0051] Fig. Figure 2 is a schematic diagram showing the system architecture of system 1 for adapting a driving condition of the in Fig. Figure 1 shows the vehicle 10. The system 1 can be integrated into the vehicle 10. The vehicle 10 includes an audio sensor assembly 40, which includes audio sensor assemblies 40a-40d for detecting an acoustic signal 41a, for example, a sound wave or acoustic wave, from an acoustic source 41 in the vicinity of the vehicle 10. Each of the acoustic sensor assemblies 40a-40d is located at a specific position on, in, or within the vehicle 10. Each of the audio sensor assemblies 40a-40d can include one or more audio sensors, for example, microphones. In the Fig. In the example shown, the acoustic source 41 is a child shouting or making a noise in the vicinity. The audio sensor arrangements 40a-40d receive the acoustic signal 41a from the person and generate acoustic signal data 41b, which is provided to the processor 44. It should be noted that, for clarity, the processor 44 and the system module 30 are shown separately from the vehicle 10; however, it is understood that in an exemplary embodiment, the processor 44 and the system module 30 are parts of the vehicle 10 or are integrated into the vehicle 10.

[0052] In the exemplary embodiment of Fig. 2 The processor 44 includes an array processing module 44a, which is configured to obtain audio data from the acoustic source 41 based on the acoustic signal data 41b from the audio sensor arrays 40a-40n. Although Fig. As illustrated by Figure 2, any number of audio sensor arrangements 40a-40d providing triangulation can be used. Based on this audio data, the processor 44 determines a receiving direction of the acoustic source 41, where the receiving direction indicates a direction of the at least one acoustic source 41 relative to the vehicle 10. The receiving direction can be measured, for example, relative to a longitudinal axis of the vehicle 10. The receiving direction therefore indicates the location of the acoustic source 41 relative to the vehicle 10, for example, using three-dimensional coordinates. In particular, two receiving directions can indicate the location of the acoustic source 41. Therefore, at least two audio sensor arrangements 40a-40d can be used to determine receiving directions of the acoustic source 41 for each of the at least two audio sensor arrangements 40a-40d in order to determine the location of the acoustic source 41.Three-dimensional coordinates can be used to determine the location of the acoustic source 41, whereby it may be possible to ignore acoustic sources 41 located above the road, while acoustic sources 41 located on the road can still be taken into account. In an exemplary embodiment, all audio sensor arrangements 40a-40d are used to determine, i.e., to locate, the acoustic source 41. This means that each audio sensor arrangement 40a-40d determines a receiving direction for the acoustic source 41, so that for each audio sensor arrangement 40a-40d, a respective receiving direction of the acoustic source 41 is obtained. These receiving directions are then used to locate the acoustic source 41 and to estimate whether it is located on the roadway or not, i.e., to estimate whether it can be ruled out that the acoustic source is located within the roadway or not.Localization is performed by the localization module 44b of the processor 44. Localization can also utilize information from an energy difference between arrays, calculated based on the intensity of the audio signals 41a received by each of the audio sensor arrays 40a-40d. In this way, a reception direction of the acoustic source 41 relative to the vehicle 10 can be determined, and therefore the acoustic source 41 can be localized such that its location relative to the vehicle 10 can be determined, for example, using three-dimensional coordinates. However, it might be preferable to determine the localization based on two reception directions. Energy differences can additionally be used to eliminate hypothetical directions.The described process can be described as a localization of acoustic sources dependent on a short-range maneuver with low latency 41.

[0053] In an exemplary embodiment of Fig. 2. The localization of the acoustic sources 41 can be performed by the localization module 44b of the processor 44 based on energy level difference elimination between arrangements. This energy level difference elimination between arrangements can include localization based on an acoustic intensity or energy determination of the acoustic sources 41 under consideration, i.e., finding the acoustic source 41 from which the strongest acoustic signal 41a is received by the various audio sensor arrangements 40a-40d.

[0054] With reference to Fig. 1 and Fig. 2. Based on the determined reception direction of the acoustic source 41, the location, direction, and speed of the vehicle 10, and map coordinates, or based on the pre-programmed or updated driving maneuver, the processor 44 estimates whether the acoustic source 41 lies within the path (not shown) of the vehicle 10. In certain embodiments, the acoustic source 41 lies within the path if it is located such that a collision event between the vehicle 10 and the acoustic source 41 would occur if the acoustic source 41 did not move and the vehicle 10 continued without driver intervention. In such an event, i.e.,If it has been estimated that the acoustic source 41 is located within or intersects the path of the vehicle 10, and therefore the processor 44 estimates that the acoustic source 41 lies within the path of the vehicle 10, the processor 44 further determines a distance between the vehicle 10 and the acoustic source 41 to confirm that the acoustic source 41 is certainly within the path of the vehicle 10. In certain embodiments, the processor 44 can determine the distance between the vehicle 10 and the acoustic source 41 when it is determined that the acoustic source 41 lies within the path of the vehicle 10.Distance determination can thus be used to provide confirmation as to whether the acoustic source 41 is indeed within the path of the vehicle 10, whereby the previously determined reception directions can only indicate which acoustic sources 41 are definitely not within the path of the vehicle 10. Therefore, it is possible to consider in the distance determination only those acoustic sources 41 that, after localization using the reception directions, cannot be ruled out as being within the path.

[0055] In an exemplary embodiment of Fig. 2. It is possible that the localization, based on the specified reception directions, provides information as to whether the acoustic source 41 can be ruled out as being within the path of the vehicle 10, or whether the acoustic source 41 cannot be ruled out as being within the path of the vehicle 10. This means that an initial estimate is performed if there is a chance that the acoustic source 41 is within the path. If this is the case, the distance between the vehicle 10 and the acoustic source 41 is determined to definitively establish whether the acoustic source 41 is within the path or not. If the reception directions for the acoustic source 41 do not indicate that the acoustic source 41 is within the path, then no distance determination is performed by the processor 44.Some reception directions may indicate that acoustic source 41 is definitely not in the maneuver path, and these will not be considered further. Therefore, it may only be possible to definitively determine whether the source is in the maneuver path after the distance has been determined.

[0056] In an exemplary embodiment of Fig. 2. The processor 44 determines the distance between the vehicle 10 and the at least one acoustic source 41 based on triangulation using at least two of the audio sensor arrangements 40a-40n. In this way, the distance, e.g., the distance between the vehicle 10 and the acoustic source 41, can be determined at a specific time.

[0057] In an exemplary embodiment of Fig. 2. The processor 44 can also receive audio data from several different acoustic sources 41 in the vicinity of the vehicle 10 and determine a reception direction for each of the several acoustic sources 41 based on the audio data. In particular, the processor 44 determines the location of each acoustic source 41. The reception directions thus indicate the respective directions of the several acoustic sources 41 with respect to the vehicle 10, providing the locations of each of the acoustic sources 41. The processor 44 then determines, for each of the acoustic sources 41, based on the pre-programmed driving maneuver and the determined reception directions, i.e., locations, of each of the several acoustic sources 41, whether it lies within the driving path of the vehicle 10.The processor 44 selects those acoustic sources 41 that are determined to lie within the path of the vehicle 10, so that the processor 44 can then determine a distance between the vehicle 10 and each of the selected acoustic sources 41. In particular, a distance or spacing is determined between each acoustic source 41 that lies on the upcoming path (selected acoustic sources 41) and the vehicle 10. The other acoustic sources 41 that are not selected and therefore do not lie within the path of the vehicle 10 are discarded. In other words, the processor 44 therefore selects all acoustic peaks in the direction of travel, i.e., all acoustic sources 41 that lie within the path of the vehicle 10, and discards all other peaks, i.e., all acoustic sources 41 that do not lie within the path of the vehicle 10.

[0058] In an exemplary embodiment of Fig. 2. The processor 44 determines a minimum distance from the determined distances between the selected acoustic sources 41 and the vehicle 10. In other words, only the selected acoustic sources 41, which have been determined to be within the path of the vehicle 10 and for which a distance has therefore been determined, are compared according to their distances, so that a single acoustic source 41 is selected from the multiple acoustic sources 41 that is closest to the vehicle 10.

[0059] In an exemplary embodiment of Fig. 2. The processor 44, for example, the array selection module 44c of the processor 44, selects a single audio sensor array 40a-40d, for example, array 40c. The selection is made by determining which of the audio sensor arrays 40a-40d receives a maximum signal-to-noise ratio from the selected single acoustic source 41, which has been chosen as being closest to the vehicle 10. In other words, the audio sensor array 40c that receives the highest acoustic signal and the lowest acoustic noise can be selected. As a result, the selected single audio sensor array 40c is further used by the processor 44 to transmit to the selected acoustic source 41 that is closest to the vehicle 10, i.e., to select an audio channel for an audio signal from an audio sensor, for example, a single audio sensor, of the selected audio sensor array 40c.This can be done by the spatial object separation module 44d of the processor 44.

[0060] In an exemplary embodiment of Fig. 2 A non-volatile, computer-readable medium 46 stores pre-trained audio models alongside the pre-programmed driving maneuver. The audio models can be descriptive of various acoustic scenarios or the properties of different types or arrangements of acoustic sources. The processor 44, in particular the pattern recognition module 44e, is then able to assign the selected audio signal to at least one of the pre-trained audio models stored on the non-volatile, computer-readable medium 46. This can be understood as a comparison between the selected audio signal and a pre-trained audio model, performed to obtain a certain probability based on which it can be assumed that the selected audio signal belongs to a specific pre-trained audio model. In other words, the selected audio signal is classified.This procedure can be performed by the type and urgency classifier module 44f of the pattern recognition module 44e. Therefore, a predetermined probability threshold can be applied, specifying the probability that must be reached to indicate that the driving scenario, in particular the acoustic source 41, has been correctly identified. The processor 44 can then determine the type of the at least one acoustic source 41 based on the comparison and probability calculation. The processor 44 can also determine an urgency estimate of a current driving scenario by analyzing the driving situation involving the vehicle 10 and the surrounding acoustic sources 41, based on the comparison and probability calculation. A positive urgency estimate can indicate an impending collision event between the acoustic source 41 and the vehicle 10.The urgency estimation can thus depend on how urgently intervention by the vehicle control system is required to avoid a collision. This can be determined based on a comparison of the selected audio data and the audio models, which provides a probability estimate for a specific current driving scenario, in particular a current situation describing the positioning and movement of the vehicle 10 and the acoustic source 41. Based on this probabilistic approach, it can be determined how urgently a change in the situation should be initiated to avoid an imminent hazardous situation or even a collision between the acoustic source 41 and the vehicle 10. Both the non-volatile, computer-readable medium 46 and the type and urgency classifier module 44f are parts of the pattern recognition module 44e of the processor 44.Urgency estimation can involve urgency-related processing of audio data based on a change in inclination under Doppler impact.

[0061] In an exemplary embodiment of Fig. 2. The process described above is iteratively repeated by processor 44 until a unique, i.e., determinable, type and / or urgency estimate is possible. A determinable urgency estimate exists when a positive or negative urgency estimate can be made. A positive urgency estimate may indicate a probable imminent collision between the acoustic source 41 and the vehicle 10, and that further verification of this urgency estimate may be necessary to provide a control intervention for the vehicle 10. A negative urgency estimate may indicate that a collision event can be ruled out.The processor 44 receives a second audio data from the at least one acoustic source 41 if the result of the urgency estimation is indeterminate or unclear, for example, if a classification of the same selected audio signal or a different selected audio signal is necessary to make the urgency decision, i.e., to make a positive or negative urgency estimation. The decision as to whether an urgency estimation is positive, negative, or indeterminate can be made by the urgency decision module 44g. If the result of the urgency estimation is indeterminate, another acoustic acquisition of the environment can be carried out to receive further audio signals and obtain a corresponding second audio signal of a different driving scenario based on the received second audio data from the at least one acoustic source 41 and the set of audio models.

[0062] In certain embodiments of Fig. 2. The processor 44 receives additional sensor data from at least one acoustic source 41 in the vicinity of the vehicle 10. This additional sensor data can be optical data from a camera 47a or a lidar sensor 47b, or data from a radar sensor 47c. The processor 44 has a sensor data fusion module 44h that provides fused data based on a fusion of the additional sensor data from the at least one acoustic source 41 with the selected audio data from the at least one acoustic source 41, in particular the selected audio signal from the selected audio channel. The data fusion can provide verification of the accuracy of the urgency estimate based on the audio data received from the audio sensors, i.e., the audio sensor array 40. The processor 44 thus verifies the urgency estimate of the current driving scenario based on the fused data.If the urgency estimate is confirmed by the additional sensor data and data fusion, the processor 44 controls the vehicle 10 based on the verified urgency estimate of the current driving scenario.

[0063] In an exemplary embodiment of Fig. System 1 provides a low-latency classification that involves a sequential evaluation of presumptive events based on sensor array detection of a direction of acoustic sources 41 relative to the vehicle 10, a maneuver of the vehicle, a distance between the acoustic sources 41 and the vehicle 10, and a urgency estimate indicating a requirement to change a driving condition of the vehicle 10. The duration of the evaluation can be incremented and performed iteratively until a predetermined detection confidence is achieved.

[0064] In an exemplary embodiment of Fig. 2. System 1 also provides maneuver-dependent spatial scanning. Events can be incrementally evaluated only in the direction of the maneuver, i.e., only for the acoustic sources 41 located on the path of the vehicle 10. Subsequently, a distance estimation is performed only for acoustic sources located in the direction of the maneuver. Furthermore, beamforming is only applied if it is determined that the acoustic sources 41 and the distance are located in the direction of the maneuver.

[0065] In an exemplary embodiment of Fig. Section 2 of the system 1 also provides an architecture for detecting a short-range event. A distributed microphone architecture is provided. Some events can be filtered out by energy differences between the audio sensor arrays 40a-40d of the audio sensor array 40, where the energy differences are based on the different intensities of various acoustic signals 41a received by the acoustic sources 41. By applying this, it is possible to use the vehicle as a blocking element for eliminating certain acoustic sources 41, which are therefore not considered for the urgency estimation.

[0066] Processor 44 is configured to distinguish emergency audio signals, such as those from ambulances, fire engines, or police vehicles. With reference to Fig. 1 and Fig. 2. The processor 44 can further determine whether the acoustic source 41, i.e., the emergency audio signal, is located in a protected location relative to the vehicle. For example, by referring to map coordinates and the location, speed, and direction of movement of the acoustic source 41, the processor can determine which lane the acoustic source 41 is on and whether this lane is separate from the vehicle 10. For example, the processor 44 can determine that the acoustic source 41 is on a different highway than the vehicle 10, such as on an overpass that is not directly connected to the location of the vehicle 10.Alternatively, the processor 44 can determine that the acoustic source 41 is on the same roadway as the vehicle 10, but that the acoustic source 41 is in the oncoming lanes, and that the roadway is a shared highway that includes a physical barrier between the acoustic source 41 and the vehicle 10. Likewise, the processor 44 can determine that the vehicle 10 is on a surface road and that the acoustic source is on a protected highway that runs through the surface roads.

[0067] In such scenarios, the direction and speed of the vehicle 10 do not need to change despite receiving emergency audio signals. The processor 44 can cause the head-up display 60 and / or the audiovisual system 80 to inform the driver that no change in vehicle operation is required. For example, the processor 40 can indicate that the emergency audio signal is being received from a source 41 that is not located in the direction of travel, i.e., the path of the vehicle 10, on the roadway.

[0068] Alternatively, the processor 44 can determine that the acoustic source 41, i.e., the emergency audio signal, is located at a more relevant location relative to the vehicle 10, such as on the roadway ahead of the vehicle 10 in the direction of travel, i.e., on the vehicle 10's path. In such a scenario, the processor 44 can determine a vehicle maneuver to best avoid the emergency acoustic source 41 or to drive safely. For example, the processor 44 can refer to map coordinates and generate driving instructions to avoid passing the emergency acoustic source 41. Or the processor 44 can refer to map coordinates and generate driving instructions to slow down and move to the opposite side of the roadway to safely pass the emergency acoustic source 41.

[0069] Processor 44 can cause such driving instructions to be displayed on the head-up display 60 and / or the audio / visual system 80. Additionally or alternatively, processor 44 can cause such driving instructions to be spoken aloud and transmitted via the speakers of the audio / visual system 80.

[0070] It should be noted that the processor 44 can receive the current audio output of the audio / visual system 80 in response to receiving an emergency audio signal, i.e., it can reduce the radio volume. In other embodiments, the processor 44 can receive the current audio output of the audio / visual system 80 when transmitting driving instructions to the driver of the vehicle 10.

[0071] In certain embodiments, the processor 44 can indicate on which side of the vehicle 10 the emergency acoustic source 41 is located. For example, a warning symbol can be displayed on one side of the head-up display 60, and / or the processor 44 can activate the haptic feedback system 70 to provide a haptic warning only on one side of the driver's seat 25 to indicate on which side of the vehicle 10 the emergency acoustic source 41 is located.

[0072] Fig. Figure 3 is a flowchart illustrating a procedure 300 for operating vehicle 10. The procedure is carried out by processor 44 of system 1. Fig. 1 and Fig. 2 carried out.

[0073] As in Fig. As shown in Figure 3, procedure 300 begins with the vehicle drive system being activated “ON” at action block 301. At action block 310, procedure 300 involves monitoring sounds outside the vehicle 10, for example, using audio sensor arrangements 40a-40n. For instance, the sensor arrangement may include an external microphone 40i in the vehicle's engine compartment, i.e., at the front of the vehicle 10, and an external microphone 40j at the rear of the vehicle 10.

[0074] At query block 320, procedure 300 uses processor 44 to determine whether the sensor array detects a noise. If the processor determines that no noise is detected, procedure 300 continues with monitoring at action block 310. If the processor determines that a noise is detected, procedure 300 continues at action block 330, defining the noise source and determining its forward or backward direction via processor 44, i.e., whether the noise source is in front of or behind the vehicle 10. For example, a decibel difference between the front and rear sensor arrays can be used to determine the forward or backward direction of the noise source.

[0075] Procedure 300 then continues in action block 340 with the monitoring of sounds inside the vehicle 10, such as with audio sensor arrangements 40a-40n, at the same frequency as the sound defined in action block 330. For example, the sensor arrangement can include an interior microphone 40k on the left side of the vehicle interior and an interior microphone 40l at the rear of the vehicle 10.

[0076] In query block 350, the procedure 300 determines the left or right direction via processor 44, i.e., whether the noise source is located to the right or left of the vehicle 10. For example, a decibel difference between the left and right sensor arrays can be used to determine the left or right direction of the noise source.

[0077] Once the location of the noise source is identified, procedure 300 can proceed to action block 360. For example, procedure 300 at action block 361 may involve transmitting a warning light or graphic via processor 44 to the head-up display 60. For example, the warning light or graphic may be located on the left or right side of the head-up display 60, corresponding to the left or right side determined at action block 350.

[0078] Furthermore, action block 360 can include, in action block 362, receiving the vehicle audio via processor 44, i.e., reducing the volume of the current audio output such as speakers inside the vehicle, and in action block 363, providing navigation or safety instructions to the driver via text-to-speech modality in processor 44.

[0079] Furthermore, action block 360 in action block 364 can include providing haptic directional warnings to the driver of vehicle 10 via processor 44 through the haptic feedback device 70. For example, the haptic feedback device 70 can provide a warning to the side of the driver on which the noise source is located.

[0080] Procedure 300 can proceed at query block 370 by determining, via processor 44, whether the sensor array is still detecting the defined noise. If processor 44 determines that the defined noise is still being detected, then the warnings are still provided at action block 360. If processor 44 determines that the defined noise is not being detected, then procedure 300 proceeds at action block 380 by terminating any warning provided at action block 360. Procedure 300 then proceeds at action block 310 by monitoring noises outside the vehicle.

[0081] Fig. Figure 4 is a flowchart illustrating a procedure 400 for operating the vehicle 10. The procedure is carried out by the processor 44 of system 1. Fig. 1 and Fig. 2 carried out.

[0082] As in Fig. As shown in Figure 4, the procedure 400 begins with the vehicle drive system being activated “ON” at action block 401.

[0083] At query block 405, procedure 400 determines whether the vehicle 10 is located in an area of ​​interest. For example, processor 44 can access the defined maps of the navigable environment stored in data storage device 32 and determine that the vehicle 10 is near a children's playground, a school, a park, or another programmed location. If the vehicle 10 is not located in an area of ​​interest, procedure 400 can proceed with the normal operation of procedure 300.

[0084] If the vehicle 10 is in an area of ​​interest, the procedure 400 continues at action block 410 with monitoring sounds outside the vehicle 10, such as with audio sensor arrangements 40a-40n. For example, the sensor arrangement may include external microphones 40m and internal microphones 40n.

[0085] In query block 415, procedure 400 uses processor 44 to determine whether the sensor array detects a sound. In procedure 400, processor 44 can limit or focus on sounds typical of a playing child or children, for example, the angle of children's voices.

[0086] If no sound is detected, procedure 400 continues monitoring at query block 405. If a sound is detected, procedure 400 continues at action block 420 to determine whether the sound is an emergency siren. For example, processor 44 can define the sound by decibel level. Furthermore, processor 44 can determine the forward or backward direction, the left or right direction, and the distance of the sound source. As shown, processor 44 can use signal inputs from external microphones 40m and internal microphones 40n.

[0087] If the processor 44 determines that the noise is not an emergency siren, then the procedure 400 at action block 425 may involve transmitting a vocalization prompt using prompt engineering, such as that enabled by a major language model 444 contained in or accessible to the processor 44. For example, the processor 44 may transmit a message through the vehicle audio system 80 indicating the presence of the noise source, the type of noise source, and / or the location of the noise source, such as "Please be careful, children are playing near the vehicle," "Please be careful, children are playing behind the vehicle," "Please be careful, children are playing ninety feet in front of the vehicle on the right," or something similar.

[0088] If processor 44 determines that the noise is an emergency siren, then procedure 400 at query block 430 determines whether the noise source is located in a position shielded from vehicle 10. For example, processor 44 may determine that the noise source is on a transition, i.e., on a different lane, or determine that the noise source is on the same lane but on the opposite side of a hard median, i.e., a divided highway.

[0089] If the noise source is shielded from the noise source, then procedure 400 at action block 425 can proceed with the transmission of a vocalization prompt using prompt engineering, as enabled, for example, by a major language model 444 contained in or accessible to processor 44. For example, processor 44 can transmit a message through the vehicle audio system 80 indicating the presence of the emergency siren source and / or the location of the noise source, such as "Emergency siren is not in lanes ahead, please proceed with caution."

[0090] If the noise source is not shielded from the noise source, then procedure 400 at action block 435 can proceed with the transmission of a vocalization prompt using prompt engineering, as enabled by a major language model 444 contained in or accessible to processor 44. For example, processor 44 can transmit a message through the vehicle audio system 80 indicating the presence of the emergency siren source and / or the location of the noise source, such as "Please move to make room for emergency vehicle" or something similar.

[0091] Furthermore, procedure 400 can continue at action block 460. For example, at action block 461, procedure 400 can involve transmitting a warning light or graphic via processor 44 to the head-up display 60. For example, the warning light or graphic can be located on the left or right side of the head-up display 60, corresponding to the left or right side determined at action block 420.

[0092] Furthermore, action block 460 in action block 442 can include receiving the vehicle audio via the processor 44, i.e. reducing the volume of the current audio output such as speakers inside the vehicle and amplifying external sound artifacts, and in action block 463 providing navigation or safety instructions to the driver via text-to-speech modality in the processor 44.

[0093] Furthermore, action block 460 in action block 464 can include providing haptic directional warnings to the driver of vehicle 10 via processor 44 through the haptic feedback device 70. For example, the haptic feedback device 70 can provide a warning to the side of the driver on which the noise source is located.

[0094] Procedure 400 can proceed at query block 470 by determining, via processor 44, whether the sensor array is still detecting the defined noise. If the defined noise is still detected, the warnings are still provided at action block 460. If the defined noise is not detected, procedure 400 proceeds at action block 480 by terminating a warning provided at action block 460. Then, procedure 400 proceeds at query block 405.

[0095] Fig. Figure 5 is a schematic and flowchart illustrating a procedure 500 for operating the vehicle 10. The procedure is carried out by the processor 44 of system 1. Fig. 1 and Fig. 2. As shown in the diagram, the method 500 can be followed under conditions where an obstacle 504, such as a ridge 504 in the roadway 502, blocks the direct line of sight 508 of a driver of the vehicle 10 or of a camera mounted on the vehicle 10. In the diagram, the vehicle 10 is located on a first section 503 of the roadway 502, and an object 11, such as a second vehicle, an emergency vehicle, a pedestrian, etc., is located on a second section 507 of the roadway 502. As shown, the ridge 504 limits the direct line of sight 508, so that the object 11 cannot be seen by the driver or a camera of the vehicle 10. While the first section 503 of the roadway is illustrated as uphill and the second section 507 as level, embodiments are not limited thereto.Furthermore, embodiments may include obstacles, such as buildings, that block the view in the lateral direction rather than in the vertical direction.

[0096] As in Fig. As shown in Figure 5, the procedure 500 begins with the vehicle drive system being activated “ON” at action block 501.

[0097] In query block 510, the procedure 500 determines whether the line of sight 508 of vehicle 10 is blocked by vehicle 10, i.e., whether the view is lost in a blind spot. For example, in some embodiments, the processor 44 can determine that vehicle 10 is driving uphill, so that the crest of the hill blocks the view. Another embodiment can use map coordinate data or data acquisition, such as from cameras.

[0098] If query block 510 determines that the line of sight 508 is not blocked, normal operation can proceed according to procedure 300. If query block 510 determines that the line of sight 508 is blocked, procedure 500 at action block 520 can proceed with monitoring sounds outside the vehicle 10, such as with audio sensor arrays 40a-40n. For example, the sensor array may include external microphones 40m and internal microphones 40n.

[0099] Procedure 500 continues at query block 525, where procedure 500 uses processor 44 to determine whether the sensor array detects a noise. If no noise is detected, procedure 500 continues at query block 510. If a noise is detected, procedure 500 continues at action block 560. For example, at action block 561, procedure 500 might involve transmitting a warning light or graphic to the head-up display 60 via processor 44. The warning light or graphic might be located on the left or right side of the head-up display 60, corresponding to the left or right side determined at action block 520.

[0100] Furthermore, action block 560 in action block 542 can include receiving the vehicle audio via processor 44, i.e. reducing the volume of the current audio output such as speakers inside the vehicle and amplifying external sound artifacts, and in action block 563 providing navigation or safety instructions to the driver via text-to-speech modality in processor 44.

[0101] Furthermore, action block 560 in action block 564 can include providing haptic directional warnings to the driver of vehicle 10 via processor 44 through the haptic feedback device 70. For example, the haptic feedback device 70 can provide a warning to the side of the driver on which the noise source is located.

[0102] Procedure 500 can proceed at query block 570 by determining, via processor 44, whether the sensor array is still detecting the defined noise. If the defined noise is still detected, the warnings are still provided at action block 560. If the defined noise is not detected, procedure 500 proceeds at action block 580 by terminating any warning provided at action block 560. Procedure 500 then proceeds at query block 510.

[0103] As described herein, embodiments can provide sound visualization in a head-up display (HUD) for a vehicle. Such embodiments can detect external sounds, such as children, emergency vehicles, human presence through speech (i.e., speaking persons, car horns), animal sounds (i.e., external pet detection, e.g., a dog barking), motorcyclists in a blind spot, etc., using an engine compartment microphone and a rear microphone. In such embodiments, the processor is configured to determine the type of detected sound, i.e., to identify that the sound originates from children, emergency vehicles, speaking persons, car horns, animal sounds, motorcyclists, etc. In such embodiments, the external microphones are the primary source of sound detection. Internal microphones are used in conjunction with the external microphones to identify the location of the sound source.Amplification can be used to provide a total processed audio buffer to support deep neural network-based classification of sound artifacts via the processor. Thus, the processor can identify a sound source as an emergency vehicle or law enforcement agency, children playing, diagnostic sounds (e.g., fuel pump, brake pads, suspension condition), a pet, or another animal.

[0104] After processing the sound input and identifying the identity and location of the sound source, the system includes a post-processing step of providing a multimodal notification to the driver. Specifically, a visual notification can be displayed on the head-up display, including text describing the identity and location of the sound source, as well as navigation or other driving instructions to mitigate or avoid the risk. Similarly, an audible notification can be delivered via a loudspeaker, describing the identity and location of the sound source, as well as navigation or other driving instructions to mitigate or avoid the risk.

[0105] In certain embodiments, the location information such as left / right / front / rear of the noise is synthesized from the detected external noise by the microphone inside the vehicle, compared, and evaluated via the amplifier.

[0106] In certain configurations, the type / identity and location of the identified noise are displayed on the head-up display along with an alarm sound. These notifications can be delivered in conjunction with haptic directional warnings through the seat and / or steering wheel.

[0107] In certain implementations, the notification to the driver uses the acoustic channel via text-to-speech-based feedback using a major language model. According to the classification, the system is intended to generate the vocalization prompt through the loudspeakers.

[0108] Certain embodiments further include, after processing the sound input and identifying the identity and location of the sound source, receiving the current audio in the vehicle cabin so that external sound artifacts are audible to the driver and occupants. For example, broadcast and streaming music are received at minimal audio levels. Furthermore, the sound artifact can be mapped to a visual symbol after classification. In certain embodiments, a glossary of visual symbols is stored in memory and assigned to identified sounds. In certain embodiments, a third notification uses the acoustic channel.

[0109] Certain implementations use text-to-speech feedback based on a major language model embedded in the processor or system. After classification, the system generates a dynamic prompt. For example, the system might generate a prompt such as, "Children playing near your vehicle detected. Please exercise extra caution while driving. Please reduce your speed to 10 mph," or "Emergency vehicle behind you, please pull over to make way."

[0110] In certain embodiments, the system and method can identify a user or driver with a disability in the vehicle. For example, the system can detect the presence of a hearing aid in the vehicle, and upon detection, it can connect the hearing aid to the infotainment head unit (IHU) via Bluetooth. This allows the audio output levels to be appropriately projected to the hearing aids.

[0111] In certain configurations, the system can identify a distracted driver in the vehicle cabin. A driver monitoring system (DMS) can measure distractions such as loud music, which are picked up by an in-vehicle microphone. If a distracted driver is identified, the system can receive the audio and warn the driver through audio alerts such as beeps and tones, or through haptic warnings.

[0112] In certain embodiments, all visual or audible notifications can be combined with haptic directional warnings. In certain embodiments, the audio warnings can be emitted by a dedicated audio source.

[0113] In certain embodiments, the system and method described herein provide a situational context with respect to a navigation scenario. In particular, a hard median of a divided roadway, or an overpass or underpass, or a surface road can be identified. As a result, appropriate mitigation strategies can be communicated to the driver.

[0114] Although not limiting, the following scenarios can generally be encountered and resolved using the system and procedure described herein. For example, if an emergency vehicle is located in the opposite lane to the vehicle and is separated from the vehicle by a physical barrier such as a median, wall, or fence—that is, the roadway is a divided lane—the procedures and systems described herein can identify the location of the emergency vehicle's source and provide the driver with a directive that the vehicle's current course need not be changed and its speed need not be reduced when the emergency vehicle is separated from the vehicle. In another scenario, the emergency vehicle is located in a different lane, such as an overpass, underpass, or adjacent non-connecting lane.In such a scenario, procedures and systems can identify the location of the emergency vehicle's source and instruct the driver that the vehicle's current course does not need to be changed and speed does not need to be reduced if the emergency vehicle is not in the same lane as the vehicle. In another scenario, an emergency vehicle emitting an emergency siren may be located in the vehicle's blind spot.In such an embodiment, methods and systems can block the audio channel, amplify the external sound, provide a warning in the form of haptic feedback on the relevant side of the vehicle, provide a warning in the form of a visual warning on the head-up display on the relevant side of the vehicle, provide a warning in the form of an audio warning on the relevant side of the vehicle, and / or provide text instructions in visual and / or audio form describing where the source of the emergency siren is located.Furthermore, methods and systems outside the blind zone of a camera or driver can detect pedestrians in the background using microphones and block the audio channel, amplify external sound, provide a warning in the form of haptic feedback on the relevant side of the vehicle, provide a warning in the form of a visual warning on the head-up display on the relevant side of the vehicle, provide a warning in the form of an audio warning on the relevant side of the vehicle, and / or provide text instructions in visual and / or audio form describing where the source of the audio is located.

Claims

[1] Vehicle encompassing: a graphic projection display; External microphones mounted on the vehicle; and a processing device that is programmed to: To receive audio signals from the external microphones; Map data can be retrieved in an area around the vehicle; to identify a sound of interest from the audio signals; to identify a location of a source of sound of interest; to determine a stationary status or a movement status of the source; If the source has a motion status, to determine a direction and speed of the source's movement; to determine a preferred driving maneuver from the map data and from the location, stationary status, movement status, direction and / or speed; to determine a graphic illustrating the preferred driving maneuver; and to display the graphic on the graphic projection display. [2] Vehicle according to claim 1, wherein the graphic projection display comprises a substantially transparent windshield head-up display comprising one of light-emitting particles and microstructures over a predefined area of ​​the windshield, enabling a luminescent display while allowing a view through it. [3] Vehicle according to claim 1, further comprising a haptic device, wherein the processing device is programmed to activate the haptic device to provide a haptic warning to a user of the source of the noise of interest. [4] Vehicle according to claim 1, further comprising an audio device, wherein the processing device is programmed to: to receive a current audio output from the audio device; and to activate the audio device to provide an audio warning to a user of the source of the noise of interest, the audio warning including a vocalization of instructions for the preferred driving maneuver. [5] Vehicle according to claim 1, further comprising interior microphones located inside the vehicle, wherein the processing device is programmed to receive interior audio signals from the interior microphones. [6] Procedure encompassing: Operating a vehicle; Retrieving map data in an area around the vehicle using a processor; Receiving an audio signal with the processor from an external microphone mounted at an external location on the vehicle; Processing, with a processor, the audio signal to determine the location of the audio signal's source, optionally to determine whether the source is moving, and, if so, to determine the direction and speed of the source's movement; Determine, from map data and from the location, direction and / or speed of a preferred driving maneuver; and Activating a warning device, using the processor, to transmit a warning to the driver of the vehicle regarding the preferred driving maneuver. [7] Method according to claim 6, wherein determining from the map data and from the location, direction and / or speed of the preferred driving maneuver comprises identifying that the location is in an oncoming lane which is separated from the vehicle by a hard median. [8] Method according to claim 6, further comprising determining, via the processor, whether the audio signal includes an emergency siren. [9] The method of claim 6, further comprising: Receiving a current audio output, wherein transmitting the warning to the driver includes transmitting an audio warning; and Using a major language model to generate the audio warning in the form of vocalized speech. [10] Method according to claim 9, wherein transmitting the warning to the driver comprises transmitting a haptic warning or displaying a visual warning as a graphic illustrating the preferred driving maneuver.

Images