Sound visualization in head-up displays for vehicles

Abstract

Sound visualization in a heads-up display for a vehicle is provided. Methods and vehicles utilizing sound visualization are provided. A vehicle includes a graphical projection display, an exterior microphone mounted to the vehicle, and a processing device programmed to receive an audio signal from the exterior microphone, retrieve map data in an area surrounding the vehicle, identify a sound of interest from the audio signal, identify a location of a source of the sound of interest, determine a stationary state or a moving state of the source, when the source has the moving state, determine a direction and a speed of movement of the source, determine a preferred driving maneuver from the map data and from the location, the stationary state, the moving state, the direction, and / or the speed, determine a graphic illustrating the preferred driving maneuver, and display the graphic on the graphical projection display.

Description

Technical Field

[0001] The technology field generally relates to systems for warning vehicle drivers, such as providing visual warnings in a head-up display in response to the detection of sound. Background Technology

[0002] As vehicles' active noise cancellation (ANC) capabilities improve, external noise may not be effectively transmitted to the driver. Furthermore, there is a greater risk that external sounds may be ignored when the driver turns up the volume or becomes distracted.

[0003] Accordingly, it is desirable to detect external sounds from outside the vehicle, identify the sound source and its location, and provide the driver with notification regarding the sound source's identity and location. Furthermore, other desirable features and characteristics of the invention will become apparent from the accompanying drawings and the foregoing technical and background information, based on the following detailed description and appended claims. Summary of the Invention

[0004] In one embodiment, a vehicle is provided, and the vehicle includes: a graphic projection display; an external microphone mounted to the vehicle; and a processing device programmed to: receive an audio signal from the external microphone; retrieve map data in an area surrounding the vehicle; identify a sound of interest from the audio signal; identify the location of the source of the sound of interest; determine whether the source is stationary or in motion; when the source is in motion, determine the direction and speed of the source's movement; determine a preferred driving maneuver based on the map data and based on the location, stationary state, in motion, direction, and / or speed; determine a graphic illustrating the preferred driving maneuver; and display the graphic on the graphic projection display.

[0005] In some embodiments of the vehicle, the graphic projection display includes a substantially transparent windshield head-up display that includes one of light-emitting particles and microstructures in a predefined area of ​​the windshield, allowing light-emitting display while allowing visual transmission.

[0006] In some embodiments, the vehicle also includes a haptic device, and the processing device is programmed to activate the haptic device to provide the user with a haptic warning about the source of a sound of interest.

[0007] In some embodiments, the vehicle also includes an audio device, and the processing device is programmed to: attenuate the current audio output from the audio device; and activate the audio device to provide the user with an audio warning about the source of the sound of interest.

[0008] In some embodiments of the vehicle, the audio warning includes the audible issuance of instructions for preferred driving maneuvers.

[0009] In some embodiments of the vehicle, the external microphones include a front external microphone mounted at the front of the vehicle and a rear external microphone mounted at the rear of the vehicle, wherein the processing device is programmed to identify the location of the source of the sound of interest based on the decibel difference between the audio signals received from the front and rear external microphones.

[0010] In some embodiments of the vehicle, the internal microphones include a left front internal microphone and a right front internal microphone mounted at the front of the vehicle, and a left rear internal microphone and a right rear internal microphone mounted at the rear of the vehicle, wherein the processing device is programmed to identify the location of the source of the sound of interest based on the decibel difference between the audio signals received from the front internal microphones and the rear internal microphones.

[0011] In some embodiments, the vehicle also includes an internal microphone located inside the vehicle, and the processing device is programmed to receive internal audio signals from the internal microphone.

[0012] In another embodiment, a method is provided, the method comprising: operating a vehicle; using a processor to retrieve map data in an area surrounding the vehicle; using the processor to receive an audio signal from an external microphone mounted at an external location on the vehicle; using the processor to process the audio signal to determine the location of a source of the audio signal, optionally determining whether the source is moving, and if so, determining the direction and speed of the movement of the source; determining preferred driving maneuvers based on the map data and based on the location, direction, and / or speed; and using the processor to actuate a warning device to convey a warning to the driver of the vehicle regarding the preferred driving maneuvers.

[0013] In some embodiments of the method, determining the preferred driving maneuver based on map data and based on position, direction and / or speed includes identifying the position in the oncoming lane separated from the vehicle by a hard central divider.

[0014] In some embodiments, the method further includes determining via a processor whether the audio signal includes an emergency siren.

[0015] In some embodiments, the method further includes attenuating the current audio output and conveying a warning to the driver, including conveying an audio warning.

[0016] In some embodiments, the method also includes using a large language model to create audio warnings in the form of spoken speech.

[0017] In some embodiments of the method, conveying a warning to the driver includes conveying a tactile warning.

[0018] In some embodiments of the method, conveying a warning to the driver includes displaying a visual warning as a graphic illustrating preferred driving maneuvers.

[0019] In another embodiment, a method is provided, the method comprising: operating a vehicle along a path; using a processor to determine that the vehicle is going uphill; using the processor to determine that visual perception of the path is obstructed by the crest of the hill; using a microphone mounted on the vehicle to detect sound from the path obstructed by the crest of the hill; using the processor to receive an audio signal from the microphone; using the processor to process the audio signal to determine the location of the source of the audio signal, to determine that the source is moving, and to determine the direction and speed of the movement of the source; determining a preferred driving maneuver based on map data and based on the location, direction and / or speed; and using the processor to actuate a warning device to convey a warning to the driver of the vehicle regarding the preferred driving maneuver.

[0020] In some embodiments, the method further includes attenuating the current audio output and conveying a warning to the driver, including conveying an audio warning.

[0021] In some embodiments, the method also includes using a large language model to create audio warnings in the form of spoken speech.

[0022] In some embodiments of the method, conveying a warning to the driver includes conveying a tactile warning.

[0023] In some embodiments of the method, conveying a warning to the driver includes displaying a visual warning as a graphic illustrating preferred driving maneuvers. Attached Figure Description

[0024] The present disclosure will be described below with reference to the following figures, wherein the same numerals denote the same elements, and wherein:

[0025] Figure 1 This is a functional block diagram of a vehicle according to an embodiment, the vehicle having a system for determining and conveying warnings in response to auditory stimuli from the vehicle's environment;

[0026] Figure 2 This illustrates an embodiment. Figure 1 A schematic diagram of the system architecture of the vehicle;

[0027] Figure 3 This illustrates operation according to an embodiment. Figure 1 A flowchart of the method for using vehicles;

[0028] Figure 4 This illustrates operation according to an embodiment. Figure 1 A flowchart of the method for using vehicles; and

[0029] Figure 5 This illustrates operation according to an embodiment. Figure 1 The flowchart of the method for the vehicle. Detailed Implementation

[0030] The following detailed description is merely exemplary in nature and is not intended to limit this disclosure or its application and use. Furthermore, it is not intended to be bound by any theory presented in the foregoing introduction or the content of the invention or the following detailed description.

[0031] As used herein, the term "module" refers to any hardware, software, firmware, electronic control components, processing logic, and / or processor device (alone or in any combination), including but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated, or grouped) and memories that execute one or more software or firmware programs, combinational logic circuits, and / or other suitable components that provide the described functionality.

[0032] This document describes embodiments of the present disclosure in terms of functional and / or logical block components and various processing steps. It should be understood that such block components can be implemented by any number of hardware, software, and / or firmware components configured to perform specified functions. For example, embodiments of the present disclosure can employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., which can perform various functions under the control of one or more microprocessors or other control devices. Furthermore, those skilled in the art will understand that embodiments of the present disclosure can be practiced in combination with any number of systems, and the systems described herein are merely exemplary embodiments of the present disclosure.

[0033] For the sake of brevity, conventional techniques related to signal processing, data transmission, signaling, control, and other functional aspects of the system (and its various operating components) are not described in detail herein. Furthermore, the connecting lines shown in the various figures included herein are intended to illustrate exemplary functional relationships and / or physical couplings between various elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in the embodiments of this disclosure.

[0034] According to one or more exemplary embodiments, systems and methods are provided for monitoring the environment around a vehicle (or other machine, device, or system requiring threat or object detection), detecting potential threats, and presenting contextualized notifications to users of the vehicle (e.g., drivers or passengers). Embodiments of the system are configured to: acquire detection data from one or more vehicle sensors and data related to vehicle dynamics (e.g., speed, direction), and identify one or more potential threats represented by detected objects. The system acquires or determines the predicted trajectory of the detected objects and generates a notification to the user that takes into account the user's attention level and threat level, providing the user with information about the predicted dynamics of the threat (or a combination of threats), providing relevant context, and guiding the user's attention. As discussed further below, the notification is tailored based on threat level and attention level to provide the user with a level of detail and sufficient stimulation to ensure that the user is aware of the threat and has sufficient contextual information to react.

[0035] In one embodiment, the system acquires data related to environmental data indicating the vehicle environment and driving context (e.g., road layout, weather, traffic, etc.). Based on this information, the system determines mitigation strategies or route instructions and uses one or more available modalities to generate contextualized notifications based on threat levels.

[0036] The notification utilizes one or more modalities, including visual modalities (graphics, text, etc.) (such as on a heads-up display), auditory modalities (e.g., sounds, tones, a series of sounds and tones, or the utterance of commands), and tactile modalities (e.g., steering wheel and / or seat vibrations). Tactile and auditory modalities can be configured as directional signals to cues the user to direct their attention to the location of the threat. Combinations and / or features of each modality are used to generate a notification that enhances the user's perception of the given context without unduly distracting them.

[0037] The embodiments described herein present numerous advantages. This system offers benefits including enhanced context awareness, providing users with relevant information in an intuitive way while effectively and promptly communicating the severity of detected threats and avoidance instructions. Therefore, compared to conventional systems, this system improves user response time and enhances incident avoidance.

[0038] The embodiments described herein recognize that as the active noise cancellation (ANC) capabilities of vehicles improve, external noise may not be effectively transmitted to the driver. Additionally, there is a risk of accidents when the driver turns up the in-vehicle volume or is distracted. The embodiments described herein provide a system that uses externally mounted microphones or other acoustic sensors to detect detectable noise around the front and rear of the vehicle. Furthermore, the embodiments can confirm the left or right location of a noise source based on internally mounted microphones located on the left and right sides of the vehicle. Microphone arrays can allow the detection of significant noises, such as human conversations or car horns, and the identification of nearby people or vehicles. Furthermore, the embodiments described herein can visually display information on a head-up display (HUD) to inform the driver of the nature of the sound and the exact location of the sound source. Tactile warnings can be used to notify the driver which side the sound is coming from.

[0039] Furthermore, embodiments of this document can identify sounds as emergency warnings, such as sirens from ambulances, fire trucks, or police cars. Using map coordinates of the area surrounding the vehicle, the system processor can identify the most desirable route to avoid the sound source, or it can identify a safe route through the sound source. Instructions regarding the most desirable or safest route can be displayed on a head-up display and / or conveyed via sound from the vehicle's speakers.

[0040] By identifying the type and location of sounds occurring around the vehicle and visually alerting the device via a head-up display that is easily recognizable to the driver, the embodiments described herein can prevent accidents in advance.

[0041] The embodiments described herein can detect ambient noise around the vehicle in front of and behind using microphones mounted externally, determine the decibel difference between the front and rear microphones, and pinpoint the exact location of the noise. The embodiments can detect the frequency band of the noise detected by the external microphones, identify the same frequency band using microphones mounted on the left and right sides inside the vehicle, and determine the left-right direction of the sound by comparing the decibel difference detected by the left and right microphones. The type and location of the sound can then be determined, and an alert can be indicated via a head-up display with a warning sound. Further notification can be provided using directional haptic warnings (towards the seat and / or steering wheel) as well as auditory output. Text-to-speech based feedback using a large language model can be used to generate audible cues via speakers, such as providing navigation instructions to safely avoid or pass the sound source.

[0042] refer to Figure 1The image illustrates a vehicle 10 according to various embodiments. The vehicle 10 generally includes a chassis 12, a body 14 surrounding a vehicle cabin 15, front wheels 16, and rear wheels 18. The body 14 is arranged on the chassis 12 and substantially surrounds the components of the vehicle 10. The body 14 and chassis 12 may collectively form a frame. The wheels 16 and 18 are each rotatably coupled to the chassis 12 near a respective corner of the body 14.

[0043] In various embodiments, vehicle 10 is operated by a driver, i.e., not an autonomous vehicle automatically controlled to transport passengers from one location to another. Vehicle 10 is depicted as a passenger car in the illustrated embodiment, but it should be understood that any other means of transportation may also be used, including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), boats, aircraft, etc.

[0044] like Figure 1 As shown, vehicle 10 typically includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one controller 34, and a communication system 36. In various embodiments, the propulsion system 20 may include an internal combustion engine, an electric motor such as a traction motor, and / or a fuel cell propulsion system. The transmission system 22 is configured to transmit power from the propulsion system 20 to wheels 16 and 18 according to a selectable speed ratio. According to various embodiments, the transmission system 22 may include a step-ratio automatic transmission, a continuously variable transmission (CVT), or other suitable transmissions. The braking system 26 is configured to provide braking torque to wheels 16 and 18. In various embodiments, the braking system 26 may include friction brakes, brake-by-wire brakes, regenerative braking systems such as electric motors, and / or other suitable braking systems. The steering system 24 affects the position of wheels 16 and 18. Although depicted as including a steering wheel for illustrative purposes, in some embodiments contemplated within the scope of this disclosure, the steering system 24 may not include a steering wheel. As shown in the figure, the steering wheel 24 is within reach of the driver's seat 25 in the cockpit 15.

[0045] like Figure 1 As shown, vehicle 10 includes a windshield 50 surrounding the cockpit 15. Furthermore, the vehicle includes a head-up display 60 configured to project light onto the windshield 50, whereby the light is converted into a visible display. The head-up display 60 is configured to present information to the operator of vehicle 10 in an effective manner by reducing operator stress through allowing the operator to reduce unnecessary eye scanning and saccades to remain focused on driving and visual tracking.

[0046] like Figure 1 As shown, vehicle 10 includes a haptic feedback system 70. The haptic feedback system 70 is configured to provide warnings or other communications via tactile feedback through vibration, motion, or other forces. As shown, the haptic feedback device 70 can provide tactile signals via the driver's seat 25. Additionally or alternatively, the haptic feedback device 70 can provide tactile signals via the steering wheel 24, via pedals (not shown), via the floor of the cabin 15, or in other ways.

[0047] like Figure 1 As shown, vehicle 10 includes an audio / visual input / output system 80. For example, the audio / visual input / output system 80 may include an input / output touchscreen for receiving input from a user and for visually displaying information and / or graphics. Furthermore, the audio / visual input / output system 80 may include speakers for conveying information and / or for issuing warnings, alarms, or other audio signals, as well as for playing music or entertainment programs.

[0048] like Figure 1 As shown, the sensor system 28 includes one or more sensing devices 40a-40n that sense observable conditions of the external and / or internal environments of the vehicle 10. Sensing devices 40a-40n may include, but are not limited to, radar, LiDAR (light detection and ranging), acoustic sensors, GPS, optical cameras, thermal imagers, ultrasonic sensors, and / or other sensors. For example, sensing device 40 may include acoustic sensors such as microphones. Figure 1 In some embodiments, the vehicle includes a left front external acoustic sensor or microphone 40a, a right front external acoustic sensor or microphone 40b, a left rear external acoustic sensor or microphone 40c, a right rear external acoustic sensor or microphone 40d, a left front internal acoustic sensor or microphone 40e, a right front internal acoustic sensor or microphone 40f, a left rear internal acoustic sensor or microphone 40g, and a right rear internal acoustic sensor or microphone 40h. Other arrangements with more or fewer microphones have also been considered. In some embodiments, the external microphones are mounted on the outer surface of the vehicle 10, while the internal microphones are mounted within the cabin 15 of the vehicle 10 or at other internal locations.

[0049] The actuator system 30 includes one or more actuator devices 42a-42n that can control one or more vehicle features, such as, but not limited to, the propulsion system 20, the transmission system 22, the steering system 24, the braking system 26, the head-up display 60, the haptic feedback system 70, or the audio / visual system 80.

[0050] In various embodiments, vehicle features may also include internal and / or external vehicle features, such as, but not limited to, doors, trunk and cabin features, such as air, lighting, etc. (not numbered).

[0051] exist Figure 1 In this embodiment, the communication system 36 is configured to wirelessly transmit information to and from other entities 48, such as, but not limited to, other vehicles (“V2V” communication), infrastructure (“V2I” communication), remote systems, and / or personal devices. In an exemplary embodiment, the communication system 36 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communication. However, additional or alternative communication methods (such as dedicated short-range communication (DSRC) channels) are also considered to be within the scope of this disclosure. A DSRC channel refers to a one-way or two-way short-to-medium-range wireless communication channel specifically designed for automotive use and corresponding set of protocols and standards.

[0052] exist Figure 1 In various embodiments, data storage device 32 stores data for facilitating the operation of vehicle 10 and / or for automatically controlling certain aspects of the operation of vehicle 10. In various embodiments, data storage device 32 stores a defined map of the navigable environment. In various embodiments, the defined map may be predefined by and obtained from a remote system. For example, the defined map may be assembled by a remote system and transmitted to vehicle 10 (wirelessly and / or via wire) and stored in data storage device 32. It is understood that data storage device 32 may be part of controller 34, separate from controller 34, or part of controller 34 and a separate system.

[0053] like Figure 1As shown, the controller 34 includes at least one processor 44 and a computer-readable storage device or medium 46. The computer-readable storage medium 46 and / or storage device 32 can store pre-programmed driving maneuvers of the vehicle 10, which can instruct the vehicle 10 on its next driving path. The processor 44 can be any custom or commercially available processor, central processing unit (CPU), graphics processing unit (GPU), auxiliary processor among several processors associated with the controller 34, semiconductor-based microprocessor (in the form of a microchip or chipset), macroprocessor, any combination thereof, or any device generally used for executing instructions. The computer-readable storage device or medium 46 can include volatile and non-volatile memory such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operational variables when the processor 44 is powered off. The computer-readable storage device or medium 46 may be implemented using any of the following known memory devices: PROM (programmable read-only memory), EPROM (electric PROM), EEPROM (electrically erasable PROM), flash memory, or any other electrical, magnetic, optical, or combined memory device capable of storing data (some of which represent executable instructions used by the controller 34 to control the operation of the vehicle 10).

[0054] exist Figure 1 In some embodiments, the instructions may include one or more separate programs, each comprising an ordered list of executable instructions for implementing logical functions. When executed by processor 44, the instructions receive and process signals from sensor system 28, perform logic, calculations, methods, and / or algorithms for automatically controlling components of vehicle 10, and generate control signals to actuator system 30 based on the logic, calculations, methods, and / or algorithms to automatically control certain components of vehicle 10. Although in Figure 1 Only one controller 34 is shown, but embodiments of vehicle 10 may include any number of controllers 34 that communicate via any suitable communication medium or combination of communication media and cooperate to process sensor signals, perform logic, calculations, methods and / or algorithms, and generate control signals to automatically control certain features of vehicle 10.

[0055] exist Figure 1In various embodiments, one or more instructions of controller 34 are embodied. The controller includes a non-transitory computer-readable medium 46 storing pre-programmed driving maneuvers for vehicle 10, which instruct the driving path of vehicle 10 and specifically instruct the driving path along which vehicle 10 will travel. The controller also includes a processor 44 configured to acquire audio data of at least one sound source 41 in the environment of vehicle 10. Sound source 41 can be any source that emits sound waves or acoustic waves, for example, that travel through the air from sound source 41 to vehicle 10. (See reference...) Figure 2 The functions of controller 34, and especially processor 44, are described in more detail.

[0056] Figure 2 This shows the adaptation. Figure 1 The diagram illustrates the system architecture of system 1 for driving conditions of vehicle 10. System 1 can be integrated into vehicle 10. Vehicle 10 includes an audio sensor arrangement 40 comprising audio sensor arrays 40a-40d for sensing acoustic signals 41a, such as sound waves, from sound sources 41 in the environment of vehicle 10. Each of the acoustic sensor arrays 40a-40d is arranged at a different location on, within, or at the vehicle 10. Each of the audio sensor arrays 40a-40d may include one or more audio sensors, such as microphones. Figure 2 In the example shown, sound source 41 is a child shouting or making noise in the environment. Audio sensor arrays 40a-40d receive acoustic signal 41a from this person and generate acoustic signal data 41b, which is provided to processor 44. It should be noted that, for clarity, processor 44 and system module 30 are depicted as separate from vehicle 10; however, it should be understood that processor 44 and system module 30 of the exemplary embodiment are part of or integrated into vehicle 10.

[0057] exist Figure 2 In an exemplary embodiment, the processor 44 includes an array processing module 44a configured to acquire audio data of the sound source 41 based on acoustic signal data 41b from audio sensor arrays 40a-40n. Although Figure 2Audio sensor arrays 40a-40d are shown, but any number of audio sensor arrays 40 providing triangulation can be used. Based on this audio data, processor 44 determines the receiving direction of sound source 41, wherein the receiving direction indicates the orientation of at least one sound source 41 relative to vehicle 10. For example, the receiving direction can be measured with reference to the longitudinal axis of vehicle 10. Thus, the receiving direction indicates the position of sound source 41 relative to vehicle 10, for example using three-dimensional coordinates. Specifically, two receiving directions can indicate the position of sound source 41. Therefore, at least two audio sensor arrays 40a-40d can be used to determine the receiving direction of sound source 41 with respect to each of the at least two audio sensor arrays 40a-40d in order to determine the position of sound source 41. The position of sound source 41 can be determined using three-dimensional coordinates, wherein sound sources 41 located above the road can be ignored, and sound sources 41 on the road surface can be further considered. In an exemplary embodiment, all audio sensor arrays 40a-40d are used to determine (i.e., locate) sound source 41. This means that each audio sensor array 40a-40d determines a receiving direction for the sound source 41, such that for each audio sensor array 40a-40d, a corresponding receiving direction for the sound source 41 is obtained. These receiving directions are then used to locate the sound source 41 and estimate whether it is within the driving path, i.e., to estimate whether the sound source can be ruled out as being within the driving path. Localization is performed by the localization module 44b of the processor 44. Localization can also utilize information from the inter-array energy difference, which is calculated based on the intensity of the audio signal 41a received by each of the audio sensor arrays 40a-40d. In this way, the receiving direction of the sound source 41 relative to the vehicle 10 can be determined, and thus the sound source 41 can be localized, allowing the position of the sound source 41 relative to the vehicle 10 to be determined, for example, using three-dimensional coordinates. However, it may be more preferable to determine localization based on two receiving directions. The energy difference can be additionally used to eliminate hypothetical directions. The described process can be referred to as low-latency, short-range maneuver-dependent localization of the sound source 41.

[0058] exist Figure 2 In an exemplary embodiment, the localization of the sound source 41 may be performed by the localization module 44b of the processor 44 based on inter-array energy level difference elimination. This inter-array energy level difference elimination may include localization based on the determination of the acoustic intensity or energy of the considered sound source 41, i.e., finding the sound source 41 from which different audio sensor arrays 40a-40d receive the strongest acoustic signal 41a.

[0059] Cross-reference Figure 1 and Figure 2The processor 44 estimates whether the sound source 41 is within the driving path (not shown) of the vehicle 10 based on the determined receiving direction of the sound source 41, and based on the position, orientation, and speed of the vehicle 10 and map coordinates, or based on pre-programmed or updated driving maneuvers. In some embodiments, the sound source 41 is within the driving path if it is positioned such that a collision event will occur between the vehicle 10 and the sound source 41 if the sound source 41 is not moved and the vehicle 10 continues without driver intervention. In this case, i.e., when it is estimated that the sound source 41 is within or intersects with the driving path of the vehicle 10 and therefore the processor 44 estimates that the sound source 41 is within the driving path of the vehicle 10, the processor 44 further determines the range between the vehicle 10 and the sound source 41 to confirm that the sound source 41 is definitely within the driving path of the vehicle 10. In some embodiments, if it is determined that the sound source 41 is within the driving path of the vehicle 10, the processor 44 may determine the range between the vehicle 10 and the sound source 41. Therefore, range determination can be used to provide confirmation of whether sound source 41 is indeed within the driving path of vehicle 10, where the predetermined receiving direction can only indicate which sound sources 41 are definitely not within the driving path of vehicle 10. Therefore, in range determination, only those sound sources 41 that were not excluded from the driving path after being located using the receiving direction can be considered.

[0060] exist Figure 2 In an exemplary embodiment, localization based on the determined receiving direction can provide information about whether the sound source 41 can be excluded from the driving path of the vehicle 10 or whether the sound source 41 cannot be excluded from the driving path of the vehicle 10. This means that when the sound source 41 is likely to be within the driving path, a first estimation is performed. If possible, the range between the vehicle 10 and the sound source 41 is determined to definitively determine whether the sound source 41 is within the driving path. If the receiving direction of the sound source 41 does not indicate that the sound source 41 is within the driving path, there is no range determination performed by the processor 44; that is, some receiving directions may indicate that the sound source 41 is definitely not in the driving path, and these cases are not considered further. Therefore, it is possible that only after the range is determined can the source be definitively determined to be within the driving path.

[0061] exist Figure 2 In an exemplary embodiment, the processor 44 uses at least two of the audio sensor arrays 40a-40n to determine the range between the vehicle 10 and at least one sound source 41 based on triangulation. In this way, the range, for example, the distance between the vehicle 10 and the sound source 41, can be determined at a certain point in time.

[0062] exist Figure 2In an exemplary embodiment, the processor 44 may also acquire audio data of multiple different sound sources 41 in the environment of the vehicle 10, and determine a receiving direction for each of the multiple sound sources 41 based on the audio data. Specifically, the processor 44 determines the position of each sound source 41. Thus, the receiving direction indicates the corresponding direction of the multiple sound sources 41 relative to the vehicle 10, which provides the position of each of the sound sources 41. Subsequently, the processor 44 determines whether each of the sound sources 41 is within the driving path of the vehicle 10 based on pre-programmed driving maneuvers and the determined receiving direction (i.e., position) of each of the multiple sound sources 41. The processor 44 selects those sound sources 41 determined to be within the driving path of the vehicle 10, such that the processor 44 can then determine the range between the vehicle 10 and each of the selected sound sources 41. Specifically, the range or distance between each sound source 41 (the selected sound source 41) on the subsequent driving path and the vehicle 10 is determined. Other sound sources 41 that are not selected and therefore not within the driving path of the vehicle 10 are discarded. In other words, the processor 44 therefore selects all acoustic peaks in the direction of operation, that is, all sound sources 41 within the driving path of the vehicle 10, and discards all other peaks, that is, all sound sources 41 not within the driving path of the vehicle 10.

[0063] exist Figure 2 In an exemplary embodiment, the processor 44 determines a minimum range from the defined range between the selected sound source 41 and the vehicle 10. In other words, only the selected sound sources 41 that are determined to be within the driving path of the vehicle 10 and whose ranges have been determined are compared according to their ranges, such that the single sound source 41 closest to the vehicle 10 is selected from the plurality of sound sources 41.

[0064] exist Figure 2 In an exemplary embodiment, processor 44 (e.g., array selection module 44c of processor 44) selects a single audio sensor array from audio sensor arrays 40a-40d, such as array 40c. This selection is made by determining which of the audio sensor arrays 40a-40d receives the maximum signal-to-noise ratio from a selected single sound source 41 that has been selected as the closest to vehicle 10. In other words, audio sensor array 40c that receives the highest acoustic signal and the lowest acoustic noise can be selected. As a result, processor 44 further uses the selected single audio sensor array 40c to perform beamforming toward the selected sound source 41 closest to vehicle 10, i.e., selecting an audio channel for the audio signals from the audio sensors (e.g., from a single audio sensor) of the selected audio sensor array 40c. This can be performed by spatial object separation module 44d of processor 44.

[0065] exist Figure 2In an exemplary embodiment, in addition to pre-programmed driving maneuvers, the non-transitory computer-readable medium 46 also stores pre-trained audio models. The audio models can describe the characteristics of different acoustic scenarios or different types or arrangements of sound sources. Subsequently, the processor 44, particularly the pattern recognition module 44e, is able to assign the selected audio signal to at least one of the pre-trained audio models stored on the non-transitory computer-readable medium 46. This can be understood as a comparison between the selected audio signal and the pre-trained audio model, performed to obtain a probability based on which it can be assumed that the selected audio signal belongs to a specific pre-trained audio model. In other words, the selected audio signal is classified. This process can be performed by the type and urgency classifier module 44f of the pattern recognition module 44e. Therefore, a predetermined probability threshold can be applied, indicating what probability must be achieved to indicate that the driving scenario, particularly the sound source 41, has been correctly identified. The processor 44 can then determine the type of at least one sound source 41 based on the comparison and probability calculation. Subsequently, the processor 44 can also determine an estimate of the urgency of the current driving scenario by analyzing the driving situation involving the vehicle 10 and the surrounding sound sources 41, based on the comparison and probability calculation. An urgency estimate can indicate an impending collision between sound source 41 and vehicle 10. Therefore, the urgency estimate can depend on the degree of urgency requiring intervention from the vehicle control system to avoid the collision. This can be determined based on a comparison of selected audio data with an audio model, which provides an indication of the probability for a specific current driving scenario (specifically, the current situation describing the location and movement of vehicle 10 and sound source 41). Based on this probabilistic approach, the degree of urgency for initiating a scenario change to avoid an impending hazardous situation or even a collision between sound source 41 and vehicle 10 can be determined. The non-transitory computer-readable medium 46 and the type and urgency classifier module 44f are both part of the pattern recognition module 44e of the processor 44. The urgency estimate can involve urgency-related processing of the audio data based on pitch variations under Doppler effects.

[0066] exist Figure 2In an exemplary embodiment, the above process is iteratively repeated by processor 44 until a clear (i.e., determinable) type and / or urgency estimate is possible. A determinable urgency estimate exists if a positive or negative urgency estimate can be made. A positive urgency estimate may indicate a possible impending collision between sound source 41 and vehicle 10, and further verification of the urgency estimate may be necessary to provide control intervention for vehicle 10. A negative urgency estimate may indicate that a collision event can be ruled out. If the outcome of the urgency estimate is indeterminate or unclear, processor 44 acquires second audio data from at least one sound source 41, for example, when it is necessary to classify the same selected audio signal or another selected audio signal to perform an urgency decision (i.e., make a positive or negative urgency estimate). The decision of whether the urgency estimate is positive, negative, or indeterminate can be performed by urgency decision module 44g. If the outcome of the urgency assessment is uncertain, another acoustic sensing of the environment can be performed to receive further audio signals, and a corresponding second audio signal for another driving scenario can be obtained based on the second audio data and audio model set obtained from at least one sound source 41.

[0067] exist Figure 2 In some embodiments, the processor 44 acquires further sensor data of at least one sound source 41 in the environment of the vehicle 10. This further sensor data may be optical data from a camera 47a or a Lidar sensor 47b, or data from a radar sensor 47c. The processor 44 has a sensor data fusion module 44h that provides fused data based on the fusion of the further sensor data of the at least one sound source 41 with selected audio data (particularly selected audio signals from selected audio channels) of the at least one sound source 41. Data fusion can provide verification of the correctness of an urgency estimate based on audio data obtained from audio sensors (i.e., audio sensor array arrangement 40). Therefore, the processor 44 verifies the urgency estimate of the current driving scenario based on the fused data. If the further sensor data and data fusion confirm the urgency estimate, the processor 44 controls the vehicle 10 based on the verified urgency estimate of the current driving scenario.

[0068] exist Figure 2 In an exemplary embodiment, system 1 provides low-latency classification, which includes continuous evaluation of a presumed event based on sensor array detection of the direction of sound source 41 relative to vehicle 10, vehicle maneuvering, the range between sound source 41 and vehicle 10, and an urgency estimate indicating a need to change driving conditions of vehicle 10. The duration of the evaluation can be incremented and performed iteratively until a predetermined detection confidence level is reached.

[0069] exist Figure 2 In an exemplary embodiment, system 1 also provides a maneuver-related spatial scan. Here, events can be evaluated incrementally only in the maneuver direction, i.e., only for sound sources 41 located in the driving path of vehicle 10. Thereafter, range estimation is performed only for sound sources located in the maneuver direction. Furthermore, beamforming is applied only when sound source 41 and range are determined to be in the maneuver direction.

[0070] exist Figure 2 In an exemplary embodiment, System 1 also provides an architecture for detecting short-range events. A distributed microphone architecture is provided. Certain events can be filtered by the energy difference between the audio sensor arrays 40a-40d of the audio sensor device 40, where the energy difference is based on the different intensities of different acoustic signals 41a received from the sound source 41. By applying this, a vehicle can be used as a blocking element to eliminate certain sound sources 41, so that these sound sources are not considered in urgency estimation.

[0071] Processor 44 is configured to identify emergency audio signals, such as those from ambulances, fire trucks, or police. (Cross-reference) Figure 1 and Figure 2 The processor 44 can also determine whether the sound source 41 (i.e., the emergency audio signal) is located in a protected position relative to the vehicle. For example, by cross-referencing map coordinates and the position, speed, and direction of movement of the sound source 41, the processor can determine which lane the sound source 41 is on and whether that lane is separate from the vehicle 10. For example, the processor 44 can determine that the sound source 41 is on a road completely different from the vehicle 10, such as on an overpass that does not directly connect to the vehicle 10. Alternatively, the processor 44 can determine that the sound source 41 is on the same lane as the vehicle 10, but the sound source 41 is in an oncoming traffic lane, and that lane is a separated roadway including a physical barrier between the sound source 41 and the vehicle 10. Similarly, the processor 44 can determine that the vehicle 10 is on a surface street, and the sound source is on a protected roadway crossing that surface street.

[0072] In this situation, despite receiving an emergency audio signal, the vehicle 10's direction of travel and speed do not need to be changed. The processor 44 can cause the head-up display 60 and / or the audiovisual system 80 to communicate to the vehicle operator that no change in vehicle operation is required. For example, the processor 40 can communicate that the emergency audio signal was received from a source 41 on the roadway that is not in the direction of travel (i.e., the driving path of the vehicle 10).

[0073] Alternatively, processor 44 can determine that the sound source 41 (i.e., the emergency audio signal) is in a more relevant location relative to vehicle 10, such as on the roadway in front of vehicle 10 in the direction of travel, i.e., on the driving path of vehicle 10. In such a scenario, processor 44 can determine vehicle maneuvers to optimally avoid or safely bypass the emergency sound source 41. For example, processor 44 can refer to map coordinates and generate driving instructions to avoid passing the emergency sound source 41. Alternatively, processor 44 can refer to map coordinates and generate driving instructions to slow down and move to the opposite side of the roadway to safely pass the emergency sound source 41.

[0074] Processor 44 may display such driving instructions on head-up display 60 and / or audio / vision system 80. Additionally or alternatively, processor 44 may amplify and convey such driving instructions via speakers of audio / vision system 80.

[0075] Note that the processor 44 may reduce the current audio output of the audio / vision system 80 in response to receiving an emergency audio signal, i.e., reduce the radio volume. In other embodiments, the processor 44 may reduce the current audio output of the audio / vision system 80 when conveying driving instructions to the operator of the vehicle 10.

[0076] In some embodiments, the processor 44 may indicate which side of the vehicle 10 the emergency sound source 41 is located on. For example, a warning symbol may be displayed on one side of the head-up display 60, and / or the processor 44 may actuate the haptic feedback system 70 to provide a haptic warning only on one side of the operator's seat 25 to indicate which side of the vehicle 10 the emergency sound source 41 is on.

[0077] Figure 3 This is a flowchart illustrating a method 300 for operating vehicle 10. The method is... Figure 1 and Figure 2 The system 1 is executed by processor 44.

[0078] like Figure 3 As shown, method 300 begins at action block 301, where the vehicle propulsion system is actuated to "on". At action block 310, method 300 includes monitoring sounds outside the vehicle 10, such as using audio sensor arrays 40a-40n. For example, the sensor array may include an external microphone 40i located in the engine compartment of the vehicle (i.e., at the front end of the vehicle 10) and an external microphone 40j located at the rear end of the vehicle 10.

[0079] At query box 320, method 300 determines, via processor 44, whether the sensor array has detected sound. If the processor determines that no sound has been detected, method 300 continues monitoring at action box 310. If the processor determines that sound has been detected, method 300 continues at action box 330, whereby the processor 44 defines the sound source and determines its front or rear direction, i.e., whether the sound source is located at the front or rear of the vehicle 10. For example, the decibel difference between the front sensor array and the rear sensor array can be used to determine the front or rear direction of the sound source.

[0080] Then, method 300 continues at action frame 340, wherein sound inside vehicle 10 is monitored, such as using audio sensor arrays 40a-40n, at the same sound frequencies defined at action frame 330. For example, the sensor array may include an internal microphone 40k on the left side of the vehicle interior and an internal microphone 40l on the rear end of vehicle 10.

[0081] At query box 350, method 300 determines the left or right direction via processor 44, i.e., whether the sound source is located on the right or left side of vehicle 10. For example, the decibel difference between the left and right sensor arrays can be used to determine the left or right direction of the sound source.

[0082] With the location of the sound source identified, method 300 can continue at action frame 360. For example, at action frame 361, method 300 may include conveying a warning light or graphic on the head-up display 60 via processor 44. For example, the warning light or graphic may be located on the left or right side of the head-up display 60, corresponding to the left or right side determined at action frame 350.

[0083] Additionally, at action frame 362, action frame 360 ​​may include attenuating vehicle audio via processor 44, i.e., reducing the volume of current audio output such as from speakers within the vehicle, and at action frame 363, providing navigation or safety instructions to the driver via text-to-speech modality in processor 44.

[0084] Furthermore, at action frame 364, action frame 360 ​​may include providing a directional tactile warning to the driver of vehicle 10 via haptic feedback device 70 through processor 44. For example, haptic feedback device 70 may provide a warning to the side where the driver's voice source is located.

[0085] Method 300 may continue at interrogation box 370, whereby processor 44 determines whether the sensor array is still detecting the defined sound. If processor 44 determines that the defined sound is still detected, a warning is still provided at action box 360. If processor 44 determines that the defined sound is not detected, method 300 continues at action box 380, where any warning provided at action box 360 is interrupted. Subsequently, method 300 continues monitoring sounds outside the vehicle at action box 310.

[0086] Figure 4 This is a flowchart illustrating a method 400 for operating vehicle 10. The method is... Figure 1 and Figure 2 The system 1 is executed by processor 44.

[0087] like Figure 4 As shown, method 400 begins at action box 401, where the vehicle propulsion system is actuated to "on".

[0088] At query box 405, method 400 determines whether vehicle 10 is located within the region of interest. For example, processor 44 can access a defined map of the navigable environment stored in data storage device 32 and determine that vehicle 10 is near a children's playground, school, park, or other preset location. When vehicle 10 is not located within the region of interest, method 400 can continue with the normal operation of method 300.

[0089] When vehicle 10 is within the region of interest, method 400 continues at action frame 410, wherein external sounds of vehicle 10 are monitored, such as using audio sensor arrays 40a-40n. For example, the sensor array may include an external microphone 40m and an internal microphone 40n.

[0090] At query box 415, method 400 determines, via processor 44, whether the sensor array has detected sound. In method 400, processor 44 may limit or focus to one or more typical sounds of children playing, such as the pitch range of a child's voice.

[0091] If no sound is detected, method 400 continues monitoring at interrogation box 405. If sound is detected, method 400 continues at action box 420, where it is determined whether the sound is an emergency siren. For example, processor 44 can limit the sound by decibel level. Furthermore, processor 44 can determine the front or rear direction and left or right direction of the sound source, as well as its distance. As shown, processor 44 can use signal input from external microphone 40m and internal microphone 40n.

[0092] When processor 44 determines that the sound is not an emergency siren, at action frame 425, method 400 may include using cue engineering to convey an audible cue, such as by a large language model 444 contained in or accessible by processor 44. For example, processor 44 may convey messages indicating the presence, type, and / or location of a sound source via vehicle audio system 80, such as “Caution, children are playing near the vehicle,” “Caution, children are playing behind the vehicle,” “Caution, children are playing ninety feet to the right of the vehicle,” and so on.

[0093] When processor 44 determines that the sound is an emergency siren, at query box 430, method 400 determines whether the sound source is located at a location separated from vehicle 10. For example, processor 44 may determine that the sound source is located on an overpass, i.e., on a different lane, or determine that the sound source is on the same lane but on the opposite side of a hard median strip, i.e., on a divided highway.

[0094] When the sound source is isolated, method 400 can continue at action box 425, where a cueing process is used to convey an audible cue, such as enabled by a large language model 444 contained in or accessible by processor 44. For example, processor 44 can convey a message indicating the presence of an hazard siren source and / or the location of the sound source via vehicle audio system 80, such as “hazard siren, not in the lane ahead, proceed with caution.”

[0095] When the sound source is not isolated, method 400 can continue at action box 435, where a cueing process is used to convey an audible cue, such as enabled by a large language model 444 contained in or accessible by processor 44. For example, processor 44 can convey a message indicating the presence and / or location of an emergency siren source via vehicle audio system 80, such as “Please pull over to make way for emergency vehicles”, etc.

[0096] Additionally, method 400 can continue at action frame 460. For example, at action frame 461, method 400 may include conveying a warning light or graphic on the head-up display 60 via processor 44. For example, the warning light or graphic may be located on the left or right side of the head-up display 60, corresponding to the left or right side determined at action frame 420.

[0097] Additionally, at action frame 462, action frame 460 may include attenuating vehicle audio via processor 44, i.e., reducing the volume of current audio output such as from speakers inside the vehicle and amplifying external sound artifacts, and at action frame 463, providing navigation or safety instructions to the driver via text-to-speech modality in processor 44.

[0098] Furthermore, at action frame 464, action frame 460 may include providing a directional tactile warning to the driver of vehicle 10 via haptic feedback device 70 through processor 44. For example, haptic feedback device 70 may provide a warning to the side where the driver's voice source is located.

[0099] Method 400 may continue at challenge box 470, whereby processor 44 determines whether the sensor array still detects the defined sound. If the defined sound is still detected, a warning is still provided at action box 460. If the defined sound is not detected, method 400 continues at action box 480, where any warning provided at action box 460 is interrupted. Subsequently, method 400 continues at challenge box 405.

[0100] Figure 5 This is a schematic diagram and flowchart illustrating a method 500 for operating vehicle 10. The method is... Figure 1 and Figure 2 The system 1 is executed by processor 44. As illustrated, method 500 can be followed when an obstacle 504 (such as a ridge 504 in a driveway 502) obstructs the direct line of sight 508 of the driver of vehicle 10 or a camera mounted to vehicle 10. In the illustration, vehicle 10 is on a first portion 503 of driveway 502, and object 11 (such as a second vehicle, emergency vehicle, pedestrian, etc.) is on a second portion 507 of driveway 502. As shown, ridge 504 restricts the direct line of sight or field of view 508 so that object 11 cannot be seen by the driver of vehicle 10 or the camera. Although the first portion 503 of the road is shown as uphill and the second portion 507 is shown as horizontal, the embodiment is not limited thereto. Furthermore, the embodiment may include obstacles such as buildings that obstruct the field of view in the lateral direction rather than the vertical direction.

[0101] like Figure 5 As shown, method 500 begins at action box 501, where the vehicle propulsion system is actuated to "on".

[0102] At query box 510, method 500 determines for vehicle 10 whether the line of sight 508 from vehicle 10 is obstructed, i.e., whether the field of vision is in a blind spot. For example, in some embodiments, processor 44 may determine that vehicle 10 is traveling uphill, so the final crest of the hill will obstruct the field of vision. Other embodiments may use map coordinate data or sensing (such as from a camera).

[0103] When interrogation box 510 determines that the line of sight 508 is not obstructed, normal operation according to method 300 can proceed. When interrogation box 510 determines that the line of sight 508 is obstructed, method 500 can continue at action box 520, wherein external sounds of the vehicle 10 are monitored, such as using audio sensor arrays 40a-40n. For example, the sensor array may include an external microphone 40m and an internal microphone 40n.

[0104] Method 500 continues at interrogation box 525, whereby method 500 determines, via processor 44, whether the sensor array has detected sound. If no sound is detected, method 500 continues at interrogation box 510. If sound is detected, method 500 continues at action box 560. For example, at action box 561, method 500 may include conveying a warning light or graphic on the head-up display 60 via processor 44. For example, the warning light or graphic may be located on the left or right side of the head-up display 60, corresponding to the left or right side determined at action box 520.

[0105] Furthermore, at action frame 562, action frame 560 may include attenuating vehicle audio via processor 44, i.e., reducing the volume of the current audio output such as from speakers inside the vehicle and amplifying external sound artifacts, and at action frame 563, providing navigation or safety instructions to the driver via text-to-speech modality in processor 44.

[0106] Furthermore, at action frame 564, action frame 560 may include providing a directional tactile warning to the driver of vehicle 10 via haptic feedback device 70 through processor 44. For example, haptic feedback device 70 may provide a warning to the side where the driver's voice source is located.

[0107] Method 500 may continue at challenge box 570, whereby processor 44 determines whether the sensor array still detects the defined sound. If the defined sound is still detected, a warning is still provided at action box 560. If the defined sound is not detected, method 500 continues at action box 580, where any warnings provided at action box 560 are interrupted. Subsequently, method 500 continues at challenge box 510.

[0108] As described herein, embodiments can provide sound visualization in a head-up display (HUD) for a vehicle. Such embodiments can detect external noises, such as children, emergency vehicles, human presence detected by voice (i.e., people speaking), car horns, animal sounds (i.e., external pet detection, such as barking dogs), motorcyclists in blind spots, etc., via engine compartment microphones and rear microphones. In such embodiments, the processor is configured to determine the type of noise detected, i.e., to identify the noise as originating from children, emergency vehicles, people speaking, car horns, animal sounds, motorcyclists, etc. In such embodiments, the external microphones are the primary source of sound capture. Internal microphones are used in conjunction with external microphones to identify the location of sound sources. Amplification can be used to provide an overall processed audio buffer to assist in the processor's classification of sound artifacts using a deep neural network. Thus, the processor can identify sound sources as emergency vehicles or law enforcement personnel, children playing, diagnostic sounds (e.g., fuel pump, brake pads, suspension health), pets, or other animals.

[0109] After processing the audio input and identifying the sound source and its location, the system includes a post-processing step of providing multimodal notifications to the driver. Specifically, visual notifications, including text describing the sound source's identity and location, as well as navigation or other driving instructions, can be displayed on the head-up display to mitigate or avoid risks. Similarly, auditory notifications describing the sound source's identity and location, as well as navigation or other driving instructions, can be conveyed via speakers to mitigate or avoid risks.

[0110] In some embodiments, based on detected external noise, the location information of the noise from the microphone inside the vehicle, such as left / right / front / rear, is synthesized, compared, and determined by an amplifier (amp).

[0111] In some embodiments, the type / identity and location of the identified noise are displayed on the head-up display along with a warning sound. These notifications may be communicated in conjunction with directional tactile warnings via the seat and / or steering wheel.

[0112] In some embodiments, notifications to the driver will utilize the auditory channel via text-to-speech feedback using a large language model. After classification, the system will generate audible cues via a speaker.

[0113] Some embodiments also include attenuating the current audio in the vehicle cabin after processing the audio input and identifying the sound source's identity and location, so that the driver and occupants in the vehicle cabin can hear external sound artifacts. For example, broadcasts and streaming music will be attenuated to a minimum audio level. Furthermore, the categorized sound artifacts can be mapped to visual icons. In some embodiments, a visual icon vocabulary is stored in memory and assigned to the identified sounds. In some embodiments, the third notification uses an auditory channel.

[0114] Some embodiments use a large language model included in the processor or system to utilize text-to-speech based feedback. After classification, the system generates dynamic prompts. For example, the system may generate prompts such as "Children have been detected playing near your vehicle. Please drive with extra caution. Please reduce your speed to 10 mph," or "There is an emergency vehicle behind you. Please pull over to make room."

[0115] In some embodiments, the system and method can identify a user or driver with a disability in the vehicle. For example, the system can detect that a hearing device is located in the vehicle, and if detected, can connect the hearing device to the infotainment head unit (IHU) via Bluetooth. The audio output level will then be appropriately projected onto the hearing device.

[0116] In some embodiments, the system can identify a distracted driver in the vehicle cabin. In addition to detecting loud music at in-vehicle microphones, the Driver Monitoring System (DMS) can also measure distraction. When a distracted driver is identified, the system can attenuate the audio and alert the driver with an audio warning such as a honk and bell, or with a tactile warning.

[0117] In some embodiments, all visual or audio notifications may be performed in conjunction with directional haptic warnings. In some embodiments, audio warnings may be issued by a dedicated, special audio source.

[0118] In some embodiments, the systems and methods described herein provide contextual background for navigation scenarios. Specifically, a hard central divider in a divided roadway, an overpass or underpass, or a surface street can be identified. Therefore, appropriate mitigation strategies can be communicated to the driver.

[0119] While not a limitation, the following scenarios are likely to be frequently encountered and can be addressed using the systems and methods described herein. For example, when an emergency vehicle is located on the opposite side of a lane and separated from other vehicles by physical barriers such as a median, wall, or fence—that is, when the lane is divided—the methods and systems of this paper can identify the location of the emergency vehicle's source and provide the driver with instructions that do not require changing the vehicle's current route or slowing down, because the emergency vehicle is separate from the other vehicles. In another scenario, the emergency vehicle is on a different lane, such as on an overpass, underpass, or adjacent non-connecting lane. In such scenarios, the methods and systems of this paper can identify the location of the emergency vehicle's source and provide the driver with instructions that do not require changing the vehicle's current route or slowing down, because the emergency vehicle is not on the same lane as the other vehicles. In yet another scenario, an emergency vehicle emitting an siren may be located in a vehicle's blind spot. In such embodiments, the method and system may attenuate the audio channel, amplify external sounds, provide warnings in the form of haptic feedback on the relevant side of the vehicle, provide warnings in the form of visual warnings on the relevant side of the vehicle's head-up display, provide warnings in the form of audio warnings on the relevant side of the vehicle, and / or provide text instructions describing the location of the source of the emergency siren in both visual and / or audio formats. Furthermore, outside the blind spots of a camera or driver, the method and system may detect pedestrians in the background using a microphone, and may attenuate the audio channel, amplify external sounds, provide warnings in the form of haptic feedback on the relevant side of the vehicle, provide warnings in the form of visual warnings on the relevant side of the vehicle's head-up display, provide warnings in the form of audio warnings on the relevant side of the vehicle, and / or provide text instructions describing the location of the audio source in both visual and / or audio formats.

[0120] While at least one exemplary embodiment has been presented in the foregoing detailed description, it should be understood that numerous variations exist. It should also be understood that the exemplary embodiments or multiple exemplary embodiments are merely examples and are not intended to limit the scope, applicability, or configuration of this disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient roadmap for implementing the exemplary embodiments or multiple exemplary embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the scope of this disclosure as set forth in the appended claims and their legal equivalents.

Claims

1. A vehicle comprising: Graphics projection display; An external microphone is installed in the vehicle; as well as The processing device is programmed to: Receive audio signals from the external microphone; Retrieve map data of the area surrounding the vehicle; Identify the sounds of interest from the audio signal; Identify the location of the source of the sound of interest; Determine whether the source is stationary or in motion; If the source has the moving state, then determine the direction and speed of the source's movement; The preferred driving maneuver is determined based on the map data and based on the location, stationary state, moving state, direction and / or speed. A graphic illustrating the preferred driving maneuver is provided. as well as The graphics are displayed on the graphics projection display.

2. The vehicle of claim 1, wherein the graphic projection display comprises a substantially transparent windshield head-up display, the head-up display comprising one of luminescent particles and microstructures on a predefined area of ​​the windshield, allowing luminescent display while allowing visual transmission.

3. The vehicle of claim 1, further comprising a haptic device, wherein the processing device is programmed to activate the haptic device to provide a user with a haptic warning about the source of the sound of interest.

4. The vehicle of claim 1, further comprising an audio device, wherein the processing device is programmed to: Reduce the current audio output from the audio device; and The audio device is activated to provide the user with an audio warning about the source of the sound of interest, wherein the audio warning includes the audible instruction of the preferred driving operation.

5. The vehicle of claim 1, further comprising an internal microphone located inside the vehicle, wherein the processing device is programmed to receive internal audio signals from the internal microphone.

6. A method comprising: Operating the vehicle; The processor is used to retrieve map data of the area surrounding the vehicle; The processor is used to receive audio signals from an external microphone installed at an external location on the vehicle. The audio signal is processed using a processor to determine the location of the source of the audio signal, optionally determining whether the source is moving, and if so, determining the direction and speed of the movement of the source; The preferred driving maneuver is determined based on the map data and based on the location, direction, and / or speed. as well as The processor is used to actuate a warning device to convey a warning to the driver of the vehicle regarding the preferred driving maneuver.

7. The method of claim 6, wherein determining the preferred driving maneuver based on the map data and based on the position, direction and / or speed includes identifying the position in an oncoming lane separated from the vehicle by a hard central divider.

8. The method of claim 6, further comprising determining, via the processor, whether the audio signal includes an emergency siren.

9. The method according to claim 6, further comprising: Reduce the current audio output, wherein conveying the warning to the driver includes conveying an audio warning; as well as The audio warning is created in the form of spoken speech using a large language model.

10. The method of claim 9, wherein conveying the warning to the driver comprises conveying a tactile warning or displaying a visual warning as a graphic illustrating the preferred driving maneuver.

Images