Emergency vehicle detected

Abstract

Techniques for determining the direction of arrival of an emergency event are described. Multiple audio sensors on a vehicle may receive audio data associated with the vehicle. A pair of audio sensors may be selected from the multiple audio sensors to generate audio data representative of sounds in the vehicle's environment. An angular spectrum associated with the pair of audio sensors may be determined based on the audio data. Features associated with the audio data may be determined based on the angular spectrum and / or the audio data itself. A direction of arrival (DoA) value associated with the sound may be determined based on the features using a machine learning model. A sound of the emergency event (e.g., a siren) may be detected in the audio data, and a direction associated with the emergency event related to the vehicle may be determined based on the features and the DoA value.

Description

Background Art

[0001] Related Applications This application was filed on October 23, 2019, claims priority to U.S. Patent Application No. 16 / 661,898, entitled "EMERGENCY VEHICLE DETECTION", and the entire disclosure of this U.S. patent application is incorporated herein by reference.

[0002] Vehicles often encounter emergency vehicles in the environment. During emergencies, emergency vehicles may travel at high speeds and use lights and / or sirens to indicate their presence. Since emergency vehicles have a higher priority when traveling, non-emergency vehicles should yield to emergency vehicles. In the context of autonomous vehicles, it can be important for autonomous vehicles to have a mechanism for detecting emergency vehicles and responding to the occurrence of emergency vehicles in their environment.

[0003] The detailed description is provided with reference to the accompanying drawings. In the figures, the leftmost digit of a reference number identifies the figure in which that reference number first appears. The use of the same reference number in different figures indicates similar or identical components or features.

Brief Description of the Drawings

[0004] [Figure 1] A schematic diagram illustrating an exemplary implementation of detecting an emergency vehicle in an environment where an autonomous vehicle is located, according to an example of the present disclosure.

[0005] [Figure 2] A diagram represented by a picture illustrating an exemplary implementation of detecting an emergency vehicle in an environment where an autonomous vehicle is located, according to an example of the present disclosure.

[0006] [Figure 3] An example diagram of selecting an audio sensor pair for detecting an emergency vehicle in an environment where an autonomous vehicle is located, according to an example of the present disclosure.

[0007] [Figure 4] A block diagram of an exemplary implementation for detecting an emergency vehicle in an environment where an autonomous vehicle is located, as shown in the example of this disclosure, is drawn.

[0008] [Figure 5] This is an example diagram illustrating exemplary features associated with audio signals detected by four audio sensor pairs, as illustrated by the present disclosure.

[0009] [Figure 6] This is an example diagram illustrating exemplary features associated with audio signals detected by four other audio sensor pairs, as illustrated by the present disclosure.

[0010] [Figure 7] This disclosure illustrates an exemplary process for detecting an emergency vehicle based on an audio signal detected by an audio sensor pair.

[0011] [Figure 8] A block diagram of an exemplary system for implementing the techniques described herein is drawn. [Modes for carrying out the invention]

[0012] This disclosure relates to a technique for determining the direction of arrival (DoA) of an emergency in a vehicle environment. The vehicle may be an autonomous vehicle that includes audio sensors for capturing sounds from the environment. In some examples, an emergency may include one or more emergency vehicles in the environment. The audio sensors may capture sounds produced by the sirens of emergency vehicles and transmit the audio data associated with the siren to a vehicle computing device implemented in the vehicle or remotely on a server to determine the DoA of the siren. Since the vehicle environment may include various types of sounds, such as car noise, sirens, human speech, music, thunder, etc., one or more pairs of audio sensors may be used to capture audio data associated with the sound in order to respond efficiently and accurately to a siren. In some examples, the audio sensor pairs may be positioned in the front area, rear area, left side, and right side of the vehicle to cover all directions around the vehicle. Each of the audio sensor pairs may be capable of covering an area at a 180-degree angle. For example, an audio sensor pair positioned in the front area of ​​the vehicle can cover the front area at a 180-degree angle, an audio sensor pair positioned in the rear area of ​​the vehicle can cover the rear area at a 180-degree angle, an audio sensor pair positioned on the left side of the vehicle can cover the left side at a 180-degree angle, and an audio sensor pair positioned on the right side of the vehicle can cover the right side at a 180-degree angle. Therefore, the four audio sensor pairs can cover all directions around the vehicle.

[0013] In some cases, audio data captured by an audio sensor pair can be analyzed to estimate the angular spectrum of the audio sensor pair. In some cases, more than two audio sensors may be used to estimate the angular spectrum. The angular spectrum may be a parameter that has information about the likelihood of a sound source being present in a direction specified by the transverse coordinate, for example, the intensity of sound transmitted to the vehicle from the DOA or TDOA. As a non-limiting example, the generalized cross correlation with phase transform (GCC-PHAT) algorithm may be implemented to estimate the angular spectrum of an audio sensor pair. However, other algorithms may also be applied to estimate the angular spectrum, such as the cross-power spectrum phase (CPS) algorithm, GCC-PHAT with Fast Fourier Transform (FFT), machine learning-based algorithms, multiple signal classification (MUSIC), state coherence transform (SCT), etc. The estimation of the angular spectrum can be based on the assumption that the DoA of each source in the auditory scene around the vehicle is represented by a single angle at which the audio signal propagates from the audio source to the audio sensor pair in the far field, and that the audio signal propagates from the audio source to each audio sensor in the free field, with a one-to-one mapping between the DoA and the time difference of arrival (TDoA). Each peak of the TDoA may represent the TDoA associated with the sound source. In some examples, the calculation can be performed for each time-frequency bin for all discrete values ​​of the difference within the range of possible TDoA. The GCC-PHAT algorithm can calculate the TDoA-by-TDoA value in the time-frequency bin associated with the audio data and, based on the audio data, output the angular spectrum of the audio sensor pair.Since the TDoA calculated for a sample can be converted to DoA for angles, the DoA of an emergency vehicle can be determined at least partially based on the peak of the TDoA. In some examples, when more than two audio sensors are implemented, DoA may be used directly instead of TDoA.

[0014] In some examples, at least one feature associated with the audio data is selected based on the angular spectrum. In some examples, at least one feature may be selected from the angular spectrum of the audio sensor pair itself, the peak value of the angular spectrum of the audio sensor pair, the TDoA value at the peak of the angular spectrum, the energy value of the audio sensor pair, etc. In some examples, more than one feature is selected from the angular spectrum to provide a more accurate DoA estimate.

[0015] In some examples, neural network components (e.g., regression or classification) may be implemented on audio data received in real time to generate DoA values. An internal buffer space may be configured to stream the audio data for processing. Audio data captured by the audio sensor may be sent to the buffer space in a first-in, first-out order, frame by frame. For example, the internal buffer space may be configured to store at least 3 seconds of data, but the buffer space may store data of any length. In some examples, the frame length can be set to 30 milliseconds, but the frame length can be set to any length. Audio data may be streamed to the internal buffer space every 30-millisecond audio frame. The internal buffer space may contain audio data corresponding to overlapping time intervals. For example, the internal buffer space may be updated every 10 milliseconds, which would create an overlapping window of 20 milliseconds of audio data. For every 30-millisecond audio frame, angular spectral estimation may be performed for the audio sensor pair, based at least partially on the data contained within the audio frame, features determined at least partially on the angular spectrum, and / or on feature fusion to generate a DoA value associated with the 30-millisecond audio frame.

[0016] In some examples, the category of an audio event may be determined based on the DoA value and an audio event classifier. For example, the audio event classifier may classify an event as an emergency event or a non-emergency event, or it may classify audio events more specifically within each broad class. The audio event classifier may be trained to determine the category of an event. Training of the audio event classifier may be at least in part based on historical data associated with past events that occurred in the vehicle's environment and / or features / parameters used to identify such past events. The features / parameters used to identify past events may be determined at least in part based on audio data and / or video data captured by the vehicle. In the case of an emergency vehicle, the audio event classifier may be a machine learning model that is pre-trained using training data. The training data may include previously determined sirens generated by the emergency vehicle, audio data associated with previously determined sirens, and features extracted from audio data used to determine previously determined sirens of the emergency vehicle. The audio event classifier may be trained periodically using feedback from identified emergencies in real time.

[0017] In some examples, an audio event can be defined at least partially based on the start time associated with the event (e.g., the event onset) and the end time of the event (e.g., the event offset). In some examples, the start time frame indicating the onset of an emergency and / or the end frame indicating the offset of an emergency can be determined based on the DoA value. In some examples, whether an emergency vehicle is approaching or moving away from a vehicle may be determined based on the DoA value. In some examples, the decision regarding whether an emergency vehicle is approaching or moving away from a vehicle may be made using a Doppler frequency shift function. In some examples, the decision regarding whether an emergency vehicle is approaching or moving away from a vehicle may be made using an audio sensor data power tracker across multiple frames. In some examples, a combination of the above elements, namely a power tracker across multiple frames, can be used to make a decision regarding whether an emergency vehicle is approaching or moving away from a vehicle. In some cases, audio data can be integrated with other perception data (e.g., image data, lidar data, radar data, time of flight data, etc.) and vehicle speed information to help classify audio events, determine the direction associated with audio events, and so on.

[0018] In some cases, multiple events occurring in an environment can be determined based on collected audio data representing various types of sounds in the environment. To eliminate ambiguity, so that vehicles can respond quickly and accurately to sirens, multiple events can be scrutinized according to predetermined rules. Emergency vehicle sirens generally have a dominant sound level when compared to any other sounds in the environment and can be heard up to several hundred meters away, even before the emergency vehicle can be seen. As a vehicle captures audio data representing an emergency vehicle in the environment, the length of the sound can be evaluated to determine whether the sound corresponds to an audio event. Occasionally, there may be other sound sources in the environment that produce siren-like sounds, or the driver of an emergency vehicle may accidentally activate the siren. To avoid confusion caused by such potentially false sirens, audio data associated with such potentially false sirens can be removed for analysis. In some cases, audio data received continuously in the time domain can be discretized into multiple audio frames. Audio events may be determined to occur in a start time frame where the onset of the sound is determined, and to end in a end time frame where the offset of the sound is determined. Multiple audio frames may be input into the internal buffer space. If the total number of audio frames associated with an audio event is below a threshold, the event may be determined to be non-event. In some examples, emergency vehicles may need to sound their sirens continuously until they reach a destination that is sometimes several miles (or kilometers) away, but there may be some intermittent silences between the sounds. To account for such potential gaps in audio, if the total number of audio frames between two adjacent events is less than a minimum frame gap (which may be predefined for siren signals), the audio frame gap may be removed and adjacent events may be merged.As an alternative solution to account for potential gaps in audio, state-based models or long-context machine learning models can be used to maintain detection states even during periods of silence. These machine learning models can be trained on data containing siren sounds with intermittent gaps, and thus can be trained to be robust to gaps.

[0019] The techniques discussed herein can improve the functionality of a vehicle's computing devices in many ways. Since a vehicle's audio sensors can capture sounds generated by a siren before other sensors such as cameras, lidar, and radar, utilizing audio data collected by audio sensors on the vehicle can improve the response speed to emergencies. For example, in a context that generates the direction of an approaching emergency, utilizing various pairs of audio sensors to collect audio data and analyzing the features associated with the angular spectra of various audio sensor pairs can improve accuracy and / or reduce latency for a vehicle responding to an emergency in its environment. Furthermore, some emergency vehicles, such as unmarked vehicles, may not provide visual indication of an emergency. The techniques discussed herein can facilitate a vehicle's response to an emergency even when the emergency vehicle is occupied and / or in the darkness of night.

[0020] The techniques described herein can be implemented in many ways. Exemplary implementations are provided below while referring to the following figures. Although discussed in the context of autonomous vehicles, the methods, apparatuses, and systems described herein can be applied to a wide variety of systems and are not limited to autonomous vehicles. In another example, the techniques can be utilized in the context of aviation or marine using audio data or in any system. Also, the techniques described herein can be used with real data (e.g., captured using sensors), simulated data (e.g., generated by a simulator), or any combination of the two.

[0021] FIG. 1 is a schematic diagram illustrating an exemplary implementation 100 for detecting an emergency vehicle in an environment where an autonomous vehicle is located, according to an embodiment of the present disclosure.

[0022] As illustrated in FIG. 1, vehicle 102 can include one or more audio sensors (e.g., 110-1, 110-2, 110-3, 110-4, 110-5, 110-6, 110-7, and 110-8) that generate audio data representative of sounds in the environment in which vehicle 102 is located.

[0023] For illustrative purposes, vehicle 102 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration of the United States Department of Transportation, which describes a vehicle that can perform all safety-critical functions during the entire journey without the expectation that a driver (or occupant) will control the vehicle at any time. In such an example, vehicle 102 can be configured to control all functions from start to stop, including any parking functions, so that vehicle 102 can be unmanned. This is merely an example, and the systems and methods described herein can be incorporated into any ground vehicle, aircraft, watercraft, including vehicles that range from vehicles that must always be manually controlled by a driver to vehicles that are partially or fully autonomously controlled. Additional details associated with vehicle 102 are described below.

[0024] In at least one example, the vehicle can be associated with a sensor system(s) 108 that can be disposed on vehicle 102. The sensor system(s) 108 can include a lidar (light detection and ranging) sensor, a radar (radio detection and ranging) sensor, ultrasonic transducers, a sound navigation and ranging (sonar) sensor, a location sensor (e.g., a global positioning system (GPS), compass, etc.), an inertial sensor (e.g., an inertial measurement unit, accelerometer, magnetometer, gyroscope, etc.), a camera (e.g., RGB, IR, intensity, depth, time of flight, etc.), a wheel encoder, an audio sensor (e.g., audio sensors 110-1, 110-2, 110-3, 110-4, 110-5, 110-6, 110-7, and 110-8), an environmental sensor (e.g., a temperature sensor, a humidity sensor, a light sensor, a pressure sensor, etc.), and the like. The sensor system(s) 108 can generate sensor data, which is utilized by a vehicle computing device(s) 110 associated with vehicle 102. It is possible

[0025] In at least one example, a vehicle computing device 110 may determine the direction of arrival (DoA) of an audio source in the environment, at least in part, based on audio data received from a sensor system 108 associated with the vehicle 102. Referring to Figure 2, the vehicle's computing device 110 may identify one or more audio sources in the environment 200. The audio sources may include at least a vehicle 204 traveling in the same direction as vehicle 102, vehicles 206 and 216 traveling in the opposite direction to vehicle 102, and thunder 218 from the sky. In some examples illustrated in Figure 2, vehicle 204 traveling in the same direction as vehicle 102 and vehicle 206 traveling in the opposite direction to vehicle 102 may be emergency vehicles. Emergency vehicles designated and certified to respond to emergencies in life-threatening situations may include, but are not limited to, police vehicles, police motorcycles, SWAT vehicles, firefighting equipment, and ambulances. Emergency vehicles may be equipped with audible and visual warning devices, which are designed to facilitate the movement of emergency vehicles through traffic to reach their destination or to provide certain protection at the scene.

[0026] The computing device 110 of vehicle 102 may receive audio data representing sounds generated by audio sources in the environment, such as sound 212 from emergency vehicle 206, sound 222 from vehicle 216, sound 220 from thunder 218, sound 208 from emergency vehicle 204, etc. Sound 222 may be sounds generated from vehicle 216, such as engine noise, belt noise, tire noise, music, human speech, dog barking, etc. Sound 212 may be sounds generated from emergency vehicle 206, such as siren sound, engine noise, belt noise, tire noise, music, human speech, dog barking, etc. Siren sound may be dominant among all sounds generated by the emergency vehicle.

[0027] Audio data from audio sources in the environment of vehicle 102, i.e., raw audio data, may be collected, captured, received, or otherwise determined by audio sensors 110-1, 110-2, 110-3, 110-4, 110-5, 110-6, 110-7, and 110-8. Raw audio data or raw data may refer to audio data or data captured by audio sensors, which may be uncompressed. Alternatively, raw audio data and raw data may refer to audio data or data captured by audio sensors, which may be compressed but otherwise unprocessed. As illustrated in Figure 1, audio sensor pair [110-1, 110-2] is positioned in the front area of ​​vehicle 102, audio sensor pair [110-3, 110-4] is positioned in the rear area of ​​vehicle 102, audio sensor pair [110-5, 110-6] is positioned on the right side of vehicle 102, and audio sensor pair [110-7, 110-8] is positioned on the left side of vehicle 102. The front area, rear area, left side, and right side relate to the direction of travel of vehicle 102. Each audio sensor pair can cover an area of ​​180 degrees, but audio sensors may be associated with any coverage range. In some examples, audio sensor pair [110-1, 110-2] can cover the front area of ​​vehicle 102 at a 180-degree angle, audio sensor pair [110-3, 110-4] can cover the rear area of ​​vehicle 102 at a 180-degree angle, audio sensor pair [110-5, 110-6] can cover the right side of vehicle 102 at a 180-degree angle, and audio pair [110-7, 110-8] can cover the left side of vehicle 102 at a 180-degree angle. Thus, the audio sensors illustrated in Figure 1 can cover all directions surrounding vehicle 102, i.e., a 360-degree angle.

[0028] It should be understood that the audio sensors in Figure 1 are for illustrative purposes only. Different numbers of audio sensors may be placed in the vehicle. The positions of those audio sensors placed in the vehicle may vary. The formation of audio sensor pairs, or sets of audio sensors comprising two or more audio sensors, may be determined according to the size and / or computing capacity of the vehicle computing device 110. It should also be understood that the environment in which the vehicle 102 is placed in Figure 2 is for illustrative purposes only. Other audio sources in the environment that similarly contribute to the generation of audio signals may exist, such as construction work on or near the driving route, or a car accident on the driving route. This disclosure is not intended to be limiting.

[0029] In some examples, the computing device 110 of the vehicle 102 may include an auditory signal processing component 118 that processes audio data or audio signals representing sounds in the environment. The auditory signal processing component 118 may include a DoA detection component 120. The DoA detection component 120 may receive audio data representing sounds from the environment from audio sensors 110-1, 110-2, 110-3, 110-4, 110-5, 110-6, 110-7, and 110-8. The DoA detection component 120 may select one or more audio sensor pairs from the four audio sensor pairs described above. In some examples, the DoA detection component 120 may form one or more additional audio sensor pairs by pairing two audio sensors selected from each of the audio sensors 110-1 to 110-8 described above. For example, the DoA detection component 120 may form an additional audio sensor pair [110-1, 110-7], where audio sensors 110-1 and 110-7 are selected from audio sensor pair [110-1, 110-2] and audio sensor pair [110-7, 110-8], respectively. The DoA detection component 120 may estimate the angular spectrum of the audio sensor pair based at least partially on the audio data. Although other techniques are discussed herein, in some examples, the DoA detection component 120 may use a generalized cross-correlation (GCC-PHAT) algorithm with phase transformation to estimate the angular spectrum of the audio sensor pair.

[0030] In some examples, the DoA detection component 120 may perform angular spectrum estimation based at least partially on the assumptions that 1) DoA is represented by a single angle at which the audio signal propagates from the audio source to the audio sensor pair in the far field, 2) the audio signal propagates from the audio source to each audio sensor in the free field, and / or 3) there is a one-to-one mapping between DoA and Time DoA (TDoA). The GCC-PHAT algorithm may, at least partially on the audio data, calculate a value for each DoA associated with the audio data and output the angular spectrum of the audio sensor pair. It should be understood that the GCC-PHAT algorithm is used for illustrative purposes only. Other algorithms, such as the whitening cross-correlation (CPS) algorithm and GCC-PHAT with Fast Fourier Transform (FFT), may also be applied to estimate the angular spectrum.

[0031] The DoA detection component 120 may determine features associated with audio data based at least partially on the angular spectrum. In some examples, features may be selected from the angular spectrum of the audio sensor pair itself, the peak value of the angular spectrum of the audio sensor pair, the TDoA value at the peak of the angular spectrum, the energy value of the audio sensor pair, etc. In some examples, the DoA detection component 120 may select one or more features from the angular spectrum to provide a more accurate DoA estimate. In some other examples, the DoA detection component 120 may determine features associated with audio data based at least partially on the raw audio data. Features may be selected from the angular spectrum of the audio sensor pair itself, the peak value of the angular spectrum of the audio sensor pair, the TDoA value at the peak of the angular spectrum, the energy value of the audio sensor pair, the frequency spectrum, the noise floor, the noise spectrum, zero crossings, the energy distribution, the cross-correlation, machine-learned embeddings, etc. The DoA detection component 120 may perform a determination of the DoA value based at least partially on one or more features. In some examples, the DoA detection component 120 may determine, at least partially, the occurrence of an emergency (or non-emergency) and the direction associated with a vehicle-related audio event in the environment, based on one or more features and a DoA value.

[0032] In some examples, the DoA detection component 120 may input audio data into a machine learning model, such as a neural network, to generate a DoA value. An internal buffer space may be configured to stream the audio data for processing. Audio data received by the audio sensor may be input into the buffer space (e.g., in a first-in, first-out (FIFO) order), thereby allowing the audio data to be associated with audio frames. For example, the internal buffer space may correspond to 3 seconds of audio data, and the audio frames may correspond to 30 milliseconds of data, but any time period can be used. For each audio frame, the DoA detection component 120 may perform an angular spectrum estimation for the audio sensor pair, based at least partially on the data contained within the audio frame, features determined at least partially on the angular spectrum, and / or feature integration, to generate a DoA value associated with the audio frame.

[0033] In some cases, the category of an emergency (or non-emergency) may be determined at least partially based on the DoA value and an audio event classifier (e.g., 410 in Figure 4). The audio event classifier may be a pre-trained component for determining the category of an event. Training of the audio event classifier may be at least partially based on historical data associated with past events that occurred in the vehicle's environment, and features / parameters used to identify past events. The features / parameters used to identify past events may be determined at least partially based on audio data, video data, lidar data, radar data, etc., captured by the vehicle. For emergency vehicles, the audio event classifier may be trained using past emergency vehicles and features associated with audio data (or other data) that were used to identify those past emergency vehicles and represent the siren of the emergency vehicle. The audio event classifier may be trained periodically using feedback from identified emergencies in real time.

[0034] In some cases, the classification of audio events may include emergency audio event classes, such as ambulance siren classes, police siren classes, and fire engine siren classes, or non-emergency audio event classes, such as engine noise classes, music noise classes, thunder noise classes, and speech noise classes.

[0035] In some cases, based on collected audio data representing various types of sounds in the environment, the DoA detection component 120 may determine multiple events. To eliminate ambiguity so that vehicles can respond quickly and accurately to sirens, the DoA detection component 120 may further analyze multiple events according to pre-set rules. Emergency vehicle sirens generally have a dominant sound level relative to any other sounds in the environment. Emergency vehicles may alert other vehicles and / or objects in the environment (e.g., pedestrians, animals, etc.) by sounding their sirens, and the length of the siren may be pre-set to exceed a certain threshold so that other vehicles can be warned. Short sirens, i.e., siren lengths below the threshold, may go unnoticed by other vehicles or cause confusion. To avoid confusion caused by such potentially false sirens, an event may be detected based on audio data, but if the total number of audio frames associated with that event is below a threshold, that event may be determined as a non-event. Once it is determined that audio data corresponds to a non-event, the audio frames associated with the non-event can be removed from further processing to determine the direction of arrival.

[0036] In some cases, emergency vehicles may need to sound their sirens continuously until they reach a destination that is sometimes several miles (or kilometers) away, but there may be some intermittent silences between the sounds. In some cases, if the total number of audio frames between two adjacent events is less than a minimum frame gap (which may be predefined for the siren signal), the potential silent frames (which may be called audio frame gaps) may be removed and the adjacent events may be merged.

[0037] In some cases, the DoA detection component 120 may determine an emergency vehicle by integrating audio data with the detection of sensor data from flashing lights associated with the emergency vehicle, such as flashing light signals 210 and 214 as illustrated in Figure 2, or from other perception systems 422 as illustrated in Figure 4. It should be understood that other perception pipelines may also be integrated with the audio data to determine the occurrence of an emergency vehicle. In one example, image data may be captured, and the emergency vehicle may be detected in that image data.

[0038] Figure 3 is an example of an exemplary audio sensor pair 300 for an implementation of detecting an emergency vehicle in an environment in which an autonomous vehicle is located, according to an embodiment of the present disclosure.

[0039] As illustrated in Figure 3, events 1 and 2 may occur in the environment of vehicle 102. Events 1 and 2 may represent siren signals (or other sounds) generated from their respective directions. To determine the direction of event 2 when event 1 is present, multiple audio sensor pairs may be formed to capture the siren signal. In some examples, the DoA detection component 120 may determine the approximate direction of the siren signal and select an audio sensor pair based at least partially on the approximate direction of the siren signal. As illustrated in Figure 3, eight audio sensor pairs, namely pairs [110-1, 110-2], [110-7, 110-8], [110-5, 110-6], [110-3, 110-4], [110-1, 110-7], [110-1, 110-6], [110-6, 110-4], and [110-4, 110-7], can be formed to determine the direction of event 2 in the presence of event 1. It should be understood that the audio sensor pairs may be formed based on any combination of audio sensors placed in vehicle 102. Selecting a specific audio sensor pair to determine the direction of a siren signal may allow vehicle 102 to achieve a rapid and efficient response to an emergency vehicle. It should be further understood that forming an audio sensor pair containing two audio sensors is for illustrative purposes only. In some examples, more than two sensors may be used to determine the direction of arrival of an audio event.

[0040] Figure 4 shows a block diagram 400 of an exemplary implementation of detecting an emergency vehicle in an environment in which an autonomous vehicle is located, according to an embodiment of the present disclosure.

[0041] As illustrated, the DoA detection component 120 may include an angle spectrum estimation component 402 for receiving audio data collected by the audio sensor 420 and estimating the angle spectrum of the audio sensor pair based at least partially on the audio data. In some examples, the DoA detection component 120 may estimate the angle spectrum of the audio sensor pair using a generalized cross-correlation (GCC-PHAT) algorithm with phase transformation. The angle spectrum estimation component 402 may perform the estimation of the angle spectrum based at least partially on the assumption that DoA is represented by a single angle at which the audio signal propagates from the audio source to the audio sensor pair in the far field, and that the audio signal propagates from the audio source to each audio sensor in the free field, and that there is a one-to-one mapping between DoA and time difference of arrival (TDoA). The angle spectrum estimation component 402 may calculate a value for each DoA associated with the audio data based at least partially on the audio data and output the angle spectrum of the audio sensor pair. It should be understood that the GCC-PHAT algorithm is used for illustrative purposes only. Other algorithms (including those involving the use of data collected from two or more microphones), such as the whitening cross-correlation (CPS) algorithm and GCC-PHAT with fast Fourier transform (FFT), may also be applied to estimate the angular spectrum.

[0042] In some examples, the DoA detection component 120 may include a feature determination component 404 that determines features associated with the audio data based at least partially on the angular spectrum and / or the audio data. In some examples, the calculation of the angular spectrum may be completely skipped as part of the DoA detection component 120. In such cases, the DoA detection component 120 may determine the DoA directly based on the audio data collected by the audio sensor 420, i.e., based on the raw audio data. In some examples, the feature determination component 404 may determine features associated with the audio data based at least partially on the angular spectrum of the audio sensor pair. In some other examples, the feature determination component 404 may determine features associated with the audio data based at least partially on the raw audio data. In some examples, features may be selected from the angular spectrum of the audio sensor pair itself, the peak value of the angular spectrum of the audio sensor pair, the TDoA value at the peak of the angular spectrum, the energy value of the audio sensor pair, the frequency spectrum, the noise floor, the noise spectrum, zero crossings, the energy distribution, the cross-correlation, the machine learning-generated embedding, etc. In some examples, feature selection may be performed using a machine learning model based at least on raw data input (e.g., audio data). In some examples, the feature determination component 404 may select one or more features from the angular spectrum to provide a more accurate DoA estimate. In some other examples, the feature determination component 404 may determine features associated with the audio data based at least partially on the raw audio data. Features may be selected from the angular spectrum of the audio sensor pair itself, the peak value of the angular spectrum of the audio sensor pair, the TDoA value at the peak of the angular spectrum, the energy value of the audio sensor pair, the frequency spectrum, the noise floor, the noise spectrum, zero crossings, the energy distribution, the cross-correlation, machine learning-based embeddings, etc.In some examples, the feature determination component 404 may construct a set of vectors as the feature set to be input to the fusion component 406 of the DoA detection component 120.

[0043] The fusion component 406 performs a fusion operation on one or more features to generate a DoA value associated with the audio. In some examples, the fusion component 406 may input audio data received in real time into a machine learning model to generate a DoA value. The DoA value may indicate the angular direction from which the sound is emitted. An internal buffer may be configured to stream audio data for processing. Audio data received by the audio sensor may be pushed into the buffer in a first-in, first-out order. The buffer may be subdivided into multiple frames. For example, the internal buffer may be configured as a 3-second data queue and subdivided into 30-millisecond frames. For each audio frame, the angular spectrum estimation component 402 may perform angular spectrum estimation for the audio sensor pair based at least partially on the data contained within the audio frame, and the feature determination component 404 may perform feature determination based at least partially on the angular spectrum and / or buffer data to construct a set of vectors as a feature set. The fusion component 406 may perform feature fusion to generate a DoA value associated with the audio frame. It should be understood that the sizes of internal buffers described herein are for illustrative purposes only. Internal buffers may be sized to store 5 seconds, 10 seconds, or any other amount of audio data. This disclosure is not intended to limit.

[0044] In some examples, the DoA detection component 120 may include a detection component 408 that determines, at least partially, the occurrence of an emergency and the direction associated with the emergency related to the vehicle, based on features and DoA values. In some examples, the detection component 408 may determine the category of the emergency based at least partially on the DoA value and an audio event classifier 410. In some examples, sensor data from another perception system 422 may be input to the detection component 408 to determine the direction of arrival of the audio sound. Sensor data from the other perception system 422 may include data captured by one or more of the following: lidar sensors, radar sensors, image sensors, time-of-flight sensors, sonar sensors, etc. In some examples, additional environmental data may be received (e.g., map data of environmental coding parts such as lanes, baselines, crosswalks, traffic devices and / or signs, etc.). The vehicle may be further controlled, at least partially, based on the direction of the emergency related to the vehicle. For example, a vehicle may be controlled to respond to an emergency by reducing its speed, stopping, moving to the shoulder, or changing lanes.

[0045] The audio event classifier 410 may be a pre-trained module for determining the category of an audio event (e.g., whether the audio involves an emergency vehicle and / or the type of emergency vehicle, i.e., an ambulance, fire truck, police siren, etc.). Training of the audio event classifier may be at least in part on training data 430, which includes historical data associated with past events (e.g., from logs of previously recorded data) and features / parameters associated with the historical data. The features / parameters used to identify past events may be determined at least in part on audio and / or video data captured by the vehicle. In some examples, the training data 430 may include past audio event classification results (e.g., whether the audio event is an emergency audio event or a non-emergency audio event, and whether the emergency audio event is related to a fire truck, ambulance, police vehicle, etc.). For emergency vehicles, the audio event classifier 410 may be trained using audio data associated with past emergency vehicles and features derived therefrom, associated with audio data representing the siren of an emergency vehicle. The audio event classifier 410 may be periodically trained using real-time feedback from identified emergencies. The detection component 408 may determine an onset time frame indicating an emergency onset based at least partially on the DoA value and an end frame indicating an offset of the emergency onset based at least partially on the DoA value. In some examples, the detection component 408 may use a Doppler frequency shift function to determine, at least partially on the DoA value, whether an emergency vehicle is approaching or moving away from vehicle 102.

[0046] In some cases, gaps between frames can be removed to combine adjacent audio events, and / or audio events associated with a time period or number of frames below a threshold can be removed as discussed herein. In some cases, such processing can be performed by detection component 408 or other components discussed herein.

[0047] In some examples, the detection component 408 may determine that an audio event includes an emergency event by integrating audio data with sensor data from other perception systems 422, as discussed herein. For example, image data may be captured and processed to determine the presence of flashing lights in the environment associated with a vehicle. The DoA value associated with the audio event may be associated with a first direction related to the vehicle in the environment. If the image data indicates that flashing lights (e.g., indicating an emergency vehicle) are positioned in the first direction, the detection component 408 may increase its confidence that the audio event is associated with an emergency event. In some examples, the color of the flashing lights may be determined to help classify the emergency class. For example, if all flashing lights are determined to be red, the emergency class may be associated with a fire truck. In another example, if the flashing lights are determined to be red and blue, the emergency class may be associated with a police car. It should be understood that other perception pipelines may also be integrated with the audio data to determine the occurrence of an emergency vehicle. For example, lidar sensor data may be received and input into a machine learning model trained to determine the presence of an emergency vehicle in the environment. A block diagram of an exemplary implementation for detecting audio and / or emergency events in an environment, as illustrated in Figure 4, may include one or more components that process other perception data for DoA estimation.

[0048] Figure 5 is an example of features associated with audio signals detected by four audio sensor pairs, according to an example of the present disclosure.

[0049] Figure 5 illustrates features associated with audio data collected by four audio sensor pairs, namely [110-1, 110-2], [110-7, 110-8], [110-3, 110-4], and [110-5, 110-6], for estimating the DoA of event 2 illustrated in Figure 3. Column 502 illustrates the two channel audio signals received by the audio sensor pair. Column 504 illustrates the spectral analysis of the two channel audio signals and their average values ​​of audio signal energy. Column 506 illustrates the angular spectrum representing the peak value and the angle at which the peak value occurs. The angular values ​​illustrated in column 506 have been converted from TDoA values ​​for illustrative purposes.

[0050] When Event 2 occurs in the forward area of ​​Vehicle 102, the audio sensor pair [110-1, 110-2] located in the forward area of ​​Vehicle 102 exhibits the strongest audio signal among the four audio sensor pairs during the time period in which Event 2 is present, as illustrated in column 502. The audio sensor pair [110-1, 110-2] located in the forward area of ​​Vehicle 102 also exhibits the highest energy level among the four audio sensor pairs during the time period in which Event 2 is present, as illustrated in column 504.

[0051] Column 506 also illustrates that the audio signal received by audio sensor pair [110-1, 110-2] peaks at an energy of 466.2 at an angle of 79.1 degrees. Audio sensor pair [110-7, 110-8], located on the left side of vehicle 102, shows the second strongest audio signal among the four audio sensor pairs and a lower energy level than audio sensor pair [110-1, 110-2] during the time period in which event 2 is present. Column 506 also illustrates that the audio signal received by audio sensor pair [110-7, 110-8] peaks at an energy of 450.6 at an angle of 90.6 degrees. Both audio sensor pairs [110-1, 110-2] and [110-7, 110-8] show dominant audio signals representing event 2, even when event 1 is also nearby, as illustrated in column 502. The audio sensor pair [110-5, 110-6] located on the right side of vehicle 102 shows a mixed audio signal representing Event 1 and Event 2. The audio sensor pair [110-5, 110-6] shows a similar energy level to the audio sensor pair [110-7, 110-8], but the audio signal received by the audio sensor pair [110-5, 110-6] peaks at an angle that deviates from that of the audio sensor pairs [110-1, 110-2] and [110-7, 110-8]. The audio sensor pair [110-3, 110-4] located in the rear area of ​​vehicle 102 shows the weakest audio signal and lowest energy level of the four audio sensor pairs, but the audio signal received by the audio sensor pair [110-3, 110-4] peaks at an angle close to that of the audio sensor pairs [110-1, 110-2] and [110-7, 110-8].

[0052] Figure 6 illustrates 600 features associated with audio signals detected by four other audio sensor pairs, according to an example of the present disclosure. Column 602 illustrates two channel audio signals received by the audio sensor pair. Column 604 illustrates a spectral analysis of the two channel audio signals and their average audio signal energies. Column 606 illustrates an angular spectrum representing the peak values ​​and angles at which the peak values ​​occur. The angular values ​​illustrated in column 506 have been converted from TDoA values ​​for illustrative purposes.

[0053] Figure 6 illustrates features associated with audio data collected by four audio sensor pairs, namely [110-1, 110-7], [110-1, 110-6], [110-6, 110-4], and [110-4, 110-7], in order to estimate the DoA of event 2 as illustrated in Figure 3. Audio sensor pair [110-1, 110-7] is formed by one audio sensor from the front area of ​​vehicle 102 and one audio sensor from the left side of vehicle 102. Compared to audio sensor pair [110-1, 110-2] located in the front area of ​​vehicle 102, audio sensor pair [110-1, 110-7] exhibits a lower energy level, and the audio signal received by audio sensor pair [110-1, 110-7] peaks at an angle that deviates from audio sensor pair [110-1, 110-2]. The audio sensor pair [110-1, 110-6] is formed by one audio sensor from the front area of ​​vehicle 102 and one audio sensor from the right side of vehicle 102. Compared to the audio sensor pair [110-1, 110-2] located in the front area of ​​vehicle 102, audio sensor [110-1, 110-6] shows a mixed audio signal representing event 1 and event 2, a lower energy level, and the audio signal received by audio sensor pair [110-1, 110-6] peaks at an angle deviating from that of audio sensor pair [110-1, 110-2]. The audio sensor pair [110-6, 110-4] is formed by one audio sensor from the right side of vehicle 102 and one audio sensor from the rear area of ​​vehicle 102. Compared to the audio sensor pair [110-1, 110-2] located in the front area of ​​vehicle 102, the audio sensor pair [110-6, 110-4] also shows a mixed audio signal representing event 1 and event 2, a lower energy level, and the audio signal received by the audio sensor pair [110-6, 110-4] peaks at an angle that deviates from that of the audio sensor pair [110-1, 110-2].The audio sensor pair [110-4, 110-7] is formed by one audio sensor from the rear area of ​​vehicle 102 and one audio sensor from the left side of vehicle 102. Compared to the audio sensor pair [110-1, 110-2] located in the front area of ​​vehicle 102, the audio sensor pair [110-4, 110-7] shows a dominant audio signal in column 502, but has a lower energy level, and the audio signal received by the audio sensor pair [110-4, 110-7] peaks at an angle that deviates from that of the audio sensor pair [110-1, 110-2].

[0054] As illustrated in Figures 5 and 6, the audio signals received by audio sensor pairs [110-1, 110-2], [110-3, 110-4], [110-1, 110-7], and [110-6, 110-4] show a single peak, while the audio signals received by audio sensor pairs [110-7, 110-8], [110-5, 110-6], [110-1, 110-6], and [110-4, 110-7] reach peaks at more than one angle. Multiple peaks introduce ambiguity when determining the DoA of an emergency vehicle. Furthermore, by comparing energy levels and peak angles, the audio sensor pair [110-1, 110-2] located in the forward area of ​​vehicle 102 provides the best estimate of the orientation in which the emergency is located. Since the audio sensor pair [110-1, 110-2] exhibits a single peak with the highest energy at one angle, and the remaining audio pair does not contradict this, the DoA component can determine the direction of sound arrival based on the audio sensor pair [110-1, 110-2]. It should be understood that the audio sensor pair can be selected or formed based on any possible combination of audio sensors placed in the vehicle. The examples illustrated in Figures 5 and 6 are for illustrative purposes only. This disclosure is not intended to limit.

[0055] Figure 7 illustrates an exemplary process 700 for detecting an emergency vehicle based on audio signals detected by an audio sensor pair, according to an embodiment of the present disclosure. The calculations described herein with respect to the process for detecting an emergency vehicle may be performed by a vehicle computing device 110, as illustrated in Figure 1. In some examples, the calculations may be performed by a direction of arrival (DoA) detection component 120 of the vehicle computing device 110, as illustrated in Figure 1.

[0056] As an example, and not an limitation, process 700 is illustrated as a logical flow graph, where each operation represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In a software context, an operation represents a computer-executable instruction stored on one or more computer-readable recording media that, when executed by one or more processors, performs the operation described. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular abstract data type. The order in which the operations are described is not intended to be interpreted as limitation, and any number of the operations described can be combined (or omitted) in any order and / or in parallel to implement the process. In some examples, multiple branches represent alternative implementations that can be used separately or in combination with other operations discussed herein.

[0057] In operation 702, the process may include receiving audio data from multiple audio sensors associated with the vehicle. In some examples, the audio data may be collected by audio sensors 110-1, 110-2, 110-3, 110-4, 110-5, 110-6, 110-7, and 110-8 located on the vehicle 102, as illustrated in Figure 1. Multiple audio sensors may be arranged to cover a 360-degree area around the vehicle. The audio data may represent sounds from various audio sources in the environment in which the vehicle 102 is located. The audio data may represent sounds from vehicles traveling in the same direction as the vehicle 102, sounds from vehicles traveling in the opposite direction to the vehicle 102, sirens from emergency vehicles, music in the environment, human speech in the environment, dog barking in the environment, construction noise in the environment, etc. In some examples, siren sounds may be dominant among all sounds generated in the environment.

[0058] In calculation 704, the process may include determining audio sensor pairs from a plurality of audio sensors. In some examples, audio sensor pairs may be arbitrarily selected or formed based on any permutation or variation of audio sensors placed on the vehicle. In some examples, calculation 704 may be performed by the angle spectrum estimation component 402 of the DoA detection component 120, as illustrated in Figure 4. In some examples, the plurality of audio sensors may be pre-configured as one or more audio sensor pairs placed on the vehicle to face different areas outside the vehicle. For example, as illustrated in Figure 1, audio sensor pair [110-1, 110-2] is placed in the front area of ​​vehicle 102, audio sensor pair [110-3, 110-4] is placed in the rear area of ​​vehicle 102, audio sensor pair [110-5, 110-6] is placed on the right side of vehicle 102, and audio sensor pair [110-7, 110-8] is placed on the left side of vehicle 102. The front area, rear area, left side, and right side relate to the direction of travel of the vehicle 102. In some examples, one or more additional audio sensor pairs may be formed by pairing two audio sensors selected from the four audio sensor pairs described above. For example, an additional audio sensor pair [110-1, 110-7] may be formed, in which case audio sensors 110-1 and 110-7 are selected from audio sensor pair [110-1, 110-2] and audio sensor pair [110-7, 110-8], respectively. It should be understood that audio sensor pairs may be formed based on any combination of audio sensors located on the vehicle 102.

[0059] In operation 730, the process may include discretizing the audio data into multiple audio frames. The length of an audio frame may be predetermined to, for example, 30 milliseconds, but the audio frames may be associated with any length of time.

[0060] In operation 732, the processing may include inputting a plurality of audio frames into a buffer space, as discussed herein. The audio data may be processed from the buffer using a first-in, first-out method.

[0061] In operation 706, the process may include estimating the angular spectrum of the audio sensor pair based at least partially on the audio data in each audio frame. In some examples, operation 706 may be performed by the angular spectrum estimation component 402 of the DoA detection component 120, as illustrated in Figure 4. In some examples, a generalized cross-correlation (GCC-PHAT) algorithm with phase transformation may be implemented to estimate the angular spectrum of the audio sensor pair. The estimation of the angular spectrum may be based at least partially on the assumption that the DoA is represented by a single angle at which the audio signal propagates from the audio source to the audio sensor pair in the far field, and that the audio signal propagates from the audio source to each audio sensor in the free field. Since the TDoA calculated for a sample can be converted to a DoA with respect to angles, the DoA of an audio event can be determined at least partially on the peaks of the TDoA. The GCC-PHAT algorithm (or other algorithms discussed herein) can be implemented to estimate the angular spectrum associated with an audio sensor pair based at least partially on audio data and to output the angular spectrum of the audio sensor pair.

[0062] In operation 708, the process may include determining features associated with the audio data, at least partially based on the angular spectrum (or the audio data itself). In some examples, operation 708 may be performed by the feature determination component 404 of the DoA detection component 120, as illustrated in Figure 4. Features associated with the audio data may be determined based at least partially on the angular spectrum. In some examples, features may be selected from the angular spectrum of the audio sensor pair itself, the peak value of the angular spectrum of the audio sensor pair, the TDoA value at the peak of the angular spectrum, the energy value of the audio sensor pair, the frequency spectrum, the noise floor, zero crossings, the energy distribution, cross-correlation, machine learning-generated embeddings, etc.

[0063] In some examples, a machine-learned component may be implemented on real-time received audio data to generate a DoA value. An internal buffer may be configured to stream the audio data for processing. Audio data received by an audio sensor may be input into the internal buffer frame by frame in a first-in, first-out order. The audio data may be streamed into the internal buffer in audio frames of a predetermined length. For each audio frame, angular spectrum estimation may be performed, and at least features may be determined based on the data contained within the audio frame or the estimated angular spectrum. In some examples, audio frame gaps may be removed to combine audio events, and / or audio events below a threshold time or threshold number of frames may be removed.

[0064] In operation 710, the process may include determining a direction of arrival (DoA) value based at least partially on the features. In some examples, operation 708 may be performed by a detection component 408 of a DoA detection component 120, as illustrated in Figure 4. Referring to the description of operation 706, the GCC-PHAT algorithm may be implemented to estimate the angular spectrum associated with an audio sensor pair based at least partially on audio data and to output the angular spectrum of the audio sensor pair. One or more features may be extracted based at least partially on the estimated angular spectrum for each audio sensor pair. In some examples, one or more features may be extracted directly from the audio data. One or more features may be input into a machine learning model, where one or more features are fused to generate a DoA value. The machine learning model may include a neural network regression (NNR) model.

[0065] In operation 712, the process may include determining the occurrence of an audio event based at least partially on features. In some examples, operation 708 may be performed by detection component 408 of DoA detection component 120, as illustrated in Figure 4. In some examples, the category of an audio event may be determined based at least partially on the DoA value and an audio event classifier.

[0066] In calculation 714, the process may include determining the direction of an audio event related to a vehicle, at least partially based on the DoA value. In some examples, calculation 708 may be performed by a detection component 408 of the DoA detection component 120, as illustrated in Figure 4. In some examples, a start time frame indicating the onset of an audio event and an end frame indicating the offset of an audio event, at least partially based on the DoA value, may be determined at least partially based on the DoA value. In some examples, the DoA detection component 120 may use a Doppler frequency shift function to determine, at least partially based on the DoA value, whether an emergency vehicle associated with an audio event is approaching vehicle 102 or moving away from vehicle 102.

[0067] In some examples, features associated with audio data may be determined based on the raw audio data. In operation 716, the process may include determining features associated with audio data based at least partially on the audio data within each audio frame.

[0068] In operation 718, the process may include receiving sensor data from a second perception sensor associated with the vehicle. The second perception sensor associated with the vehicle may include one or more of the following: a lidar sensor, a radar sensor, an image sensor, a time-of-flight sensor, a sonar sensor, and a location sensor. In some examples, the sensor data may also include environmental data captured by environmental sensors. In some examples, operation 718 may include receiving other data such as segmentation information, classification information, detection results, object size, object velocity, etc. In some examples, operation 718 may include receiving environmental map data, which may enable the determination of locations where an emergency vehicle is likely to be located.

[0069] In operation 720, the process may include determining the occurrence of an audio event based at least in part on sensor data and features. In some examples, sensor data and features associated with audio data may be treated as different parameters for determining the occurrence of an emergency. Different weights may be assigned to the decision based on sensor data and the decision based on features associated with audio data in order to calculate the likelihood, e.g., probability value, of an emergency occurring. In at least one example, image data may be applied to determine the occurrence of an emergency. The DoA detection component 120 may determine whether an image of an emergency vehicle is captured based on the image data. In another example, a light signal, i.e., a flashing light, may be applied to determine the occurrence of an emergency. The color of the flashing light may be further used to determine the emergency class. It should be understood that the examples described above are for illustrative purposes only and this disclosure is not intended to limit.

[0070] In operation 722, the processing may include controlling the vehicle based at least in part on the direction of audio events related to the vehicle. For example, the vehicle may be controlled to slow down or stop at its current location, move to one side of the road, or change lanes to allow an emergency vehicle to pass. In some examples, operation 722 may include generating a trajectory that the vehicle should follow. In some examples, operation 722 may include receiving commands from a remote computing device (e.g., a teleoperator) to control the vehicle.

[0071] It should be understood that process 700 is for illustrative purposes only. This disclosure is not intended to be limiting. In some examples, multiple emergency audio events may be detected. Process 700 may further include calculations for determining a DoA or TDoA for each of the multiple emergency audio events.

[0072] It should be understood that the audio sensor pair used to detect the DoA of an emergency vehicle may be selected according to real-time needs. Different locations where emergency vehicles appear may result in different pairings of audio sensors for detection. Therefore, once detection of an emergency vehicle is performed, for example, calculation 714 may be completed, and processing may return to calculation 704 to prepare for the detection of future emergencies.

[0073] Figure 8 shows a block diagram of an exemplary system 800 for implementing the techniques described herein.

[0074] The vehicle 102 may include one or more vehicle computing devices 804 (also referred to as vehicle computing device 804, or vehicle computing device(s) 804), one or more sensor systems 806, one or more emitters 808, one or more communication connections 810, at least one direct connection 812, and one or more drive systems 814.

[0075] The vehicle computing device 804 may include one or more processors 816 and a memory 818 that is communicated with the one or more processors 816. In the illustrated example, the vehicle 102 is an autonomous vehicle, but the vehicle 102 may be any other type of vehicle. In the illustrated example, the memory 818 of the vehicle computing device 804 stores a localization component 820, a perception component 822, one or more maps 824, one or more system controllers 826, and an auditory signal processing component 828 which includes at least a DoA detection component 830 and a machine learning component 832. In Figure 8, the localization component 820, the perception component 822, one or more maps 824, one or more system controllers 826, the auditory signal processing component 828, the DoA detection component 830, and the machine learning component 832 are depicted as residing in the memory 818 for illustrative purposes, but it is assumed that they may additionally or alternatively be accessible to the vehicle 102 (e.g., stored remotely).

[0076] In at least one example, the localization component 820 may include functionality for receiving data from a sensor system 806 that determines the position and / or orientation of the vehicle 102 (e.g., one or more of x, y, z positions, roll, pitch, or yaw). For example, the localization component 820 may include and / or request / receive a map of the environment and continuously determine the location and / or orientation of the autonomous vehicle within the map. In some examples, the localization component 820 may accurately determine the location of the autonomous vehicle by receiving image data, lidar data, radar data, IMU data, GPS data, wheel encoder data, etc., using SLAM (simultaneous localization and mapping), CLAMS (calibration, localization and mapping, simultaneously), relative SLAM, bundle adjustment, nonlinear least-squares optimization, etc. In some examples, the localization component 820 may provide data to various components of the vehicle 102 to determine the initial position of the autonomous vehicle in order to generate a trajectory and / or generate or receive map data, as discussed herein.

[0077] In some examples, the perception component 822 may include functionality for performing object detection, segmentation, and / or classification. In some examples, the perception component 822 may provide processed sensor data indicating the presence of entities near the vehicle 102, and / or classification of entities as entity types (e.g., automobile, pedestrian, cyclist, animal, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional or alternative examples, the perception component 822 may provide processed sensor data indicating one or more characteristics associated with the detected entity (e.g., the tracked object) and / or the environment in which the entity is located. In some examples, characteristics associated with an entity may include, but are not limited to, x-position (global and / or local position), y-position (global and / or local position), z-position (global and / or local position), orientation (e.g., roll, pitch, yaw), entity type (e.g., classification), entity velocity, entity acceleration, entity (size) range, etc. Characteristics associated with an environment may include, but are not limited to, the presence of other entities in the environment, the state of other entities in the environment, time of day, day of the week, season, weather, darkness / light indication, etc.

[0078] Memory 818 may further include one or more maps 824 that can be used by the vehicle 102 to navigate within the environment. For the purposes of this discussion, a map can be any number of data structures modeled in two, three, or N dimensions that can provide information about the environment, including, but not limited to, topology (such as intersections), streets, mountain ranges, roads, terrain, and the environment in general. In some examples, a map may include, but is not limited to, texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV / HSL color information), intensity information (e.g., LIDAR information, RADAR information, etc.), spatial information (e.g., image data projected onto a mesh, individual "surfaces" (e.g., polygons associated with individual colors and / or intensities)), and reflection information (e.g., specularity information, retroreflection information, BRDF information, BSSRDF information, etc.). In one example, a map may include a three-dimensional mesh of the environment. In some examples, the maps may be stored in a tiled format, where individual tiles of the map represent distinct parts of the environment, and may be loaded into working memory as needed, as discussed herein. In at least one example, one or more maps 824 may include at least one map (e.g., an image and / or mesh). In some examples, the vehicle 102 may be controlled at least in part based on the map(s) 824. In some examples, one or more maps 824 may be stored on a remote computing device (e.g., computing device(s) 842) accessible via a network(s) 840. In some examples, multiple maps 824 may be stored based on characteristics (e.g., entity type, time of day, day of the week, season of the year, etc.). Storing multiple maps 824 may have similar memory requirements but may increase the speed at which data in the maps can be accessed.

[0079] In at least one example, the vehicle computing device 804 may include one or more system controllers 826 that may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 102. The system controllers 826 described above may communicate with and / or control the corresponding systems of the drive system 814 and / or other components of the vehicle 102.

[0080] In some examples, the auditory signal processing component 828 may process audio data received by a plurality of audio sensors associated with the vehicle 102 and determine, at least in part, the occurrence of an emergency and the direction of the emergency relating to the vehicle. Such an auditory signal processing component 828 may include a DoA detection component (e.g., DoA detection component 120 in Figures 1, 4, etc.) as described herein. Given various pairs of sensor data as described herein, the DoA detection component 830 may receive audio data collected by audio sensors in the sensor system 806(or more) to determine the direction of arrival.

[0081] In some examples, the auditory signal processing component 828 may further comprise a machine learning component 832 (also referred to as a machine-learned model component 832), which may include functionality for receiving one or more features associated with audio data from the DoA detection component 830 (and / or from any one or more sensors), as discussed herein, and generating a DoA value.

[0082] In some cases, some or all aspects of the components discussed herein may include any model, algorithm, and / or machine learning algorithm. For example, in some cases, the components in memory 818 (and memory 846 discussed below) may be implemented as a neural network.

[0083] As described herein, an exemplary neural network is a biologically inspired algorithm that passes input data through a series of connected layers to produce an output. Each layer in the neural network may include another neural network, or any number of layers (whether convolutional or not). As can be understood in the context of this disclosure, the neural network may utilize machine learning, which can refer to a broad class of such algorithms in which an output is produced based on learned parameters.

[0084] Although discussed in the context of neural networks, any type of machine learning that is not inconsistent with this disclosure may be used. For example, machine learning algorithms include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally estimated scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic nets, least-angle regression (LARS)), decision tree algorithms (e.g., classification and regression tree (CART), iterative dichotomiser 3 (ID3), chi-squared automatic interaction detection (CHAID), decision stump, conditional decision tree), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, polynomial naive Bayes, mean-1 dependent estimator (AODE)). One-dependence estimators), Bayesian belief networks (BNN), Bayesian networks), clustering algorithms (e.g., k-means, k-intermediate, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptron, backpropagation, Hopfield network, radial basis function network (RBFN)), deep learning algorithms (e.g., deep Boltzmann machine (DBM), deep belief networks (DBN))Networks, Convolutional Neural Networks (CNNs), Stacked Autoencoders, Dimensionality Reduction Algorithms (e.g., Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Summon Mapping, Multidimensional Scaling (MDS), Projection Tracking, Linear Discriminant Analysis (LDA), Mixture Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis (FDA)), Ensemble Algorithms (e.g., Boosting, Bootstrap Aggregation (Bagging), AdaBoost, Stack Generalization (Blending), Gradient Boosting Machines (GBMs), Gradient Boosted Regression Trees (GBRTs)) This may include, but is not limited to, Trees, Random Forests, SVMs (Support Vector Machines), supervised learning, unsupervised learning, semi-supervised learning, etc.

[0085] Additional architectural examples include neural networks such as ResNet50, ResNet101, VGG, DenseNet, and PointNet.

[0086] In at least one example, the sensor system 806 may include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, location sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement unit (IMU), accelerometer, magnetometer, gyroscope, etc.), cameras (e.g., RGB, IR, intensity, depth, etc.), time-of-flight sensors, audio sensors, wheel encoders, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.). The sensor system 806 may include multiple examples of each of these or other types of sensors. For example, the lidar sensors may include individual lidar sensors located at the corners, front, rear, sides, and / or top of the vehicle 102. In another example, the camera sensors may include multiple cameras located at various locations on the exterior and / or interior of the vehicle 102. The sensor system 806 may provide input to the vehicle computing device 804. Additionally, or alternatively, the sensor system 806(or more) may send sensor data to one or more computing devices via one or more networks 840 at a specific frequency, after a predetermined time period has elapsed, or in near real-time.

[0087] Vehicle 102 may also include one or more emitters 808 for emitting light and / or sound, as described above. In this example, emitters 808 include internal audio and visual emitters for communicating with the occupants of vehicle 102. Internal emitters may include, but are not limited to, speakers, lighting, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, headrest positioners, etc.). In this example, emitters 808 also include external emitters. For example, but not limited to, the external emitters in this example include, exemplary, lighting or other indicators of vehicle action for indicating direction of travel (e.g., indicator lights, signs, lighting arrays, etc.), and one or more audio emitters for audible communication with pedestrians or other nearby vehicles (e.g., speakers, speaker arrays, horns, etc.), one or more of the one or more audio emitters including auditory beam steering technology.

[0088] The vehicle 102 may also include one or more communication connections 810 that enable communication between the vehicle 102 and one or more other local or remote computing devices. For example, the communication connections 810 may facilitate communication with other local computing devices and / or drive systems 814 on the vehicle 102. The communication connections 810 may also enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic lights, etc.). Furthermore, the communication connections 810 may also enable the vehicle 102 to communicate with remote teleoperation computing devices or other remote services.

[0089] A communication connection(s) 810 may include physical and / or logical interfaces for connecting a vehicle computing device 804 to another computing device or to a network(s) 840. For example, a communication connection(s) 810 may enable Wi-Fi-based communication via frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communication protocol that allows each computing device to interface with other computing devices.

[0090] In at least one example, the vehicle 102 may include one or more drive systems 814. In some examples, the vehicle 102 may have a single drive system 814. In at least one example, if the vehicle 102 has multiple drive systems 814, the individual drive systems 814 may be located at opposing ends of the vehicle 102 (e.g., front and rear). In at least one example, the drive system(s) 814 may include one or more sensor systems for detecting the conditions of the drive system(s) 814 and / or the surroundings of the vehicle 102. For example, but not limited to, the sensor systems may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of the drive system's wheels, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive system, cameras or other image sensors, ultrasonic sensors, lidar sensors, radar sensors, etc. for audibly detecting objects around the drive system. Some sensors, such as wheel encoders, may be specific to the drive system 814. In some cases, the sensor system on the drive system 814 may overlap with or supplement the corresponding system on the vehicle 102 (e.g., sensor system 806).

[0091] The drive system(s) 814 may include many of the vehicle systems, including a high-voltage battery, a motor for propelling the vehicle, an inverter for converting DC from the battery to AC for use by other vehicle systems, a steering system including a steering motor and steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking force to mitigate traction loss and maintain control, an HVAC system, lighting (e.g., headlights / taillights for illuminating the exterior of the vehicle), and one or more other systems (e.g., a cooling system, safety systems, an on-board charging system, other electrical components such as DC / DC converters, high-voltage contacts, high-voltage cables, a charging system, a charging port, etc.). Additionally, the drive system(s) 814 may include a drive system controller that receives and preprocesses data from sensor systems(s) and controls the operation of various vehicle systems. In some examples, the drive system controller may include one or more processors and memory communicably coupled to one or more processors. The memory may store one or more components for performing various functionalities of the drive system(s) 814. Furthermore, the drive system(s) 814 also include one or more communication connections that enable each drive system to communicate with one or more other local or remote computing devices.

[0092] In at least one example, the direct connection 812 may provide a physical interface for coupling one or more drive systems 814 to the body of the vehicle 102. For example, the direct connection 812 may enable the transmission of energy, fluid, air, data, etc., between the drive system 814 and the vehicle. In some examples, the direct connection 812 may further detachably secure the drive system 814 to the body of the vehicle 102.

[0093] In some examples, the vehicle 102 may send sensor data to one or more computing devices 842 via a network 840. In some examples, the vehicle 102 may send raw sensor data to the computing devices 842. In other examples, the vehicle 102 may send processed sensor data and / or a representation of the sensor data to the computing devices 842. In some examples, the vehicle 102 may send sensor data to the computing devices 842 at a specific frequency, after a predetermined time period, or in near real-time. In some cases, the vehicle 102 may send sensor data (raw or processed) to the computing devices 842 as one or more log files.

[0094] The computing device(s) 842 may include a processor(s) 844 and memory 846 for storing training components 848.

[0095] In some examples, the training component 848 may include functionality for training a machine learning model to generate DoA values ​​associated with detected emergency vehicles. For example, the training component 848 may receive a set of audio data associated with an emergency event, and features / parameters associated with them. In some cases, the set of audio data may represent sounds in the environment over a period of time, such as 5 minutes, 5 hours, 1 day, 1 week, etc. At least a portion of the set of audio data may be used as input for training a machine learning model, and at least a portion of the set of audio data may be used as ground truth information for training a machine learning model. As a non-limiting example, a first set of sequences of audio data frames (e.g., three, four, five, or more) may be input to a machine learning model. A second set of audio data in the sequence of audio data frames immediately preceding the first set (or, for example, by extracting angular spectra, peaks in the angular spectra, and / or associated feature information from those peaks, etc.) may then be used as ground truth information for training the model. Therefore, by providing audio data of emergency vehicles previously detected in the environment, the DoA detection component can efficiently detect the occurrence and orientation of emergency vehicles, as discussed herein.

[0096] In some examples, the training component 848 may include training data generated by the simulator. For example, the simulated training data may represent an example of testing an audio source in an environment to provide additional training examples.

[0097] The processor(s) 816 of vehicle 102 and the processor(s) 844 of computing device(s) 842 may be any suitable processor capable of processing data and executing instructions for performing operations as described herein. For example, but not limited to, the processor(s) 816 and 844 may comprise one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or part of a device that processes electronic data and converts that electronic data into other electronic data that can be stored in registers and / or memory. In some examples, integrated circuits (e.g., ASICs), gate arrays (e.g., FPGAs), and other hardware devices may also be considered processors, insofar as they are configured to implement encoded instructions.

[0098] Memory 818 and 846 are examples of non-temporary computer-readable media. Memory 818 and 846 may store an operating system and one or more software applications, instructions, programs and / or data for implementing the methods and functions belonging to various systems described herein. In various implementations, memory may be implemented using any suitable memory technology, such as static random-access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash-type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which those shown in the accompanying figures are merely examples relevant to the discussion herein.

[0099] In some examples, memory 818 and 846 may include at least working memory and storage memory. For example, working memory may be a limited-capacity, high-speed memory (e.g., cache memory) used to store data to be manipulated by processors 816 and 844. In some examples, memory 818 and 846 may include storage memory, which may be a relatively large-capacity, slower memory used for long-term storage of data. In some cases, processors 816 and 844 may not have to directly manipulate the data stored in storage memory, and the data may need to be loaded into working memory in order to perform data-based operations, as discussed herein.

[0100] Although Figure 8 illustrates a distributed system, it should be noted that in alternative examples, the components of vehicle 102 may be associated with computing device(s) 842, and / or the components of computing device(s) 842 may be associated with vehicle 102. That is, vehicle 102 may perform one or more of the functions associated with computing device(s) 842, and vice versa.

[0101] Exemplary clauses A. A system comprising one or more processors and one or more computer-readable media storing instructions that can be executed by one or more processors, wherein the instructions, when executed, cause the system to perform an operation including: receiving audio data from a pair of audio sensors associated with a vehicle; determining angular spectral data based at least in part on a portion of the audio data; determining features associated with the audio data based at least in part on the angular spectral data; inputting features into a machine-learned model; receiving direction of arrival (DoA) values ​​associated with the audio data from the machine-learned model; determining the occurrence of sounds associated with an emergency vehicle based at least in part on the audio data; and determining the direction of an emergency vehicle related to the vehicle based at least in part on the DoA values.

[0102] B. The system described in paragraph A, wherein the audio data includes multiple audio frames captured over time, determining angular spectral data includes determining multiple angular spectra, the angular spectra among the multiple angular spectra are associated with the audio frames among the multiple audio frames, determining features includes determining multiple features, the features among the multiple features are associated with the angular spectra among the multiple angular spectra, and inputting features into a machine learning model includes inputting multiple features into a machine learning model.

[0103] C. The system described in paragraph A or B, wherein the features include at least one of the following: the distribution of angular spectral data, the peak value of the angular spectral data, the time difference of arrival (TDoA), or the energy value of the audio data.

[0104] D. The system described in any one of paragraphs A to C, wherein the audio data is discretized into multiple audio frames, and determining the occurrence of sound includes inputting at least a portion of the audio data into a classifier and receiving a sound classification from the classifier, the classification including one or more of the following: ambulance siren class, police siren class, or fire engine siren class, and the operation further includes determining an onset time frame indicating the onset of sound and determining an end time frame indicating the offset of sound.

[0105] E. The system described in any one of paragraphs A to D, further comprising: determining an event that starts at the onset of sound and ends at the offset of sound; determining the total number of audio frames associated with the event; and removing the event when the total number of audio frames associated with the event is less than a threshold.

[0106] A method comprising: receiving audio data from a first audio sensor and a second audio sensor associated with a vehicle; determining features associated with the audio data; determining a direction of arrival (DoA) value associated with the audio data, at least in part, based on the features; determining the occurrence of an audio event, at least in part, based on the audio data; determining the direction of an audio event related to the vehicle, at least in part, based on the DoA value; and controlling the vehicle, at least in part, based on the direction of the audio event related to the vehicle.

[0107] G. The first and second audio sensors associated with the vehicle comprises at least one of the following: two audio sensors associated with the front area of ​​the vehicle, two audio sensors associated with the left side of the vehicle, two audio sensors associated with the rear area of ​​the vehicle, or two audio sensors associated with the right side of the vehicle, wherein the front area, rear area, left side, and right side are relative to the direction of travel of the vehicle, as described in paragraph F.

[0108] The method according to paragraph F or G, further comprising H. determining that an audio event includes an emergency event, and controlling a vehicle based at least in part on the audio event including the emergency event, wherein controlling the vehicle includes at least one of stopping the vehicle or changing the vehicle to another lane.

[0109] I. The method according to any one of paragraphs F to H, further comprising determining the angular spectrum at least partially on one of the following: a generalized cross-correlation phase transform algorithm (GCC-PHAT), a multiple signal classification (MUSIC) algorithm, or a state coherence transform (SCT) algorithm, wherein the features associated with the audio data are determined at least partially on the angular spectrum.

[0110] J. The method according to any one of paragraphs F to I, wherein the audio data comprises a plurality of audio frames, and the method further comprises determining the angular spectra of a first audio sensor and a second audio sensor based at least partially on an audio frame among the plurality of audio frames, determining one or more parameters associated with the audio frame based at least partially on the angular spectra, and determining the direction of arrival (DoA) value associated with the audio frame based at least partially on one or more parameters.

[0111] K. The method according to any one of paragraphs F to J, wherein one or more parameters include at least one of the following: a distribution associated with an angular spectrum, a peak value associated with an angular spectrum, a time difference of arrival (TDoA) value, an energy value of audio data, a frequency spectrum, a noise floor, a noise spectrum, zero crossings, an energy distribution, a cross-correlation, or a machine learning-generated embedding.

[0112] L. The method according to any one of paragraphs F to K, wherein audio data is discretized into multiple audio frames, and determining the occurrence of an audio event includes inputting at least a portion of the audio data into a classifier and receiving a classification of the audio event from the classifier, the classification including one or more of ambulance siren classes, police siren classes, or fire engine siren classes, and the method further includes determining a start audio frame indicating the onset of the audio event and determining a end audio frame indicating the offset of the audio event.

[0113] M. The method according to any one of paragraphs F to L, further comprising determining a first audio event that occurred between a first start audio frame and a first end audio frame, determining the total number of audio frames associated with the first audio event, and removing the first audio event when the total number of audio frames associated with the first audio event is less than a threshold.

[0114] N. The method according to any one of paragraphs F to M, further comprising: determining a second audio event that occurred between a second start audio frame and a second end audio frame; determining a third audio event that occurred between a third start audio frame and a third end audio frame; determining the total number of audio frames between the second audio event and the third audio event; and removing the total number of audio frames between the second audio event and the third audio event when the total number of audio frames is less than a threshold.

[0115] The method according to any one of paragraphs F to N, further comprising determining at least one of an audio event approaching the vehicle or an audio event progressing away from the vehicle, at least in part on the Doppler frequency shift.

[0116] P. The method according to any one of paragraphs F to O, wherein the audio event includes at least one of the following emergency sound classes: ambulance siren class, police siren class, or fire truck siren class, or at least one of the following non-emergency sound classes: engine sound class, music sound class, thunder sound class, or speech sound class.

[0117] Q. A non-temporary computer-readable medium for storing instructions, wherein, when executed, the instructions cause one or more processors to perform an operation including: receiving audio data from a first audio sensor and a second audio sensor associated with a vehicle; determining features associated with the audio data; determining a direction of arrival (DoA) value associated with the audio data, at least in part on the features; determining the occurrence of an audio event, at least in part on the audio data; determining the direction of an audio event relating to the vehicle, at least in part on the DoA value; and controlling the vehicle, at least in part on the direction of the audio event relating to the vehicle.

[0118] R. Audio data comprises multiple audio frames, and the operation further comprises determining the angular spectra of a first audio sensor and a second audio sensor based at least partially on the audio frames among the multiple audio frames; determining one or more parameters associated with the audio frame based at least partially on the angular spectra; and determining the direction of arrival (DoA) value associated with the audio frame based at least partially on one or more parameters, as described in paragraph Q.

[0119] S. A non-temporary computer-readable medium as described in paragraph Q or R, in which one or more parameters include at least one of the following: angular spectral distribution, angular spectral peak value, time-to-arrival (TDoA) value, energy value of audio data, frequency spectrum, noise floor, noise spectrum, zero crossing, energy distribution, cross-correlation, or machine-learned embedding.

[0120] T. Audio data is discretized into multiple audio frames, and determining the occurrence of an audio event includes inputting at least a portion of the audio data into a classifier and receiving a classification of the audio event from the classifier, the classification including one or more of the following: ambulance siren class, police siren class, or fire engine siren class, and the operation further includes determining a start time frame indicating the onset of the audio event and determining an end time frame indicating the offset of the audio event, as described in any one of paragraphs Q to S.

[0121] While the exemplary clauses described above illustrate one specific implementation, it should be understood that, in the context of this document, the contents of the exemplary clauses may also be implemented through methods, devices, systems, computer-readable media, and / or other implementations. Additionally, any one of Examples A through T may be implemented alone or in combination with any one or more of the other Examples A through T.

[0122] conclusion While one or more examples of the techniques described herein have been described, various modifications, additions, substitutions and their equivalents are included within the scope of the techniques described herein. In the description of the examples, references are made to the accompanying drawings, which form part of this specification, illustrating specific examples of the claimed subject matter. It should be understood that other examples may be used, and that modifications or alterations, such as structural changes, may be made. Such examples, modifications or alterations do not necessarily deviate from the intended scope of the claimed subject matter. The steps herein may be presented in a fixed order, but in some cases the ordering may be altered so that certain inputs are provided at different times or in different orders without altering the function of the systems and methods described. The disclosed procedures may also be performed in different orders. Additionally, the various calculations herein do not need to be performed in the disclosed order, and other examples using alternative orderings of calculations can be readily implemented. In addition to being rearranged, calculations may be decomposed into subcalculations having the same results.

Claims

1. It is a system, One or more processors, One or more computer-readable media storing instructions that can be executed by the one or more processors, wherein when an instruction is executed, the system Receiving audio data from a pair of audio sensors associated with the vehicle, Determining angular spectral data based at least partially on a portion of the aforementioned audio data, Determining features associated with the audio data based at least partially on the angular spectral data, Inputting the aforementioned features into a machine learning model, The process involves receiving direction of arrival (DoA) values ​​associated with the audio data from the machine-learned model, wherein the DoA values ​​are associated with time difference (TDoA) values ​​at the peaks of the angular spectral data. Based at least partially on the aforementioned audio data, the occurrence of sound associated with the audio event is determined. Based at least partially on the DoA value, the direction of the audio event related to the vehicle is determined. One or more computer-readable media that perform an operation including the following The audio data is discretized into multiple audio frames, The aforementioned operation is, Determining the start time frame that indicates the onset of the aforementioned sound, Determining the end time frame that indicates the offset of the sound, Determining an event that starts at the onset of the sound and ends at the offset of the sound, To determine the total number of audio frames associated with the aforementioned event, The event is removed when the total number of audio frames associated with the event is less than a threshold. A system that further includes this.

2. The audio data includes a plurality of audio frames captured over time, the angular spectrum data includes a plurality of angular spectra, and the angular spectra among the plurality of angular spectra are associated with an audio frame among the plurality of audio frames. Determining the aforementioned features includes determining a plurality of features, wherein one of the plurality of features is associated with the angular spectrum among the plurality of angular spectra. Inputting the features into the machine-learned model includes inputting the multiple features into the machine-learned model. The system according to claim 1.

3. The aforementioned features are, Distribution of angular spectral data, The peak value of the aforementioned angular spectral data, The aforementioned time difference (TDoA) value, or Energy value of the aforementioned audio data Includes at least one of the following: The previous audio event was, Emergency sound class including at least one of the following: ambulance siren class, police siren class, or fire truck siren class, Non-emergency sound class that includes at least one of the following: engine sound class, music sound class, thunder sound class, or speech sound class. Includes at least one of the following The system according to claim 1 or 2.

4. Determining the generation of the sound is Inputting at least a portion of the aforementioned audio data into the classifier, Receiving the classification of the sound from the classifier, wherein the classification includes one or more of the following: ambulance siren class, police siren class, or fire truck siren class. The system according to any one of claims 1 to 3, including the system described in any one of claims 1 to 3.

5. Receiving audio data from a first audio sensor and a second audio sensor associated with the vehicle, Determining angular spectral data based at least partially on a portion of the aforementioned audio data, Determining features associated with the audio data based at least partially on the angular spectral data, Determining the direction of arrival (DoA) value associated with the audio data, at least in part, based on the aforementioned features, wherein the DoA value is associated with the time difference of arrival (TDoA) value at the peak of the angular spectral data, The determination of the occurrence of an audio event based at least partially on the aforementioned audio data, Based at least partially on the DoA value, the direction of the audio event related to the vehicle is determined, Controlling the vehicle based at least partially on the direction of the audio event related to the vehicle. A method comprising, wherein the audio data is discretized into a plurality of audio frames, and the method Determining the starting audio frame that indicates the onset of the aforementioned audio event, Determining the end audio frame that indicates the offset of the aforementioned audio event, To determine the first audio event that occurred between the first start audio frame and the first end audio frame, To determine the total number of audio frames associated with the first audio event, The first audio event is removed when the total number of audio frames associated with the first audio event is less than a threshold. A way to prepare further.

6. The first audio sensor and the second audio sensor associated with the vehicle are, Two audio sensors associated with the front area of ​​the aforementioned vehicle, Two audio sensors associated with the left side of the aforementioned vehicle, Two audio sensors associated with the rear area of ​​the vehicle, or Two audio sensors associated with the right side of the aforementioned vehicle Includes at least one of the following: The method according to claim 5, wherein the front area, the rear area, the left side, and the right side relate to the direction of travel of the vehicle.

7. (i) determining that the audio event includes an emergency event, and controlling the vehicle based at least in part on the audio event including the emergency event, wherein controlling the vehicle includes at least one of stopping the vehicle or changing the vehicle to another lane, (ii) Determining, at least in part, the audio event approaching the vehicle or the audio event progressing away from the vehicle, based on the Doppler frequency shift. The method according to claim 5 or 6, further comprising one of the following.

8. Further comprising determining the angular spectrum at least partially based on one of the following algorithms: Generalized Cross-Correlation Phase Transform (GCC-PHAT), Multiple Signal Classification (MUSIC) algorithm, or State Coherence Transform (SCT) algorithm, The method according to claim 5 or 6, wherein the features associated with the audio data are determined at least partially based on the angular spectrum.

9. The method described above is: The angular spectra of the first audio sensor and the second audio sensor are determined based at least partially on the audio frames among the plurality of audio frames, Determining one or more parameters associated with the audio frame based at least partially on the angular spectrum, Determining the direction of arrival (DoA) value associated with the audio frame based at least partially on one or more of the above parameters: The method according to claim 5 or 6, further comprising:

10. The one or more of the above parameters are, Distribution associated with the aforementioned angular spectrum, The peak value associated with the angular spectrum, The aforementioned time difference (TDoA) value, The energy value of the aforementioned audio data, frequency spectrum, Noise floor, Noise spectrum, Zero crossing, Energy distribution, Cross-correlation, or Machine learning-based embeddings The method according to claim 9, comprising at least one of the following.

11. Determining the occurrence of the audio event is: Inputting at least a portion of the aforementioned audio data into the classifier, Receiving a classification of the audio event from the classifier, wherein the classification includes one or more of the following: ambulance siren class, police siren class, or fire engine siren class. including, The method according to claim 5 or 6.

12. To determine the second audio event that occurred between the second start audio frame and the second end audio frame, To determine the third audio event that occurred between the third start audio frame and the third end audio frame, To determine the total number of audio frames between the second audio event and the third audio event, When the total number of audio frames is less than a threshold, the total number of audio frames between the second audio event and the third audio event is removed. The method according to claim 11, further comprising:

13. A computer program that, when executed on a computer, includes encoded instructions that implement the method claimed in any one of claims 5 to 12.

Images