Sound recognition system and sound recognition method for transportation means

Abstract

This invention proposes a sound recognition system and a sound recognition method. The method includes: receiving multiple sound signals in a driving environment via multiple microphones installed in a vehicle; identifying whether these sound signals contain sound signals belonging to a preset vehicle category based on a machine learning model; and if the sound signals contain sound signals belonging to the preset vehicle category, calculating spatial information of the sound source based on the sound signals of the preset vehicle category, and displaying the spatial information on a display device on the vehicle.

Description

Technical Field

[0001] This disclosure relates to a sound recognition system and method installed on a vehicle to assist in identifying the presence of sound sources such as ambulances, police cars, and fire trucks in the driving environment. Prior Technology

[0002] While driving, drivers are frequently exposed to environmental noise, such as engine noise from other vehicles, road noise, and music or conversations inside the car. These factors can affect a driver's ability to recognize important external sounds, especially the sirens of emergency vehicles like ambulances or police cars. Furthermore, while soundproofing in the vehicle effectively reduces external noise interference for passengers, it further decreases the driver's chances of hearing emergency vehicle sirens. If a driver fails to recognize an emergency vehicle in time, they may be unable to avoid it, obstructing emergency vehicles and delaying emergency medical services / rescue efforts. Current technology primarily relies on drivers to judge the location and approach speed of emergency vehicles by hearing, but as mentioned above, relying on human hearing presents various problems. Summary of the Invention

[0003] This invention proposes a sound recognition system and method that can identify vehicles with emergency missions, such as ambulances, police cars, or fire trucks, in a driving environment and present them as images to remind the driver.

[0004] This invention proposes a sound recognition system, comprising a vehicle, a microphone, a display device, and a computing module. The microphone is mounted on the vehicle to receive multiple sound signals in a driving environment. The display device is also mounted on the vehicle. The computing module is communicatively connected to the microphone and the display device to acquire the sound signals and, based on a machine learning model, identify whether the sound signals contain sound signals belonging to a preset vehicle category. If the sound signals contain sound signals belonging to the preset vehicle category, the computing module calculates the spatial information of the sound source based on the sound signals of the preset vehicle category and displays the spatial information on the display device.

[0005] In one embodiment of the present invention, the computing module is further configured to perform time-domain to frequency-domain conversion on the sound signal to obtain multiple frequency-domain signals, and extract multiple feature vectors from the frequency-domain signals. The aforementioned machine learning model is a neural network, and the computing module is configured to input the feature vectors into the neural network to determine whether the sound signal contains a sound signal belonging to a preset vehicle category.

[0006] In one embodiment of the present invention, the computing module is used to perform triangulation to calculate the direction of the sound source and also to calculate the distance of the sound source based on the Doppler effect, wherein the spatial information of the sound source includes direction and distance.

[0007] In one embodiment of the present invention, the above-mentioned computing module is also used to display multiple lanes on a display device and mark one of the lanes according to the spatial information of the sound source.

[0008] In one embodiment of the present invention, the aforementioned preset vehicle category includes ambulance, police car, or fire truck categories, and the sound source is a vehicle in the driving environment.

[0009] From another perspective, embodiments of the present invention propose a sound recognition method, executed by a computing module installed on a vehicle. This sound recognition method includes: receiving multiple sound signals in a driving environment via multiple microphones installed on the vehicle; identifying, according to a machine learning model, whether the sound signals contain sound signals belonging to a preset vehicle category; and if the sound signals contain sound signals belonging to the preset vehicle category, calculating spatial information of the sound source based on the sound signals of the preset vehicle category, and displaying the spatial information on a display device on the vehicle.

[0010] In one embodiment of the present invention, the aforementioned machine learning model is a neural network. The sound recognition method further includes: performing a time-domain to frequency-domain conversion on the sound signal to obtain multiple frequency-domain signals, and extracting multiple feature vectors from the frequency-domain signals; inputting the feature vectors into the neural network to determine whether the sound signal contains a sound signal belonging to a preset vehicle category.

[0011] In one embodiment of the present invention, the above-described sound recognition method further includes: performing triangulation to calculate the direction of the sound source; and calculating the distance of the sound source based on the Doppler effect, wherein the spatial information of the sound source includes direction and distance.

[0012] In one embodiment of the present invention, the above-described sound recognition method further includes: displaying multiple lanes on a display device and marking one of the lanes according to the spatial information of the sound source.

[0013] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings for detailed explanation. Simple Explanation of the Diagram

[0014] Figure 1 is a schematic diagram illustrating a voice recognition system according to one embodiment. Figure 2 is a flowchart illustrating a sound recognition method according to one embodiment. Figure 3 is a schematic diagram illustrating sound source identification according to one embodiment. Figure 4 is a schematic diagram illustrating the screen displayed by the display device 120 according to one embodiment. Figure 5 is a schematic diagram illustrating sound source identification according to another embodiment. Figure 6 is a schematic diagram illustrating the screen displayed by the display device 120 according to another embodiment. Implementation

[0015] Some embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Component symbols used in the following description are considered identical or similar when they appear in different drawings. These embodiments are only a part of the present invention and do not disclose all possible implementations of the invention. More precisely, these embodiments are merely examples of systems and methods within the scope of the present invention's patent application.

[0016] The terms "first," "second," etc., used in this article do not specifically refer to order or sequence; they are merely used to distinguish elements or operations described using the same technical terms.

[0017] Figure 1 is a schematic diagram illustrating a voice recognition system according to an embodiment. Referring to Figure 1, the voice recognition system 100 includes a vehicle 110, a display device 120, a computing module 130, and microphones 141-143, wherein the display device 120, the computing module 130, and the microphones 141-143 are all mounted on the vehicle 110. Furthermore, the computing module 130 is communicatively connected to the display device 120 and the microphones 141-143, and this communication connection can be achieved through any suitable wired or wireless communication means.

[0018] In this embodiment, the vehicle 110 is a vehicle traveling on a road, but in other embodiments it can be any suitable vehicle such as a motorcycle, bicycle, bus, or truck. The display device 120 can include a liquid crystal display panel, an organic light-emitting diode panel, an electronic paper panel, etc., and this invention is not limited thereto. In this embodiment, the display device 120 is a dashboard or central control panel inside the vehicle, but in other embodiments, the display device 120 can also be a rearview mirror, a separate display, or an augmented reality device or mixed reality device mounted on the windshield or helmet. The computing module 130 can be a central processing unit, a processor installed in the vehicle, or any circuit (or device) with computing capabilities. Microphones 141-143 are used to receive sound signals in the driving environment. In this embodiment, microphones 141-143 form a microphone array, and these microphones 141-143 can be positioned in front, behind, or on both sides of the vehicle 110. This invention does not limit the number or placement of microphones 141-143.

[0019] In this embodiment, the sound source 150 is identified based on the sound signal to determine whether it belongs to a preset vehicle category. For example, the sound source 150 is a vehicle in a driving environment, and the preset vehicle categories include vehicles with emergency missions such as ambulances, police cars, and fire trucks. When the sound source 150 is identified as an ambulance, police car, or fire truck, the computing module 130 calculates the spatial information of the sound source 150 and displays this spatial data on the display device 120.

[0020] Figure 2 is a flowchart illustrating a sound recognition method according to an embodiment. Referring to Figure 2, steps 201-207 are executed by the computing module 130, and will not be described again below. In step 201, multiple sound signals are acquired through microphones 141-143.

[0021] In step 210, a machine learning model is used to identify whether the aforementioned sound signals contain sound signals belonging to a preset vehicle category. In some embodiments, each sound signal can be segmented into multiple frames, and these frames can be directly input into the machine learning model. The machine learning model can output one or more numerical values ​​to represent whether sound signals belonging to the preset vehicle category are contained. For example, ambulance, police car, and fire truck categories can each correspond to three numerical values, or a single numerical value can be used to represent these vehicles with emergency missions. In some embodiments, the machine learning model can also output an additional vector sequence representing the sound signals of the preset vehicle category separated from the input sound signals.

[0022] In the embodiment shown in Figure 2, step 210 includes steps 202-204. In step 202, a time-domain to frequency-domain transformation is performed on the sound signal to obtain multiple frequency-domain signals. This time-domain to frequency-domain transformation is, for example, a Fast Fourier Transform (FFT). In step 203, multiple feature vectors are extracted from the frequency-domain signals. For example, Mel-Frequency Cepstral Coefficients (MFCCs) can be calculated based on the frequency-domain signals, but this invention is not limited to this. In step 204, the feature vectors are input into a machine learning model to identify whether the original sound signal contains a sound signal belonging to a preset vehicle category.

[0023] In other words, step 210 may include time-domain to frequency-domain conversion and feature extraction, but in some embodiments, the sound signal may be directly input into the machine learning model without time-domain to frequency-domain conversion and feature extraction. The aforementioned machine learning model may be a neural network, such as a convolutional neural network, a recurrent neural network (RNN), a transformer, etc., and this invention is not limited thereto.

[0024] If the existing sound signal contains a sound signal belonging to a preset vehicle category, continue to step 220 to calculate the spatial information of the sound source based on the sound signal of the preset vehicle category. As mentioned above, the sound source is a vehicle in the driving environment, such as an ambulance, police car, or fire truck. The aforementioned spatial information may include position, direction relative to the sound source, relative speed, distance, direction of movement of the sound source, etc.

[0025] In the embodiment shown in Figure 2, step 220 includes steps 205 and 206. In step 205, triangulation is performed to calculate the direction of the sound source. Generally speaking, the time difference between the arrival times of the sound signals at the microphone arrays can be calculated based on the sound signals received by the arrays. The distance difference can then be calculated from the time difference, and triangulation can be applied to determine the direction of the sound source. Those skilled in the art will understand the triangulation method, and it will not be described in detail here.

[0026] In step 206, the distance and relative speed of the sound source are calculated based on the Doppler effect. In some embodiments, after time-domain to frequency-domain conversion, sound signals belonging to a preset vehicle category can be obtained from a specific frequency range. The direction of movement and relative speed of the sound source can be calculated by analyzing the frequency changes of these sound signals based on the Doppler effect. As time progresses, the angle change of the sound source relative to the microphone array can be recalculated using triangulation, and the distance the sound source moves during this time can be calculated by multiplying the relative speed of the sound source by the time. Finally, by combining the angle change and the distance moved by the sound source, the actual distance of the sound source relative to the microphone array can be estimated. In some embodiments, if the intensity of the sound signal emitted by the sound source is known, the distance of the sound source relative to microphones 141-143 can also be estimated based on the intensity of the sound signal received by microphones 141-143. For example, according to the inverse-square law, the intensity of sound decreases inversely with the square of the distance. Therefore, the distance of the sound source relative to the microphone 141~143 can be deduced from the ratio between the intensity of the emitted sound signal and the intensity of the received sound signal.

[0027] The direction, position, and relative velocity calculated in steps 205 and 206 can be referred to as the spatial information of the sound source. In step 207, the spatial information is displayed on the display device 120.

[0028] Figure 3 is a schematic diagram illustrating sound source identification according to one embodiment. Referring to Figure 3, in this scenario, sound source 150 is an ambulance approaching from directly behind vehicle 110, and microphones 141-143 are arranged at the rear of vehicle 110 to form a microphone array. The direction and position of sound source 150 can be identified using the method shown in Figure 2, and this spatial information can then be displayed on display device 120. For example, Figure 4 is a schematic diagram illustrating the screen displayed on display device 120 according to one embodiment. Referring to Figure 4, computing module 130 displays multiple lanes 401-403 on display device 120 and marks the middle lane 402 based on the spatial data of the sound source. For example, a graphic object 410 can be provided to represent vehicle 110 itself, a graphic object 420 can be provided in lane 402 to represent the ambulance, and finally a graphic object 430 can be provided to represent the direction of travel of the ambulance. This can remind the driver that the ambulance is approaching from directly behind and that it is necessary to change lanes to allow the ambulance to pass.

[0029] In the embodiment shown in Figure 3, the microphone is located at the rear of the vehicle, but in other embodiments it can be located at the front or other positions. For example, in the embodiment shown in Figure 5, microphones 141-143 are located at the front of the vehicle to form a microphone array, used to identify a sound source 150 from a cross lane at an intersection, such as an ambulance. The screen displayed on the display device 120 is shown in Figure 6, which similarly depicts multiple lanes 601-603 and graphic objects 610, 620, and 630. Lanes 601 and 602 represent cross lanes. Graphic object 610 represents the vehicle 110 itself, graphic object 620 is located in lane 601 representing the ambulance, and graphic object 630 represents the direction of the ambulance. This can remind drivers to stop at intersections (even on a green light) and wait for the ambulance to pass.

[0030] The graphic objects described above can be represented by any suitable pattern, and the present invention does not limit the color, texture, size, position, etc. of these patterns. In other embodiments, any text, numbers, colors, or patterns can also be used to represent the spatial information of the sound source, and the present invention is not limited to the above embodiments.

[0031] In the aforementioned system and method, a microphone array combined with a machine learning model can provide real-time alerts to the driver of an ambulance (or other vehicles with emergency missions). Even if the driver cannot hear external sounds, they can still determine the direction and location of the ambulance through visual information. In one application scenario, where the driver is hearing impaired, the method can assist them in obtaining spatial information such as the direction and location of the ambulance.

[0032] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0033] 100: Voice Recognition System 110: Transportation 120: Display device 130: Computing Module 141~143: Microphone 150: Sound source 201~207,210,220: Steps Lanes 401~403, 601~603: 410, 420, 430, 610, 620, 630: Graphic objects

Claims

1. A sound recognition system, comprising: a vehicle; a plurality of microphones disposed on the vehicle for receiving a plurality of sound signals in a driving environment; a display device disposed on the vehicle; a computing module communicatively connected to the microphones and the display device, for acquiring the sound signals, performing a time-domain to frequency-domain conversion on the sound signals to obtain a plurality of frequency-domain signals, extracting a plurality of feature vectors from the frequency-domain signals, and inputting the feature vectors into a neural network to determine whether the sound signals contain sound signals belonging to a preset vehicle category; if the sound signals contain sound signals belonging to the preset vehicle category, the computing module calculates spatial information of a sound source based on the sound signals of the preset vehicle category and displays the spatial information on the display device, wherein the computing module performs a triangulation method to calculate the direction of the sound source, and calculates the distance of the sound source based on a Doppler effect, wherein the spatial information of the sound source includes the direction and the distance. The computing module is also used to display multiple lanes on the display device, mark one of the lanes according to the spatial information of the sound source, display a first graphic object to represent the vehicle, display a second graphic object to represent the sound source, and display a third graphic object to represent the direction of the sound source.

2. The sound recognition system as described in claim 1, wherein the preset vehicle category includes an ambulance category, a police car category, or a fire truck category, and the sound source is a vehicle in the driving environment.

3. A sound recognition method, executed by a computing module installed on a vehicle, the sound recognition method comprising: receiving multiple sound signals in a driving environment through multiple microphones installed on the vehicle; performing a time-domain to frequency-domain conversion on the sound signals to obtain multiple frequency-domain signals, extracting multiple feature vectors from the frequency-domain signals, and inputting the feature vectors into a neural network to determine whether the sound signals contain sound signals belonging to a preset vehicle category; If the sound signals include sound signals belonging to the preset vehicle category, spatial information of a sound source is calculated based on the sound signals of the preset vehicle category, including performing a triangulation method to calculate the direction of the sound source, calculating the distance of the sound source based on a Doppler effect, wherein the spatial information of the sound source includes the direction and the distance; and displaying the spatial information on a display device on the vehicle, including displaying multiple lanes on the display device, marking one of the lanes according to the spatial information of the sound source, displaying a first graphic object to represent the vehicle on the display device, displaying a second graphic object to represent the sound source, and displaying a third graphic object to represent the direction of the sound source.

4. The sound recognition method as described in claim 3, wherein the preset vehicle category includes an ambulance category, a police car category, or a fire truck category, and the sound source is a vehicle in the driving environment.

Images