Emergency vehicle detection method based on acoustic spectrum-time domain information fusion

Abstract

The invention belongs to the field of vehicle-mounted acoustic signal processing and recognition, and particularly relates to an emergency vehicle detection method based on acoustic spectrum-time domain information fusion, which comprises the following steps: acquiring an audio signal from an environment through vehicle-mounted microphone equipment, and respectively expressing the audio signal by a frequency domain and a time domain; introducing a coordinate attention network to perform fusion calculation on the features in the time domain representation and the frequency domain representation to obtain attention fused features; and inputting the attention fused features into a lightweight detection neural network model, and calculating whether siren sound exists in the current signal. According to the invention, through fusion of time domain and frequency domain information and combination of a coordinate attention mechanism, high identification precision can be maintained in high-speed driving or other scenes with large noise interference; especially under the condition that the influence of background noise such as engine noise, wind noise and traffic flow is large, the siren sound of the emergency vehicle can still be accurately detected, and the robustness of the system is enhanced.

Description

Technical Field

[0001] The invention belongs to the field of vehicle-mounted acoustic signal processing and recognition, and in particular relates to an emergency vehicle detection method based on acoustic spectrum-time domain information fusion. Background Art

[0002] In recent years, with the rapid development of the field of smart cars, AI-based vehicle environment perception technology has played an important role in improving driving safety and optimizing driving experience. With the help of on-board visual and auditory sensors, drivers can have a clearer understanding of road conditions, thereby effectively improving driving safety. Among them, the siren is a special road acoustic signal, which is usually emitted by emergency vehicles such as fire trucks, police cars and ambulances to remind and warn pedestrians and vehicles on the road.

[0003] Traditional siren detection is mainly based on spectrum analysis, which is achieved by analyzing the frequency components in the spectrum that change with a specific periodic pattern. Siren sounds usually exist in the form of sawtooth waves in the spectrum diagram and have a certain periodicity. Although sirens have stable frequency components, the frequency distribution of the collected sound signals may become unstable when driving at high speeds due to the influence of the Doppler effect, environmental noise, etc., and spectrum analysis has limited processing capabilities for such non-steady-state signals.

[0004] Siren detection based on deep learning usually takes the frequency domain features of the siren signal, such as the logarithmic Mel spectrogram or Mel cepstral coefficients, as input, and uses a neural network to classify the spatial correlation of its frequency domain features to determine whether the siren is present. However, in some cases, frequency domain features alone may not be sufficient to accurately distinguish sirens from other similar audio signals (such as whistles, etc.). If the ambient noise has a strong component in the frequency range of the siren, spectral analysis will have difficulty distinguishing the siren from the noise, especially in complex urban roads or highway environments.

[0005] Most models use a stationary microphone when recording samples, ignoring the effects of wind noise, tire noise and other environmental interference when the vehicle is traveling at high speed, as well as the Doppler effect between the vehicle and the siren source. Relying only on frequency domain features will ignore the dynamic changes of the sound signal in the time domain, which are crucial for identifying sirens and other similar sounds, especially in a noisy environment where vehicles are traveling at high speeds. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides an emergency vehicle detection method based on acoustic spectrum-time domain information fusion, comprising:

[0007] S1: collect audio signals from the environment through the vehicle-mounted microphone device and represent them in frequency domain and time domain respectively;

[0008] S2: Introduce the coordinate attention network to fuse the features in the time domain representation and the frequency domain representation to obtain the features after attention fusion;

[0009] S3: Input the attention-fused features into the lightweight detection neural network model to calculate whether there is a siren sound in the current signal.

[0010] Beneficial effects of the present invention:

[0011] The present invention combines time domain and frequency domain information and utilizes the multi-dimensional characteristics of acoustic signals. The spectrum characteristics can identify the frequency pattern of the siren sound, while the time domain characteristics further capture the dynamic changes of the siren sound, thereby improving the detection accuracy of emergency vehicles.

[0012] By fusing time domain and frequency domain information and combining it with a coordinate attention mechanism, the present invention can maintain a high recognition accuracy in high-speed driving or other scenarios with large noise interference. In particular, when the background noise such as engine noise, wind noise and traffic flow has a large impact, the sirens of emergency vehicles can still be accurately detected, thereby enhancing the robustness of the system.

[0013] The present invention adopts a lightweight deep learning model, which can automatically extract high-dimensional features compared to traditional methods, and can also optimize performance through end-to-end learning, greatly improving the system's response speed and processing efficiency, and meeting the high real-time requirements of the vehicle-mounted system.

[0014] The present invention can detect emergency vehicles in time at a long distance, and can significantly improve the response speed of the vehicle navigation system and the automatic driving system to emergency vehicles. In an emergency, the system can promptly identify the emergency vehicle and notify the driver or the automatic driving system to make an avoidance decision, effectively reducing traffic accidents and conflicts and ensuring road safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0016] Figure 2 It is a schematic diagram of frequency domain representation and time domain representation calculation in the present invention;

[0017] Figure 3 Schematic diagram of the attention fusion network in the present invention;

[0018] Figure 4 Schematic diagram of the classification and detection model in the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Figure 1 This is a schematic diagram of the overall structure of an emergency vehicle detection method based on acoustic spectrum-time domain information fusion. The figure records the process of collecting environmental sound signals from the on-board microphone, model processing and analysis of sound signals, detecting siren signals and the appearance of emergency vehicles, and finally alerting the driver under high-speed driving conditions, including:

[0021] S1: collect audio signals from the environment through the vehicle-mounted microphone device and represent them in frequency domain and time domain respectively;

[0022] S2: Introduce the coordinate attention network to fuse the features in the time domain representation and the frequency domain representation to obtain the features after attention fusion;

[0023] S3: Input the attention-fused features into the lightweight detection neural network model to calculate whether there is a siren sound in the current signal.

[0024] Preferably, the audio signal is represented in the frequency domain and the time domain respectively, including:

[0025] In the frequency domain branch, the signal is transformed into the frequency domain using short-time Fourier transform, and then a Mel filter bank and logarithmic operation are used to obtain the frequency domain representation of the signal - the logarithmic Mel spectrogram;

[0026] In the time domain branch, a one-dimensional convolution layer (Conv1D) with a large-size convolution kernel is used to expand the dimension of the signal in the time domain. Then, the convolution layer composed of LayerNorm, LeakyReLU and Conv1D encodes the expanded vector to calculate the time domain feature encoding of the signal.

[0027] Furthermore, Figure 2 It is a schematic diagram of the calculation of frequency domain representation and time domain representation in the present invention. The sound signal collected in the figure is respectively calculated through the frequency domain branch and the time domain branch to obtain the frequency domain features and time domain features of the sound signal, and finally obtains the preliminary fusion features; the Mel filter bank and the logarithmic operation are used to obtain the frequency domain representation of the signal - the logarithmic Mel spectrogram, including:

[0028] Perform short-time Fourier transform (STFT) on the sound waveform to obtain the time domain representation of the signal:

[0029] F M =STFT(x)

[0030] Among them, STFT(·) is the short-time Fourier transform operation, F M is the transformed spectrum diagram, and x represents the sound waveform;

[0031] By squaring the amplitude of the STFT result, we can get the power spectrum P of the signal. M , that is, the energy distribution of the signal at each frequency component:

[0032] P M =‖F M ‖ 2

[0033] The power spectrum is processed using a filter bank Mel(·) consisting of M Mel filters to calculate the Mel spectrogram f of the signal S :

[0034] f S =Mel(P M )

[0035] Mel-spectrogram f S Perform logarithmic operation to obtain the logarithmic Mel spectrum F of the signal S :

[0036] F S =log(f S )

[0037] Among them, the logarithmic Mel spectrum of the signal Where N is the number of time units.

[0038] Preferably, Figure 3 : is a schematic diagram of the attention fusion network in the present invention. In the figure, the input signal is pooled from the two dimensions of time and frequency to suppress the influence of noise, and then different feature information is fused through a shared convolution layer. Finally, the siren sound feature in the signal is enhanced through convolution attention, that is, the coordinate attention network is introduced to fuse the features in the time domain representation and the frequency domain representation to obtain the features after attention fusion, including:

[0039] The coordinate attention network includes: two parallel average pooling modules, a parameter-sharing convolutional layer, and two independent convolutional layers;

[0040] The obtained time domain and frequency domain representations are spliced ​​to obtain the fused feature spectrum F ST ;

[0041] Use two independent average pooling kernels (H, 1) and (1, W) to fused feature spectrum F ST Modeling position information along different directions to obtain the characteristics of different channels;

[0042] The features of different channels are concatenated and transformed and interacted using the convolutional layer to obtain the intermediate feature f containing spatial information in N and M directions;

[0043] Split f into two separate vectors f through two separate convolutional layers n and f m , and use CNN and Sigmoid activation functions to process them and make their shapes similar to F ST Keep it consistent and get the attention weight matrix g n and g n ;

[0044] The output attention weight matrix g n and g n Reapply to input F ST , get the features after attention fusion

[0045] Furthermore, two independent average pooling kernels (H, 1) and (1, W) are used to fused the feature spectrum F ST Position information is modeled along different directions to obtain features of different channels, including:

[0046]

[0047] in, and Respectively represent the feature information in the N and M directions, F ST (n,i) represents the calculation of different frequency information at the same time along the concatenated feature spectrum vector, F ST (j,m) indicates the calculation of different time information along the same frequency of the concatenated feature spectrum vector.

[0048] Furthermore, the features of different channels are concatenated and transformed and interacted using the convolutional layer to obtain the intermediate features f containing spatial information in the N and M directions, including:

[0049]

[0050] Where f represents the intermediate feature containing spatial information in N and M directions, F1(·) represents CNN calculation, δ(·) represents batch normalization operation, and Represent the feature information in the N and M directions respectively.

[0051] Furthermore, CNN and Sigmoid activation function are used to transform the vector f n and f m Process it and make it look like F STKeep it consistent and get the attention weight matrix g n and g m ,include:

[0052] g n =σ(F n (f n ))

[0053] g m =σ(F m (f m ))

[0054] Among them, g n , g m Represents the attention weight matrices in the N and M directions respectively, F n represents the convolution layer calculation in N directions, F m represents the convolution layer calculation in the M direction, σ(·) represents the Sigmoid activation function, and f n 、f m Represent the independent eigenvectors in the N and M directions respectively.

[0055] Furthermore, the output attention weight matrix g n and g n Reapply to input F ST , get the features after attention fusion include:

[0056]

[0057] Among them, F ST represents the characteristic spectrum after fusion, g n , g m Represent the attention weight matrices in the N and M directions respectively.

[0058] Preferably, Figure 4 This is a schematic diagram of the classification and detection model in the present invention. The classification network calculates the score of the siren signal in the current feature, and compares the output score with the threshold value set by the system. If the output is greater than the threshold, it indicates that the siren is present, and finally the driver is reminded. If the output value is less than the threshold, it means that the siren does not appear, that is, the feature after attention fusion is input into the lightweight detection neural network model to calculate whether there is a siren in the current signal, including:

[0059] The lightweight detection neural network model includes: three layers of 2D convolutional layers with consistent structures and one LogSoftMax layer;

[0060] The features after attention fusion Input three layers of 2D convolutional layer with the same structure, and we get;

[0061]

[0062] Among them, F m represents the output of a 2D convolutional layer, Conv2D(·) represents a 2D convolutional layer, ReLU(·) represents a linear rectification activation function, and Norm(·) represents a layer normalization operation;

[0063] F m Input the LogSoftMax layer and calculate the detection score through LogSoftMax:

[0064]

[0065] Among them, Score represents the detection score, log(·) represents the logarithm, and exp(·) represents the exponential operation;

[0066] Finally, the detection score is compared with the set threshold to determine whether a siren sound is present; the detection threshold is set to 0.5. If the detection score is greater than or equal to the threshold, it indicates that a siren sound is present in the current sound sample, otherwise it does not.

[0067] Experimental environment and equipment:

[0068] On-board sensors: Use an on-board microphone array to collect acoustic signals inside and outside the vehicle. The microphone array can effectively collect external noise sources (such as other vehicles, engine noise, road noise, etc.) and sirens from emergency vehicles when the vehicle is driving at high speed.

[0069] Data preprocessing:

[0070] Signal acquisition and preprocessing: First, the audio signal collected by the vehicle microphone array is preprocessed, including gain adjustment and signal smoothing to improve signal quality.

[0071] Model deployment:

[0072] Deploy the model to the vehicle system and read the microphone signal as the model input.

[0073] Real-time detection and reasoning:

[0074] Detection process: The processed audio signal is fed into a trained deep learning model for real-time inference. The model will output the detection result, that is, whether the siren of the emergency vehicle is detected.

[0075] Detection process: If a siren is detected, the system will transmit the information to the vehicle's navigation or autonomous driving system to alert the driver or the autonomous driving system to evade the vehicle.

[0076] System response: In an autonomous driving environment, the system can adjust the driving route or actively slow down according to the approach of the emergency vehicle to ensure safe avoidance.

[0077] System Response: In traditional vehicles, the system alerts the driver to their surroundings through voice prompts or visual warnings.

[0078] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An emergency vehicle detection method based on acoustic spectrum-time domain information fusion, characterized in that: include: S1: collect audio signals from the environment through the vehicle-mounted microphone device and represent them in frequency domain and time domain respectively; S2: Introduce the coordinate attention network to fuse the features in the time domain representation and the frequency domain representation to obtain the features after attention fusion; S3: Input the attention-fused features into the lightweight detection neural network model to calculate whether there is a siren sound in the current signal.

2. The emergency vehicle detection method based on acoustic spectrum-time domain information fusion according to claim 1 is characterized in that: The audio signal is represented in the frequency domain and the time domain, including: In the frequency domain branch, the signal is transformed into the frequency domain using short-time Fourier transform, and then a Mel filter bank and logarithmic operation are used to obtain the frequency domain representation of the signal - the logarithmic Mel spectrogram; In the time domain branch, a one-dimensional convolution layer (Conv1D) with a large-size convolution kernel is used to expand the dimension of the signal in the time domain. Then, the convolution layer composed of LayerNorm, LeakyReLU and Conv1D encodes the expanded vector to calculate the time domain feature encoding of the signal.

3. The emergency vehicle detection method based on acoustic spectrum-time domain information fusion according to claim 2 is characterized in that: Use a Mel filter bank and take the logarithm operation to obtain the frequency domain representation of the signal - the log Mel spectrogram, including: Perform short-time Fourier transform (STFT) on the sound waveform to obtain the time domain representation of the signal: F M =STFT(x) Among them, STFT(·) is the short-time Fourier transform operation, F M is the transformed spectrum diagram, and x represents the sound waveform; By squaring the amplitude of the STFT result, we can get the power spectrum P of the signal. M , that is, the energy distribution of the signal at each frequency component: P M =‖F M ‖ 2 The power spectrum is processed using a filter bank Mel(·) consisting of M Mel filters to calculate the Mel spectrogram f of the signal S : f S =Mel(P M ) Mel-spectrogram f S Perform logarithmic operation to obtain the logarithmic Mel spectrum F of the signal S : F S =log(f S ) Among them, the logarithmic Mel spectrum of the signal Where N is the number of time units.

4. The emergency vehicle detection method based on acoustic spectrum-time domain information fusion according to claim 1 is characterized in that: The coordinate attention network is introduced to fuse the features in the time domain representation and the frequency domain representation to obtain the features after attention fusion, including: The coordinate attention network includes: two parallel average pooling modules, a parameter-sharing convolutional layer, and two independent convolutional layers; The obtained time domain and frequency domain representations are spliced ​​to obtain the fused feature spectrum F ST ; Use two independent average pooling kernels (H, 1) and (1, W) to fused feature spectrum F ST Modeling position information along different directions to obtain the characteristics of different channels; The features of different channels are concatenated and transformed and interacted using the convolutional layer to obtain the intermediate feature f containing spatial information in N and M directions; Split f into two separate vectors f through two separate convolutional layers n and f m , and use CNN and Sigmoid activation functions to process them and make their shapes similar to F ST Keep it consistent and get the attention weight matrix g n and g n ; The output attention weight matrix g n and g n Reapply to input F ST , get the features after attention fusion 5. The emergency vehicle detection method based on acoustic spectrum-time domain information fusion according to claim 4 is characterized in that: Use two independent average pooling kernels (H, 1) and (1, W) to fused feature spectrum F ST Position information is modeled along different directions to obtain features of different channels, including: in, and Respectively represent the feature information in the N and M directions, F ST (n,i) represents the calculation of different frequency information at the same time along the concatenated feature spectrum vector, F ST (j,m) indicates the calculation of different time information along the same frequency of the concatenated feature spectrum vector.

6. The emergency vehicle detection method based on acoustic spectrum-time domain information fusion according to claim 4 is characterized in that: The features of different channels are concatenated and transformed and interacted using the convolutional layer to obtain the intermediate features f containing spatial information in N and M directions, including: Where f represents the intermediate feature containing spatial information in N and M directions, F1(·) represents CNN calculation, δ(·) represents batch normalization operation, and Represent the feature information in the N and M directions respectively.

7. The method for emergency vehicle detection based on acoustic spectrum-time domain information fusion according to claim 4, characterized in that: Use CNN and Sigmoid activation function to transform the vector f n and f m Process it and make it look like F ST Keep it consistent and get the attention weight matrix g n and g m ,include: g n =σ(F n (f n )) g m =σ(F m (f m )) Among them, g n , g m Represents the attention weight matrices in the N and M directions respectively, F n represents the convolution layer calculation in N directions, F m represents the convolution layer calculation in the M direction, σ(·) represents the Sigmoid activation function, and f n 、f m Represent the independent eigenvectors in the N and M directions respectively.

8. The method for emergency vehicle detection based on acoustic spectrum-time domain information fusion according to claim 4, characterized in that: The output attention weight matrix g n and g n Reapply to input F ST , get the features after attention fusion include: Among them, F ST represents the characteristic spectrum after fusion, g n , g m Represent the attention weight matrices in the N and M directions respectively.

9. The emergency vehicle detection method based on acoustic spectrum-time domain information fusion according to claim 1 is characterized in that: The attention fused features are input into the lightweight detection neural network model to calculate whether there is a siren sound in the current signal, including: The lightweight detection neural network model includes: three layers of 2D convolutional layers with consistent structures and one LogSoftMax layer; The features after attention fusion Input three layers of 2D convolutional layer with the same structure, and we get; Among them, F m represents the output of a 2D convolutional layer, Conv2D(·) represents a 2D convolutional layer, ReLU(·) represents a linear rectification activation function, and Norm(·) represents a layer normalization operation; F m Input the LogSoftMax layer and calculate the detection score through LogSoftMax: Score=F m -log(∑exp(F m )) Among them, Score represents the detection score, log(·) represents the logarithm, and exp(·) represents the exponential operation; Finally, the detection score is compared with the set threshold to determine whether a siren sound is present; the detection threshold is set to 0.

5. If the detection score is greater than or equal to the threshold, it indicates that a siren sound is present in the current sound sample, otherwise it does not.

Images