Emergency vehicle and road surface state combined acoustic identification method and related device

Abstract

The application provides an emergency vehicle and road surface state combined acoustic identification method and related device, and relates to the technical field of acoustic signal processing and deep learning. The method obtains a multi-channel environmental audio signal, and obtains a multi-channel log-mel spectrum feature map after preprocessing; the feature map is input into a causal shared backbone network for feature extraction, and a shared feature is obtained; the shared feature is input into parallel vehicle classification heads and road surface classification heads respectively, and an emergency vehicle type identification result and a road surface state identification result are obtained. The application extracts a shared feature by constructing a causal shared backbone network, and inputs the shared feature into parallel vehicle classification heads and road surface classification heads respectively, independently outputs an emergency vehicle type identification result and a road surface state identification result from two classification heads, thereby simultaneously completes two tasks of emergency vehicle classification and road surface state identification from the same audio stream, and meets the causal real-time constraint and the lightweight vehicle deployment requirement.

Description

Technical Field

[0001] This invention relates to the fields of acoustic signal processing and deep learning technology, specifically to a method and related device for joint acoustic recognition of emergency vehicles and road conditions. Background Technology

[0002] With the iterative upgrades of intelligent connected vehicle technology and autonomous driving systems, vehicle environmental perception capabilities, as the core foundation of autonomous driving decision-making and control, require continuously increasing precision, real-time performance, and scenario adaptability. Current vehicle perception systems primarily rely on vision and millimeter-wave radar. However, visual sensors are susceptible to interference from changes in lighting, rain, snow, fog, and other adverse weather conditions, and have blind spots. Millimeter-wave radar is costly and lacks the ability to identify non-obstacle targets such as road surfaces. Acoustic sensor-based environmental perception methods can effectively address these shortcomings. In autonomous driving safety management scenarios, acoustic recognition of emergency vehicles and road conditions are two core perception tasks. Accurately identifying the siren signals of emergency vehicles such as ambulances, fire trucks, and police cars can assist the autonomous driving system in making timely avoidance decisions; accurately identifying whether the road surface is dry or slippery can support the dynamic adaptation of braking distance prediction and vehicle stability control strategies.

[0003] In related technologies, vehicle-mounted acoustic recognition typically employs a single-task, single-branch independent recognition architecture, using separate single-task models to handle emergency vehicle identification and road condition detection. For emergency vehicle acoustic recognition, some studies extract fundamental frequency feature sequences from ambient sound data, perform correlation calculations with various emergency vehicle sound templates, and identify target vehicles based on correlation coefficients and threshold comparisons. For road condition acoustic recognition, related schemes extract Mel-Frequency Cepstral Coefficients (MFCC) and Log-Mel spectral features from vehicle road noise frequencies, fusing Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTMs) networks to construct a hybrid neural network to identify road and weather conditions in multiple scenarios. However, these schemes are insufficient to meet the comprehensive requirements of multi-task joint perception, real-time causal reasoning, and lightweight vehicle deployment in autonomous driving scenarios. Summary of the Invention

[0004] This invention provides a method and related device for joint acoustic recognition of emergency vehicles and road conditions, which is used to improve the problem that related technologies are difficult to meet the comprehensive requirements of multi-task joint perception, real-time causal reasoning and lightweight vehicle deployment in autonomous driving scenarios.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides a method for joint acoustic recognition of emergency vehicles and road conditions, comprising:

[0007] Acquire multi-channel environmental audio signals;

[0008] The multi-channel environmental audio signal is preprocessed to obtain a multi-channel log-Mel spectrum feature map. The preprocessing includes pre-emphasis processing, causal short-time Fourier transform, and Mel filtering.

[0009] Multi-channel log-Melogram spectral feature maps are input into a causal shared backbone network for feature extraction to obtain shared features. The causal shared backbone network consists of multiple causal multi-scale multi-channel channel attention modules (CMMCA) cascaded in a funnel-like multi-stage manner, with the number of channels in the feature map gradually increasing and the frequency dimension gradually decreasing along the cascading direction. All convolutional layers in the causal shared backbone network employ a zero-padding strategy on the left side of the time axis to satisfy causal constraints. The shared features are common acoustic representations that are simultaneously related to the sound source features of emergency vehicles and the acoustic features of road conditions.

[0010] The shared features are input into the parallel vehicle classification head and road classification head respectively to obtain the emergency vehicle type identification result and the road condition identification result.

[0011] In one possible embodiment, the multi-channel log-Mel spectrum feature map is input into a causal shared backbone network for feature extraction, including: inputting the multi-channel log-Mel spectrum feature map into the entry convolutional layer of the causal shared backbone network for preliminary feature extraction, resulting in an initial feature map with doubled channel number and reduced frequency and time dimensions; inputting the initial feature map sequentially into at least three cascaded stages for hierarchical feature extraction, resulting in shared features; in the at least three cascaded stages, the first stage includes CMMCA and does not enable multi-scale channel attention, and the number of channels and resolution of the output feature map remain unchanged, used for extracting low-level acoustic features; the middle stages include CMMCA and enable multi-scale channel attention, and the number of channels of the output feature map is increased, while the frequency and time dimensions are reduced, used for extracting mid-level semantic features; the last stage includes CMMCA and enables multi-scale channel attention, and the number of channels of the output feature map is increased, the frequency dimension is reduced, while the time dimension remains unchanged, used for extracting high-level semantic features.

[0012] In one possible embodiment, CMMCA employs an inverse residual bottleneck structure and processes the input features as follows: Channel expansion of the input features is performed using an up-dimensional convolutional layer to obtain up-dimensional features; convolution operations are performed on the up-dimensional features using a causal-depth-separable convolutional layer to obtain convolutionally processed features; wherein the convolution operation includes: symmetrical padding only in the frequency dimension and no padding in the time dimension; the number of channels of the convolutionally processed features is compressed to the target output channel number using a dimensionality-reduced projection layer to obtain dimensionality-reduced features; adaptive recalibration of the channel dimensions of the dimensionality-reduced features is performed using a multi-scale channel attention layer to obtain attention-recalibrated features; and the attention-recalibrated features are element-wise added to the original input of CMMCA using a residual connection layer to obtain the output features of CMMCA.

[0013] In one possible embodiment, the multi-scale channel attention layer includes a global frequency domain pooling branch and a local feature extraction branch. The multi-scale channel attention layer is used to adaptively recalibrate the channel dimensions of the dimensionality-reduced features, including: inputting the dimensionality-reduced features into the global frequency domain pooling branch and the local feature extraction branch respectively; performing global frequency domain pooling on the input feature map along the frequency dimension through the global frequency domain pooling branch, compressing the frequency information into a single value, and then performing a bottleneck transformation to generate a global channel description; directly performing a bottleneck transformation on the input feature map through the local feature extraction branch to generate a local channel description; adding the global channel description and the local channel description element-wise and then normalizing to obtain a channel attention weight map; multiplying the channel attention weight map element-wise with the dimensionality-reduced features to obtain the attention-recalibrated features.

[0014] In one possible embodiment, before inputting the shared features into the parallel vehicle classification head and road classification head respectively, the method further includes: performing adaptive average pooling on the shared features along the frequency dimension to obtain pooled features; performing grouped cross-frequency convolution on the pooled features to fuse each frequency point across frequencies to obtain cross-frequency fused features; performing a squeeze operation on the cross-frequency fused features to eliminate the frequency dimension, obtaining a one-dimensional temporal feature sequence; and using the one-dimensional temporal feature sequence as input to the vehicle classification head and road classification head.

[0015] In one possible embodiment, shared features are input into a parallel vehicle classification head and a road surface classification head to obtain emergency vehicle type identification results and road surface condition identification results. This includes: inputting a one-dimensional temporal feature sequence into the vehicle classification head, and outputting frame-by-frame predicted vehicle type bottom-level eight-class classification results through the vehicle classification head. The vehicle type bottom-level eight-class classification includes four subclasses: ambulance, fire truck, police car, engineering rescue vehicle, and background noise; aggregating the frame-by-frame predicted results of the four police car subclasses into the confidence score of the police car category, and outputting vehicle type identification results for ambulance, fire truck, police car, engineering rescue vehicle, and background noise; and inputting a one-dimensional temporal feature sequence into the road surface classification head, and outputting frame-by-frame predicted road surface condition binary classification results through the road surface classification head. The road surface condition binary classification results include dry and slippery conditions.

[0016] In one possible embodiment, both the vehicle classification head and the road classification head are causal dilated convolutional classification heads.

[0017] In one possible embodiment, preprocessing of the multi-channel environmental audio signal includes: pre-emphasizing the multi-channel environmental audio signal to obtain a pre-emphasized audio signal; performing a causal short-time Fourier transform on the pre-emphasized audio signal to obtain a frequency domain signal; wherein the causal short-time Fourier transform disables center alignment; performing dynamic frequency range Mel filtering on the frequency domain signal to obtain a Mel spectrum; and performing logarithmic compression and linear normalization on the Mel spectrum to obtain a multi-channel logarithmic Mel spectrum feature map.

[0018] In one possible embodiment, the causal shared backbone network, vehicle classification head, and road classification head are obtained through dual-task joint training. The dual-task joint training includes: acquiring a training sample set, where each sample contains multi-channel audio signals, vehicle type labels, and road state labels; inputting the training sample set into the causal shared backbone network, vehicle classification head, and road classification head for forward propagation; using a combination of vehicle recognition loss and road detection loss as the total loss function, and optimizing all parameters of the causal shared backbone network, vehicle classification head, and road classification head simultaneously through backpropagation based on the total loss function.

[0019] In one possible embodiment, the combination of vehicle identification loss and road surface detection loss is a weighted combination; the total loss function is the sum of vehicle identification loss and road surface detection loss weighted by road surface loss weighting coefficient.

[0020] In one possible implementation, both vehicle recognition loss and road surface detection loss are binary cross-entropy losses.

[0021] In one possible embodiment, before inputting the training sample set into the causal shared backbone network, the vehicle classification head, and the road classification head for dual-task joint training, the method further includes: performing hybrid data augmentation processing on the training sample set, wherein the hybrid data augmentation includes at least one of spectral masking enhancement, signal-to-noise ratio perturbation superposition, time rolling enhancement, device impulse response convolution enhancement, frequency domain statistics hybrid enhancement, and frequency random perturbation enhancement.

[0022] Secondly, the present invention provides an acoustic recognition device for emergency vehicles and road conditions, comprising:

[0023] The acquisition module is used to acquire multi-channel environmental audio signals;

[0024] The preprocessing module is used to preprocess multi-channel environmental audio signals to obtain multi-channel logarithmic Mel spectrum feature maps. The preprocessing includes pre-emphasis processing, causal short-time Fourier transform, and Mel filtering.

[0025] The feature extraction module is used to input multi-channel log-Mel spectrum feature maps into the causal shared backbone network for feature extraction to obtain shared features. The causal shared backbone network is composed of multiple CMMCAs cascaded in a funnel-shaped multi-stage manner, with the number of channels of the feature map gradually increasing and the frequency dimension gradually compressing along the cascading direction. All convolutional layers in the causal shared backbone network adopt a zero-padding strategy on the left side of the time axis to meet causal constraints. The shared features are common acoustic representations that are simultaneously related to the sound source features of emergency vehicles and the acoustic features of road conditions.

[0026] The identification module is used to input shared features into the parallel vehicle classification head and road classification head to obtain emergency vehicle type identification results and road condition identification results.

[0027] In one possible embodiment, the feature extraction module can be specifically used to: input a multi-channel log-Mel spectrum feature map into the entry convolutional layer of a causal shared backbone network for preliminary feature extraction, resulting in an initial feature map with doubled channel count and reduced frequency and time dimensions; input the initial feature map sequentially into at least three cascaded stages for hierarchical feature extraction, resulting in shared features; in the at least three cascaded stages, the first stage includes CMMCA and does not enable multi-scale channel attention, and the number of channels and resolution of the output feature map remain unchanged, used for extracting low-level acoustic features; the middle stages include CMMCA and enable multi-scale channel attention, and the number of channels of the output feature map is increased, while the frequency and time dimensions are reduced, used for extracting mid-level semantic features; the last stage includes CMMCA and enables multi-scale channel attention, and the number of channels of the output feature map is increased, the frequency dimension is reduced, while the time dimension remains unchanged, used for extracting high-level semantic features.

[0028] In one possible embodiment, CMMCA employs an inverse residual bottleneck structure. The feature extraction module can also be used to: expand the channels of the input features using an up-dimensional convolutional layer to obtain up-dimensional features; perform convolution operations on the up-dimensional features using a causal-depth-separable convolutional layer to obtain convolution-processed features; wherein the convolution operation includes: symmetrical padding only in the frequency dimension and no padding in the time dimension; compress the number of channels of the convolution-processed features to the target output channel number using a dimensionality-reduction projection layer to obtain dimensionality-reduced features; perform adaptive recalibration of the channel dimensions of the dimensionality-reduced features using a multi-scale channel attention layer to obtain attention-recalibrated features; and add the attention-recalibrated features to the original input of CMMCA element-wise using a residual connection layer to obtain the output features of CMMCA.

[0029] In one possible embodiment, the multi-scale channel attention layer includes a global frequency domain pooling branch and a local feature extraction branch. The feature extraction module can also be used to: input the dimensionality-reduced features into the global frequency domain pooling branch and the local feature extraction branch respectively; perform global frequency domain pooling on the input feature map along the frequency dimension through the global frequency domain pooling branch, compress the frequency information into a single value, and then perform a bottleneck transformation to generate a global channel description; directly perform a bottleneck transformation on the input feature map through the local feature extraction branch to generate a local channel description; add the global channel description and the local channel description element by element and then normalize to obtain a channel attention weight map; multiply the channel attention weight map element by element with the dimensionality-reduced features to obtain the attention-recalibrated features.

[0030] In one possible embodiment, before inputting the shared features into the parallel vehicle classification head and road classification head respectively, the feature extraction module can also be used to: perform adaptive average pooling on the shared features along the frequency dimension to obtain pooled features; perform grouped cross-frequency convolution on the pooled features to fuse each frequency point across frequencies to obtain cross-frequency fused features; perform a squeeze operation on the cross-frequency fused features to eliminate the frequency dimension, obtaining a one-dimensional temporal feature sequence; and use the one-dimensional temporal feature sequence as input to the vehicle classification head and road classification head.

[0031] In one possible embodiment, the identification module can be specifically used to: input a one-dimensional temporal feature sequence into a vehicle classification head, and output a frame-by-frame predicted eight-class classification result of vehicle type, the eight-class classification including four subclasses: ambulance, fire truck, police car, engineering rescue vehicle, and background noise; aggregate the frame-by-frame prediction results of the four police car subclasses into the confidence score of the police car category, and output the vehicle type identification results of ambulance, fire truck, police car, engineering rescue vehicle, and background noise; and input a one-dimensional temporal feature sequence into a road surface classification head, and output a frame-by-frame predicted two-class classification result of road surface condition, the two-class classification result of road surface condition including dry and wet / slippery.

[0032] Thirdly, the present invention provides an acoustic recognition system for emergency vehicles and road conditions, comprising:

[0033] The vehicle microphone array is deployed in the front and rear bumpers and left and right rearview mirrors of the vehicle to collect multi-channel environmental audio signals.

[0034] The on-board computing unit is used to execute the joint acoustic recognition method for emergency vehicles and road conditions as provided in the first aspect.

[0035] Fourthly, the present invention provides an in-vehicle computing unit, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executed instructions;

[0036] The processor executes computer execution instructions stored in memory to implement the joint acoustic recognition method for emergency vehicles and road conditions as provided in the first aspect.

[0037] Fifthly, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the joint acoustic recognition method for emergency vehicles and road conditions as provided in the first aspect.

[0038] In a sixth aspect, the present invention provides a computer program product, comprising: a computer program that, when executed by a processor, implements the joint acoustic recognition method for emergency vehicles and road conditions as provided in the first aspect.

[0039] The beneficial effects of this invention are:

[0040] This invention provides a method and related apparatus for joint acoustic recognition of emergency vehicles and road conditions. The method includes: acquiring multi-channel environmental audio signals; preprocessing the multi-channel environmental audio signals to obtain multi-channel log-Mel spectrum feature maps, the preprocessing including pre-emphasis processing, causal short-time Fourier transform, and Mel filtering; inputting the multi-channel log-Mel spectrum feature maps into a causal shared backbone network for feature extraction to obtain shared features; wherein, the causal shared backbone network is composed of multiple CMMCAs cascaded in a funnel-shaped multi-stage manner, with the number of channels of the feature map gradually increasing and the frequency dimension gradually compressing along the cascading direction; all convolutional layers in the causal shared backbone network adopt a left-side zero-padding strategy to satisfy causal constraints; the shared features are common acoustic representations that are simultaneously related to the acoustic features of emergency vehicle sound sources and road conditions; and inputting the shared features into parallel vehicle classification heads and road condition classification heads to obtain emergency vehicle type recognition results and road condition recognition results. This invention constructs a causal shared backbone network to extract features from preprocessed multi-channel audio signals, obtaining unified shared features. These shared features are then input into parallel vehicle and road classification heads, which independently output emergency vehicle type identification and road condition identification results. Compared to related technologies that build independent models for two perception tasks, this invention uses a single backbone network to provide shared feature representations for both tasks, eliminating redundant calculations and model parameter redundancy in the feature extraction stage, reducing system computing power consumption and storage overhead, and better meeting the lightweight deployment requirements of automotive embedded platforms. Furthermore, the parallel dual-branch classification head enables synchronous inference output for both tasks, overcoming the shortcomings of related technologies where the two tasks are independent and cannot be jointly perceived. Simultaneously, all convolutional layers in the causal shared backbone network employ a left-side zero-padding strategy on the time axis, ensuring that the output of each frame depends only on the current and historical frames, not on future frame information, eliminating inference delays caused by waiting for future frames, thus meeting the stringent real-time requirements of autonomous driving systems. Attached Figure Description

[0041] Figure 1 A flowchart illustrating the joint acoustic recognition method for emergency vehicles and road conditions provided in an embodiment of the present invention;

[0042] Figure 2 A schematic diagram illustrating the dimensional changes of the feature map at various stages, as provided in embodiments of the present invention;

[0043] Figure 3 Another schematic diagram of the joint acoustic recognition method for emergency vehicles and road conditions provided in this embodiment of the invention;

[0044] Figure 4 This is a schematic diagram of the structure of the emergency vehicle and road condition combined acoustic recognition device provided in an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of the overall architecture of the emergency vehicle and road condition joint acoustic recognition system provided in an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of the structure of the vehicle-mounted computing unit provided in an embodiment of the present invention.

[0047] The accompanying drawings have illustrated specific embodiments of the invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0048] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0049] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0050] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present invention.

[0051] The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in the process, method, product, or apparatus that includes elements is not excluded. For example, the use of terms such as "first," "second," etc., to indicate names does not imply any particular order.

[0052] In related technologies, when using independent single-task models to handle emergency vehicle identification and road condition detection separately, a single-task, single-branch independent identification architecture is generally adopted. Independent acoustic perception models and inference systems need to be built for emergency vehicle identification and road condition identification respectively. However, the independent deployment of dual perception tasks usually leads to redundant model parameters and repeated calculations of feature extraction, resulting in huge system computing power consumption and low resource utilization, which is difficult to adapt to deployment scenarios where the computing power of vehicle terminals is limited. Moreover, the two independent inference processes result in low integration and poor coupling of perception modules, which greatly increases the development, deployment and operation and maintenance costs of vehicle perception systems and is not conducive to the lightweight and integrated implementation of vehicle perception devices. Meanwhile, most existing mainstream acoustic recognition models adopt non-causal structures such as Transformer and non-causal convolution. These models rely on historical, current and future acoustic signals from multiple frames to complete feature reasoning and recognition, resulting in inherent reasoning delays. Autonomous driving is a dynamic scenario with high real-time and high safety requirements. The decision-making system needs to achieve millisecond-level streaming frame-by-frame real-time perception output. Any reasoning delay that relies on future frame data will cause vehicle decision lag, which can easily lead to safety hazards such as untimely avoidance and delayed updates of road adaptation strategies, and cannot meet the stringent requirements of real-time driving control in autonomous driving.

[0053] Specifically, some studies collect multi-period ambient sound data, extract fundamental frequency feature sequences and various emergency vehicle sound templates for correlation calculations, and use correlation coefficients and threshold comparisons to identify targets, achieving accurate identification of surrounding emergency vehicles. However, this approach relies on manually designed feature extraction processes, has limited generalization ability under complex environmental noise, and is only for vehicle detection in a single task, without addressing road condition detection. Other studies extract MFCC and Log-Mel features from vehicle road noise frequencies, fuse CNN and LSTM to construct a hybrid neural network to train a detection model for identifying road weather conditions in multiple scenarios. However, this approach only addresses the single task of road weather, and the parallel fusion network structure of CNN and LSTM differs from the causal shared backbone dual-branch architecture of this invention. Still other studies build image-audio cross-modal retrieval models to achieve feature interaction, use bidirectional sequence models for multi-granularity spatiotemporal correlation and remove redundant features, and use attention-based adaptive feature fusion to output vehicle task status and recognition confidence. While this approach achieves good recognition accuracy, it requires the coordination of two perception systems—image and audio—resulting in high hardware costs, and does not address road condition detection. Therefore, there is an urgent need for a multi-task acoustic recognition solution that can simultaneously classify emergency vehicles and identify road conditions from the same audio stream, while also meeting the requirements of real-time causal constraints and lightweight vehicle deployment.

[0054] Based on the technical problems existing in related technologies, this invention provides a joint acoustic recognition scheme for emergency vehicles and road conditions. After acquiring multi-channel environmental audio signals, the multi-channel environmental audio signals are preprocessed to convert them into log-Mel spectrogram feature maps to adapt to the frequency-time domain representation required for acoustic recognition tasks. Subsequently, the multi-channel log-Mel spectrogram feature map is input into a causal shared backbone network. Utilizing the strict causal constraints achieved by the zero-padding strategy on the left side of the time axis in all its internal convolutional layers, the feature extraction process is made independent frame by frame. The output of each frame depends only on the current and historical frames, thereby eliminating the dependence on subsequent frames and inherent inference delays. The causal shared backbone network outputs shared features that can be used by multiple tasks. These shared features are fed to the parallel emergency vehicle classification head and road condition classification head, respectively, so that the two recognition tasks share the same feature extraction path but complete classification inference independently. This reduces model parameter redundancy and feature duplication calculations while achieving synchronous joint perception of emergency vehicle type and road condition. The above processing method simultaneously completes two tasks, emergency vehicle type identification and road condition detection, within the same network, taking into account the comprehensive needs of multi-task joint perception, real-time causal reasoning, and lightweight vehicle deployment.

[0055] The application scenarios of the embodiments of the present invention will be described below first.

[0056] The joint acoustic recognition method for emergency vehicles and road conditions provided in this invention is applicable to environmental perception scenarios for autonomous and intelligent connected vehicles. Specifically, it utilizes onboard acoustic sensors as non-visual perception sources to perform real-time analysis and recognition of environmental audio signals collected during driving in diverse driving environments such as urban roads, highways, and tunnels. The onboard computing unit synchronously collects multi-channel environmental audio signals through microphone arrays deployed at the front bumper, rear bumper, and left and right rearview mirrors. The joint acoustic recognition model outputs vehicle type recognition results and road condition recognition results in parallel, and feeds back the recognition results in real time to the autonomous driving decision-making system or advanced driver assistance system (ADAS) to support the dynamic adaptation of emergency vehicle avoidance decisions and driving safety strategies.

[0057] Autonomous driving and intelligent connected vehicles belong to the interdisciplinary fields of vehicle engineering, acoustic signal processing, and artificial intelligence, and are typically applied in diverse driving scenarios such as urban roads, highways, tunnels, and residential areas. In these application scenarios, the onboard computing unit needs to perform millisecond-level streaming frame-by-frame processing of continuously collected environmental audio signals while the vehicle is traveling at high speed. It must simultaneously complete emergency vehicle type identification and road surface wetness / dryness detection on a low-power, low-computing-power onboard embedded platform, so as to trigger avoidance decisions in a timely manner when emergency vehicles approach, or dynamically adjust the vehicle stability control strategy when the road surface is slippery, to ensure driving safety.

[0058] Specifically, intelligent connected vehicles typically employ an operational architecture consisting of an audio acquisition module, a preprocessing module, a feature extraction module, and a recognition module. The audio acquisition module collects ambient audio signals (including emergency vehicle alarms and tire-road friction sounds) from the vehicle's surroundings using onboard acoustic sensors (such as multi-channel microphone arrays), covering the front, rear, left, and right directions to achieve omnidirectional acoustic perception. The preprocessing module preprocesses the acquired multi-channel audio signals, converting them into multi-channel log-Melogram spectral features. The feature extraction module extracts multi-scale acoustic features through a shared backbone network. The recognition module uses parallel vehicle and road classification heads to simultaneously output vehicle type and road condition recognition results. Furthermore, the recognition results are pushed in real-time to the autonomous driving decision-making module for dynamic adaptation of strategies such as emergency vehicle avoidance, braking distance prediction, and vehicle stability control. Because autonomous driving scenarios are extremely sensitive to perception latency, even a millisecond-level decision lag can lead to serious consequences. Given the limited computing resources of onboard computing units and the extremely high requirements for recognition accuracy and response speed, how to simultaneously classify emergency vehicles and detect road conditions based on the same audio stream while meeting causal real-time constraints has become a critical technical problem that urgently needs to be solved in the field of environmental perception for intelligent connected vehicles. This scenario is not limited to laboratory simulation verification but directly addresses real-world vehicle deployment and operation on open roads. It requires the acoustic recognition scheme to operate stably in complex noise environments (such as urban traffic noise, wind and rain noise, and simultaneous alarms from multiple vehicles), possess full-link causal real-time inference capabilities, and have a model parameter set that meets the lightweight deployment constraints of the onboard embedded platform, providing reliable perception information support for the driving safety decisions of the autonomous driving system.

[0059] The following describes in detail the specific implementation of the joint acoustic recognition method for emergency vehicles and road conditions provided by the present invention, taking the on-board computing unit as the execution subject and combining specific embodiments.

[0060] Figure 1 This is a schematic flowchart of a joint acoustic recognition method for emergency vehicles and road conditions provided in an embodiment of the present invention. Figure 1 As shown, the specific implementation of this joint acoustic recognition method for emergency vehicles and road conditions may include the following steps:

[0061] S101, acquires multi-channel environmental audio signals.

[0062] The multi-channel environmental audio signal in this step serves as the raw input to the joint acoustic recognition method, carrying acoustic information from the vehicle's surrounding environment for subsequent extraction of features related to the emergency vehicle's sound source and road conditions. "Multi-channel" means that the audio does not originate from a single acquisition location, but is synchronously acquired by multiple acoustic acquisition channels at the same time reference, thus preserving acoustic change information corresponding to spatial location differences. In practice, environmental audio can be synchronously acquired through multiple acoustic acquisition channels on the vehicle, forming a multi-channel environmental audio signal. To ensure the success of subsequent processing, each channel can be aligned based on a unified time reference, allowing acoustic responses at different locations at the same time to correspond. During acquisition, the input signal may contain environmental interference such as wind noise, vehicle noise, and road splashing sounds; however, the acquisition of multi-channel environmental audio is still maintained to uniformly extract information relevant to the recognition task in subsequent processing. For example, multi-channel environmental audio signals are synchronously acquired using microphone arrays deployed on the vehicle's front bumper, rear bumper, and left and right rearview mirrors, with a sampling rate of, for example, 16 kHz and a quantization bit depth of, for example, 16 bits. Based on the above analysis, it can be seen that by synchronously acquiring multi-channel environmental audio signals, a continuous acoustic input sequence can be established, providing a unified data foundation for the subsequent joint identification of emergency vehicle sound sources and road conditions.

[0063] It should be noted that, in this invention, emergency vehicles refer to vehicles that have priority on the road due to performing emergency tasks, such as, but not limited to, fire trucks, ambulances, or police cars; siren sounds refer to specific sound signals emitted by emergency vehicles to warn other road users to give way, such as, but not limited to, sirens.

[0064] S102 preprocesses the multi-channel environmental audio signal to obtain a multi-channel logarithmic Mel spectrum feature map. The preprocessing includes pre-emphasis processing, causal short-time Fourier transform, and Mel filtering.

[0065] The multi-channel log-Mel spectrum feature map in this step is used to convert the original time-domain audio signal into a time-frequency feature representation suitable for neural network processing, highlighting the frequency structure relevant to the recognition task while preserving multi-channel information. In practice, the preprocessing process can be performed channel by channel, followed by alignment and combination along the feature dimension. For each channel's input audio, time-frequency transformation, Mel mapping, logarithmic compression, and normalization can be used to convert the original audio into corresponding log-Mel spectrum features. The log-Mel spectrum feature maps of multiple channels are organized according to the channel dimension to form a multi-channel log-Mel spectrum feature map. Based on the above analysis, this step, by preprocessing the multi-channel environmental audio signal, transforms the original audio of each channel into a standardized feature representation suitable for subsequent network processing, while preserving the time and frequency dimension discrimination structures. This provides consistent input for the subsequent shared backbone network to simultaneously serve two recognition tasks.

[0066] In some embodiments, preprocessing of the multi-channel environmental audio signal includes: pre-emphasizing the multi-channel environmental audio signal to obtain a pre-emphasized audio signal; performing a causal short-time Fourier transform on the pre-emphasized audio signal to obtain a frequency domain signal; wherein the causal short-time Fourier transform disables center alignment; performing dynamic frequency range Mel filtering on the frequency domain signal to obtain a Mel spectrum; and performing logarithmic compression and linear normalization on the Mel spectrum to obtain a multi-channel logarithmic Mel spectrum feature map.

[0067] In practice, multi-channel environmental audio signals can be synchronously acquired by an on-board microphone array, and each channel signal is aligned and then enters the pre-emphasis unit. The pre-emphasis unit can be implemented using a first-order differential high-pass structure. Each channel is processed by subtracting the previous sample from the current sample, with coefficients set to 0.95 to 0.98 to preserve high-frequency details and suppress low-frequency DC bias. The causal short-time Fourier transform can use parameters with a window length of 20ms to 40ms and a frame shift of 10ms to 20ms. The window function can be either a Hamming window or a Hanning window. After center alignment is turned off, the frequency domain calculation only uses the data in the current frame and its historical window, without introducing future samples. The dynamic frequency range Mel filter can establish a Mel filter bank in the range of 300Hz to 8000Hz, and dynamically set the lower and upper limits according to the frequency bands required for vehicle recognition and road condition recognition to preserve the frequency structure related to the differences in alarm harmonics and tire noise. Logarithmic compression can add a very small constant to the filtered energy value and then take the logarithm. Linear normalization can map the result to a unified numerical range by channel, thereby forming a multi-channel logarithmic Mel spectrum feature map.

[0068] In this embodiment of the invention, the original audio signal is converted into a log-Mel spectral feature map by sequentially performing pre-emphasis, causal short-time Fourier transform, dynamic frequency range Mel filtering, logarithmic compression, and linear normalization on the multi-channel environmental audio signal. Specifically, the causal short-time Fourier transform, by disabling center alignment, ensures that the calculation of each frame's frequency domain signal strictly depends on the current frame and historical frames, avoiding the introduction of future frame information. This provides a spectral input that satisfies causal constraints for the subsequent frame-by-frame streaming inference of the causal shared backbone network. Simultaneously, the dynamic frequency range Mel filtering and logarithmic compression make the spectral features more consistent with human auditory perception characteristics, enhancing the feature representation of key acoustic events such as vehicle alarm sounds and tire road noise, and compressing the signal's dynamic range, thus improving the robustness of features in complex noise environments. Through the above preprocessing, the original multi-channel audio signal is converted into a scale-uniform time-frequency representation, laying the foundation for efficient feature extraction and accurate recognition in subsequent networks.

[0069] S103, the multi-channel log-Mel spectrum feature map is input into the causal shared backbone network for feature extraction to obtain shared features; wherein, the causal shared backbone network is composed of multiple CMMCAs cascaded in a funnel-shaped multi-stage manner, with the number of channels of the feature map gradually increasing and the frequency dimension gradually compressing along the cascading direction; all convolutional layers in the causal shared backbone network adopt a zero-padding strategy on the left side of the time axis to meet causal constraints; the shared features are common acoustic representations that are simultaneously related to the sound source features of emergency vehicles and the acoustic features of road conditions.

[0070] The causal shared backbone network is used to extract shared features from multi-channel log-Melogram spectral feature maps, generating shared features for subsequent classification heads to reduce redundant computation of underlying acoustic patterns between emergency vehicle identification and road condition identification tasks. All convolutional layers in the causal shared backbone network employ a left-side zero-padding strategy to ensure that the output at any given time depends only on historical and current frames, not on future frame information. Specifically, if the convolutional kernel has a length of k in the time dimension, k-1 zero-value positions are padded to the left of the input feature's time dimension, while the right side remains unpadded, thus ensuring that the convolution operation is independent of data points after the current time step. Other convolutional processing layers also maintain this left-padding constraint along the time dimension. With this construction, the entire path from input to output satisfies causality, allowing direct deployment in online vehicle operation scenarios.

[0071] In practical implementation, the causal shared backbone network can perform causal convolutional feature extraction on the input multi-channel log-Mel spectrum feature map and output shared features. These shared features are essentially common acoustic representations related to both vehicle category and road surface condition, including sound source features of emergency vehicles and acoustic features related to road surface condition. Based on the above analysis, this step completes feature extraction uniformly through a single causal shared backbone network, improving upon the repetitive extraction problem caused by building independent models in existing solutions. Furthermore, since all convolutional layers satisfy temporal causality constraints, the entire inference process can output intermediate representations in real-time for the current frame, making it suitable for scenarios with limited onboard resources and high real-time requirements.

[0072] S104. The shared features are input into the parallel vehicle classification head and road classification head respectively to obtain the emergency vehicle type identification result and the road condition identification result.

[0073] The vehicle classification head outputs emergency vehicle type identification results based on shared features, while the road surface classification head outputs road surface condition identification results based on the same shared features. These two heads work in parallel rather than sequentially. In practice, the shared features output by the causal shared backbone network can be directly fed into the two classification branches, or they can first pass through a unified time-dimensional aggregation module before being fed into the two classification heads respectively. If the shared features are time-frequency tensors, the vehicle and road surface classification heads can each contain one or more layers of causal convolution, frequency-dimensional aggregation layers, time-dimensional pooling layers, and fully connected output layers. If the shared features are already a one-dimensional time-series feature sequence, the two classification heads can adopt a lightweight one-dimensional causal convolution plus linear classifier structure. The vehicle classification head outputs identification results corresponding to multiple vehicle categories, and the road surface classification head outputs identification results corresponding to multiple road surface conditions.

[0074] In streaming inference scenarios, the two classification heads receive the shared features output by the causal shared backbone network for the latest time position and perform synchronous calculations without waiting for future frames to complete the process. After the recognition results are generated, they can be sent to the vehicle perception decision module, alarm module, or vehicle control module. Based on the above analysis, this step, by setting up parallel vehicle classification heads and road surface classification heads on the same shared features, enables emergency vehicle type recognition and road surface condition recognition to be completed synchronously after a single feature extraction, avoiding the need to establish separate complete backend networks for the two tasks. This parallel structure, in conjunction with the aforementioned causal shared backbone network, forms a continuous online processing link from audio acquisition, time-frequency feature generation, shared feature extraction to dual-task classification output, enabling the output of emergency vehicle type recognition results and road surface condition recognition results without relying on future frames.

[0075] The emergency vehicle and road condition joint acoustic recognition method provided in this invention constructs a causal shared backbone network to extract features from preprocessed multi-channel audio signals, obtaining unified shared features. These shared features are then input into parallel vehicle classification heads and road condition classification heads, which independently output emergency vehicle type recognition results and road condition recognition results. Compared to related technologies that build independent models for two perception tasks, this invention uses a single backbone network to provide shared feature representations for both tasks, eliminating redundant calculations and model parameter redundancy in the feature extraction stage, reducing system computing power consumption and storage overhead, and better meeting the lightweight deployment requirements of in-vehicle embedded platforms. Furthermore, the parallel dual-branch classification heads enable synchronous inference output for both tasks, overcoming the shortcomings of related technologies where the two tasks are independent and cannot be jointly perceived. Simultaneously, all convolutional layers in the causal shared backbone network employ a left-side zero-padding strategy on the time axis, ensuring that the output of each frame depends only on the current and historical frames, not on future frame information, eliminating inference delays caused by waiting for future frames, thus meeting the stringent real-time requirements of autonomous driving systems.

[0076] In some embodiments, the multi-channel log-Mel spectrum feature map is input into a causal shared backbone network for feature extraction, including: inputting the multi-channel log-Mel spectrum feature map into the entry convolutional layer of the causal shared backbone network for preliminary feature extraction, resulting in an initial feature map with doubled channel number and reduced frequency and time dimensions; inputting the initial feature map sequentially into at least three cascaded stages for hierarchical feature extraction, resulting in shared features; in the at least three cascaded stages, the first stage includes CMMCA and does not enable multi-scale channel attention, and the number of channels and resolution of the output feature map remain unchanged, used to extract low-level acoustic features; the middle stages include CMMCA and enable multi-scale channel attention, and the number of channels of the output feature map is increased, while the frequency and time dimensions are reduced, used to extract mid-level semantic features; the last stage includes CMMCA and enables multi-scale channel attention, and the number of channels of the output feature map is increased, the frequency dimension is reduced, while the time dimension remains unchanged, used to extract high-level semantic features.

[0077] In this embodiment, the ingress convolutional layer is, for example, a causal Stem convolutional layer. Specifically, it can be a two-dimensional convolution with a kernel size of 3×3 and a stride of 2, and a zero value of one time frame width is padded to the left of the time dimension so as to expand the number of channels to twice the number of input channels while reducing the frequency and time dimensions, resulting in an initial feature map with doubled channel number and reduced frequency and time dimensions. Then, the initial feature map is input into at least three cascaded stages, each stage containing at least one CMMCA. The number of CMMCAs in each stage can be set to 1, 2, or 3, etc., depending on the computational resources and accuracy requirements, and is not limited to a specific number. The initial stage does not enable multi-scale channel attention. In this stage, CMMCA only retains the backbone convolutional branch, keeping the number of channels and resolution of the output feature map unchanged. This stage is used to extract low-level acoustic features such as edges, textures, and harmonic structures. The middle stages enable multi-scale channel attention, generating channel attention weight maps through global frequency domain pooling and local feature extraction branches. Adaptive recalibration is performed on the responses of each channel. In this stage, CMMCA downsamples and expands the number of channels in both the frequency and time dimensions, used to extract mid-level semantic features. The final stage enables multi-scale channel attention. In this stage, CMMCA only compresses the frequency dimension while maintaining the time dimension resolution and further expanding the number of channels. This allows high-level semantics to be aggregated while preserving temporal continuity, used to extract high-level semantic features. All convolutional layers in each stage employ a zero-padding strategy on the left side of the time axis to ensure that the feature extraction process meets strict causal constraints.

[0078] For example, suppose the causal sharing backbone network consists of multiple CMMCAs cascaded in a funnel-shaped three-stage configuration, with each stage containing two CMMCAs. Accordingly, during the process of inputting the initial feature map into the funnel-shaped three-stage CMMCAs for multi-scale feature extraction, the first stage's two CMMCAs do not enable multi-scale channel attention, maintaining the feature map resolution and number of channels, and extracting low-level acoustic features such as edges, textures, and harmonic structures. The second stage enables multi-scale channel attention for the first time; its first CMMCA downsamples the frequency and time dimensions and expands the number of channels, while the second CMMCA performs refined feature extraction in the expanded channel space. The third stage continues to enable multi-scale channel attention, where the first CMMCA only downsamples the frequency dimension and continues to expand the number of channels, while maintaining the time dimension resolution, and the second CMMCA performs final high-level semantic abstraction. These three stages together constitute a funnel-shaped pyramid feature hierarchy of "gradual channel expansion and gradual frequency compression."

[0079] It is understood that the above example of three stages with two CMMCAs per stage is only one specific implementation method. Those skilled in the art can flexibly adjust the number of stages and the number of CMMCAs within each stage according to actual needs. For example, when using four stages, the first stage still does not enable multi-scale channel attention, keeping the feature map resolution and number of channels unchanged to extract low-level acoustic features; several stages in the middle enable multi-scale channel attention, downsampling the frequency and time dimensions, allowing the network to gradually refine mid-level semantic features at more levels; the last stage enables multi-scale channel attention, compressing only the frequency dimension while keeping the time dimension resolution unchanged, to complete high-level semantic aggregation while preserving temporal continuity; when a stage contains three CMMCAs, the multiple CMMCAs are connected sequentially along the time dimension, maintaining the downsampling strategy of that stage as a whole. All convolutional layers in each stage adopt a zero-padding strategy on the left side of the time axis to ensure that the feature extraction process meets strict causal constraints and does not depend on any future frame information.

[0080] The network first transforms the multi-channel log-Mel spectrum feature map into an initial feature map, and then refines the shared features through at least three stages. The first stage outputs low-level acoustic information, the middle stage outputs mid-level semantic information, and the last stage outputs high-level semantic information, integrating the information from each layer into a unified shared representation. Because each stage progressively expands the channels and compresses the frequency dimension along a funnel-shaped structure, the network can complete cross-scale acoustic modeling while maintaining causality.

[0081] In this embodiment of the invention, by adopting the above-mentioned funnel-shaped multi-stage structure, acoustic features from low to high level can be extracted in layers. The low-level features required by the two types of tasks can be reused in the same backbone, reducing redundant calculations. Strict causal constraints are maintained during channel amplification and frequency compression, achieving effective unification of real-time reasoning and cross-scale acoustic modeling. This provides a shared feature foundation with strong discriminative power for subsequent vehicle type recognition and road condition detection.

[0082] Optionally, CMMCA employs an inverse residual bottleneck structure and processes the input features as follows: Channel expansion of the input features is performed using an upscaling convolutional layer to obtain upscaling features; convolution operations are performed on the upscaling features using a causal-depth-separable convolutional layer to obtain convolutionally processed features; wherein, the convolution operation includes: symmetrical padding only in the frequency dimension and no padding in the time dimension; the number of channels of the convolutionally processed features is compressed to the target output channel number using a dimensionality reduction projection layer to obtain dimensionality-reduced features; adaptive recalibration of the channel dimensions of the dimensionality-reduced features is performed using a multi-scale channel attention layer to obtain attention-recalibrated features; and the attention-recalibrated features are added element-wise to the original input of CMMCA using a residual connection layer to obtain the output features of CMMCA.

[0083] In the specific implementation, the input features can be a two-dimensional time-frequency feature map output by the previous layer, where the channel dimension corresponds to different feature maps, the frequency dimension corresponds to the Mel frequency band, and the time dimension corresponds to a continuous frame sequence. The dimension-upgrading convolutional layer can be implemented using pointwise convolution, with a kernel size of 1×1 or an equivalent form. The number of output channels is set to twice the number of input channels or a preset multiple to form the extended channels in the inverse residual bottleneck. The causal-depth-separable convolutional layer uses grouped convolution, with the number of groups equal to the number of expanded channels. The kernel size can be set according to actual needs. The frequency dimension uses symmetrical padding to maintain the center alignment of the frequency band, and the time dimension uses left-side zero padding (i.e., only zero values ​​are padded on the left side of the current frame, and no padding on the right side), so that the current output depends only on the current and historical frames, not on future frame data, thus satisfying the causal constraint. The stride is determined by the external stage configuration. The dimension-reducing projection layer can also use pointwise convolution to compress the number of channels to the target output channel number, which can be set to the original number of channels, twice the number of channels, or other preset values ​​according to the stage configuration. The kernel size, channel ratio, and number of output channels in this application can all be configured according to the on-board computing power and feature scale, and this invention does not limit them.

[0084] Furthermore, the multi-scale channel attention layer consists of two parallel branches for adaptive recalibration of the channel dimensions. Assuming the causal shared backbone network is composed of multiple CMMCAs cascaded in a funnel-shaped three-stage configuration, the multi-scale channel attention layer is only activated in the second and third stages, and not in the first stage. The residual connection layer sums the recalibrated result element-wise with the original features from the CMMCA input to obtain the output features, which are then fed into subsequent networks. This structure completes channel expansion, causal convolution extraction, channel selection, and residual fusion within a single module, enabling the causal shared backbone network to maintain strong frequency discrimination capabilities under real-time inference conditions.

[0085] For example, suppose the input feature map X has dimensions (B, C_in, F, T), where B is the batch size, C_in is the number of input channels, F is the frequency dimension, and T is the time dimension. The input feature map X first undergoes a dimensionality-upgrading convolutional layer to expand the number of channels from C_in to C_exp, followed by batch normalization and PReLU activation. Further, the upgraded features are input to a causal-depth-separable convolutional layer for spatial feature extraction. The number of groups in this convolutional layer is equal to the number of expanded channels C_exp. Symmetrical padding is performed only in the frequency dimension during the convolution operation, with no padding in the time dimension. Afterwards, it also undergoes batch normalization and PReLU activation. The ReLU activation function is used for processing; then, the number of channels is compressed from C_exp to the target output channel number C_out through a dimensionality reduction projection layer, and batch normalization is performed; then, it enters a switchable multi-scale channel attention layer, which is skipped in the first stage but enabled in the second and third stages to adaptively recalibrate the channel dimensions; if the number of input channels C_in is equal to the number of output channels C_out, the input feature map X is added element-wise to the main path output through a conditional residual connection; finally, the final feature map Y is output through a channel-wise PReLU activation function (which can learn negative slopes), with dimensions (B, C_out, F, T).

[0086] In this embodiment of the invention, by employing an inverse residual bottleneck structure combined with causal depthwise separable convolution, effective feature extraction can be achieved with a low number of parameters and computational cost. By filling only the frequency dimension and not the time dimension, the output features satisfy temporal causal constraints. By adaptively recalibrating the dimensionality-reduced features through multi-scale channel attention, the acoustic channel response related to vehicle type recognition and road condition recognition can be enhanced. After residual fusion, the module output retains the input information and enhances the effective feature expression, thereby improving the stability and real-time performance of joint acoustic recognition.

[0087] In some embodiments, the multi-scale channel attention layer includes a global frequency domain pooling branch and a local feature extraction branch. The multi-scale channel attention layer is used to adaptively recalibrate the channel dimensions of the dimensionality-reduced features, including: inputting the dimensionality-reduced features into the global frequency domain pooling branch and the local feature extraction branch respectively; performing global frequency domain pooling on the input feature map along the frequency dimension through the global frequency domain pooling branch, compressing the frequency information into a single value, and then performing a bottleneck transformation to generate a global channel description; directly performing a bottleneck transformation on the input feature map through the local feature extraction branch to generate a local channel description; adding the global channel description and the local channel description element-wise and then normalizing to obtain a channel attention weight map; multiplying the channel attention weight map element-wise with the dimensionality-reduced features to obtain the attention-recalibrated features.

[0088] In practical implementation, the dimensionality-reduced features can be output by the previous inverse residual bottleneck structure, whose tensor size is C×F×T, corresponding to the number of channels, frequency dimension and time dimension, respectively. The feature is input into the global frequency domain pooling branch and the local feature extraction branch respectively. The global frequency domain pooling branch performs average pooling along the frequency dimension, compressing the frequency information of each channel into a single value to obtain the channel description of the global receptive field. Then, it generates the global channel description through a bottleneck transformation composed of one-dimensional convolution or fully connected layers. The channel compression ratio r in the bottleneck transformation can be set to 16. The local feature extraction branch directly performs a bottleneck transformation on the original feature, retaining the local frequency band texture and harmonic edge discrimination information to generate the local channel description. The two outputs are superimposed element-wise along the channel dimension and normalized to the [0,1] interval by the Sigmoid function to form a channel attention weight map with the same number of input channels, ensuring that the discrimination information of the frequency dimension is not lost after the global and local features are fused. Furthermore, the channel attention weight map is multiplied with the dimensionality-reduced feature channel by channel to complete the adaptive recalibration of the channel dimension and output the recalibrated feature.

[0089] For example, the input feature map X has dimensions (B, C, F, T), where B is the batch size, C is the number of channels, F is the frequency dimension, and T is the time dimension. This multi-scale channel attention layer consists of two parallel branches. Branch one is a global frequency domain pooling branch, which performs global frequency domain pooling on the input feature map X along the frequency dimension, compressing the frequency information of each channel into a single value to obtain the channel description of the global receptive field. Then, it passes through a convolutional layer with a kernel size of 1×1 to compress the number of channels from C to C / 16. After batch normalization and PReLU activation, it passes through another convolutional layer with a kernel size of 1×1 to restore the number of channels to C, and then passes through batch normalization to obtain the global channel description. Branch two is a local feature extraction branch, which directly processes the input feature map X through the same two 1×1 convolutions, batch normalization, and PReLU activation to generate local channel descriptions. The outputs of the two branches are added element by element and then normalized to the [0,1] interval by the Sigmoid function to obtain the channel attention weight map. Finally, the channel attention weight map is multiplied element by element with the input feature map X to complete the adaptive recalibration of the channel dimension and output feature map Y with dimensions (B, C, F, T).

[0090] This multi-scale channel attention layer simultaneously utilizes global frequency distribution and local frequency band details to weight each channel during operation, enabling channels sensitive to alarm sound harmonics and tire noise frequency band differences to obtain higher responses, while the responses of redundant channels are suppressed, thereby outputting feature representations that are more suitable for subsequent vehicle classification heads and road classification heads to share.

[0091] With this specific implementation, the channel weights no longer rely solely on a single global statistic, but take into account both the overall spectrum and the local frequency band structure, resulting in more complete retention of frequency domain discrimination information and more stable feature recalibration, thereby improving the recognition accuracy and noise resistance in complex vehicle acoustic scenarios.

[0092] Considering that existing acoustic recognition schemes, when converting two-dimensional time-frequency feature maps (frequency × time) into one-dimensional time-series features, typically directly use global average pooling to compress the frequency dimension into a single value or directly flatten it before connecting it to a fully connected layer, this leads to the loss of structured discriminative information within the frequency dimension and the inability to adaptively learn the differences in importance of different frequency points for the classification task, thus limiting further improvement in recognition accuracy. Therefore, in some embodiments, before inputting the shared features into the parallel vehicle classification head and road classification head respectively, the following steps are included: adaptively average pooling the shared features along the frequency dimension to obtain pooled features; performing grouped cross-frequency convolution on the pooled features to fuse each frequency point across frequencies to obtain cross-frequency fused features; eliminating the frequency dimension of the cross-frequency fused features through a Squeeze operation to obtain a one-dimensional time-series feature sequence; and using the one-dimensional time-series feature sequence as input to the vehicle classification head and road classification head.

[0093] In its implementation, the grouped cross-frequency convolution groups the pooled features by channel, with each group corresponding to several adjacent or non-adjacent frequency points. The convolution kernel width is set in the frequency direction to allow local interaction across multiple frequency bands. The convolution kernel can be a one-dimensional frequency convolution, and the convolution weights are shared within a group but set independently between groups. This structure allows for explicit modeling of the frequency band correlation between alarm sound harmonics, tire contact noise, and road texture noise. The cross-frequency fused features are then compressed along the frequency dimension by a Squeeze operation into a one-dimensional temporal feature sequence with unchanged time length. The vehicle classification head and road surface classification head can each consist of a fully connected layer and a classification output layer, respectively, and receive this one-dimensional temporal feature sequence as a common input. In practical applications, other network structures can also be selected for the classification head; this invention does not limit this.

[0094] In this embodiment of the invention, adaptive average pooling is first applied to the shared features along the frequency dimension, compressing the frequency dimension to representative frequency points while preserving the relative response intensity of each frequency band. Subsequently, cross-frequency convolution is used to establish information interaction and dependencies between different frequency bands, allowing the features at each frequency point to complement each other during the fusion process. Finally, a squeeze operation eliminates the frequency dimension, outputting a compact one-dimensional temporal feature sequence, which serves as the shared input for both the vehicle classification head and the road classification head. Through this processing, the structured discriminative information in the frequency dimension is preserved during compression, enabling the classification head to obtain a more suitable input representation for discrimination. Simultaneously, the two tasks share the same feature sequence to complete their respective classification inferences, reducing redundant computations, maintaining the feature compactness required for real-time inference, and enhancing the representation ability of effective acoustic patterns in complex noisy environments, thereby effectively improving recognition accuracy.

[0095] Correspondingly, the feature map undergoes a systematic pyramid-shaped hierarchical evolution from input to output, characterized by "gradual channel expansion and frequency compression." The specific evolution process is as follows: After the input multi-channel log-Mel spectrum feature map passes through a causal Stem layer, the number of channels doubles, while both the frequency and time dimensions decrease. After the first stage, the number of channels and resolution remain unchanged to preserve low-level acoustic details. After the second stage, the number of channels continues to expand, and the frequency and time dimensions decrease again, allowing mid-level semantic features to begin forming, with an attention mechanism enhancing the channel representation of discriminative features. After the third stage, the number of channels expands again, the frequency dimension decreases while the time dimension remains unchanged, completing high-level semantic abstraction. In the frequency feature fusion layer, the frequency dimension is first compressed using adaptive average pooling, then cross-frequency fusion is performed using grouped cross-frequency convolution, and finally, the frequency dimension is eliminated through a Squeeze operation, outputting a one-dimensional temporal feature sequence. The entire funnel-shaped design achieves gradual compression of the frequency dimension, cumulative compression of the time dimension, and gradual expansion of the number of channels, thus completing an efficient feature transformation from a two-dimensional time-frequency feature map to a one-dimensional temporal feature sequence.

[0096] Figure 2 This is a schematic diagram illustrating the dimensional changes of the feature map at various stages, as provided in embodiments of the present invention. Figure 2 As shown, the input multi-channel log-Melogram spectral feature map is amplified in the number of channels and reduced in both frequency and time dimensions after passing through a causal Stem convolutional layer. It then passes through two CMMCAs in the first stage, where the number of channels, frequency dimension, and time dimension remain unchanged to preserve low-level acoustic details. Next, it passes through two CMMCAs in the second stage, where the first CMMCA amplifies the number of channels and reduces the frequency and time dimensions, while the second CMMCA keeps the dimensions unchanged, allowing mid-level semantic features to begin forming. Then, it passes through two CMMCAs in the third stage, where the first CMMCA amplifies the number of channels and reduces the frequency dimension while keeping the time dimension unchanged, while the second CMMCA keeps the frequency dimension unchanged, completing high-level semantic abstraction. Finally, it passes through a frequency feature fusion layer, where adaptive average pooling further reduces the frequency dimension, then grouped cross-frequency convolution fuses the frequency points and eliminates the frequency dimension through a Squeeze operation, outputting a one-dimensional temporal feature sequence. This sequence is fed into parallel vehicle classification heads and road classification heads, resulting in emergency vehicle type recognition and road condition recognition results.

[0097] Based on the above embodiments, in some embodiments, shared features are input into parallel vehicle classification heads and road classification heads respectively to obtain emergency vehicle type identification results and road condition identification results. This includes: inputting a one-dimensional temporal feature sequence into the vehicle classification head, and outputting frame-by-frame predicted vehicle type bottom-level eight-class classification results through the vehicle classification head. The bottom-level eight-class classification of vehicle types includes four subcategories: ambulance, fire truck, police car, engineering rescue vehicle, and background noise; aggregating the frame-by-frame prediction results of the four police car subcategories into the confidence level of the police car category, and outputting vehicle type identification results for ambulance, fire truck, police car, engineering rescue vehicle, and background noise; and inputting a one-dimensional temporal feature sequence into the road classification head, and outputting frame-by-frame predicted road condition binary classification results through the road classification head. The road condition binary classification results include dry and slippery conditions.

[0098] The one-dimensional temporal feature sequence is obtained by adaptive average pooling along the frequency dimension, grouped cross-frequency convolution, and Squeeze compression of shared features. The grouped cross-frequency convolution encodes the local differences and cross-frequency correlations between adjacent frequency points into temporal representations, facilitating the subsequent frame-by-frame output of results by the classification head. The vehicle classification head and road classification head can each consist of a temporal convolutional layer, a normalization layer, and a fully connected prediction layer. The vehicle classification head outputs eight probability distributions for each frame and sums or weighted sums the probabilities of the four police car subclasses to obtain the confidence score for the police car category. The road classification head outputs two probability distributions for each frame: dry and wet / slippery. The two classification heads share the same input temporal features but independently complete the discrimination, enabling the vehicle alarm acoustic mode and the road contact acoustic mode to be modeled separately based on unified features while maintaining the temporal resolution of frame-level prediction. In practical applications, this structure can be executed by an onboard processor. The convolutional kernel size, number of channels, and fully connected layer dimension of the classification head can be set according to the computing power configuration; this invention does not impose any limitations on these settings.

[0099] The above method enables the simultaneous output of emergency vehicle category and road condition results without introducing additional feature extraction branches, and allows the police vehicle subclass results and the major category results to converge within the same prediction link, thereby improving the consistency and real-time performance of joint identification.

[0100] Optionally, both the vehicle classification head and the road classification head are causal extended convolutional integral classification heads.

[0101] In practical implementation, the vehicle classification head receives a one-dimensional temporal feature sequence, extracts cross-time correlations through several layers of causal dilated convolution, and then maps the convolution results to frame-by-frame vehicle category probabilities. The road classification head performs causal dilated convolution on the input one-dimensional temporal feature sequence in the same way and outputs frame-by-frame road state probabilities. Since both use causal convolution, the output at any given time depends only on the features at that time and before, and inference can be completed without waiting for subsequent frames, thus meeting the real-time requirements of online vehicle recognition. The kernel size, number of layers, and dilation rate in the causal dilated convolution classification head can be configured according to the vehicle's computing power and recognition accuracy requirements. For example, three layers of convolution can be stacked with the dilation rate increasing layer by layer to take into account both local details and longer temporal dependencies.

[0102] After the vehicle classification head and road classification head complete convolutional feature aggregation, the output layer normalizes the response values ​​corresponding to different categories to obtain vehicle type recognition results and road condition recognition results. Because dilated convolutions create a larger receptive range in the temporal dimension, the classification head can capture acoustic patterns over a longer time span from shorter input segments, thus enabling stable discrimination of the acoustic features related to ambulances, fire trucks, police cars, and engineering rescue vehicles, as well as the differences in contact sound between dry and wet road surfaces. This structure also synergizes with the preceding causal shared backbone network, allowing shared features to directly perform temporal discrimination after entering the classification head, without requiring additional future frame buffering or offline smoothing processing.

[0103] In this embodiment of the invention, by limiting both the vehicle classification head and the road classification head to causal dilated convolutional classification heads, both heads can expand the temporal modeling range while maintaining causal constraints, and complete multi-task classification with relatively small parameter overhead. This reduces online recognition latency and improves classification stability in complex acoustic scenarios. Since the two types of classification heads have the same structure, they are easily deployed uniformly on the vehicle computing platform, thereby improving the consistency of engineering implementation and operational reliability of multi-task acoustic perception.

[0104] In some embodiments, the causal shared backbone network, vehicle classification head, and road classification head are obtained through dual-task joint training. The dual-task joint training includes: acquiring a training sample set, where each sample contains multi-channel audio signals, vehicle type labels, and road state labels; inputting the training sample set into the causal shared backbone network, vehicle classification head, and road classification head for forward propagation; using a combination of vehicle recognition loss and road detection loss as the total loss function, and optimizing all parameters of the causal shared backbone network, vehicle classification head, and road classification head simultaneously through backpropagation based on the total loss function.

[0105] In practical implementation, the acquired multi-channel environmental audio signals are first preprocessed to form multi-channel log-Melogram spectral feature maps. These maps are then input into a causal shared backbone network to obtain shared features. These shared features are then fed into the vehicle classification head and road classification head respectively to complete dual-task prediction. The causal shared backbone network can adopt a multi-stage cascaded structure, while the vehicle and road classification heads can employ a causal dilated convolutional classification structure. During training, both share front-end feature representations and are supervised separately for different labels. During training, cross-entropy loss can be used to construct the vehicle recognition loss and road detection loss, and the combination of these two losses is used as the total loss function to ensure that the two tasks converge collaboratively within the same optimization framework.

[0106] During operation, audio samples in the training set are first forward-propagated to obtain prediction results for both tasks. Then, the total loss function is calculated based on the prediction results and corresponding labels. Subsequently, backpropagation is used to simultaneously adjust the parameters of the causal shared backbone network and the two classifiers, ensuring that the shared features meet the representation requirements of both vehicle acoustic patterns and road contact acoustic patterns. Through this joint training method, the two recognition tasks can share underlying acoustic features and converge together during the same parameter update process.

[0107] With this implementation method, the causal shared backbone network, vehicle classification head and road classification head can be optimized synchronously under a unified training framework, which helps to reduce parameter redundancy caused by repeated feature learning, and establishes a consistent feature basis for vehicle type recognition and road condition recognition, thereby improving the model convergence stability and consistency of dual-task recognition.

[0108] In one possible embodiment, the combination of vehicle identification loss and road surface detection loss is a weighted combination; the total loss function is the sum of vehicle identification loss and road surface detection loss weighted by road surface loss weighting coefficient.

[0109] In the dual-task joint training process, the vehicle recognition loss characterizes the deviation between the output of the vehicle classification head and the vehicle type label, while the road detection loss characterizes the deviation between the output of the road classification head and the road state label. Both are calculated from their respective cross-entropy losses or other loss functions adapted to the classification task. The total loss function is formed by summing the vehicle recognition loss with the road detection loss weighted by the road loss weight coefficient, and is used to update the parameters of the causal shared backbone network, the vehicle classification head, and the road classification head through backpropagation. For example, the total loss function satisfies the following formula:

[0110]

[0111] in, Total loss; and These are vehicle recognition loss and road surface detection loss, respectively. This is the road surface loss weight coefficient. This road surface loss weight coefficient can be set to a fixed value based on the distribution of the two types of labels in the training samples, the task convergence speed, and the classification difficulty, or it can be preset in the early stage of training and remain unchanged.

[0112] By employing a weighted combination approach, the total loss function enables the vehicle recognition and road detection tasks to achieve differentiated optimization intensities within the same shared feature space. This allows for control over the bias of one task during training and maintains relatively stable gradient contributions from both classifiers during joint training. Since the road detection loss is explicitly adjusted by the road loss weight coefficient, even with a large number of vehicle category samples or a small number of road state samples, adjusting this coefficient can still maintain a convergence balance between the two tasks.

[0113] By adopting this total loss function, dual-task training can simultaneously constrain the outputs of vehicle recognition and road detection within the causal shared backbone network. This allows the shared features to take into account the discriminative information of both tasks and reduces the interference caused by over-optimization of one task on the other. Since the total loss is composed of a weighted sum, the direction of training parameter updates is more likely to converge around the common goal of the two tasks, thereby further improving the stability and multi-task consistency of the joint acoustic recognition model.

[0114] Optionally, both vehicle recognition loss and road surface detection loss are binary cross-entropy losses.

[0115] In implementation, vehicle recognition loss and road surface detection loss can be written into the joint training framework in the form of binary cross-entropy, and calculated element-wise with the label tensor of the corresponding sample. Then, the losses of each frame in the time dimension are summed or averaged to form a stable task supervision signal. Since both tasks use the same type of loss function, the shared features extracted by the causal shared backbone network can be simultaneously supervised by vehicle category and road surface state under a unified discrimination criterion, thus making the shared features more suitable for multi-task joint optimization.

[0116] During operation, the vehicle classification head and the road classification head output probability values ​​respectively. During training, these values ​​are fed into the binary cross-entropy loss calculation module along with the real labels to obtain two losses, which are then combined in the total loss function. The parameters of the causal shared backbone network and the two classification heads are updated through backpropagation.

[0117] In this embodiment of the invention, by limiting both vehicle recognition loss and road detection loss to binary cross-entropy loss, the supervision representation of the two tasks is consistent, the deviation between the label and the prediction can be directly quantified, thereby making the joint training process stable and convergent, and making the classification boundary of vehicle recognition and road detection easier to learn.

[0118] Furthermore, in some embodiments, before inputting the training sample set into the causal shared backbone network, the vehicle classification head, and the road classification head for dual-task joint training, the method further includes: performing hybrid data augmentation processing on the training sample set, wherein the hybrid data augmentation includes at least one of spectral masking enhancement, signal-to-noise ratio perturbation superposition, time rolling enhancement, device impulse response convolution enhancement, frequency domain statistics hybrid enhancement, and frequency random perturbation enhancement.

[0119] In practical implementation, the training sample set can consist of multi-channel environmental audio and its corresponding labels. The samples undergo enhancement processing before entering the causal shared backbone network. In the above hybrid data enhancement processing, each enhancement method can be implemented in the following ways: For spectral mask enhancement, the audio can first be converted into a multi-channel log-Mel spectrum feature map, and then a frequency range can be randomly selected and zeroed according to a preset mask width (e.g., the frequency mask width is randomly set within the range of [0, 48]) to simulate a scenario where some frequency bands are blocked or filtered; For signal-to-noise ratio perturbation superposition, pure background noise of the BACKGROUND category can be extracted from the noise library as an interference source, and the target signal-to-noise ratio can be randomly sampled within the range of [5, 20] dB and then analyzed using the root mean square (RMS). The RMS (Real Squared) energy matching algorithm performs amplitude scaling and then superimposes it onto the original audio sample by sample to address class imbalance caused by the excessive proportion of BACKGROUND class samples in the dataset. For time-rolling enhancement, the audio waveform or spectrum can be randomly shifted along the time axis (e.g., shifted within the range of [-4000, +4000] sample points) to simulate the random offset at the start of an audio segment, forcing the model to focus on the essential spectral features of the sound event rather than its absolute position on the time axis. For device impulse response convolution enhancement, an impulse response sequence corresponding to the vehicle microphone can be selected to perform channel-by-channel FFT-convolve convolution on the audio (e.g., setting the activation probability p to 0.6). The system simulates the acoustic filtering characteristics of different recording devices. For frequency domain statistical enhancement, the Mel band mean, variance, or energy distribution of two samples can be linearly mixed (e.g., activation probability of 0.4, mixing weights following a Beta(0.3,0.3) distribution) to implicitly augment the domain distribution of the training data without changing the semantics of the audio content. For frequency random perturbation enhancement, random perturbations can be applied to the passband range of the Mel filter in each training iteration (e.g., f_min randomly increases from 0 to f_min_aug_range, f_max randomly increases or decreases within the range of the base value ± f_max_aug_range / 2) to simulate the frequency response differences of different microphone hardware. These enhancement methods can be used individually or in combination according to preset probabilities.

[0120] This processing method allows for richer variations in amplitude, frequency distribution, temporal location, and device response characteristics in the training samples, thus exposing the causal shared backbone network to diverse acoustic inputs that more closely resemble real-world in-vehicle environments during the training phase. Because the enhanced samples retain the key discriminative structures of vehicle alarm sounds and road contact sounds, dual-task joint training can still stably learn the shared features, which are then fed into the vehicle classification head and road classification head respectively for parameter optimization.

[0121] In this embodiment of the invention, by performing hybrid data augmentation processing on the training sample set, the robustness of the model to complex noise, different acquisition devices, and local frequency missingness is improved, the effective coverage of the training samples is expanded, overfitting is reduced and the generalization ability to unseen environments is enhanced, thereby enabling the subsequent vehicle type recognition results and road surface condition recognition results to maintain more stable output consistency under complex road conditions.

[0122] Figure 3 This is another schematic diagram of the joint acoustic recognition method for emergency vehicles and road conditions provided in an embodiment of the present invention. Figure 3 As shown, a specific implementation of the joint acoustic recognition method for emergency vehicles and road conditions may include the following steps:

[0123] S301 acquires multi-channel ambient audio signals.

[0124] S302 preprocesses the multi-channel environmental audio signal to obtain a multi-channel log-Mel spectrum feature map.

[0125] In some embodiments, preprocessing of the multi-channel environmental audio signal includes: pre-emphasizing the multi-channel environmental audio signal to obtain a pre-emphasized audio signal; performing a causal short-time Fourier transform on the pre-emphasized audio signal to obtain a frequency domain signal; wherein the causal short-time Fourier transform disables center alignment; performing dynamic frequency range Mel filtering on the frequency domain signal to obtain a Mel spectrum; and performing logarithmic compression and linear normalization on the Mel spectrum to obtain a multi-channel logarithmic Mel spectrum feature map.

[0126] S303, input the multi-channel log-Mel spectrum feature map into the entry convolutional layer of the causal shared backbone network for preliminary feature extraction, and obtain an initial feature map with doubled channel number and reduced frequency and time dimensions.

[0127] S304, the initial feature map is sequentially input into three cascaded stages for hierarchical feature extraction to obtain shared features.

[0128] For example, suppose the causal sharing backbone network consists of multiple CMMCAs cascaded in a funnel-shaped three-stage configuration, with each stage containing two CMMCAs. Accordingly, during the multi-scale feature extraction process of inputting the initial feature map into the funnel-shaped three-stage CMMCAs, the first two CMMCAs do not apply channel attention, maintaining the feature map resolution and number of channels, and extracting low-level acoustic features such as edges, textures, and harmonic structures. The second stage initiates multi-scale channel attention, with the first CMMCA downsampling and expanding the number of channels in both the frequency and time dimensions, and the second CMMCA performing refined feature extraction in the expanded channel space. The third stage continues to apply attention, with the first CMMCA downsampling only the frequency dimension while maintaining the time dimension resolution and further expanding the number of channels, and the second CMMCA performing the final high-level semantic abstraction. These three stages form a funnel-shaped pyramid feature hierarchy of "gradual channel expansion and gradual frequency compression."

[0129] S305, frequency feature fusion is performed on the shared features along the frequency dimension to obtain a one-dimensional time series feature sequence.

[0130] For example, adaptive average pooling is performed on the shared features along the frequency dimension to obtain pooled features; grouped cross-frequency convolution is performed on the pooled features to fuse the frequency points across frequencies to obtain cross-frequency fused features; the cross-frequency fused features are then subjected to a Squeeze operation to eliminate the frequency dimension, resulting in a one-dimensional temporal feature sequence; the one-dimensional temporal feature sequence is used as the input to the vehicle classification head and the road classification head.

[0131] S306, input the one-dimensional time-series feature sequence into the parallel vehicle classification head and road classification head respectively to obtain the emergency vehicle type identification result and the road condition identification result.

[0132] For example, a one-dimensional temporal feature sequence is input into a vehicle classification head, which outputs frame-by-frame predicted eight-class classification results for vehicle types. The eight-class classification of vehicle types includes four subcategories: ambulance, fire truck, police car, engineering rescue vehicle, and background noise. The frame-by-frame prediction results of the four police car subcategories are aggregated into the confidence score of the police car category, and the vehicle type identification results for ambulance, fire truck, police car, engineering rescue vehicle, and background noise are output. In addition, a one-dimensional temporal feature sequence is input into a road surface classification head, which outputs frame-by-frame predicted binary classification results for road surface conditions, including dry and wet / slippery conditions.

[0133] Furthermore, to verify the accuracy of the present invention's solution in identifying emergency vehicle types and road conditions, this verification example is tested based on a real-world urban road environment audio dataset containing four types of emergency vehicles (e.g., ambulances (AMB), fire trucks (FIR), police cars (POL), and engineering rescue vehicles (PRO)) and background noise. Each sample is simultaneously labeled with a vehicle type label (eight-class classification) and a road surface wet / dry label (two-class classification), covering various real-world driving scenarios such as urban roads, highways, tunnels, residential areas, and different weather conditions (sunny and rainy days). The model performance is analyzed below from three dimensions: vehicle type recognition, road condition recognition, and overall model performance.

[0134] For example, Table 1 illustrates the performance indicators of the four-channel emergency vehicle identification eight-class classification provided by the embodiments of the present invention. As shown in Table 1, under the four-channel configuration, the overall accuracy of the model's bottom-level eight-class classification reaches 97.38%, and the macroscopic average F1-score is 96.09%, demonstrating excellent classification performance. Among them, the background noise category (BACKGROUND), which has the most significant acoustic features, has particularly outstanding recognition accuracy, with a precision of 100%, a recall rate of 99.93%, and an F1-score of 99.96%, indicating that the model has extremely strong binary discrimination ability for "target present / no target". The recall rate of ambulances is 90.71%, slightly lower than other categories. From the confusion matrix analysis, the main misclassification flow is to the police car subclass POL2, because the dual-tone alternation pattern of some ambulance siren sounds is similar to the acoustic features of the POL2 siren sound. The F1-scores for all four police vehicle subcategories exceeded 95.7%. Slight confusion between categories mainly stemmed from misclassification of different siren acoustic patterns, but the recall rates for each subcategory were all above 95.3%. The recall rate for engineering rescue vehicles was 94.35%. A small number of samples were misclassified as fire trucks or police vehicle subcategory POL3, primarily because the spectral overlap between some engineering vehicle siren sounds and ambulance / siren frequency sweep patterns.

[0135]

[0136] Furthermore, to verify the performance gain of multi-channel compared to single-channel, this invention compared the vehicle recognition accuracy under different microphone channel configurations, and the results are shown in Table 2. The best accuracy for a single channel was 91.02% (channel 3), while the accuracy of the four-channel combined configuration reached 97.38%, an improvement of 6.36 percentage points compared to the best single channel and an improvement of 11.75 percentage points compared to the worst single channel (channel 2, 85.63%). This significant improvement verifies the key role of multi-channel microphone arrays in capturing spatially differentiated acoustic information, namely, microphones at different locations can collect direct and reflected alarm sounds from different directions and intensities. The complementary information between channels, after being fused by a causal sharing backbone network, effectively enhances the acoustic discrimination capability of the model.

[0137]

[0138] Accordingly, the test results for binary classification of road surface conditions (dry and wet) are shown in Table 3. With the four-channel configuration, the overall accuracy of the model reached 97.67%, and the macro-average F1-score was 97.66%. Specifically, the recall rate for dry roads was as high as 99.86%, with a very small number of dry samples being misclassified as slippery, demonstrating the model's near-perfect detection capability for dry roads. The recall rate for slippery roads was 95.76%, with some slippery samples being misclassified as dry. This was mainly because the tire noise characteristics of some slightly damp roads were not significantly different from those of completely dry roads, and the attenuation of high-frequency tire noise components was limited, leading to ambiguity in the classification of boundary samples. The accuracy for slippery roads reached 99.88%, indicating that once the model classifies a road surface as slippery, its confidence level is extremely high, and the risk of false alarms is extremely low. This characteristic is particularly important for autonomous driving safety strategies—false alarms about slippery surfaces only cause the vehicle to adopt unnecessary conservative strategies, which is still within a safe and controllable range; while missed alarms about dry surfaces may lead to insufficient braking distance estimation, directly threatening driving safety.

[0139]

[0140] Correspondingly, Table 4 compares the road condition recognition accuracy under different microphone channel configurations. As shown in Table 4, the optimal accuracy for a single channel is 87.22% (Channel 2), while the four-channel combined configuration reaches 97.67%, an improvement of more than 10 percentage points. It should be noted that the single-channel accuracy of Channel 3 and Channel 4 is only 63.04% and 66.81% respectively, significantly lower than that of Channel 1 (87.09%) and Channel 2 (87.22%). This reflects a significant difference in the acquisition quality of road tire noise signals by microphones installed at different locations. Specifically, microphones at the front / rear positions (Channels 1 and 2) are closer to the tire-road contact area and acquire stronger tire noise signals; while microphones at the left and right rearview mirror positions (Channels 3 and 4) are farther from the tires and are more susceptible to wind noise interference. However, the four-channel combined configuration effectively overcomes the problem of uneven quality in single channels, achieving optimal detection performance through the complementary fusion of multi-channel information.

[0141]

[0142] In summary, the causal shared network dual-branch model in this invention has 215K parameters and a single inference computation of 46.4M MACs (based on 1 second 4-channel audio and 128×30 Mel spectrum input), which fully meets the resource constraints of edge devices and is suitable for lightweight deployment in vehicle-mounted embedded platforms. It balances real-time performance and recognition accuracy under limited computing power conditions.

[0143] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the embodiments of the apparatus of the present invention, please refer to the embodiments of the method of the present invention.

[0144] Figure 4 This is a schematic diagram of the structure of the emergency vehicle and road condition combined acoustic recognition device provided in an embodiment of the present invention. Figure 4 As shown, the emergency vehicle and road condition combined acoustic recognition device 40 includes an acquisition module 41, a preprocessing module 42, a feature extraction module 43, and a recognition module 44.

[0145] Among them, the acquisition module 41 is used to acquire multi-channel environmental audio signals;

[0146] Preprocessing module 42 is used to preprocess the multi-channel environmental audio signal to obtain a multi-channel logarithmic Mel spectrum feature map. The preprocessing includes pre-emphasis processing, causal short-time Fourier transform and Mel filtering.

[0147] Feature extraction module 43 is used to input multi-channel log-Mel spectrum feature maps into the causal shared backbone network for feature extraction to obtain shared features. The causal shared backbone network is composed of multiple CMMCAs cascaded in a funnel-shaped multi-stage manner, with the number of channels of the feature map gradually increasing and the frequency dimension gradually compressing along the cascading direction. All convolutional layers in the causal shared backbone network adopt a zero-padding strategy on the left side of the time axis to meet causal constraints. The shared features are common acoustic representations that are simultaneously related to the sound source features of emergency vehicles and the acoustic features of road conditions.

[0148] The identification module 44 is used to input the shared features into the parallel vehicle classification head and road classification head respectively to obtain the emergency vehicle type identification result and the road condition identification result.

[0149] In one possible implementation, the feature extraction module 43 can be specifically used to: input the multi-channel log-Mel spectrum feature map into the entry convolutional layer of the causal shared backbone network for preliminary feature extraction, obtaining an initial feature map with doubled channel number and reduced frequency and time dimensions; input the initial feature map sequentially into at least three cascaded stages for hierarchical feature extraction, obtaining shared features; in the at least three cascaded stages, the first stage includes CMMCA and does not enable multi-scale channel attention, and the number of channels and resolution of the output feature map remain unchanged, used to extract low-level acoustic features; the middle stages include CMMCA and enable multi-scale channel attention, and the number of channels of the output feature map is increased, while the frequency and time dimensions are reduced, used to extract mid-level semantic features; the last stage includes CMMCA and enables multi-scale channel attention, and the number of channels of the output feature map is increased, the frequency dimension is reduced, while the time dimension remains unchanged, used to extract high-level semantic features.

[0150] In one possible embodiment, CMMCA employs an inverse residual bottleneck structure. The feature extraction module 43 can also be used to: expand the channels of the input features using an up-dimensional convolutional layer to obtain up-dimensional features; perform convolution operations on the up-dimensional features using a causal-depth-separable convolutional layer to obtain convolution-processed features; wherein the convolution operation includes: symmetrical padding only in the frequency dimension and no padding in the time dimension; compress the number of channels of the convolution-processed features to the target output channel number using a dimensionality-reduction projection layer to obtain dimensionality-reduced features; perform adaptive recalibration of the channel dimensions of the dimensionality-reduced features using a multi-scale channel attention layer to obtain attention-recalibrated features; and add the attention-recalibrated features to the original input of CMMCA element-wise using a residual connection layer to obtain the output features of CMMCA.

[0151] In one possible embodiment, the multi-scale channel attention layer includes a global frequency domain pooling branch and a local feature extraction branch; the feature extraction module 43 can also be used to: input the dimensionality-reduced features into the global frequency domain pooling branch and the local feature extraction branch respectively; perform global frequency domain pooling on the input feature map along the frequency dimension through the global frequency domain pooling branch, compress the frequency information into a single value, and then perform a bottleneck transformation to generate a global channel description; directly perform a bottleneck transformation on the input feature map through the local feature extraction branch to generate a local channel description; add the global channel description and the local channel description element by element and then normalize to obtain a channel attention weight map; multiply the channel attention weight map and the dimensionality-reduced features element by element to obtain the attention-recalibrated features.

[0152] In one possible embodiment, before inputting the shared features into the parallel vehicle classification head and road classification head respectively, the feature extraction module 43 can also be used to: perform adaptive average pooling on the shared features along the frequency dimension to obtain pooled features; perform grouped cross-frequency convolution on the pooled features to fuse each frequency point across frequencies to obtain cross-frequency fused features; eliminate the frequency dimension of the cross-frequency fused features through a Squeeze operation to obtain a one-dimensional temporal feature sequence; and use the one-dimensional temporal feature sequence as input to the vehicle classification head and road classification head.

[0153] In one possible embodiment, the identification module 44 may be specifically used to: input a one-dimensional temporal feature sequence into a vehicle classification head, and output a frame-by-frame predicted eight-class classification result of vehicle type, wherein the eight-class classification of vehicle type includes four subclasses: ambulance, fire truck, police car, engineering rescue vehicle, and background noise; aggregate the frame-by-frame prediction results of the four subclasses of police car into the confidence score of the police car category, and output the vehicle type identification results of ambulance, fire truck, police car, engineering rescue vehicle, and background noise; and input a one-dimensional temporal feature sequence into a road surface classification head, and output a frame-by-frame predicted two-class classification result of road surface condition, wherein the two-class classification result of road surface condition includes dry and slippery.

[0154] In one possible embodiment, both the vehicle classification head and the road classification head are causal dilated convolutional classification heads.

[0155] In one possible embodiment, the preprocessing module 42 may be specifically used to: pre-emphasize the multi-channel environmental audio signal to obtain a pre-emphasized audio signal; perform a causal short-time Fourier transform on the pre-emphasized audio signal to obtain a frequency domain signal; wherein the causal short-time Fourier transform disables center alignment; perform dynamic frequency range Mel filtering on the frequency domain signal to obtain a Mel spectrum; and perform logarithmic compression and linear normalization on the Mel spectrum to obtain a multi-channel logarithmic Mel spectrum feature map.

[0156] In one possible embodiment, the causal shared backbone network, vehicle classification head, and road classification head are obtained through dual-task joint training. The dual-task joint training includes: acquiring a training sample set, where each sample contains multi-channel audio signals, vehicle type labels, and road state labels; inputting the training sample set into the causal shared backbone network, vehicle classification head, and road classification head for forward propagation; using a combination of vehicle recognition loss and road detection loss as the total loss function, and optimizing all parameters of the causal shared backbone network, vehicle classification head, and road classification head simultaneously through backpropagation based on the total loss function.

[0157] In one possible embodiment, the combination of vehicle identification loss and road surface detection loss is a weighted combination; the total loss function is the sum of vehicle identification loss and road surface detection loss weighted by road surface loss weighting coefficient.

[0158] In one possible implementation, both vehicle recognition loss and road surface detection loss are binary cross-entropy losses.

[0159] In one possible embodiment, before inputting the training sample set into the causal shared backbone network, the vehicle classification head, and the road classification head for dual-task joint training, the preprocessing module 42 can also be used to: perform hybrid data augmentation processing on the training sample set, wherein the hybrid data augmentation includes at least one of spectral masking enhancement, signal-to-noise ratio perturbation superposition, time rolling enhancement, device impulse response convolution enhancement, frequency domain statistics hybrid enhancement, and frequency random perturbation enhancement.

[0160] The acquisition module collects multi-channel environmental audio signals from the vehicle-mounted microphone array, preserving acoustic differences under different orientations and propagation paths, thus enhancing the characterization of alarm sounds, tire noise, and road contact sounds. The preprocessing module converts the raw audio into a multi-channel log-Melogram spectral feature map, compressing the data size while retaining key time-frequency information, making it easier for subsequent networks to extract stable discriminative features. The feature extraction module employs a causal shared backbone network, and all convolutional layers use a left-side zero-padding strategy on the time axis, ensuring that the current output depends only on current and historical information, enabling the device to meet the requirements of real-time online inference in vehicles. Furthermore, by inputting the shared features into parallel vehicle and road classification heads, simultaneous identification of emergency vehicle types and road conditions is achieved, reducing redundant feature extraction caused by separate modeling and balancing computational efficiency, recognition accuracy, and stability under complex conditions.

[0161] The emergency vehicle and road condition joint acoustic recognition device provided in this embodiment of the invention can be used to execute the method steps of the above method embodiment. The specific implementation and technical effects are similar, and will not be repeated here.

[0162] This invention also provides an acoustic recognition system for emergency vehicles and road conditions, comprising:

[0163] The vehicle microphone array is deployed in the front and rear bumpers and left and right rearview mirrors of the vehicle to collect multi-channel environmental audio signals.

[0164] The on-board computing unit is used to execute the joint acoustic recognition method for emergency vehicles and road conditions as provided in the above embodiments.

[0165] The vehicle-mounted microphone array is deployed in the front bumper, rear bumper, and behind the left and right rearview mirrors, covering ambient sound sources in all directions and acquiring multi-channel ambient audio signals with spatial differences. The controller executes a joint acoustic recognition method based on the acquired multi-channel audio, simultaneously analyzing alarm sounds, tire noise, and road contact sounds. Because the onboard computing unit performs multi-channel preprocessing, shared feature extraction, and parallel classification directly on the vehicle, emergency vehicle type identification and road condition identification can be output in real time under a unified architecture, reducing the computational cost of repetitive feature extraction. Therefore, in complex scenarios such as urban roads, highways, tunnels, and rain / snow, this system can improve the timeliness, stability, and accuracy of environmental perception, providing more timely acoustic evidence for vehicle avoidance decisions and vehicle control.

[0166] For example, Figure 5 This is a schematic diagram of the overall architecture of the joint acoustic recognition system for emergency vehicles and road conditions provided in an embodiment of the present invention. Figure 5As shown, the multi-channel environmental audio signal, after preprocessing, yields a four-channel log-Mel spectrum feature map, which serves as the input to the causal shared backbone network. This feature map first undergoes preliminary feature extraction through a causal Stem convolutional layer, achieving channel expansion and reduction of frequency and time dimensions while satisfying causal constraints. Subsequently, it is sequentially input into three cascaded CMMCA stages for hierarchical feature extraction. The first stage is an information-preserving layer, containing two CMMCAs without multi-scale channel attention, maintaining the feature map resolution and channel number unchanged, used to extract low-level acoustic detail features. The second stage is a time-frequency synchronization compression layer, containing two CMMCAs with multi-scale channel attention, synchronously downsampling the frequency and time dimensions and expanding the channel number, used to extract mid-level semantic features. The third stage is a frequency depth compression layer, containing two CMMCAs with multi-scale channel attention, compressing only the frequency dimension while maintaining the time dimension resolution unchanged, used to extract high-level semantic features. These three stages together constitute a funnel-shaped pyramid feature hierarchy of "gradual channel expansion and gradual frequency compression," enabling the network to complete cross-scale acoustic modeling while maintaining causality. After the three CMMCA stages are stacked, the feature map is processed by a frequency feature fusion layer. Adaptive average pooling is used to compress the frequency dimension to representative frequencies, grouped cross-frequency convolution is used to fuse these frequencies, and a squeeze operation is performed to eliminate the frequency dimension. Finally, the two-dimensional time-frequency feature map is compressed into a one-dimensional time-series feature sequence, which is then fed into parallel vehicle and road classification heads. The vehicle classification head outputs the bottom-level eight-class frame-by-frame prediction results (including four subclasses: ambulance (AMB), fire truck (FIR), police car (POL1-POL4), engineering rescue vehicle (PRO), and background noise (BACKGROUND)). The four police car subclasses are then fused using mean fusion, ultimately outputting five-class recognition results: ambulance, fire truck, police car, engineering rescue vehicle, and background noise. The road classification head outputs two-class frame-by-frame prediction results for dry and wet conditions.

[0167] Figure 6 This is a schematic diagram of the structure of an on-board computing unit provided in an embodiment of the present invention. Figure 6 As shown, the on-board computing unit 60 includes at least one processor 61 and a memory 62.

[0168] Optionally, the on-board computing unit 60 also includes a communication component 63. The processor 61, memory 62, and communication component 63 are connected via a bus 64.

[0169] In a specific implementation, at least one processor 61 executes computer execution instructions stored in memory 62, causing at least one processor 61 to perform the above-described method.

[0170] The specific implementation process of processor 61 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0171] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0172] The memory may include random access memory (RAM) and non-volatile memory (NVM), such as at least one disk storage device.

[0173] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0174] This onboard computing unit, through the collaborative execution of computer instructions by the processor and memory, can perform preprocessing of multi-channel environmental audio, construction of log-Mel spectrum, extraction of causal shared backbone network features, and parallel classification output on the onboard platform, thereby unifying emergency vehicle identification and road condition identification within the same control architecture. Because the shared backbone network reuses underlying acoustic features, redundant computations are reduced, resulting in lower onboard computing power consumption. Simultaneously, causal constraints avoid reliance on future frame information, thus better adapting to the real-time inference needs of vehicle operation. Combined with parallel vehicle and road classification heads, it can simultaneously output stable identification results in complex road conditions, occlusion, low light, and rain / snow scenarios to assist driving decisions and vehicle control.

[0175] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0176] This invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0177] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0178] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0179] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0180] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0182] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0184] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for joint acoustic recognition of emergency vehicles and road conditions, characterized in that, include: Acquire multi-channel environmental audio signals; The multi-channel environmental audio signal is preprocessed to obtain a multi-channel logarithmic Mel spectrum feature map. The preprocessing includes pre-emphasis processing, causal short-time Fourier transform, and Mel filtering. The multi-channel log-Melogram spectral feature map is input into a causal shared backbone network for feature extraction to obtain shared features. The causal shared backbone network consists of multiple causal multi-scale multi-channel attention modules (CMMCA) cascaded in a funnel-shaped multi-stage manner, with the number of channels in the feature map gradually increasing and the frequency dimension gradually decreasing along the cascading direction. All convolutional layers in the causal shared backbone network employ a left-side zero-padding strategy on the time axis to satisfy causal constraints. The shared features are common acoustic representations that are simultaneously related to the sound source features of emergency vehicles and the acoustic features of road conditions. The shared features are input into the parallel vehicle classification head and road classification head respectively to obtain the emergency vehicle type identification result and the road condition identification result.

2. The method for joint acoustic recognition of emergency vehicles and road conditions according to claim 1, characterized in that, The step of inputting the multi-channel log-Mel spectrum feature map into the causal shared backbone network for feature extraction includes: The multi-channel log-Mel spectrum feature map is input into the entry convolutional layer of the causal shared backbone network for preliminary feature extraction, resulting in an initial feature map with doubled channel number and reduced frequency and time dimensions. The initial feature map is sequentially input into at least three cascaded stages for hierarchical feature extraction to obtain shared features; Of the at least three cascaded stages, the first stage includes CMMCA and does not enable multi-scale channel attention, and the number of channels and resolution of the output feature map remain unchanged, which is used to extract low-level acoustic features. The intermediate stages include CMMCA and enable multi-scale channel attention, which increases the number of channels in the output feature map and reduces the frequency and time dimensions to extract mid-level semantic features. The final stage includes CMMCA and enables multi-scale channel attention, which increases the number of channels in the output feature map, reduces the frequency dimension, and keeps the time dimension unchanged, in order to extract high-level semantic features.

3. The method for joint acoustic recognition of emergency vehicles and road conditions according to claim 2, characterized in that, The CMMCA employs an inverted residual bottleneck structure and processes the input features in the following manner: The input features are channel-expanded using a dimension-up convolutional layer to obtain the dimension-up features. The causal depth separable convolutional layer is used to perform convolution operations on the up-dimensional features to obtain convolution-processed features; wherein, the convolution operation includes: symmetrical padding only in the frequency dimension and no padding in the time dimension; The number of channels of the convolutional features is compressed to the target number of output channels using a dimension reduction projection layer to obtain the dimension-reduced features; The multi-scale channel attention layer is used to adaptively recalibrate the channel dimensions of the dimensionality-reduced features to obtain the attention-recalibrated features. The attention-recalibrated features are added element-wise to the original input of the CMMCA using a residual connection layer to obtain the output features of the CMMCA.

4. The method for joint acoustic recognition of emergency vehicles and road conditions according to claim 3, characterized in that, The multi-scale channel attention layer includes a global frequency domain pooling branch and a local feature extraction branch; The adaptive recalibration of the channel dimensions of the dimensionality-reduced features using a multi-scale channel attention layer includes: The dimensionality-reduced features are respectively input into the global frequency domain pooling branch and the local feature extraction branch; The input feature map is globally pooled along the frequency dimension through the global frequency pooling branch. After the frequency information is compressed into a single value, it is transformed by the bottleneck to generate a global channel description. The local feature extraction branch directly transforms the input feature map through a bottleneck transformation to generate a local channel description. The global channel description and the local channel description are added element by element and then normalized to obtain the channel attention weight map; The channel attention weight map is multiplied element-wise with the dimensionality-reduced features to obtain the attention-recalibrated features.

5. The method for joint acoustic recognition of emergency vehicles and road conditions according to any one of claims 1 to 4, characterized in that, Before inputting the shared features into the parallel vehicle classification head and road classification head respectively, the method further includes: The shared features are subjected to adaptive average pooling along the frequency dimension to obtain the pooled features; The pooled features are grouped and cross-frequency convolutions are performed to fuse the frequency points across frequencies, resulting in cross-frequency fused features. The frequency dimension of the cross-frequency fused features is eliminated by a squeezing operation to obtain a one-dimensional time-series feature sequence. The one-dimensional temporal feature sequence is used as the input to the vehicle classification head and the road classification head.

6. The joint acoustic recognition method for emergency vehicles and road conditions according to claim 5, characterized in that, The shared features are input into the parallel vehicle classification head and road surface classification head, respectively, to obtain emergency vehicle type identification results and road surface condition identification results, including: The one-dimensional temporal feature sequence is input into the vehicle classification head, and the vehicle classification head outputs the frame-by-frame predicted vehicle type bottom-level eight classification results. The bottom-level eight classification includes four subclasses: ambulance, fire truck, police car, engineering rescue vehicle, and background noise. The frame-by-frame prediction results of the four subclasses of police cars are aggregated into the confidence score of the major category of police cars, and the vehicle type recognition results of ambulance, fire truck, police car, engineering rescue vehicle and background noise are output. Furthermore, the one-dimensional temporal feature sequence is input into the road surface classification head, and the road surface classification head outputs a frame-by-frame predicted binary classification result of the road surface state, which includes dry and slippery conditions.

7. The method for joint acoustic recognition of emergency vehicles and road conditions according to claim 6, characterized in that, Both the vehicle classification head and the road classification head are causal extended convolutional integral classification heads.

8. The method for joint acoustic recognition of emergency vehicles and road conditions according to any one of claims 1 to 4, characterized in that, The preprocessing of the multi-channel environmental audio signal includes: The multi-channel environmental audio signal is pre-emphasized to obtain a pre-emphasized audio signal; A causal short-time Fourier transform is performed on the pre-emphasized audio signal to obtain a frequency domain signal; wherein, the causal short-time Fourier transform disables center alignment. The frequency domain signal is subjected to dynamic frequency range Mel filtering to obtain the Mel spectrum; The Mel spectrum is subjected to logarithmic compression and linear normalization to obtain a multi-channel logarithmic Mel spectrum feature map.

9. The method for joint acoustic recognition of emergency vehicles and road conditions according to any one of claims 1 to 4, characterized in that, The causal sharing backbone network, the vehicle classification head, and the road classification head are obtained through dual-task joint training. The dual-task joint training includes: Obtain a training sample set, wherein each sample in the training sample set contains a multi-channel audio signal, a vehicle type label, and a road surface condition label; The training sample set is input into the causal shared backbone network, the vehicle classification head, and the road classification head for forward propagation; The combination of vehicle recognition loss and road surface detection loss is used as the total loss function, and all parameters of the causal shared backbone network, the vehicle classification head, and the road surface classification head are simultaneously optimized through backpropagation based on the total loss function.

10. The method for joint acoustic recognition of emergency vehicles and road conditions according to claim 9, characterized in that, The combination of vehicle recognition loss and road detection loss is a weighted combination; the total loss function is the sum of vehicle recognition loss and road detection loss weighted by road loss weighting coefficient.

11. The method for joint acoustic recognition of emergency vehicles and road conditions according to claim 9, characterized in that, Both the vehicle recognition loss and the road surface detection loss are binary cross-entropy losses.

12. The joint acoustic recognition method for emergency vehicles and road conditions according to claim 9, characterized in that, Before inputting the training sample set into the causal shared backbone network, the vehicle classification head, and the road classification head for dual-task joint training, the method further includes: The training sample set is subjected to hybrid data augmentation processing, which includes at least one of spectral masking enhancement, signal-to-noise ratio perturbation superposition, time-rolling enhancement, device impulse response convolution enhancement, frequency domain statistics hybrid enhancement, and frequency random perturbation enhancement.

13. An acoustic recognition device for emergency vehicles and road conditions, characterized in that, include: The acquisition module is used to acquire multi-channel environmental audio signals; The preprocessing module is used to preprocess the multi-channel environmental audio signal to obtain a multi-channel logarithmic Mel spectrum feature map. The preprocessing includes pre-emphasis processing, causal short-time Fourier transform, and Mel filtering. The feature extraction module is used to input the multi-channel log-Mel spectrum feature map into the causal shared backbone network for feature extraction to obtain shared features. The causal shared backbone network is composed of multiple causal multi-scale multi-channel attention modules (CMMCA) cascaded in a funnel-shaped multi-stage manner, with the number of channels in the feature map gradually increasing and the frequency dimension gradually compressing along the cascading direction. All convolutional layers in the causal shared backbone network employ a left-side zero-padding strategy on the time axis to satisfy causal constraints. The shared features are common acoustic representations that are simultaneously related to the sound source features of emergency vehicles and the acoustic features of road conditions. The identification module is used to input the shared features into the parallel vehicle classification head and road classification head respectively to obtain the emergency vehicle type identification result and the road condition identification result.

14. The emergency vehicle and road condition combined acoustic recognition device according to claim 13, characterized in that, The feature extraction module can be specifically used for: The multi-channel log-Mel spectrum feature map is input into the entry convolutional layer of the causal shared backbone network for preliminary feature extraction, resulting in an initial feature map with doubled channel number and reduced frequency and time dimensions. The initial feature map is sequentially input into at least three cascaded stages for hierarchical feature extraction to obtain shared features; Of the at least three cascaded stages, the first stage includes CMMCA and does not enable multi-scale channel attention, and the number of channels and resolution of the output feature map remain unchanged, which is used to extract low-level acoustic features. The intermediate stages include CMMCA and enable multi-scale channel attention, which increases the number of channels in the output feature map and reduces the frequency and time dimensions to extract mid-level semantic features. The final stage includes CMMCA and enables multi-scale channel attention, which increases the number of channels in the output feature map, reduces the frequency dimension, and keeps the time dimension unchanged, in order to extract high-level semantic features.

15. The emergency vehicle and road condition combined acoustic recognition device according to claim 14, characterized in that, The CMMCA employs an inverted residual bottleneck structure, and the feature extraction module can also be used for: The input features are expanded using a dimension-up convolutional layer to obtain the dimension-up features. The causal depth separable convolutional layer is used to perform convolution operations on the up-dimensional features to obtain convolution-processed features; wherein, the convolution operation includes: symmetrical padding only in the frequency dimension and no padding in the time dimension; The number of channels of the convolutional features is compressed to the target number of output channels using a dimension reduction projection layer to obtain the dimension-reduced features; The multi-scale channel attention layer is used to adaptively recalibrate the channel dimensions of the dimensionality-reduced features to obtain the attention-recalibrated features. The attention-recalibrated features are added element-wise to the original input of the CMMCA using a residual connection layer to obtain the output features of the CMMCA.

16. The emergency vehicle and road condition combined acoustic recognition device according to claim 15, characterized in that, The multi-scale channel attention layer includes a global frequency domain pooling branch and a local feature extraction branch; The feature extraction module can also be used for: The dimensionality-reduced features are respectively input into the global frequency domain pooling branch and the local feature extraction branch; The input feature map is globally pooled along the frequency dimension through the global frequency pooling branch. After the frequency information is compressed into a single value, it is transformed by the bottleneck to generate a global channel description. The local feature extraction branch directly transforms the input feature map through a bottleneck transformation to generate a local channel description. The global channel description and the local channel description are added element by element and then normalized to obtain the channel attention weight map; The channel attention weight map is multiplied element-wise with the dimensionality-reduced features to obtain the attention-recalibrated features.

17. The emergency vehicle and road condition combined acoustic recognition device according to any one of claims 13 to 16, characterized in that, Before inputting the shared features into the parallel vehicle classification head and road classification head respectively, the feature extraction module can also be used for: The shared features are subjected to adaptive average pooling along the frequency dimension to obtain the pooled features; The pooled features are grouped and cross-frequency convolutions are performed to fuse the frequency points across frequencies, resulting in cross-frequency fused features. The frequency dimension of the cross-frequency fused features is eliminated by a squeezing operation to obtain a one-dimensional time-series feature sequence. The one-dimensional temporal feature sequence is used as the input to the vehicle classification head and the road classification head.

18. The emergency vehicle and road condition combined acoustic recognition device according to claim 17, characterized in that, The identification module can be specifically used for: The one-dimensional temporal feature sequence is input into the vehicle classification head, and the vehicle classification head outputs the frame-by-frame predicted vehicle type bottom-level eight classification results. The bottom-level eight classification includes four subclasses: ambulance, fire truck, police car, engineering rescue vehicle, and background noise. The frame-by-frame prediction results of the four subclasses of police cars are aggregated into the confidence score of the major category of police cars, and the vehicle type recognition results of ambulance, fire truck, police car, engineering rescue vehicle and background noise are output. Furthermore, the one-dimensional temporal feature sequence is input into the road surface classification head, and the road surface classification head outputs a frame-by-frame predicted binary classification result of the road surface state, which includes dry and slippery conditions.

19. An acoustic recognition system for emergency vehicles and road conditions, characterized in that, include: The vehicle microphone array is deployed in the front and rear bumpers and left and right rearview mirrors of the vehicle to collect multi-channel environmental audio signals. The on-board computing unit is used to execute the joint acoustic recognition method for emergency vehicles and road conditions as described in any one of claims 1 to 12.

20. An on-board computing unit, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the joint acoustic recognition method for emergency vehicles and road conditions as described in any one of claims 1 to 12.

21. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the joint acoustic recognition method for emergency vehicles and road conditions as described in any one of claims 1 to 12.

22. A computer program product, characterized in that, include: A computer program, when executed by a processor, implements the joint acoustic recognition method for emergency vehicles and road conditions as described in any one of claims 1 to 12.

Images