Sound event positioning and detecting method based on multi-scale adaptive feature fusion

Abstract

The invention discloses a sound event positioning and detecting method based on multi-scale adaptive feature fusion. The method comprises the following steps: preprocessing a to-be-detected multi-channel audio signal, generating a Mel spectrogram, calculating an effective intensity vector, constructing an input feature matrix, inputting the input feature matrix into an encoder for multi-scale feature extraction, and outputting multi-layer spatial features; frequency self-adaptive adjustment is carried out on the multi-layer spatial features through the selective frequency sensing feature fusion module, the global features are smoothed, and boundary details are enhanced. And fusing the shallow layer features and the deep layer features by using a layered multi-scale feature integration module, carrying out time modeling by combining a bidirectional gating circulation unit, and outputting a detection result. According to the method, the problem of detection precision reduction caused by positioning instability and boundary fuzziness under multi-source overlapping sound source interference and noise influence can be solved, the problem of time frequency detail loss caused by feature confusion can be solved, and the positioning and detection performance can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to sound event localization and detection technology, and in particular to a sound event localization and detection method using multi-scale adaptive feature fusion, which is suitable for sound source localization and event recognition in complex acoustic scenes and belongs to the field of artificial intelligence and signal processing technology. Background Art

[0002] Sound Event Localization and Detection (SELD) is a key technology in the field of audio signal analysis, which aims to identify the category and time boundary of sound events and determine their location in space. With the improvement of multi-channel audio acquisition equipment and computer performance, SELD technology has been widely used in smart home, surveillance, autonomous driving, robot hearing and other fields. However, the robustness and generalization ability of existing technologies in complex acoustic scenes still have many shortcomings.

[0003] At present, the mainstream SELD methods can be divided into physical model-based methods and deep learning-based methods. Physical model methods such as Generalized Cross-Correlation with Phase Transform (GCC-PHAT) rely on acoustic principles for positioning and detection. This type of method has high interpretability when dealing with simple sound sources, but it is often difficult to deal with multi-source overlap and strong noise interference in complex scenes.

[0004] With the rapid development of deep learning technology, SELD methods based on neural networks have become a research focus. These methods achieve more accurate modeling of acoustic scenes through data-driven methods combined with end-to-end optimization. For example, SELDNet proposed by Adavanne et al. combines convolutional neural networks with bidirectional long short-term memory networks for the first time to achieve the joint task of sound event detection and direction estimation. However, SELDNet and its subsequent methods still face the following technical challenges:

[0005] 1. Multi-source overlapping problem: When there are multiple sound sources in the same time frame, the fixed receptive field of traditional CNN limits its ability to capture dynamic time-frequency features, resulting in increased error in sound source location estimation.

[0006] 2. Noise interference: Real scenes are often accompanied by strong background noise. Especially under low signal-to-noise ratio conditions, the network is easily affected by high-frequency noise, resulting in reduced positioning and detection performance.

[0007] 3. Boundary fuzziness: The start and end time of a sound event is often difficult to determine accurately due to fuzzy features, especially for short-term, rapidly changing events such as clicks or alarms.

[0008] In addition, existing convolutional neural network architectures usually use pooling and downsampling operations to improve computational efficiency, but these operations inevitably compress the temporal and frequency resolution of features, resulting in the loss of key detail information. This loss of detail is particularly detrimental to the modeling of spatial and temporal features of sound events. To address this problem, some studies have attempted to introduce multi-scale feature fusion and attention mechanisms. For example, multi-level feature pyramid structures are used to enhance the hierarchical expression of time-frequency features, and fusion methods based on multi-channel Mel spectrograms and spatial covariance matrices have also made some progress. However, these methods still have limitations in complex acoustic scenes, mainly manifested in feature confusion and insufficient global perception capabilities: in multi-source overlapping scenes, features of different sound sources may interfere with each other, resulting in reduced feature differentiation; at the same time, most methods lack effective modeling of global time and frequency change patterns, making it difficult to capture rapidly changing high-frequency details. Summary of the invention

[0009] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a sound event localization and detection method with multi-scale adaptive feature fusion. The present invention can solve the problems of interference from multiple overlapping sound sources, positioning instability under the influence of noise, decreased detection accuracy caused by blurred boundaries, and loss of time and frequency details due to feature confusion through multi-module collaborative design and optimization of the training process. It can not only significantly improve the positioning and detection performance, but also effectively reduce the computational complexity, thereby meeting the real-time requirements in practical applications.

[0010] The technical solution of the present invention is achieved in this way:

[0011] A method for localizing and detecting sound events by multi-scale adaptive feature fusion, the steps are as follows:

[0012] S1. Preprocess the multi-channel audio signal to be detected, generate a Mel spectrogram and calculate the effective intensity vector, and construct an input feature matrix;

[0013] S2, input the input feature matrix into the encoder for multi-scale feature extraction, and output multi-layer spatial features;

[0014] S3, frequency adaptively adjusts the multi-layer spatial features output by the encoder through the selective frequency-aware feature fusion module, smoothes the global features and enhances the boundary details, so that the features have higher boundary clarity and feature consistency;

[0015] S4. Use the hierarchical multi-scale feature integration module to fuse shallow features with deep features, and combine the bidirectional gated recurrent unit for temporal modeling to output the detection results.

[0016] Furthermore, step S1 is specifically processed as follows:

[0017] First, the multi-channel audio signal to be detected is preprocessed, including short-time Fourier transform to obtain the audio spectrum, and the appropriate window size and step size are selected; then, the spectrum is converted into a logarithmic Mel spectrum to capture the characteristics of frequency distribution perceived by the human ear; at the same time, the effective intensity vector is calculated, which is obtained by taking the real part of the product of the omnidirectional signal and the directional signal and spatially averaging; finally, the Mel spectrum and the effective intensity vector are spliced ​​to obtain the input feature matrix.

[0018] Furthermore, in step S2, multi-scale feature extraction is specifically as follows:

[0019] In the encoder, multi-layer convolution operations are used to gradually downsample the input feature matrix to extract multi-scale features in the time and frequency dimensions; each layer of the encoder captures spatial features of different scales through the sliding operation of the convolution kernel; the time and frequency dimensions are further compressed through pooling operations to improve the computational efficiency of the features; finally, the encoder outputs multi-layer nested features, which are the multi-layer spatial features.

[0020] Further, in step S3, frequency adaptive adjustment is performed as follows:

[0021] In the selective frequency perception feature fusion module, a low-pass filter and a high-pass filter are designed to dynamically and adaptively adjust the multi-layer spatial features output by the encoder; the low-pass filter adaptively generates a low-pass filter parameter matrix according to the frequency distribution to smooth the global low-frequency features and suppress high-frequency noise interference; the high-pass filter adaptively restores high-frequency details through filter inversion operation; through the processing of the low-pass filter and the high-pass filter, the output features ultimately have higher boundary clarity and feature consistency.

[0022] Furthermore, the multi-layer spatial features include and The high-level features composed of and The shallow features composed of the selective frequency perception feature fusion modules are the same as the two, one of which is a selective frequency perception feature fusion module with the same and Connection to extract global semantic information from high-level features, the output is Another selective frequency-aware feature fusion module and feature and The connection is used to preserve and optimize the local details of shallow features, ensuring that the boundary features and time-frequency details of sound events can be preserved during the fusion process.

[0023] Further, step S4 is implemented as follows:

[0024] The hierarchical multi-scale feature integration module combines the fine-grained temporal frequency changes captured by shallow features with the global context information extracted by deep features through a feature pyramid structure, and realizes the step-by-step fusion of multi-scale features through up and down sampling operations; at the same time, the hierarchical multi-scale feature integration module introduces a context-aware multi-scale aggregation mechanism, and uses different pooling scales and dilated convolutions to extract global and local features; then, the fused features are input into a bidirectional gated recurrent unit, and the dynamic change characteristics of sound events are captured through the temporal modeling capability of the bidirectional gated recurrent unit, and finally the joint prediction results of the sound event category, temporal boundary and spatial position are output.

[0025] Furthermore, the context-aware multi-scale aggregation mechanism is implemented by a context-aware multi-scale aggregation module, which uses the feature map output by the encoder is the input, where H and W are the spatial dimensions respectively, and C is the number of channels; the context-aware multi-scale aggregation module first scales the feature map to different scales through adaptive average pooling operation, and then upsamples the feature map back to the original spatial dimension to capture multi-scale global features; at the same time, the context-aware multi-scale aggregation module performs spatial convolution operations with different expansion rates on the input feature map to further extract multi-scale context information; finally, the pooled features, convolution features and original input features are merged to form a multi-scale feature expression that integrates global context and fine-grained details.

[0026] Compared with the existing sound event localization and detection technology, the multi-scale adaptive feature fusion model proposed in this invention shows significant advantages in multi-source overlap, strong noise interference and complex acoustic scenes. The beneficial effects include the following aspects:

[0027] (1) Significantly improve positioning accuracy and event boundary detection capabilities: The present invention effectively reduces the interference of high-frequency noise on the boundaries of sound events through the low-frequency smoothing and high-frequency enhancement mechanism of the selective frequency-aware feature fusion module (SFFB), while enhancing the high-frequency feature expression capability of the boundary area. Compared with traditional methods, this module has higher accuracy in detecting the start and end time of sound events, especially in fast-changing short-term events (such as alarm sounds and click sounds).

[0028] (2) Enhanced multi-source overlapping processing capabilities in complex scenes: The Hierarchical Multi-Scale Feature Integration Module (HFI) significantly improves the model's feature differentiation capabilities in multi-source overlapping scenes and reduces feature confusion by integrating shallow fine-grained features with deep global features. This module combines multi-scale feature fusion strategies to simultaneously retain detail information and global characteristics in both time and frequency dimensions, thereby achieving accurate positioning of multiple sound events in the same scene.

[0029] (3) Improve model robustness and generalization ability: The multi-scale feature extraction and fusion strategy designed by the present invention enables the model to adaptively adjust the feature extraction method to adapt to the complexity of different acoustic environments. For example, in a low signal-to-noise ratio environment, the model can reduce noise interference through low-frequency smoothing; in a high-dynamic change scene, the integrity of the detailed features is maintained through high-frequency enhancement. This design enables the model to have good robustness and generalization ability in different environments, and can be effectively applied to sound event detection in indoor monitoring, smart home, and traffic scenes.

[0030] (4) Optimizing computational efficiency and real-time performance: The present invention uses modular design to fully utilize the temporal modeling capabilities of the encoder-decoder structure and the bidirectional gated recurrent unit (BiGRU), significantly reducing computational complexity. While maintaining high positioning and detection accuracy, it supports tasks with high real-time requirements. In addition, the combined design of hierarchical feature extraction and context fusion effectively reduces redundant calculations and improves the overall operating efficiency of the model.

[0031] (5) Practical application potential: The model proposed in this paper can be widely used in a variety of practical scenarios such as smart home, autonomous driving, monitoring system and robot hearing, and has good application prospects. For example, in smart home, it can be used to accurately identify emergency sound events in the home environment (such as fire alarms, glass breaking, etc.); in the field of autonomous driving, it can accurately locate various sound sources in the road environment (such as horns, alarms), providing auxiliary information for vehicle decision-making; in the monitoring system, it can realize real-time detection and positioning of abnormal sound events, improving security capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is the overall architecture diagram of the present invention.

[0033] Figure 2 for Figure 1 Architecture diagram of the CAFFNet module.

[0034] Figure 3 It is a schematic diagram of the selective frequency perception feature fusion module of the present invention.

[0035] Figure 4 Schematic diagram of the context-aware multi-scale aggregation module of the present invention.

[0036] Figure 5 This is a visual comparison chart of the frequency adaptive low-pass filter provided by the present invention and other methods on a data set.

[0037] Figure 6 This is a visual comparison chart of the frequency adaptive high-pass filter provided by the present invention and other methods on a data set.

[0038] Figure 7 This is a visual comparison chart of the detection effects of the method proposed in the present invention and other methods (residual neural network) on three different sound categories (music, woman’s voice and footsteps) on the data set. DETAILED DESCRIPTION

[0039] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] In view of the shortcomings of existing sound event localization and detection (SELD) methods in complex acoustic scenes, this paper aims to propose a new multi-scale adaptive feature fusion model, which solves the following core technical problems through multi-module collaborative design: interference from multiple overlapping sound sources, positioning instability under the influence of noise, decreased detection accuracy caused by blurred boundaries, and loss of time-frequency details caused by feature confusion. The core goal of this invention is to comprehensively improve the robustness and accuracy of the SELD system by optimizing feature expression and fusion strategies.

[0041] Specifically, the present invention proposes an innovative model architecture based on multi-scale feature fusion and frequency perception, referred to as CASE-DLM. By designing a selective frequency-aware feature fusion block (SFFB), the low-frequency smoothing and high-frequency enhancement of features are dynamically adjusted to reduce the interference of high-frequency noise and improve the retention of key boundary features, thereby solving the positioning error and event boundary blurring caused by noise. At the same time, through the hierarchical multi-scale feature integration module (Hierarchical Multi-Scale Feature Integration, HFI), shallow fine-grained features are fused with deep global features to enhance the multi-scale feature expression in time and frequency, effectively alleviating the feature confusion problem in the case of multi-source overlap.

[0042] In addition, the present invention adopts an encoder-decoder structure in the overall model design, and further improves the model's adaptability to complex acoustic environments by combining multi-level feature extraction and context modeling. Combined with the temporal modeling capability of the bidirectional gated recurrent unit (BiGRU), the multi-scale adaptive feature fusion model can accurately capture the dynamic change characteristics of sound events and realize the joint prediction of event categories, temporal boundaries, and spatial positions.

[0043] The present invention is mainly composed of the following modules:

[0044] 1. Overall structure of the model

[0045] The multi-scale adaptive feature fusion model CASE-DLM proposed in the present invention includes the following key modules: a selective frequency-aware feature fusion module (SFFB), a hierarchical multi-scale feature integration module (HFI) and a bidirectional gated recurrent unit (BiGRU) for time modeling. The overall model adopts an encoder-decoder framework, with the input being the processed multi-channel audio signal and the output being the category, time boundary and spatial position of the sound event. Encoder part: a multi-layer convolution operation is used to extract features from the input Mel spectrogram and effective intensity vector (Effective Intensity Vector, EIV), gradually compress the time and frequency dimensions, and extract multi-scale spatial features. After feature extraction, the multi-layer features output by the encoder are sent to the SFFB module for frequency adaptive processing. In the architecture of the present invention, the SFFB module is applied twice, acting on high-level features ( and ) and shallow features ( and ) to achieve feature enhancement and optimization. In the first application, SFFB extracts global semantic information from high-level features, especially for low-resolution features (such as The size is H / 8×W / 8×8C) to enhance the overall modeling ability of the network for complex acoustic environments. At the same time, medium-resolution features (such as The size is H / 4×W / 4×4C) and its semantic expression is further enriched by enhancement. In the second application, SFFB retains and optimizes local details of shallow features, and by size H×W×C) and medium and high resolution features (such as The details of the sound event are enhanced by the HFI module (size is H / 2×W / 2×2C) to ensure that the boundary features and time frequency details of the sound event can be preserved during the fusion process. The two applications effectively coordinate the expression capabilities of shallow and high-level features through complementary design, providing comprehensive semantic support and fine feature expression for the detection and positioning of sound events, thereby significantly improving the overall detection and positioning accuracy of the model. Decoder part: The features extracted by the encoder are reconstructed by combining the HFI module and the bidirectional GRU. The multi-scale features are cross-modally enhanced and fused through CMA to improve the effectiveness and global consistency of feature expression. The fused features are sequentially input into three cascaded CSP modules, which combine feature segmentation, feature fusion and layer-by-layer processing. Through the collaborative work of the backbone path and the direct path (the backbone path refers to the processing path of the input features after deep convolution operations and nonlinear activation functions; the direct path refers to the retention of original information), the spatial resolution of the feature map is gradually restored while maintaining computational efficiency. Finally, the upsampled multi-scale features are spliced ​​in the channel dimension to generate a multi-scale fused feature map with rich semantic information. Subsequently, the feature map is passed to the bidirectional gated recurrent unit, which further extracts temporal features by capturing the temporal correlation of the sequence data to provide support for subsequent task outputs. In the task output module, the network branch simultaneously completes two tasks: sound event detection is used to identify different sound event categories in the input signal, while sound direction localization is responsible for predicting the spatial direction information of the sound signal, including the angle or position coordinates on the x, y, and z axes, thereby achieving comprehensive detection and positioning of sound events. The HFI module integrates shallow fine-grained features and deep global features to output the category, boundary, and location of the sound event. See the overall architecture of the model CASE-DLM. Figure 1 .

[0046] 2. Selective frequency-aware feature fusion module (SFFB) structure

[0047] SFFB module such as Figure 3 As shown in the figure, the frequency adaptive low-pass and high-pass filters (FLPB and FHPB) are used to dynamically adjust the different frequency characteristics. The SFFB module can be described as follows:

[0048]

[0049] In this case, X l ∈R C×2H×2W and Y l ∈R C×H×W They represent high-resolution feature maps from the shallow layer of the backbone network and low-resolution feature maps from the deep layer of the backbone network. FLPB and FHPB are designed to address challenges such as intra-class inconsistency and boundary ambiguity, while F UP Refers to the bilinear interpolation operation used for upsampling.

[0050] FLPB adaptively generates a low-pass filter parameter matrix based on the frequency distribution of the input features, and suppresses high-frequency noise interference by smoothing the global low-frequency features. This module uses convolution operations to generate a weight matrix, and combines the spatial attention mechanism to dynamically adjust the smoothing degree for different areas of the Mel spectrogram, thereby effectively improving the robustness of the model in a noisy environment.

[0051] The FLPB module first compresses and fuses the features Z l As input, a spatially adaptive low-pass filter is generated through an adaptive filtering mechanism. Specifically, this module uses a 3×3 convolutional layer to extract preliminary features, and then forms an adaptive filter matrix by ensuring that the filter weights are positive and their sum is 1 through a softmax normalization process. The filter weight calculation formula is as follows:

[0052] V l =Conv3×3(Z l )

[0053]

[0054] Among them, i,j represents the spatial index of the output feature map, represents the spatially varying filter weight, and K represents the kernel size of the low-pass filter. Contains K×K filters at each spatial position, forming an adaptive low-pass filter matrix Adaptive smoothing can be performed on the spectrogram area, effectively alleviating the boundary blur problem. Next, the FLPB module reconstructs and amplifies higher-level features Yl∈R through sub-pixel upsampling technology. C×H×W Specifically, the low-pass filter Applied to the spectrogram region, generating spatially adaptive weights for smoothing and upsampling. The upsampling operation directly reconstructs and rearranges the filtered features so that their spatial dimensions align with the target resolution, resulting in the upsampled features. This process enhances the time and frequency resolution while maintaining boundary sharpness without dividing the filters into multiple independent groups. The upsampled features are calculated as follows:

[0055]

[0056]

[0057] At the same time, FHPB adaptively restores high-frequency details, especially boundary feature information, through filter inversion operations during feature reconstruction. This module can extract high-frequency features lost during downsampling, and strengthen the feature expression of sound event boundaries through high-frequency enhancement technology, significantly improving the accuracy and stability of boundary detection. The design of the SFFB module achieves a dynamic balance between low-frequency noise suppression and high-frequency detail recovery, providing the model with more accurate feature representation capabilities.

[0058] The FHPB module uses the compressed and fused features As input, a series of high-pass filtering operations are performed to restore high-frequency detail information. The core steps include a 3×3 convolution layer, a softmax layer, and a filter inversion operation. The specific process is as follows:

[0059]

[0060] in, represents the initial weight matrix of the high-pass filter, and E is the unit filter. When the filter kernel size K = 3, its weight is set to E = [[0,0,0], [0,1,0], [0,0,0]] to ensure that high-frequency information is preserved. The high-pass filter generates a low-pass kernel through the softmax operation, and then generates the high-pass filter kernel by subtracting the low-pass kernel from the unit filter E. Finally, the FHPB module applies the generated high-pass filter to the input feature X l , to restore the high-frequency details in the spectrogram, the formula is as follows:

[0061]

[0062] 3. Hierarchical Multi-Scale Feature Integration Module (HFI) Structure

[0063] HFI modules such as Figure 2As shown in the figure, the feature confusion problem in complex acoustic scenes is solved through the multi-scale feature fusion strategy, and the resolution of time and frequency dimensions is significantly enhanced. The module adopts a feature pyramid structure and combines the complementary advantages of shallow and deep features. Shallow features are used to capture fine-grained time and frequency changes, while deep features extract global context information. Through layer-by-layer feature upsampling and downsampling operations, the HFI module realizes the step-by-step fusion of shallow detail features and deep global features, significantly improving the feature expression ability. At the same time, the HFI module introduces the context-aware multi-scale aggregation (CMA) mechanism, combined with adaptive average pooling and spatial convolution operations, and enhances the model's perception of complex time and frequency patterns through multi-scale feature extraction. The CMA module uses multiple expansion rates (such as 3, 6, 9) and different pooling scales (such as 1, 3) to achieve a balance between the context aggregation of global features and the retention of local details, thereby effectively capturing multi-scale information and further improving the model's feature modeling capabilities in multi-source overlapping and rapidly changing scenes. Through the above design, the HFI module can significantly alleviate the feature confusion problem and provide a more accurate and robust feature expression for the localization and detection of sound events in complex acoustic scenes.

[0064] 4. Context-aware multi-scale aggregation module

[0065] CMA modules such as Figure 4 As shown in the figure, the context-aware multi-scale aggregation module (CMA) in the present invention enhances the model's perception of global features by expanding the receptive field, effectively solving the problem of key fine-grained feature loss caused by the reduction of time and frequency resolution in the downsampling process of the traditional convolutional model. This module realizes multi-scale feature extraction by combining adaptive average pooling and spatial convolution operations, which can not only enhance the perception of global context information, but also retain the fine-grained features on the time and frequency axis.

[0066] Specifically, the CMA module uses the feature map output by the encoder is the input, where H and W are the spatial dimensions and C is the number of channels. The module first scales the feature map to different scales through adaptive average pooling, and then upsamples it back to the original spatial dimension (H×W) to capture multi-scale global features. At the same time, the module performs spatial convolution operations with different expansion rates (such as 3, 6, 9) on the input feature map to further extract multi-scale contextual information. The pooled features and convolutional features are finally merged with the original input features to form a multi-scale feature expression that integrates global context and fine-grained details. The CMA calculation formula is as follows:

[0067]

[0068]

[0069]

[0070] In this case, Conv 1×1 (·) and Conv 3×3 Represent 1×1 convolution and 3×3 dilated convolution respectively. U, and δ represent upsampling operation, batch normalization and ReLU activation function respectively. Concat and APool represent the combination of multi-scale features and adaptive global pooling operations at different scales respectively.

[0071] Based on the above module introduction, the detection process of the present invention is as follows: (taking RGB face video image as an example)

[0072] S1: Preprocess the multi-channel audio signal to be detected, generate Mel spectrogram and EIV, and construct the input feature matrix.

[0073] S2: Input feature matrix into encoder for multi-scale feature extraction and output multi-layer spatial features.

[0074] S3: The SFFB module is used to adaptively adjust the frequency of multi-layer features, smooth global features and enhance boundary details.

[0075] S4: Use the HFI module to fuse shallow and deep features, and combine it with the bidirectional GRU for temporal modeling to output the detection results.

[0076] The preprocessing of the audio signal in S1 specifically includes the following steps: The preprocessing object here, in addition to the audio signal to be processed, also includes sample data for model training. First, the multi-channel audio data is preprocessed, including short-time Fourier transform (STFT) to obtain the audio spectrum, and select an appropriate window size and step size (such as 0.04 second window, 0.02 second step). Subsequently, the spectrum is converted into a logarithmic Mel spectrum to capture the characteristics of the human ear's perception of frequency distribution. At the same time, the effective intensity vector (EIV) is calculated. The vector is obtained by taking the real part of the product of the omnidirectional signal and the directional signal and spatially averaging it, which enhances the spatial feature representation. Finally, the Mel spectrogram (4 channels) and the EIV (3 channels) are spliced ​​into a 7-channel input feature matrix and input into the model for subsequent processing.

[0077] The multi-scale feature extraction method of the encoder module in S2 is as follows: In the encoder part, multi-layer convolution operations are used to gradually downsample the input feature matrix to extract multi-scale features in time and frequency dimensions. Each layer of the encoder captures spatial features of different scales through the sliding operation of the convolution kernel; the pooling operation further compresses the time and frequency dimensions to improve the computational efficiency of the features. Finally, the encoder outputs multi-layer nested spatial features, laying the foundation for subsequent frequency adaptive adjustment.

[0078] The design principle and implementation of the selective frequency-aware feature fusion module in S3 are as follows: In the SFFB module, the multi-layer features output by the encoder are dynamically adjusted through the frequency-adaptive low-pass filter (FLPB) and high-pass filter (FHPB). FLPB generates a low-pass filter parameter matrix based on the frequency distribution, smoothes the global low-frequency features, and suppresses high-frequency noise interference; FHPB restores high-frequency details, especially the boundary information of sound events, through filter inversion operations. The features processed by the SFFB module have higher boundary clarity and feature consistency, providing support for positioning and detection in complex acoustic scenes.

[0079] The design principle and implementation of the hierarchical multi-scale feature integration module in S4 are as follows: The HFI module combines the temporal frequency details captured by shallow features with the global context information of deep features through a feature pyramid structure. This module realizes the step-by-step fusion of multi-scale features through up- and down-sampling operations, and extracts global and local features using different pooling scales and dilated convolutions through the context-aware multi-scale aggregation (CMA) mechanism. Subsequently, the fused features are input to the bidirectional GRU module, which captures the dynamic characteristics of sound events through its temporal modeling capabilities, and finally outputs the joint prediction results of event categories, temporal boundaries, and spatial positions.

[0080] Before the model is used in actual detection, it needs to be trained using existing data sets. Through training, the model parameters are continuously optimized so that the model meets the designed performance requirements, including evaluating the positioning error, classification accuracy, and boundary detection performance, and finally completing the model structure.

[0081] Specifically, the present invention uses a joint loss function to optimize parameters during the model training phase. The loss function includes classification loss (cross entropy loss), positioning loss (mean square error) and boundary detection loss (based on F 20° After training, the model performance is evaluated in multiple dimensions, including positioning error (LE), positioning recall rate (LR CD ), error rate (ER), and event boundary detection performance (F 20°). The SELD score is a comprehensive metric that combines all four metrics to provide a single indicator of overall system performance. After the evaluation metrics show the superiority of the model in complex acoustic scenarios, further optimization of the structure is completed to complete the system deployment.

[0082] Example 1: Model performance verification on a public dataset

[0083] The present invention has been experimentally verified in detail on the public dataset STARSS22. This dataset covers a variety of complex acoustic scenes, including multi-source overlap, dynamically changing sound sources, and strong background noise, which fully reflects the challenges of sound event localization and detection in real environments. The experiment aims to verify the localization and detection performance of the multi-scale adaptive feature fusion model proposed in the present invention in different complex acoustic scenes.

[0084] This experiment uses the first-order Ambisonic (FOA) signal of the STARSS22 dataset as input. The dataset contains 4 channels, including omnidirectional signal W and directional signals X, Y, and Z, covering a variety of sound events, such as speech, alarms, and environmental sounds. The data labels provide the category, time boundary, and spatial location of the sound event, providing comprehensive support for positioning and detection tasks.

[0085] In the data preprocessing stage, the audio signal is first subjected to a short-time Fourier transform (STFT) to extract spectral features with a window length of 40 milliseconds and a step length of 20 milliseconds to generate a time-frequency representation. Subsequently, 64 Mel filters are used to convert the spectrogram into a logarithmic Mel spectrogram to capture frequency characteristics that are closer to human auditory perception. At the same time, the effective intensity vector (EIV) is calculated to further enrich the feature expression by enhancing the directional information. Finally, the Mel spectrogram (4 channels) and the EIV (3 channels) are concatenated into a 7-channel input feature matrix as the input of the model.

[0086] The model adopts a temporal modeling method based on a bidirectional gated recurrent unit (BiGRU), and combines the selective frequency-aware feature fusion module (SFFB) and the hierarchical multiscale feature integration module (HFI) to dynamically adjust and fuse features at multiple scales to improve the localization and detection performance in complex acoustic scenes. The model training was performed for a total of 50 epochs, and the initial learning rate was set to 3×10 -4 , and gradually decays to 1×10 -6 A joint loss function is used in the optimization process, including classification loss, localization loss, and boundary detection loss, to balance the comprehensive performance of the model on multiple tasks.

[0087] In order to comprehensively evaluate the performance of the present invention in the task of sound event localization and detection (SELD), the present invention adopts a joint evaluation method, combining sound event detection (SED) and localization accuracy, and evaluates them from the aspects of time boundary detection, classification accuracy, spatial localization accuracy, etc. The main performance indicators include the following categories: error rate (ER) and F score (F 20° ). Both indicators are combined with spatial positioning, taking the spatial distance between the predicted result and the true value as the calculation basis, and only when the angle difference between the predicted and the true value is less than 20°, the true positive (TruePositive, TP) is counted. The positioning performance is measured by the positioning error (LE) and the positioning recall rate (LR CD ) are two indicators that reflect the accuracy and coverage of the model in spatial positioning. The SELD score is a comprehensive indicator that combines all four indicators and provides a single indicator of overall system performance. Parameters is an important indicator for measuring the scale and complexity of deep learning models.

[0088] Table 1 shows the comparison results of the CASE-DLM proposed in this paper and other methods on the STARS22 dataset (the best is shown in bold). The experimental results show that the performance of the multi-scale adaptive feature fusion model proposed in this paper on the STARSS22 dataset is better than that of the existing methods. The following is a comparative analysis of specific performance indicators: CASE-DLM is better than the comparison model in all indicators, especially in F 20° , LE and LR CD Among them, CASE-DLM's F 20° Reached 55.8%, LR CD The results show that CASE-DLM has a higher retrieval accuracy and positioning coverage than other advanced methods. In addition, the positioning error (LE) of CASE-DLM in complex acoustic environments is reduced to 17.8°, and the error rate (ER) is reduced to 0.63, demonstrating its robustness in noise interference and multi-source overlap scenarios. CASE-DLM finally achieved a SELD score of 0.373, verifying its superior comprehensive performance in positioning and detection tasks.

[0089] Table 1 Comparison results of CASE-DLM and other methods on the STARS22 dataset (best in bold)

[0090]

[0091] Table 2 shows the results of the CASE-DLM ablation study of the present invention on the STARS22 dataset (the underlined results indicate the optimal results). Table 2 verifies the independent contributions of the key modules (SFFB, HFI, and CMA) in the CASE-DLM model and their synergistic effects. The Baseline model uses a common residual network as the basic framework for feature extraction, and only performs basic convolution operations and jump connections. It has the weakest performance in various indicators, indicating that it has obvious limitations in processing complex acoustic scenes. The SFFB module performs most significantly in reducing LE, and can effectively solve boundary ambiguity and intra-class inconsistency problems; the HFI module significantly improves ER and F through hierarchical multi-scale feature fusion. 20° , suitable for dealing with complex multi-source scenarios; the CMA module is slightly less effective when used alone, but it further enhances the global feature perception ability after working in synergy with other modules. In the complete CASE-DLM model that integrates all modules, all indicators are optimized, with the ER score reduced to 0.63, the LE reduced to 17.811, and the F 20° Improved to 55.8, while achieving the highest F 20° (55.8) and LRCD (67.7), which fully verifies the high robustness and high precision performance of the model in complex acoustic scenarios.

[0092] Table 3 compares the effects of the proposed model and other different feature fusion methods on the performance of the SFFB module on the STARS22 dataset. PA is the traditional feature fusion method, and SA is the SFFB module after removing the FHPB submodule. The experimental results highlight the significant contribution of the SFFB module in improving noise suppression and positioning accuracy. Compared with the traditional splicing fusion method (PA), the SFFB module achieved a 6.5% reduction in LE and a 5.7% reduction in SELD score, demonstrating its superior performance in processing complex overlapping sound sources. The simplified configuration (SA, i.e. after removing the FHPB submodule) has a significant impact on the F 20° The indicators are slightly improved compared to PA (increased by 0.4%), but the LE and SELD scores have increased, indicating that its effect in capturing fine spatial boundaries has decreased. This result further highlights the important role of the FHPB submodule in enhancing high-frequency details and maintaining the robustness of positioning performance. The full SFFB configuration (integrating FLPB and FHPB at the same time) outperforms other configurations in all key indicators. Compared with PA, F 20° The improvement is 7.3% and the LE is reduced by 9.5%. These results verify the effectiveness of the frequency-adaptive design of the SFFB module in improving boundary definition and enhancing robustness in dynamic acoustic environments.

[0093] Table 2 CASE-DLM ablation study on STARS22 dataset (optimal underline)

[0094]

[0095] Table 3. Impact of different feature fusion methods on the performance of SFFB module.

[0096]

[0097] Figure 5 This is a visual comparison chart of the frequency adaptive low-pass filter provided by the present invention and other methods on a data set. Figure 5 The results of (a) show that the traditional bilinear interpolation method is prone to intra-class feature inconsistency when locating sound events, especially near event boundaries, which can lead to a decrease in positioning accuracy or unstable positioning results. In scenes with overlapping sound sources or complex environments, convolution operations may introduce intra-class inconsistencies during feature extraction, which will have a negative impact on the accuracy of the model's estimation of the spatial locations of similar events, ultimately increasing positioning errors. In contrast, Figure 5 (b) shows the improved features, highlighting the effect of the FLPB module on significantly enhancing the consistency of intra-class features through adaptive low-pass filtering technology. This improvement is particularly evident in the spatial location estimation of sound events. The FLPB module can adaptively smooth different spectral components, reduce intra-class inconsistencies, thereby improving the accuracy of the model and enhancing the stability of locating similar events.

[0098] Figure 6 A visual comparison chart of the frequency adaptive high-pass filter provided by the present invention and other methods on the data set. The core contribution of the FHPB module is that it can restore the high-frequency details lost in the downsampling process, which is crucial for the accurate detection of sound events. In the spectrogram, high-frequency information usually corresponds to the start, end, and rapidly changing characteristics of the sound event. Without a high-pass filter, these high-frequency details may be aliased or lost in the downsampling process, resulting in blurred event boundaries. Figure 6 As shown in Figure 1, the FHPB module performs well in enhancing the boundary details of sound events. For example, the original spectrogram is difficult to accurately depict the boundary details of complex sound events, but after the introduction of the FHPB module, these details are significantly improved, making the feature representation in the spectrogram clearer.

[0099] Example 2: Performance verification in a multi-source overlapping scenario

[0100] To further verify the performance of the present invention in multi-source overlapping scenarios, an audio clip containing three overlapping sound sources in the STARSS22 data set was selected for testing. The preprocessing steps of the audio signal are consistent with those in Example 1, including short-time Fourier transform (STFT), logarithmic Mel spectrogram generation, and calculation of effective intensity vector (EIV), and a 7-channel input feature matrix is ​​constructed. The model parameter setting is also the same as in Example 1, using a time modeling method based on a bidirectional gated recurrent unit (BiGRU), and combining a selective frequency-aware feature fusion module (SFFB) and a hierarchical multi-scale feature integration module (HFI) to ensure the unity of feature extraction and fusion. The SFFB module adaptively suppresses high-frequency noise interference in overlapping sound sources by dynamically adjusting frequency features, thereby improving the recognition of features; the HFI module effectively distinguishes the feature representations of different sound sources through a multi-scale feature fusion strategy, alleviating the confusion caused by multi-source overlap; at the same time, the bidirectional GRU combines the timing modeling capability to accurately capture the dynamically changing positions of multiple sound sources. The experimental results show that the model of the present invention performs well in locating overlapping sound sources, and can accurately identify the positions of different sound sources, with the positioning error kept within 20°. In addition, in terms of classification and time boundary detection, the model can effectively distinguish the event categories of different sound sources and accurately mark their time ranges, which fully verifies the robustness and detection ability of the present invention in multi-source overlapping scenarios. Figure 7 This is a visual comparison chart of the detection effects of the method proposed in the present invention and other methods (residual neural network) on three different sound categories (music, woman’s voice and footsteps) on the dataset of Example 2.

[0101] The SELD method proposed in the present invention has the characteristics of high robustness and high precision in real complex acoustic environments, which makes up for the shortcomings of existing technologies and provides more efficient technical solutions for application scenarios such as smart homes, autonomous driving, and robot hearing. Through multi-module collaborative design and optimized training processes, the present invention can not only significantly improve positioning and detection performance, but also effectively reduce computational complexity, thereby meeting the real-time requirements in practical applications. Experimental results show that the present invention has shown significant performance advantages in multi-source overlap, noise interference, and dynamic event detection scenarios.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the applicant has described the present invention in detail with reference to the preferred embodiments, ordinary technicians in the field should understand that modifications or equivalent substitutions of the technical solution of the present invention without departing from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. A method for localizing and detecting sound events using multi-scale adaptive feature fusion, characterized in that: The steps are as follows, S1. Preprocess the multi-channel audio signal to be detected, generate a Mel spectrogram and calculate the effective intensity vector, and construct an input feature matrix; S2, input the input feature matrix into the encoder for multi-scale feature extraction, and output multi-layer spatial features; S3, frequency adaptively adjusts the multi-layer spatial features output by the encoder through the selective frequency-aware feature fusion module, smoothes the global features and enhances the boundary details, so that the features have higher boundary clarity and feature consistency; S4. Use the hierarchical multi-scale feature integration module to fuse shallow features with deep features, and combine the bidirectional gated recurrent unit for temporal modeling to output the detection results.

2. The method for locating and detecting sound events by multi-scale adaptive feature fusion according to claim 1, characterized in that: Step S1 is specifically processed as follows: First, the multi-channel audio signal to be detected is preprocessed, including short-time Fourier transform to obtain the audio spectrum, and the appropriate window size and step size are selected; then, the spectrum is converted into a logarithmic Mel spectrum to capture the characteristics of frequency distribution perceived by the human ear; at the same time, the effective intensity vector is calculated, which is obtained by taking the real part of the product of the omnidirectional signal and the directional signal and spatially averaging; finally, the Mel spectrum and the effective intensity vector are spliced ​​to obtain the input feature matrix.

3. The method for locating and detecting sound events by multi-scale adaptive feature fusion according to claim 1, characterized in that: In step S2, multi-scale feature extraction is specifically as follows: In the encoder, multi-layer convolution operations are used to gradually downsample the input feature matrix to extract multi-scale features in the time and frequency dimensions; each layer of the encoder captures spatial features of different scales through the sliding operation of the convolution kernel; the time and frequency dimensions are further compressed through pooling operations to improve the computational efficiency of the features; finally, the encoder outputs multi-layer nested features, which are the multi-layer spatial features.

4. The method for locating and detecting sound events by multi-scale adaptive feature fusion according to claim 1, characterized in that: In step S3, frequency adaptive adjustment is performed as follows: In the selective frequency perception feature fusion module, a low-pass filter and a high-pass filter are designed to dynamically and adaptively adjust the multi-layer spatial features output by the encoder; the low-pass filter adaptively generates a low-pass filter parameter matrix according to the frequency distribution to smooth the global low-frequency features and suppress high-frequency noise interference; the high-pass filter adaptively restores high-frequency details through filter inversion operation; through the processing of the low-pass filter and the high-pass filter, the output features ultimately have higher boundary clarity and feature consistency.

5. The method for locating and detecting sound events by multi-scale adaptive feature fusion according to claim 4, characterized in that: The multi-layer spatial features include and The high-level features composed of and The shallow features composed of the selective frequency perception feature fusion modules are the same as the two, one of which is a selective frequency perception feature fusion module with the same and Connection to extract global semantic information from high-level features, the output is Another selective frequency-aware feature fusion module and feature and The connection is used to preserve and optimize the local details of shallow features, ensuring that the boundary features and time-frequency details of sound events can be preserved during the fusion process.

6. The method for locating and detecting sound events by multi-scale adaptive feature fusion according to claim 1, characterized in that: Step S4 is implemented as follows: The hierarchical multi-scale feature integration module combines the fine-grained temporal frequency changes captured by shallow features with the global context information extracted by deep features through a feature pyramid structure, and realizes the step-by-step fusion of multi-scale features through up and down sampling operations; at the same time, the hierarchical multi-scale feature integration module introduces a context-aware multi-scale aggregation mechanism, and uses different pooling scales and dilated convolutions to extract global and local features; then, the fused features are input into a bidirectional gated recurrent unit, and the dynamic change characteristics of sound events are captured through the temporal modeling capability of the bidirectional gated recurrent unit, and finally the joint prediction results of the sound event category, temporal boundary and spatial position are output.

7. The method for locating and detecting sound events by multi-scale adaptive feature fusion according to claim 6, characterized in that: The context-aware multi-scale aggregation mechanism is implemented by a context-aware multi-scale aggregation module, which uses the feature map output by the encoder is the input, where H and W are the spatial dimensions respectively, and C is the number of channels; the context-aware multi-scale aggregation module first scales the feature map to different scales through adaptive average pooling operation, and then upsamples the feature map back to the original spatial dimension to capture multi-scale global features; at the same time, the context-aware multi-scale aggregation module performs spatial convolution operations with different expansion rates on the input feature map to further extract multi-scale context information; finally, the pooled features, convolution features and original input features are merged to form a multi-scale feature expression that integrates global context and fine-grained details.

8. The method for locating and detecting sound events by multi-scale adaptive feature fusion according to claim 7, characterized in that: The multi-scale features fused by the context-aware multi-scale aggregation module are sequentially input into three cascaded CSP modules. The CSP module combines feature segmentation, feature fusion and layer-by-layer processing. Through the collaborative work of the trunk path and the direct path, the spatial resolution of the feature map is gradually restored while maintaining computational efficiency. Finally, the upsampled multi-scale features are spliced ​​in the channel dimension to generate a multi-scale fused feature map with rich semantic information. The multi-scale fused feature map is passed to the bidirectional gated recurrent unit to further extract temporal features by capturing the temporal correlation of the sequence data.

Images