Small sample sound event detection method based on multi-scale feature aggregation
By combining a multi-scale feature aggregation network and a meta-learning framework, the problems of incomplete feature extraction and insufficient model adaptability in small sample sound event detection are solved, enabling rapid and accurate identification of complex sound events under data-scarce conditions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-12
- Publication Date
- 2026-04-03
AI Technical Summary
Existing small-sample sound event detection technologies suffer from incomplete feature extraction, weak model generalization ability, and poor rapid adaptation under conditions of scarce data, making them unable to effectively identify complex, multi-scale sound events.
The design incorporates a network based on multi-scale feature aggregation. It extracts time, frequency, and time-frequency features through three parallel convolutional paths and combines them with a meta-learning framework to achieve explicit feature separation and aggregation, thereby improving the model's ability to quickly adapt to new event categories.
In situations where data is scarce, the ability to quickly and accurately identify and classify new sound events improves the model's generalization ability and adaptability.
Smart Images
Figure CN121789712A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of audio signal processing and machine learning, specifically to a few-sample sound event detection method based on deep learning and meta-learning, which is particularly suitable for fast and accurate identification of audio events in data-scarce scenarios. Background Technology
[0002] Sound event detection, as one of the core tasks of computational auditory scene analysis, aims to automatically identify the occurrence of specific types of sound events from a continuous audio stream and accurately pinpoint their start and end times. This technology serves as a crucial bridge for machines to perceive the auditory dimension of the physical world and is essential for building intelligent systems with comprehensive environmental perception capabilities.
[0003] Traditional sound event detection methods typically follow a paradigm of "feature engineering + shallow model." At the feature level, they heavily rely on acoustic expert knowledge, manually designing time-frequency domain features such as Mel-frequency cepstral coefficients, zero-crossing rates, spectral centroids, and spectral roll-off points to capture discriminative information about the sound. At the model level, they often employ Gaussian mixture models, hidden Markov models, or combinations thereof, as well as classic machine learning algorithms such as support vector machines. These methods have achieved some success in laboratory settings with limited categories and controllable background noise databases. However, their inherent limitations become apparent when facing the complex and open real world: First, the representational power of hand-crafted features is limited, making it difficult to characterize the high-dimensional, nonlinear, and complex patterns of sound events, especially when the signal-to-noise ratio is low or multiple events overlap, causing a rapid decline in discriminative power; second, the model's capacity and flexibility are insufficient, with almost zero generalization ability to newly emerging event categories not seen in the training set; finally, the entire process is highly dependent on the experience of domain experts, resulting in low scalability and automation.
[0004] The revolutionary advancements in deep learning have brought about a paradigm shift in Sound Evidence Detection (SED). Deep models, represented by convolutional neural networks (CNNs), recurrent neural networks (RNNs), and their hybrid models, can learn hierarchical feature representations directly from raw waveforms or spectrograms in an end-to-end manner. These deep features automatically contain rich information from low-level edges to high-level semantics, significantly surpassing handcrafted features. Driven by large-scale labeled datasets, deep learning-based SED systems have achieved a huge leap in detection accuracy and robustness. However, deep learning models are inherently "data-hungry," and their superior performance is built on massive amounts of high-quality, balanced labeled data. This premise is difficult to meet in many real-world scenarios: 1) Scarcity: Many sound events themselves have low occurrence rates, such as specific types of mechanical failures, rare wildlife calls, and specific dangerous events in the home environment; 2) Labeling costs: Fine-grained temporal boundary labeling of audio requires a large amount of professional manpower, which is time-consuming, labor-intensive, and costly; 3) Long-tailed distribution: Real-world sound categories follow a long-tailed distribution, with many categories having very few samples, while mainstream categories have abundant data. Direct training can lead to a model that is severely biased towards the head categories.
[0005] To address the problem of data scarcity, few-shot learning (FSL) has been introduced into the audio domain. Its core idea is to enable models to quickly learn new concepts from a very small number of samples. Among the many FSL methods, meta-learning has shown great potential. The strategy of meta-learning is not to directly learn a specific classifier during the training phase, but rather to allow the model to learn on a series of simulated "few-shot tasks," thereby mastering the "meta-knowledge" or inductive bias that allows it to quickly adapt to new tasks. Model-independent meta-learning (MAML) is one of the most influential frameworks. Its core idea is to optimize the initial state of the model parameters so that, starting from this initial point, excellent performance can be achieved for any new task with only a few gradient updates. The model-independent nature of MAML allows it to be flexibly combined with various neural network architectures.
[0006] However, directly applying meta-learning frameworks such as MAML to few-sample sound event detection still faces significant challenges. The performance bottleneck primarily stems from the coarse-grained nature of feature extraction and the homogeneity of task modeling.
[0007] Feature extraction suffers from dimensional confusion and a lack of specificity: Most existing research simply treats the time-frequency representation of audio as a regular two-dimensional image, directly applying standard CNNs designed for image classification. This approach ignores the distinct physical meanings and statistical patterns of audio signals across two dimensions. The time dimension characterizes the dynamic evolution of events, temporal dependencies, and duration patterns; the frequency dimension reflects the essential spectral properties of events, such as pitch, harmonic structure, and formant distribution. Using isotropic square convolutional kernels for hybrid convolutions forces the network to process time and frequency information in the same way. While this might be implicitly learned through network depth and a large number of parameters in conventional learning with ample data, it becomes extremely problematic with small sample sizes and limited model parameter updates. This ambiguous approach leads to insufficient discriminative power in the extracted features, failing to efficiently focus on the most crucial temporal dynamics or spectral details for identifying new events.
[0008] Insufficient multi-scale perception and simplistic task adaptation: Sound events inherently possess multi-scale characteristics. For example, a "knocking sound" contains transient impact components and possible echo attenuation; an "alarm sound" exhibits a regular periodic pattern. Different categories may have different sensitivities to time or frequency scales. Existing meta-learning methods based on a single architecture typically use stacked convolutional layers with fixed-size receptive fields, making it difficult to effectively capture these cross-scale patterns simultaneously in a single forward propagation. In the rapid adaptation of the meta-learning inner loop, the single model structure limits its ability to flexibly adjust feature focus based on a small number of supporting samples for new tasks, resulting in inefficient adaptation to diverse new event categories.
[0009] In summary, the core contradiction of current few-sample sound event detection technologies lies in the fact that limited data budgets require features to possess extremely high discriminative power and information density, while general, coarse-grained feature extraction methods cannot meet this requirement. Therefore, this invention aims to fundamentally resolve this contradiction by designing a representation learning network specifically tailored to the time-frequency characteristics of audio, capable of explicitly separating and aggregating multi-dimensional, multi-scale features, and deeply integrating it with an efficient meta-learning framework. This enables powerful generalization and rapid adaptation to new sound event categories under the strict constraint of scarce data. Summary of the Invention
[0010] To address the shortcomings of the existing technologies, the present invention aims to provide a small-sample sound event detection method based on multi-scale feature aggregation, thereby solving the technical problems of incomplete sound event feature extraction, weak model generalization ability, and poor rapid adaptability of traditional methods under data scarcity conditions.
[0011] To achieve the above objectives, the present invention adopts the following technical solution:
[0012] A method for detecting few-sample sound events based on multi-scale feature aggregation includes the following steps:
[0013] Step 1: Audio signal preprocessing and feature extraction. The original audio signal is acquired, pre-emphasized, framed, and windowed. A Fast Fourier Transform is performed on each frame to obtain its spectrum. The spectrum is then passed through a Mel filter bank, and its logarithmic energy is taken to generate a Mel spectrogram as the basic acoustic feature.
[0014] Step 2: Construct and train a multi-scale feature aggregation network based on meta-learning. The network includes a feature aggregation module and a meta-learning training framework.
[0015] The core of the feature aggregation module consists of three parallel convolutional paths:
[0016] The first path uses a 3×1 convolution kernel that slides along the time dimension to extract long-range temporal dependencies and dynamic pattern features of sound events.
[0017] The second path uses a 1×3 convolution kernel that slides along the frequency dimension to extract the spectral contour and frequency band correlation features of sound events.
[0018] The third path uses a 3×3 convolution kernel that slides across the time-frequency plane to extract the local time-frequency structure and transient features of sound events.
[0019] The output features of the three paths are concatenated along the channel dimension, then fused by 1×1 convolution, followed by batch normalization, nonlinear activation and pooling operations, to output the aggregated high-dimensional feature representation.
[0020] The meta-learning training framework employs a model-independent meta-learning method: In the meta-training phase, multiple N-way K-shot few-shot tasks are constructed from the training dataset, each task containing a support set and a query set; in the inner loop, the network performs fast gradient updates on the task support set to adapt to the specific task; in the outer loop, meta-gradients are calculated based on the loss of each task on the query set, thereby updating the network's global initialization parameters, enabling the network to gain rapid adaptability across tasks.
[0021] Step 3: Rapid Recognition of Small Sample Sound Events. The trained network model is applied to a scenario with a small number of samples containing the new category. Using these few samples as the support set, the model parameters are rapidly adjusted through the network's internal adaptation mechanism, enabling the model to recognize the new sound event and thus accurately classify unknown audio segments.
[0022] Furthermore, in step 1, the number of Mel filter banks is 40, and the dimension of the generated Mel spectrogram is time frame number × 40.
[0023] Furthermore, in step 2, the number of inner loop update steps for model-independent meta-learning is 1 to 5, cross-entropy is used as the loss function, and Adam is used as the outer loop optimizer. Attached Figure Description
[0024] Figure 1 This is a schematic diagram of the overall process of the method of the present invention.
[0025] Figure 2 This is a schematic diagram of the structure of the multi-scale feature aggregation network in this invention.
[0026] Figure 3 This is a schematic diagram of the model-independent meta-learning training process used in this invention. Detailed Implementation
[0027] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0028] Step S1: Audio preprocessing and feature extraction
[0029] First, the original audio signal is acquired and resampled to a uniform sampling rate of 16kHz. Preprocessing is then performed: a pre-emphasis filter is applied to compensate for high-frequency attenuation; the signal is segmented into overlapping short frames; and a Hamming window is applied to each frame to reduce spectral leakage. A Fast Fourier Transform (FFT) is then performed on each windowed frame to obtain the amplitude spectrum. This amplitude spectrum is passed through a set of triangular Mel filter banks, and the logarithm of the output of each filter is taken to form the Mel spectrogram, which serves as the input feature for subsequent networks. This feature conforms to the characteristics of human hearing and is a commonly used representation in audio processing.
[0030] Step S2: Design of Multi-Scale Feature Aggregation Network
[0031] like Figure 2 As shown, the core of this invention is a multi-scale feature aggregation network. The network input is a T×F Mel spectrum (T is the number of time frames, and F is the number of Mel bands). The network front end uses three independent convolutional paths for parallel processing:
[0032] Temporal path: Using multiple 3×1 convolutional kernels, this shape of kernel slides only in the time dimension, specifically designed to capture long-range dependencies and dynamic change patterns of sound events over time.
[0033] F time =ReLU(W time *X+b time (1)
[0034] Frequency path: Using multiple 1×3 convolution kernels, this kernel shape slides only in the frequency dimension, specifically designed to analyze the correlation between different frequency bands and capture the spectral profile features of sound events.
[0035] F freq =ReLU(W freq *X+b freq (2)
[0036] Time-frequency path: Uses multiple 3×3 two-dimensional convolution kernels to capture joint features in local time-frequency regions, and is sensitive to transient characteristics of events and subtle time-frequency patterns.
[0037] F joint =ReLU(W joint *X+b joint (3)
[0038] The output feature maps from the three paths are concatenated along the channel dimension to form a composite feature tensor that integrates multi-scale and multi-dimensional information. Subsequently, a 1×1 convolutional layer is used to perform dimensionality reduction and deep fusion on the concatenated features. Finally, after several layers of standard 2D convolution, batch normalization, ReLU activation, and pooling operations, high-level semantic features are further refined, outputting the aggregated feature representation.
[0039] Step S3: MAML-based meta-learning training
[0040] like Figure 3 As shown, the network constructed in step S2 is trained using the model-independent meta-learning framework.
[0041] Task Construction: A large number of meta-tasks are constructed by sampling from the training dataset. Each meta-task T i Simulate an N-way K-shot few-shot classification problem, containing a support set S. i (for fast adaptation) and a queryset Q i (Used for evaluation and meta-optimization).
[0042] Inner loop adaptation: For each task T i The current global parameters θ of the network are used as initial values. In the task's support set S... i Perform one or more steps of gradient descent to obtain temporary parameters θ′ adapted to the task. i .
[0043]
[0044] This process teaches the network how to quickly adjust using a small number of samples.
[0045] Outer loop optimization: Calculate the network in terms of parameter θ′ i Next, in task T i query set Q iThe sum of the query set losses for all sampling tasks constitutes the meta-loss. The gradient of the meta-loss with respect to the initial parameters θ (meta-gradient) is calculated, and θ is updated using optimizers such as Adam.
[0046]
[0047] The aim of this process is to find a parameter initialization point that is friendly to various new tasks, so that good performance can be achieved from that point through a small number of steps in the inner loop.
[0048] Step S4: Small Sample Testing and Evaluation
[0049] After training, the global parameters θ of the network are fixed. When faced with a small number of samples containing new sound event categories, these samples are simply used as the support set for the new task. The inner loop adaptation process in step S3 (usually requiring only a few gradient updates) is then executed to obtain a classifier for the new category. Subsequently, this classifier can be used to detect and recognize new audio segments.
Claims
1. A method for detecting small-sample sound events based on multi-scale feature aggregation, characterized in that, Includes the following steps: Step S1: Acquire audio signals and perform preprocessing to extract Mel spectrogram features; Step S2: Construct a multi-scale feature aggregation network. The network receives the Mel spectrogram as input and extracts local features in the time dimension, frequency dimension, and time-frequency joint dimension through three parallel convolutional paths. The features extracted by the three paths are then concatenated and fused in the channel dimension to output the aggregated feature representation. Step S3: The multi-scale feature aggregation network is trained using a model-independent meta-learning framework. The training task is divided into multiple meta-tasks, each of which contains a support set and a query set. In the inner loop, the network performs fast adaptation on the support set. In the outer loop, the meta gradient is calculated based on the loss on the query set, and the global initialization parameters of the network are updated. Step S4: Use the trained model to quickly adapt to and identify new sound event categories with only a small number of samples.
2. The method according to claim 1, characterized in that, The specific process of extracting the Mel spectrogram in step S1 includes: The original audio signal is pre-emphasized, framed, and subjected to Hamming windowing. A fast Fourier transform is performed on each frame to obtain the spectrum. The spectrum is then passed through a set of Mel-scale filters, and the logarithm of the output energy of each filter is taken to obtain the final Mel spectrum.
3. The method according to claim 1, characterized in that, The three parallel convolutional paths in step S2 are specifically as follows: The first path uses a 3×1 one-dimensional convolutional kernel to extract long-term dependent features along the time axis; The second path uses a 1×3 one-dimensional convolution kernel to extract frequency band correlation features along the frequency axis; The third path uses a 3×3 two-dimensional convolution kernel to extract joint features in the local time-frequency region.
4. The method according to claim 3, characterized in that, After the features from the three paths are concatenated, the multi-scale feature aggregation network also includes, in sequence: a 1×1 convolutional layer for feature dimensionality reduction and fusion, and multiple standard convolutional modules consisting of batch normalization, ReLU activation function and pooling layer for further extraction of high-level abstract features.
5. The method according to claim 1, characterized in that, The model-independent meta-learning framework in step S3 uses cross-entropy loss as the loss function updated in the inner loop and gradient descent to update task-specific parameters. The outer loop uses the sum of the losses of all meta-tasks on the query set as the meta-objective, calculates the meta-gradient through the second derivative, and uses the Adam optimizer to update the global initialization parameters of the network.
6. A sound event detection system, characterized in that, include: The preprocessing and feature extraction unit is used to acquire audio signals and perform preprocessing to extract Mel spectrogram features. A multi-scale feature aggregation network unit is used to construct a multi-scale feature aggregation network. The network receives a Mel spectrogram as input, extracts local features in the time dimension, frequency dimension, and time-frequency joint dimension through three parallel convolutional paths, and concatenates and fuses the features extracted by the three paths in the channel dimension to output the aggregated feature representation. The meta-learning training engine unit is used to train the multi-scale feature aggregation network using a model-independent meta-learning framework. The training task is divided into multiple meta-tasks, each of which contains a support set and a query set. In the inner loop, the network performs fast adaptation on the support set. In the outer loop, the meta gradient is calculated based on the loss on the query set, and the global initialization parameters of the network are updated. The small-sample fast classification unit is used to quickly adapt to and identify new sound event categories with only a small number of samples using a trained model.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.