Sound event detection method based on recursive gated convolution and self-attention mechanism

Abstract

The invention discloses a sound event detection method based on recursive gating convolution and a self-attention mechanism, and the method comprises the steps: collecting a to-be-detected audio signal, and constructing a sound event detection model; inputting the audio signal into a sound event detection model, and extracting a time-frequency feature through a preprocessing module to obtain a logarithmic Mel spectrum feature; inputting the logarithmic Mel spectrum features into a convolution module, and performing spatial feature fusion through recursive gating convolution to obtain spatial fusion features; inputting the spatial fusion features into a time domain modeling module, and performing context information interaction through a self-attention mechanism to obtain global information features; and inputting the global information features into a KANLinear classifier, and carrying out frame-by-frame classification to obtain each sound event category in the audio and the occurrence time period thereof. According to the method, the accuracy of sound event detection can be remarkably improved on the premise that the number of parameters is small and the number of floating point operation times is low.

Description

Technical Field

[0001] This invention belongs to the field of sound signal processing technology, and in particular relates to a sound event detection method based on recursive gated convolution and self-attention mechanism. Background Technology

[0002] In daily life, people frequently encounter various sound events, such as car horns, thunder, birdsong, and baby cries. These sounds not only help people perceive their surroundings but also carry rich environmental information. To enable computers to recognize and understand these sounds, sound event detection technology has emerged, aiming to simulate the human auditory perception process through artificial intelligence systems. The core task of this technology is to analyze and classify audio signals frame by frame, determining the occurrence category and time interval of various sound events, thereby achieving automatic recognition and analysis of complex soundscapes. Compared with visual detection, sound signals have the significant advantage of being unaffected by lighting and occlusion, allowing sound event detection technology to operate stably in complex environments. Therefore, sound event detection has broad research prospects and significant application value in fields such as intelligent monitoring, smart cities, healthcare, and human-computer interaction.

[0003] Early sound event detection methods were primarily based on traditional machine learning models, such as Gaussian Mixture and Hidden Markov Models (GMM-HMM), Support Vector Machines (SVM), and Nonnegative Matrix Factorization (NMF). These methods achieved some success in monophonic environments, but their performance in polyphonic detection was limited due to their inability to effectively handle sound overlap. With the rapid development of deep learning and the emergence of large-scale datasets, deep learning-based sound event detection has gradually become the mainstream research method. Early research used feedforward neural networks (FFN) for sound classification, which could improve detection accuracy to some extent. However, due to the fixed network structure, it was difficult to adapt to changes in time and frequency, and it could only process short-term information, resulting in insufficient performance in modeling long-term dependencies. Convolutional Neural Networks (CNN) were used to extract time-frequency features of audio, achieving translation invariance of features through local receptive fields. Subsequently, Recurrent Neural Networks (RNN) and their improved forms, Long Short-Term Memory Networks (LSTM) and Gated Recurrent Units (GRU), were used to model temporal contextual dependencies. The Convolutional Recurrent Neural Network (CRNN) structure, combining the advantages of CNN and RNN, has become the mainstream architecture and has achieved significant results in the DCASE International Acoustic Challenge. Building upon the CRNN framework, subsequent research has continuously improved and expanded it. Li et al. proposed the DilatedCRNN structure based on dilated convolution, which expands the receptive field by dilating the convolutional kernel, capturing longer temporal dependencies without increasing the number of parameters. Adavanne et al. further constructed the C3RNN model using three-dimensional convolution (3DCNN), enabling the network to learn intra-channel and inter-channel features simultaneously from multi-channel inputs. Drossos et al. introduced teacher forcing and scheduled sampling mechanisms, proposing the TFCRNN model. By gradually using the model's own prediction output during the training phase, it effectively alleviated the mismatch between the training and inference phases, thereby improving the model's temporal modeling ability.

[0004] However, RNN-based models still suffer from problems such as large parameter count, vanishing gradients, and low computational efficiency, making it difficult to meet the requirements of lightweight and real-time detection. To address these issues, some studies have attempted to combine depthwise separable convolutions with dilated convolutions to replace traditional convolutional and RNN structures, achieving progress in reducing parameter count and training time. However, while dilated convolutions expand the receptive field to capture longer temporal dependencies by inserting intervals, they may also cause local information loss. When the dilation coefficient is large, a large amount of zero-padding is required to maintain the temporal dimension, leading to the dilution of effective features and performance degradation. Furthermore, these models still have shortcomings in capturing global contextual information. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a lightweight sound event detection method based on recursive gated convolution and self-attention mechanisms. This invention significantly reduces the number of model parameters and computational complexity while maintaining detection accuracy, exhibits high generalization ability, and can be widely applied to sound event detection tasks in complex acoustic environments.

[0006] To achieve the above objectives, this invention provides a sound event detection method based on recursive gated convolution and self-attention mechanism, comprising: Collect the audio signal to be detected and build a sound event detection model; The audio signal is input into the sound event detection model, and the time-frequency features are extracted through the preprocessing module to obtain the log-Mel spectrum features; The log-Mel spectrum features are input into the convolution module, and spatial features are fused through recursive gated convolution to obtain spatial fused features; The spatial fusion features are input into the temporal modeling module, and contextual information is exchanged through a self-attention mechanism to obtain global information features; The global information features are input into the KANLinear classifier for frame-by-frame classification to obtain the category of each sound event in the audio and its occurrence time.

[0007] Preferably, the sound event detection model is constructed using a lightweight network architecture consisting of a recursive gated convolutional network, a self-attention network, and a KANLinear classifier. The lightweight network architecture achieves adaptive spatial mixing through recursive gated convolutional layers, enhances temporal relationship modeling through a self-attention mechanism, and implements nonlinear mapping based on spline functions through a KANLinear classifier.

[0008] Preferably, the process of extracting time-frequency features through a preprocessing module to obtain log-Mel spectrum features includes: The audio signal is pre-emphasized and filtered to obtain the pre-emphasized signal; The pre-emphasized signal is subjected to frame segmentation processing to obtain a set of frame signals; A Hamming window is applied to each frame signal in the frame signal set to obtain a windowed signal; Perform a short-time Fourier transform on the windowed signal to extract spectral information and obtain spectral features; The spectral features are filtered by a bandpass filter, and the logarithmic energy is calculated to obtain the log-Mel spectral features.

[0009] Preferably, the convolution module includes three recursive gated convolutional layers stacked in sequence, with a nonlinear activation layer, a batch normalization layer and a max pooling layer sequentially arranged after each recursive gated convolutional layer; The process of obtaining spatial fused features through recursive gated convolution includes: The log-Mel spectrum features are channel-expanded and then input into the first recursive gated convolutional layer to obtain the first layer features. The output of the previous recursive gated convolutional layer is used as the input of the next recursive gated convolutional layer. This process is repeated for three layers to obtain deep fusion features. After processing through each recursive gated convolutional layer, max pooling, batch normalization, and ReLU activation functions are applied sequentially to obtain pooling activation features. The output of the third max pooling layer is dimensionally compressed to obtain spatial fusion features.

[0010] Preferably, the max pooling operation only applies to the feature dimension, while the time dimension remains unchanged; After three layers of convolution and pooling, the output feature dimension is gradually compressed from the initial T×F×C to T×1×C, where T represents the number of frames, F represents the initial feature dimension, and C represents the number of channels. The output of T×1×C dimensions is compressed to T×C dimensions to obtain spatial fusion features.

[0011] Preferably, the process by which the recursively gated convolutional layer performs high-order spatial interactions on the input features includes: Linear projection is performed on the input features to obtain the projected feature set; Recursively perform gated convolution operations on the projected feature group to obtain recursively gated features; The last recursive result is fused at the channel level through a projection layer to obtain the spatially fused output features.

[0012] Preferably, the process of obtaining global information features through contextual information interaction via a self-attention mechanism includes: Based on the temporal dimension of the spatial fusion features, a location code is generated for each time frame to obtain location code information; The spatial fusion features are linearly mapped by a linear layer to adjust the dimensions to the dimensions required by the multi-head self-attention mechanism, and multi-head partitioning is performed to obtain the query matrix, key matrix, and value matrix. Based on the query matrix, key matrix, and value matrix, the attention scores between different time frames are calculated to obtain the attention weights; Multiply the attention weights by the value matrix to obtain the weighted features; The outputs of each attention point are concatenated and linearly transformed using a weight matrix to obtain global information features.

[0013] Preferably, the process of calculating attention scores between different time frames based on the query matrix, key matrix, and value matrix, and obtaining attention weights, includes: Calculate the dot product of the query matrix and the transpose of the key matrix to obtain the first relevance score; Calculate the dot product of the query matrix and the relative position code to obtain the second relevance score; The first relevance score and the second relevance score are added together and normalized using the softmax function to obtain the attention weights.

[0014] Preferably, the KANLinear classifier includes a main path and a residual path; The process of inputting the global information features into the KANLinear classifier and performing frame-by-frame classification includes: The context interaction features are input into the B-spline transform of the main path and the linear transform of the residual path, respectively. Based on the main path, nonlinear mapping features are obtained through B-spline basis function calculation, weighted summation, and dimensional transformation; Based on the residual path, linear mapping features are obtained through linear layer calculation; The nonlinear mapping feature is added to the linear mapping feature to obtain the fused feature; The fused features are input into the Sigmoid activation function to obtain frame-by-frame classification results.

[0015] Preferably, the process of obtaining nonlinear mapping features includes: Based on the preset node sequence and spline order, the B-spline basis function value corresponding to each element in the input feature is calculated recursively to obtain the basis function matrix; The basis function matrix is ​​weighted and summed using a learnable weight matrix to obtain weighted features; By performing dimensional transformation on the weighted features through a linear layer, nonlinear mapping features are obtained.

[0016] Compared with the prior art, the present invention has the following advantages and technical effects: This invention extracts log-Melogram spectral features from audio signals through a preprocessing module, effectively characterizing the time-frequency properties of audio and providing an initial feature representation that conforms to human auditory perception for subsequent processing. Spatial feature fusion is achieved through recursive gated convolution in the convolution module, enhancing the ability to extract local spatial information and realizing dynamic adaptive mixing of information from different spatial dimensions. Contextual information interaction is achieved through the self-attention mechanism of the temporal modeling module, capturing the long- and short-term temporal dependencies in the audio sequence. Frame-by-frame classification is performed using a KANLinear classifier, and fine nonlinear mapping is achieved using spline functions, improving the fitting accuracy of complex feature mapping relationships. Compared with traditional methods, this invention significantly improves the accuracy of sound event detection with fewer parameters and fewer floating-point operations. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall network architecture of the method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the network structure of the recursive gated convolution module in an embodiment of the present invention; Figure 4 This is a schematic diagram of the network structure of the self-attention mechanism in an embodiment of the present invention; Figure 5 This is a schematic diagram of the network structure of the KANLinear classifier in an embodiment of the present invention; Figure 6 This is a comparison chart of the present invention and existing methods on the TUT-SED Synthetic 2016 dataset. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0020] like Figure 1 As shown, this embodiment provides a sound event detection method based on recursive gated convolution and self-attention mechanism, including: Collect the audio signal to be detected and build a sound event detection model; The audio signal is input into the sound event detection model, and the time-frequency features are extracted through the preprocessing module to obtain the log-Mel spectrum features; Log-Mel spectrum features are input into the convolution module, and spatial features are fused through recursive gated convolution to obtain spatial fused features; Spatial fusion features are input into the temporal modeling module, and contextual information is exchanged through a self-attention mechanism to obtain global information features; The global information features are input into the KANLinear classifier for frame-by-frame classification to obtain the category of each sound event in the audio and its occurrence time.

[0021] In the network architecture proposed in this embodiment, RGC achieves deep feature extraction through high-order spatial interactions, enhancing the ability to interact with contextual information during temporal modeling, thereby effectively capturing long- and short-term dependencies in complex soundscapes. Simultaneously, the KANLinear classifier enhances its fitting ability by introducing B-spline transformation, improving the model's ability to express complex feature mapping relationships. This approach improves the accuracy and robustness of sound event detection while maintaining low computational complexity.

[0022] Furthermore, such as Figure 2 As shown, the sound event detection model forms a lightweight network architecture by constructing a recursive gated convolutional network, a self-attention network, and a KANLinear classifier; The lightweight network architecture achieves adaptive spatial mixing through recursive gated convolutional layers, enhances temporal relationship modeling through a self-attention mechanism, and implements nonlinear mapping based on spline functions through a KANLinear classifier.

[0023] Furthermore, the sound event detection model in this embodiment includes an audio signal preprocessing module. This module measures the audio in the acoustic scene using a recording device and extracts its log-Mel spectral features. An input audio record is first segmented into a set of frames. The log-Mel spectral energy of each frame is calculated and considered as the feature of that frame. The process of obtaining the log-Mel spectral energy of the audio record is as follows: pre-emphasis, frame segmentation, windowing, spectral analysis, and bandpass filtering. After the audio signal passes through the preprocessing module to obtain the audio features, it is input into the convolution module for high-order spatial feature processing. The inter-processing module consists of three stacked RGC layers, each followed by a non-linear activation layer, a batch normalization layer, and a max pooling layer. The interacting features are then fed into a multi-head self-attention layer for contextual information exchange, before being fed into a KANLinear classifier for frame-by-frame classification. This classifier is constructed using linear transformation and B-spline transformation residual connections. Finally, a SoftMax function normalizes the classifier's output to obtain the predicted probability of each event in each frame. By selecting an appropriate threshold, binarization is performed to obtain the final detection result.

[0024] The network architecture proposed in this embodiment includes a convolutional module, a temporal modeling module, and a high-level classifier, which capture local and contextual global features, respectively, thereby enhancing the network's representation learning ability.

[0025] Furthermore, the process of extracting time-frequency features through a preprocessing module to obtain log-Mel spectrum features includes: The audio signal is pre-emphasized and filtered to obtain the pre-emphasized signal; The pre-emphasized signal is subjected to frame segmentation processing with a frame overlap rate of 50% to obtain a set of frame signals. A Hamming window is applied to each frame signal in the frame signal set to obtain a windowed signal; Perform a short-time Fourier transform on the windowed signal to extract spectral information and obtain spectral characteristics; The spectral features are filtered by a bandpass filter, and the logarithmic energy is calculated to obtain the log-Mel spectral features.

[0026] Each pre-emphasized audio segment is divided into a set of overlapping frames. To prevent frequency leakage, the frame shift is 50%. This embodiment enhances the high-frequency components of the audio signal and balances the spectral distribution through pre-emphasis filtering; effectively reduces spectral leakage and maintains the temporal continuity between frames by using frame segmentation with a 50% overlap rate and combining it with Hamming window weighting; accurately extracts time-varying spectral information through short-time Fourier transform; and obtains log-Mel spectrum features that conform to the characteristics of human auditory perception by filtering with a bandpass filter and combining it with logarithmic energy calculation, thereby enhancing feature discrimination.

[0027] Furthermore, the convolution module includes three recursive gated convolutional layers stacked in sequence, with a non-linear activation layer, a batch normalization layer, and a max pooling layer set sequentially after each recursive gated convolutional layer; The process of obtaining spatial fused features through recursive gated convolution includes: The log-Mel spectrum features are channel-expanded and then input into the first recursive gated convolutional layer to obtain the first layer features. The output of the previous recursive gated convolutional layer is used as the input of the next recursive gated convolutional layer. This process is repeated for three layers to obtain deep fusion features. After processing through each recursive gated convolutional layer, max pooling, batch normalization, and ReLU activation functions are applied sequentially to obtain pooling activation features. The output of the third max pooling layer is dimensionally compressed to obtain spatial fusion features.

[0028] This embodiment achieves hierarchical feature extraction and progressive abstraction by stacking three recursively gated convolutional layers, resulting in rich high-level semantic representations. By sequentially adding nonlinear activation layers, batch normalization layers, and max pooling layers after each layer, the nonlinear expressive power of the model is enhanced, training convergence is accelerated, and the risk of overfitting is reduced. By expanding the input features with 1×1 convolutional channels, the initial feature representation is enriched, providing sufficient feature dimensions for subsequent high-order interactions.

[0029] Furthermore, max pooling only operates on the feature dimension, while the time dimension remains unchanged; After three layers of convolution and pooling, the output feature dimension is gradually compressed from the initial T×F×C to T×1×C, where T represents the number of frames, F represents the initial feature dimension, and C represents the number of channels. The output of T×1×C dimensions is compressed to T×C dimensions to obtain spatial fusion features.

[0030] Furthermore, as a preferred implementation, the convolutional module involved in this embodiment is composed of three recursively gated convolutional layers (RGCs) stacked sequentially, with a nonlinear activation layer, a batch normalization layer, and a max pooling layer sequentially set after each layer. The log-Mel spectrum features (with a shape of T×F, where T represents the number of audio frames and F represents the number of features per frame) are channel-expanded and then input into the first RGC layer. The expansion is achieved through 1×1 convolution to meet the number of channels required by the RGC. Subsequent RGC layers take the output of the previous convolutional layer as input to extract deeper features; Each RGC layer is followed by max pooling, batch normalization, and ReLU activation function in sequence. Max pooling is performed only in the feature dimension to keep the time dimension unchanged. After three layers of convolution and pooling, the output feature dimension is gradually compressed from T×F×C to T×F3×C; since F=40, and the max pooling and stride are respectively... Furthermore, pooling is non-overlapping, therefore F3=1; The output of the third pooling layer is compressed from T×1×C to T×C to obtain the fused spatial feature representation H.

[0031] Furthermore, the process by which recursively gated convolutional layers perform high-order spatial interactions on input features includes: Linear projection is performed on the input features to obtain the projected feature set; Recursively perform gated convolution operations on the projected feature groups to obtain recursively gated features; The last recursive result is fused at the channel level through a projection layer to obtain the spatially fused output features.

[0032] The audio signal is processed by a preprocessing module to output features, which are then input into a convolutional module for spatial feature interaction. The convolutional module consists of four parts: an RGC convolutional layer, a non-linear activation layer, a batch normalization layer, and a max pooling layer.

[0033] Specifically, Mel-spectral features interact recursively in the convolutional layers. When the interaction order is 1, RGC is similar to labeled gated convolution. Input features Output features The output of this convolutional layer can be represented as: in and This represents two 1×1 convolutional layers, responsible for processing features at the input and output stages, respectively. The purpose is to double the number of input channels, thus providing a more comprehensive feature representation for subsequent feature interactions. During the interaction, the depthwise convolution f focuses on extracting spatial features, but since it operates independently on each channel, it cannot directly achieve channel feature fusion. Therefore, a new feature is introduced... This achieves channel fusion, thereby ensuring effective integration of spatial and channel information at the output stage. Element-wise multiplication introduces adjacent features. and Interactions between them.

[0034] When the interaction order exceeds 2, RGC enhances the model's ability to hierarchically integrate information in the feature space by introducing higher-order interactions. For example, when the interaction order is set to 3, the structure of RGC is as follows: Figure 3 As shown, this structure can be extended to arbitrarily higher orders. Given input Output features The steps described are as follows: The first step is through linear mapping. A set of projection features is obtained and : in and .

[0035] The second step is to recursively perform gated convolutions: in, It is a hyperparameter used to scale the output. It is a set of depthwise convolutions, similar to the first-order depthwise convolution. It is primarily responsible for capturing local spatial information from features. It is a set of 1×1 standard convolutions used to dynamically adjust feature channels during recursion, which ensures that the features at each level of the recursive convolution can effectively interact and integrate.

[0036] The third step is to process the last recursive result. Through the projection layer Channel-level feature fusion is achieved to obtain the final output.

[0037] This embodiment obtains a projected feature group by linearly projecting the input features, mapping the features to a high-dimensional interaction space, and providing a basis for recursive operations; by recursively executing gated convolution operations, multi-level feature interaction is realized, enhancing the model's ability to integrate hierarchical information in the feature space; by fusing the last recursive result at the channel level, multi-channel spatial information is effectively integrated, and a structured deep feature representation is obtained.

[0038] Furthermore, the process of obtaining global information features through contextual information interaction via self-attention mechanisms includes: Based on the temporal dimension of the spatial fusion features, a location code is generated for each time frame to obtain the location code information; By linearly mapping the spatial fusion features through a linear layer, the dimensions are adjusted to the dimensions required by the multi-head self-attention mechanism, and multi-head partitioning is performed to obtain the query matrix, key matrix, and value matrix. Attention scores are calculated across different time frames based on the query matrix, key matrix, and value matrix to obtain attention weights; Multiply the attention weights by the value matrix to obtain the weighted features; The outputs of each attention point are concatenated and linearly transformed using a weight matrix to obtain global information features.

[0039] This embodiment generates positional codes for each time frame of spatially fused features, preserving the temporal positional information of the sequence and enhancing the model's ability to perceive the sequence order. It maps features to generate query matrices, key matrices, and value matrices through linear layers, transforming the features into a form suitable for attention calculation. It quantifies the correlation between different time frames by calculating attention scores, highlighting key temporal dependencies. By concatenating multi-head outputs and performing linear transformations, it integrates contextual information from multiple perspectives, obtaining enhanced global feature representation.

[0040] Furthermore, the process of calculating attention scores across different time frames based on the query matrix, key matrix, and value matrix, and obtaining attention weights, includes: Calculate the dot product of the query matrix and the transpose of the key matrix to obtain the first relevance score; Calculate the dot product of the query matrix and the relative position code to obtain the second relevance score; The first relevance score and the second relevance score are added together and normalized using the softmax function to obtain the attention weights.

[0041] Furthermore, after the sound features are fused by the convolution module, they are input into the temporal modeling module to interact with contextual information and obtain features containing global information.

[0042] Specifically, in this embodiment, the working principle of time-domain modeling is as follows: Figure 4 As shown, this module uses a multi-head self-attention network, which can measure the correlation between time frames by calculating attention scores to generate contextual representations. The following is a detailed description of this module: The first step is to process the output features from the convolutional module. The query is generated through three different linear transformations. ),key( ) and value ( ), where the weight matrix and All shapes are ,matrix All shapes are .here and All of them are integers that are divisible by the number of attention heads.

[0043] The second step is to determine the number of self-focused heads. , and It is divided into multiple parts and sent to different attention heads for parallel computation. For each attention head, the computation is performed... and The dot product, and the result is passed through Scaling is performed. Then, calculations are performed. The dot product of the relative position encoding and the result is multiplied Then add the two results together and normalize them using softmax. Finally, through... Multiplying this by the score gives the attention score.

[0044] The third step is to concatenate the outputs of the multiple heads mentioned above and combine them with the output weight matrix. Multiply them to obtain the final output of the module.

[0045] After the sound features are fused by the convolution module, they are input into the temporal modeling module to interact with contextual information and obtain features containing global information.

[0046] This embodiment captures the content relevance between frames by calculating the dot product of the query matrix and the transpose of the key matrix; it enhances the model's ability to model temporal positional relationships by introducing relative positional encoding to calculate the dot product; and it obtains a stable and interpretable attention weight distribution by normalizing the summed scores using the softmax function, ensuring the reliability of the attention mechanism.

[0047] Furthermore, the KANLinear classifier includes the main path and the residual path; The process of inputting global information features into the KANLinear classifier for frame-by-frame classification includes: The context interaction features are input into the B-spline transform of the main path and the linear transform of the residual path, respectively. Based on the main path, nonlinear mapping features are obtained through B-spline basis function calculation, weighted summation, and dimension transformation; Based on the residual path, linear mapping features are obtained through linear layer calculation; The nonlinear mapping features are added to the linear mapping features to obtain the fused features; The fused features are input into the Sigmoid activation function to obtain frame-by-frame classification results.

[0048] Specifically, B-spline transformation is used to capture the fine and non-linear relationships between features, while linear transformation is used to fit the simple linear mapping relationship between features. After adding the results of the two transformations, the results are processed by the Sigmoid activation function to obtain the final classification result Y.

[0049] This embodiment captures complex nonlinear relationships through the main path B-spline transformation of the KANLinear classifier, and fits simple mapping relationships by combining the linear transformation of the residual path, thus realizing parallel modeling of relationships at different levels of features. By adding and fusing the two transformation results, nonlinear and linear features are combined, improving classification robustness. Through processing with the Sigmoid activation function, a normalized probability output is obtained, which facilitates subsequent threshold decision processing.

[0050] Furthermore, the process of obtaining the nonlinear mapping features includes: Based on the preset node sequence and spline order, the B-spline basis function value corresponding to each element in the input feature is calculated recursively to obtain the basis function matrix; Weighted features are obtained by summing the basis function matrix using a learnable weight matrix. By performing dimensional transformation on the weighted features through a linear layer, nonlinear mapping features are obtained.

[0051] Furthermore, the features after temporal modeling contain global contextual information and need to be fed into a KANLinear classifier for prediction and classification. This classifier has a residual structure; the main path uses B-spline transformation, and the residual path uses linear transformation. Specifically, in this embodiment, the working principle of the KANLinear classifier is as follows: Figure 5 As shown.

[0052] The main path B-spline transform uses piecewise polynomial functions to approximate complex nonlinear relationships. When performing the B-spline transform, a set of node parameters and the order of the spline curve need to be preset. Nodes are used to define the B-spline basis functions. The input data range is divided into several intervals, each corresponding to a basis function. The order determines the degree of the polynomial and the smoothness of the curve. A higher order results in a more complex curve, enabling more accurate fitting of complex relationships, but also increases computational cost and complexity. The main steps of the B-spline transform are as follows: The first step is to compute the basis functions. For a given input... Calculate each element using recursion. The corresponding basis functions extend up to the specified order. A basis function is a set of piecewise defined polynomials, denoted as... ,in For the index of the basis functions, Let be the corresponding order. The recursive definition of the basis functions is as follows: 0th order basis functions: K-order basis functions (recursive definition): in, It is a sequence of nodes. The number of basis functions depends on the number of nodes and the order of the spline curve, typically the number of nodes minus the spline order minus one. This step is mainly responsible for generating the B-spline basis function value corresponding to each input point.

[0053] The second step is the weighted summation of the basis functions. After calculating the B-spline basis functions, these basis functions are weighted and summed using a set of learnable weight parameters: in, It is a learnable weight matrix. It is the number of basis functions recursively up to the highest order, which depends on the spline order. The number of nodes.

[0054] The third step is dimensional transformation. The result of the spline transformation... The dimensional transformation is performed using a linear layer, and the specific calculation is as follows: in, It is a weight matrix. For bias terms, It is the target dimension.

[0055] The residual path first performs nonlinear processing on the input, and then calculates the output through a linear layer. : in, and Represents the weights and biases of the linear layer. This is the SiLU activation function.

[0056] The final output of the KANlinear classifier is obtained by adding the residual output and the main path output, and then passing it through a Sigmoid activation function.

[0057] This embodiment calculates B-spline basis functions recursively, constructing a flexible piecewise polynomial function approximation capability to adapt to complex feature distributions; it uses a learnable weight matrix to perform weighted summation of the basis functions, adaptively adjusting the strength and shape of the nonlinear mapping; and it uses a linear layer to perform dimensional transformation, making the output dimension match the classifier requirements, ensuring effective feature transfer.

[0058] This embodiment preserves the temporal resolution and avoids the loss of temporal information by applying max pooling only to the feature dimension while keeping the time dimension unchanged. It achieves effective spatial dimension reduction by progressively compressing the feature dimension to T×1×C through three layers of convolution and pooling. Finally, by compressing the T×1×C dimension output to T×C dimension, a compact spatiotemporal feature representation is obtained, laying the foundation for subsequent temporal modeling.

[0059] This embodiment significantly reduces the number of parameters while ensuring feature extraction performance by leveraging the adaptive spatial mixing capability of recursively gated convolutional layers. It enhances temporal relationship modeling through a self-attention mechanism, replacing the traditional recurrent neural network structure, thus avoiding the gradient vanishing problem and improving the efficiency of long-range dependency modeling. Furthermore, it reduces the parameter scale of fully connected layers through the KANLinear classifier's spline function-based nonlinear mapping, thereby reducing computational complexity while enhancing nonlinear expressive power and achieving a lightweight design for the overall network architecture.

[0060] To obtain event activity prediction results, the output probability values ​​of the prediction module are filtered by using a pre-set threshold.

[0061] In addition, this embodiment also verifies a sound event detection method based on RGC and self-attention networks: To evaluate the effectiveness of the proposed method, the TUT-SEDSynthetic2016 dataset was used, a dataset specifically created for sound event detection tasks. This dataset contains isolated sound samples from 16 sound event categories, which were artificially mixed to generate mixed signals. The dataset contains a total of 100 mixed audio samples, with a total duration of approximately 566 minutes. These mixed signals have different polyphony and random silent regions to simulate the complexity of sound events in the real world. Sound event categories include alarms, baby cries, bird calls, bus sounds, cat meows, clapping, cheers, dog barks, footsteps, glass breaking sounds, gunshots, horse moos, etc. Samples for each sound event category are detailedly annotated, including the start and end times of the event. These annotations are optimized based on an automatic signal energy annotation method to improve the accuracy and consistency of the annotations. The dataset was divided into training, validation, and testing parts, accounting for 60%, 20%, and 20%, respectively. To evaluate the model's performance, the most authoritative evaluation metrics for sound event detection tasks were used: segment-based F1 score and error rate (ER). In addition, precision and recall were introduced as supplementary evaluation metrics in some experiments.

[0062] Figure 6 A comparison of the proposed method's F1 score and ER with other deep learning-based sound event detection methods on this dataset is provided. First, as baseline models, TFCRNN and CRNN achieve F1 scores of 0.54 and 0.59, respectively, and ER scores of 0.61 and 0.54, respectively. These results indicate that while both models demonstrate some ability to capture temporal features of audio data, their relatively low F1 scores and high ER scores suggest limitations in handling overlapping events and complex dependencies. Notably, TFCRNN's performance degrades compared to CRNN. ​​Although the teacher-forced and plan sampling mechanisms introduced in TFCRNN enhance learning in some natural language processing tasks, these improvements fail to translate effectively into the SED task. Specifically, these mechanisms cause the model to over-rely on sequence prediction, thus diverting attention from frame-level event prediction and limiting accurate feature modeling. The DWS-DIL model enhances feature extraction capabilities by introducing DWSConv and DILConv, improving its F1 score to 0.63 and reducing its error rate to 0.50. This demonstrates that optimizing the convolutional structure is effective in capturing spatial information. However, although DILConov expands the receptive field, it is still insufficient to fully capture long-range temporal dependencies, limiting the model's ability to comprehensively model the context and thus restricting further performance improvements. These results demonstrate the effectiveness of the proposed method in improving the accuracy of sound event detection and reducing the error rate.

[0063] This embodiment not only focuses on performance improvement but also places great emphasis on lightweight characteristics. Therefore, the parameters and floating-point operation counts (FLOPs) of each model were recorded, and... Figure 6 The document provides a comparison of their parameters and FLOPs. From... Figure 6 As can be seen, both CRNN and TFCRNN models have nearly 3.7 million parameters and over 17.5 billion FLOPs. This indicates that although these two benchmark models possess certain feature extraction and time series modeling capabilities, their high parameter count and computational requirements significantly increase resource consumption. Such high computational demands make them unsuitable for deployment in resource-constrained environments. The DWS-DIL model successfully reduces the number of parameters to 580,000 and the number of floating-point operations to 5.36 trillion, significantly reducing the computational load while maintaining reasonable feature extraction capabilities. The proposed RGC-SA (KANLinear) model, with 57 million parameters and 4.85 trillion floating-point operations, is currently the lightest model. Compared to the DWS-DIL model, these values ​​are reduced by 1.72% and 9.51%, respectively. These results demonstrate that the proposed RGC-SA (KANLinear) achieves a more efficient architecture with lower computational and storage requirements, highlighting its potential in practical applications, especially in real-time scenarios or on resource-constrained devices.

[0064] As an additional implementation, this embodiment also proposes a novel lightweight SED method that combines RGC and self-attention mechanisms to enhance the network's representation learning capabilities, thereby accurately detecting multiple sound events within each time frame. Log-Melband energy features obtained from the audio signal are used as input to a convolutional module containing three stacked RGC layers for high-level feature extraction. Each RGC layer performs recursive spatial interactions to capture complex temporal and spatial dependencies and internally uses 1×1 convolutions to fuse multi-channel information. The progressive stacking of these three layers progressively refines the feature representation, resulting in useful features with fine temporal granularity. These high-level features are then processed by self-attention layers to capture short-range and long-range contextual relationships between sound events. By focusing on different temporal locations, self-attention effectively models global dependencies, thereby improving the ability to identify overlapping and sequential events. Finally, the enhanced features are fed into a KANLinear classifier for frame-level prediction. Unlike traditional linear classifiers, KANLinear improves detection accuracy by fitting complex decision boundaries through spline-based nonlinear mappings.

[0065] The method described in this embodiment acquires raw audio signals and extracts their time-frequency features. These features are then input into a sound event detection model to obtain the categories of sound events and their occurrence time periods in the audio. The sound event detection model is constructed from a convolutional module, a self-attention module, and a KANLinear classifier. The convolutional module consists of three stacked RGC layers, dynamically fusing information from different spatial dimensions through the adaptive spatial mixing capability of the RGC to obtain a structured and information-rich feature representation. A relative position encoding mechanism is introduced in the self-attention module to enhance the model's ability to model the temporal relationships and contextual dependencies between sequence elements. The features after temporal modeling are input into the KANLinear classifier, which uses spline functions to implement nonlinear mapping for frame-by-frame classification. Compared with traditional methods, this embodiment significantly improves the accuracy of sound event detection with fewer parameters and fewer floating-point operations.

[0066] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A sound event detection method based on recursive gated convolution and self-attention mechanism, characterized in that, include: Collect the audio signal to be detected and build a sound event detection model; The audio signal is input into the sound event detection model, and the time-frequency features are extracted through the preprocessing module to obtain the log-Mel spectrum features; The log-Mel spectrum features are input into the convolution module, and spatial features are fused through recursive gated convolution to obtain spatial fused features; The spatial fusion features are input into the temporal modeling module, and contextual information is exchanged through a self-attention mechanism to obtain global information features; The global information features are input into the KANLinear classifier for frame-by-frame classification to obtain the category of each sound event in the audio and its occurrence time.

2. The method according to claim 1, characterized in that, The sound event detection model is constructed using a lightweight network architecture consisting of a recursive gated convolutional network, a self-attention network, and a KANLinear classifier. The lightweight network architecture achieves adaptive spatial mixing through recursive gated convolutional layers, enhances temporal relationship modeling through a self-attention mechanism, and implements nonlinear mapping based on spline functions through a KANLinear classifier.

3. The method according to claim 1, characterized in that, The process of extracting time-frequency features through a preprocessing module to obtain log-Mel spectrum features includes: The audio signal is pre-emphasized and filtered to obtain the pre-emphasized signal; The pre-emphasized signal is subjected to frame segmentation processing to obtain a set of frame signals; A Hamming window is applied to each frame signal in the frame signal set to obtain a windowed signal; Perform a short-time Fourier transform on the windowed signal to extract spectral information and obtain spectral features; The spectral features are filtered by a bandpass filter, and the logarithmic energy is calculated to obtain the log-Mel spectral features.

4. The method according to claim 1, characterized in that, The convolution module includes three recursive gated convolutional layers stacked in sequence, with a non-linear activation layer, a batch normalization layer, and a max pooling layer set sequentially after each recursive gated convolutional layer. The process of obtaining spatial fused features through recursive gated convolution includes: The log-Mel spectrum features are channel-expanded and then input into the first recursive gated convolutional layer to obtain the first layer features. The output of the previous recursive gated convolutional layer is used as the input of the next recursive gated convolutional layer. This process is repeated for three layers to obtain deep fusion features. After processing through each recursive gated convolutional layer, max pooling, batch normalization, and ReLU activation functions are applied sequentially to obtain pooling activation features. The output of the third max pooling layer is dimensionally compressed to obtain spatial fusion features.

5. The method according to claim 4, characterized in that, The max pooling operation only applies to the feature dimension, while the time dimension remains unchanged; After three layers of convolution and pooling, the output feature dimension is gradually compressed from the initial T×F×C to T×1×C, where T represents the number of frames, F represents the initial feature dimension, and C represents the number of channels. The output of T×1×C dimensions is compressed to T×C dimensions to obtain spatial fusion features.

6. The method according to claim 4, characterized in that, The process by which the recursively gated convolutional layer performs high-order spatial interactions on the input features includes: Linear projection is performed on the input features to obtain the projected feature set; Recursively perform gated convolution operations on the projected feature group to obtain recursively gated features; The last recursive result is fused at the channel level through a projection layer to obtain the spatially fused output features.

7. The method according to claim 1, characterized in that, The process of obtaining global information features through contextual information interaction via self-attention mechanisms includes: Based on the temporal dimension of the spatial fusion features, a location code is generated for each time frame to obtain location code information; The spatial fusion features are linearly mapped by a linear layer to adjust the dimensions to the dimensions required by the multi-head self-attention mechanism, and multi-head partitioning is performed to obtain the query matrix, key matrix, and value matrix. Based on the query matrix, key matrix, and value matrix, the attention scores between different time frames are calculated to obtain the attention weights; Multiply the attention weights by the value matrix to obtain the weighted features; The outputs of each attention point are concatenated and linearly transformed using a weight matrix to obtain global information features.

8. The method according to claim 7, characterized in that, The process of calculating attention scores across different time frames based on the query matrix, key matrix, and value matrix, and obtaining attention weights, includes: Calculate the dot product of the query matrix and the transpose of the key matrix to obtain the first relevance score; Calculate the dot product of the query matrix and the relative position code to obtain the second relevance score; The first relevance score and the second relevance score are added together and normalized using the softmax function to obtain the attention weights.

9. The method according to claim 1, characterized in that, The KANLinear classifier includes a main path and a residual path; The process of inputting the global information features into the KANLinear classifier and performing frame-by-frame classification includes: The context interaction features are input into the B-spline transform of the main path and the linear transform of the residual path, respectively. Based on the main path, nonlinear mapping features are obtained through B-spline basis function calculation, weighted summation, and dimensional transformation; Based on the residual path, linear mapping features are obtained through linear layer calculation; The nonlinear mapping feature is added to the linear mapping feature to obtain the fused feature; The fused features are input into the Sigmoid activation function to obtain frame-by-frame classification results.

10. The method according to claim 9, characterized in that, The process of obtaining nonlinear mapping features includes: Based on the preset node sequence and spline order, the B-spline basis function value corresponding to each element in the input feature is calculated recursively to obtain the basis function matrix; The basis function matrix is ​​weighted and summed using a learnable weight matrix to obtain weighted features; By performing dimensional transformation on the weighted features through a linear layer, nonlinear mapping features are obtained.

Images