A sound event detection method and system based on group feature calibration

Abstract

This invention provides a sound event detection method and system based on grouped feature calibration, comprising: acquiring audio feature data of the sound event to be detected; inputting the audio feature data into a time-frequency learning network, obtaining a time-spectrum graph through a convolutional neural network, performing grouped feature learning based on the intermediate representations of the time-spectrum graph from multiple dimensions to obtain grouped enhanced features, performing task-aware activation on the grouped enhanced features to obtain adaptive features; inputting the adaptive features into a context modeling network to obtain the time-domain correlation features of the audio signal, classifying the time-domain correlation features of the audio signal, and obtaining the sound event category detection result. This invention introduces a grouped feature calibration module based on the time-frequency characteristics of different types of audio in the sound event detection task, enhancing the feature representation capability of the sound event detection network for various types of audio, with a small number of parameters and strong versatility, and introducing it into existing mainstream sound event detection models with a relatively low computational cost while improving their performance.

Description

Technical Field

[0001] This invention relates to the field of digital media processing technology, and in particular to a method and system for detecting sound events based on grouped feature calibration. Background Technology

[0002] In daily life, sound signals contain a wealth of scene and event information. Sound Event Detection (SED) technology analyzes the semantic features of audio signals to uncover abnormal events in audio and video signals, and has important application value in fields such as urban security monitoring and cyberspace content security review.

[0003] SED (Search Engine Optimization) comprises two subtasks: event classification and boundary detection. These subtasks aim to identify the categories of various concurrent events within a sound segment and the temporal boundaries of each event, respectively. Currently, the most commonly used architecture for SED models based on deep neural networks is the Convolutional Recurrent Neural Network (CRNN), such as... Figure 1 As shown, for the front-end audio features input to the network, firstly, time-frequency learning networks such as Convolutional Neural Networks (CNNs) are used to extract high-level features with translation invariance from the time-frequency spectrogram and progressively compress the frequency dimension. Then, context modeling networks such as Recurrent Neural Networks (RNNs) are used to learn the long-term correlation of the audio signal in the time domain. Because sound events exhibit complex and varied behavior in the natural world, with different events having different durations and frequency distributions, constituting complex time-frequency dynamics, extracting effective high-dimensional time-frequency representations is crucial for improving the model's detection capabilities.

[0004] Existing work largely focuses on structurally optimizing time-frequency learning networks to obtain more effective feature representations. Optimization of time-frequency learning networks aims to achieve more effective time-frequency relationship modeling and can be broadly categorized into feature selection-based methods and representation learning-based methods. Among feature selection-based methods, a representative structure is a pooling-based attention module, such as... Figure 2As shown, this module learns attention information from the channel dimension or time-frequency dimension and then reweights the network representation output by each CNN layer to obtain a more discriminative representation that is more favorable for downstream classification tasks. Since attention mechanisms along only a single dimension cannot adapt to the time-frequency representation requirements of different events, most works stack and fuse attention modules of different dimensions through parallel or serial methods. However, stacking modules will lead to high computational complexity and cannot be integrated into existing network structures at a low cost. In addition, the serial or parallel stacking method will lead to the loss of feature information of different dimensions to a certain extent, and the serial method needs to further consider the impact of the order of module stacking. Among the representation learning-based methods, a representative structure is the Selective Kernel (SK) module. This module is a structure that splits and then fuses, changing the CNN module from a single-branch computation with a fixed receptive field to a multi-branch computation with multiple convolutional kernels of different sizes, and learning the fusion weights of different branches through a squeeze excitation (SE) structure, so that the receptive field size can be adaptively adjusted within a certain range for different inputs. In the SED task, this structure can capture sound events with complex time-frequency dynamics locally, significantly improving the detection performance of short-term non-stationary events, but its performance degrades on some stationary categories. Another representative structure based on representation learning methods is frequency dynamic convolution, such as... Figure 3 As shown, this structure adaptively learns the convolution kernel parameters based on the frequency band information of different inputs, solving the problem that the translation invariance of CNN structures with fixed convolution kernels is not suitable for audio frequency band modeling. In the SED task, this structure can effectively improve the detection performance of the SED model for non-stationary sound events, while the detection performance for sound events with stable frequency components decreases slightly.

[0005] It can be seen that existing SED models often struggle to simultaneously adapt to the time-frequency representation requirements of different events. Events with stable time-frequency relationships have frequency components that remain approximately constant over time, while events with unstable time-frequency relationships require more flexible time-frequency modeling. Furthermore, there are conflicting requirements between the event classification and boundary detection subtasks of SED. Event classification requires deeper and more global representations, integrating audio contextual information through a larger receptive field, while boundary detection relies more on detailed information and requires relatively lower-level, higher temporal resolution representations.

[0006] Therefore, to further improve the detection performance of the model, it is necessary to study more effective time-frequency feature representation methods to improve the accuracy of sound event detection. Summary of the Invention

[0007] This invention provides a sound event detection method and system based on group feature calibration, which addresses the shortcomings of existing SED networks in representing complex time-frequency dynamics in audio.

[0008] In a first aspect, the present invention provides a sound event detection method based on grouped feature calibration, comprising:

[0009] Obtain the audio feature data of the sound event to be detected;

[0010] The audio feature data is input into a time-frequency learning network, and a time-frequency spectrogram is obtained through a convolutional neural network. Based on multiple dimensions, the intermediate representations of the time-frequency spectrogram are grouped to learn grouped enhanced features. The grouped enhanced features are then activated by task awareness to obtain adaptive features.

[0011] The adaptive features are input into the context modeling network to obtain the temporal correlation features of the audio signal. The temporal correlation features of the audio signal are then classified to obtain the sound event category detection results.

[0012] According to the present invention, a sound event detection method based on grouped feature calibration obtains grouped enhancement features by learning grouped features from the intermediate representation of the time-spectrum graph based on multiple dimensions, including:

[0013] Obtain any training data from the time-spectrum graph, and determine that the feature tensor output by the intermediate feature representation layer from the training data includes channel dimension, time dimension, and frequency dimension;

[0014] Based on the channel dimension, the feature tensor is divided into features to be calibrated and features to be retained according to a preset ratio;

[0015] The features to be calibrated are divided into time-domain features to be calibrated, frequency-domain features to be calibrated, global features to be calibrated, and local features to be calibrated based on the average channel dimension.

[0016] Feature learning and reweighting are performed on the time-domain features to be calibrated, the frequency-domain features to be calibrated, the global features to be calibrated, and the local features to be calibrated to obtain grouped calibration features;

[0017] The grouping calibration feature is connected with the retention feature to obtain the grouping enhancement feature.

[0018] According to the present invention, a sound event detection method based on grouped feature calibration performs feature learning and reweighting on the temporal features to be calibrated, including:

[0019] One-dimensional frequency domain pooling is used to compress the frequency dimension of the time-domain feature to be calibrated to one dimension, thus obtaining the intermediate time-domain feature.

[0020] The feature channel dimension of the intermediate temporal feature is compressed and expanded by two concatenated one-dimensional convolutions respectively, so as to obtain the temporal context-dependent features of the intermediate temporal feature;

[0021] The temporal context-dependent features are reweighted using the sigmoid activation function and expanded to the input dimension of the temporal domain features to be calibrated. Then, they are multiplied with the temporal domain features to be calibrated to obtain the temporal domain learning features.

[0022] According to the present invention, a sound event detection method based on grouped feature calibration performs feature learning and reweighting on the frequency domain features to be calibrated, including:

[0023] One-dimensional time-domain pooling is used to compress the time dimension of the frequency domain feature to be calibrated to one dimension, thus obtaining the intermediate frequency domain feature.

[0024] The graph node representation set of the intermediate features in the frequency domain is obtained through a graph attention network;

[0025] After multiplying the element-wise representations of any two original nodes in the graph node representation set, the hyperbolic tangent function is obtained, and then multiplied with the learnable weight parameter. The attention weight between any two nodes is calculated using the normalized exponential function.

[0026] The aggregated feature of any node is obtained by summing the product of the attention weights between any two nodes and the representation of any original node.

[0027] By performing a residual connection between the aggregated features of any node and the representation of any original node through a batch normalization layer, the graph node output features are obtained.

[0028] The frequency domain learning features are obtained by reweighting the output features of the graph nodes using the sigmoid activation function and expanding them to the input dimension of the frequency domain features to be calibrated.

[0029] According to the present invention, a sound event detection method based on grouped feature calibration performs feature learning and reweighting on the global features to be calibrated, including:

[0030] Two-dimensional global pooling is used to compress the time dimension and frequency dimension of the global features to be calibrated into one dimension, respectively, to obtain the global context channel representation features;

[0031] Global channel dimension-dependent features are obtained by using one-dimensional convolution to represent global context channel features.

[0032] The global channel dimension-dependent features are reweighted using the sigmoid activation function and expanded to the input dimension of the global features to be calibrated. The resulting global learning features are then multiplied by the global features to be calibrated to obtain the global learning features.

[0033] According to the present invention, a sound event detection method based on grouped feature calibration performs feature learning and reweighting on the local features to be calibrated, including:

[0034] The local features to be calibrated are split into convolutional features of different scales through a convolutional transformation containing multiple convolutional branches;

[0035] The convolutional features at different scales are added element-wise to obtain fused features. Global pooling is then performed on the fused features to compress the frequency dimension and time dimension of the fused features into one dimension, thereby obtaining global context channel representation features.

[0036] The channel dimension of the global context channel representation features is compressed to one-quarter of the initial channel dimension by squeezing the excitation structure. The compressed global context channel representation features are then expanded to the initial channel dimension by fully connected layers corresponding to the multiple convolutional branches, resulting in global context channel representation features for different branches.

[0037] The weights of different branch channels are obtained by calculating the same channels of the global context channel representation features of different branches using a normalized exponential function;

[0038] The local learning features are obtained by weighting and summing the different branch channel weights with the different branch global context channel representation features.

[0039] According to the present invention, a sound event detection method based on grouped feature calibration is provided, wherein the grouped enhancement features are subjected to task-aware activation to obtain adaptive features, including:

[0040] The grouped enhancement features are split into positive features and negative features. Global pooling is used to compress the frequency dimension and time dimension of the positive features and negative features to one dimension, resulting in compressed positive features and compressed negative features.

[0041] The connection feature is obtained by connecting the compressed positive feature and the compressed negative feature based on the channel dimension. The connection feature is then input into the extrusion excitation structure to obtain the excitation feature.

[0042] The activation features are split according to the channel dimension to obtain positive feature weights and negative feature weights;

[0043] The adaptive feature is obtained by multiplying the positive feature by its weight and then adding the product of the negative feature and its weight.

[0044] Secondly, the present invention also provides a sound event detection system based on grouped feature calibration, comprising:

[0045] The acquisition module is used to acquire the audio feature data of the sound event to be detected;

[0046] The calibration module is used to input the audio feature data into the time-frequency learning network, obtain the time-frequency spectrum through the convolutional neural network, learn grouped features based on the intermediate representations of the time-frequency spectrum from multiple dimensions to obtain grouped enhanced features, and perform task-aware activation on the grouped enhanced features to obtain adaptive features.

[0047] The detection module is used to input the adaptive features into the context modeling network, obtain the temporal correlation features of the audio signal, classify the temporal correlation features of the audio signal, and obtain the sound event category detection result.

[0048] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the sound event detection method based on group feature calibration as described above.

[0049] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sound event detection method based on group feature calibration as described above.

[0050] The sound event detection method and system based on grouped feature calibration provided by this invention introduces a grouped feature calibration module according to the time-frequency characteristics of different types of audio in the sound event detection task. It performs grouped feature learning on intermediate representations from multiple dimensions and adopts task-aware activation to achieve adaptive feature selection, which enhances the feature representation capability of the sound event detection network for various types of audio. It has the characteristics of small number of parameters and strong versatility, and is introduced into the existing mainstream sound event detection model with a small computational cost and improves its performance. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0052] Figure 1 This is a general framework diagram of the SED model based on the CRNN architecture provided by existing technologies;

[0053] Figure 2 This is a general structural diagram of the attention module in SED provided by existing technology;

[0054] Figure 3 It is a frequency dynamic convolution structure diagram provided by existing technology;

[0055] Figure 4 This is a flowchart illustrating the sound event detection method based on grouped feature calibration provided by the present invention;

[0056] Figure 5 This is an overall structural diagram of the group feature calibration module in the SED model provided by the present invention;

[0057] Figure 6 This is a structural diagram of the group feature learning submodule provided by the present invention;

[0058] Figure 7 This is a structural diagram of the graph attention network in frequency domain feature learning provided by the present invention;

[0059] Figure 8 This is a structural diagram of the selective core module provided by the present invention;

[0060] Figure 9 This is a structural diagram of the task-aware activation submodule provided by the present invention;

[0061] Figure 10 This is a comparison chart of PSDS1 scores for each category in the DCASE dataset when using a single feature learning branch, as provided by this invention.

[0062] Figure 11 This is a comparison chart of PSDS2 scores for each category in the DCASE dataset when using a single feature learning branch, as provided by this invention.

[0063] Figure 12 This is a schematic diagram of the sound event detection system based on grouped feature calibration provided by the present invention;

[0064] Figure 13 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0066] To address the problem that existing SED networks lack the ability to represent complex time-frequency dynamics in audio, this invention proposes a sound event detection method based on grouped feature calibration.

[0067] Figure 4 This is a flowchart illustrating the sound event detection method based on grouped feature calibration provided in an embodiment of the present invention, as shown below. Figure 4 As shown, it includes:

[0068] Step 100: Obtain the audio feature data of the sound event to be detected;

[0069] Step 200: Input the audio feature data into the time-frequency learning network, obtain the time-frequency spectrogram through the convolutional neural network, learn grouped features based on the intermediate representations of the time-frequency spectrogram from multiple dimensions to obtain grouped enhanced features, and perform task-aware activation on the grouped enhanced features to obtain adaptive features;

[0070] Step 300: Input the adaptive features into the context modeling network to obtain the temporal correlation features of the audio signal, classify the temporal correlation features of the audio signal, and obtain the sound event category detection results.

[0071] Based on the analysis of the time-frequency relationship differences between the two sub-tasks of SED and different event classes, this invention proposes a plug-and-play Group Feature Calibration (GFC) module. This module refers to the idea of ​​grouped convolution and performs group feature learning on the intermediate representation output of the SED feature representation network from four dimensions: time domain, frequency domain, global and local. It also uses task-aware activation to achieve adaptive feature selection, thereby enhancing the SED network's feature representation capability for various types of audio.

[0072] Specifically, the overall structure of GFC is as follows: Figure 5 As shown, for any SED network, this module can be placed after the CNN block of each time-frequency learning network layer to calibrate the intermediate feature representation of the CNN block output layer by layer from multiple perspectives. This module includes two sub-modules: grouped feature learning and task-aware activation. Figure 1 Based on the existing SED model framework, a group feature learning module and a task-aware activation module are added after the CNN block in the time-frequency learning network. Subsequent context modeling networks and classification... Figure 1 The structure is the same. It's understandable that the data is N-dimensional in time-frequency learning networks, M-dimensional in context modeling networks, and transformed into one-dimensional data in classification.

[0073] The GFC module proposed in this invention has the characteristics of small number of module parameters and strong versatility. It can be applied to various existing mainstream SED models with low computational cost and improve model performance.

[0074] Based on the above embodiments, grouped enhancement features are obtained by performing grouped feature learning on the intermediate representations of the time-spectrum graph based on multiple dimensions, including:

[0075] Obtain any training data from the time-spectrum graph, and determine that the feature tensor output by the intermediate feature representation layer from the training data includes channel dimension, time dimension, and frequency dimension;

[0076] Based on the channel dimension, the feature tensor is divided into features to be calibrated and features to be retained according to a preset ratio;

[0077] The features to be calibrated are divided into time-domain features to be calibrated, frequency-domain features to be calibrated, global features to be calibrated, and local features to be calibrated based on the average channel dimension.

[0078] Feature learning and reweighting are performed on the time-domain features to be calibrated, the frequency-domain features to be calibrated, the global features to be calibrated, and the local features to be calibrated to obtain grouped calibration features;

[0079] The grouping calibration feature is connected with the retention feature to obtain the grouping enhancement feature.

[0080] The process of performing feature learning and reweighting on the time-domain features to be calibrated includes:

[0081] One-dimensional frequency domain pooling is used to compress the frequency dimension of the time-domain feature to be calibrated to one dimension, thus obtaining the intermediate time-domain feature.

[0082] The feature channel dimension of the intermediate temporal feature is compressed and expanded by two concatenated one-dimensional convolutions respectively, so as to obtain the temporal context-dependent features of the intermediate temporal feature;

[0083] The temporal context-dependent features are reweighted using the sigmoid activation function and expanded to the input dimension of the temporal domain features to be calibrated. Then, they are multiplied with the temporal domain features to be calibrated to obtain the temporal domain learning features.

[0084] The process of performing feature learning and reweighting on the frequency domain features to be calibrated includes:

[0085] One-dimensional time-domain pooling is used to compress the time dimension of the frequency domain feature to be calibrated to one dimension, thus obtaining the intermediate frequency domain feature.

[0086] The graph node representation set of the intermediate features in the frequency domain is obtained through a graph attention network;

[0087] After multiplying the element-wise representations of any two original nodes in the graph node representation set, the hyperbolic tangent function is obtained, and then multiplied with the learnable weight parameter. The attention weight between any two nodes is calculated using the normalized exponential function.

[0088] The aggregated feature of any node is obtained by summing the product of the attention weights between any two nodes and the representation of any original node.

[0089] By performing a residual connection between the aggregated features of any node and the representation of any original node through a batch normalization layer, the graph node output features are obtained.

[0090] The frequency domain learning features are obtained by reweighting the output features of the graph nodes using the sigmoid activation function and expanding them to the input dimension of the frequency domain features to be calibrated.

[0091] The process of performing feature learning and reweighting on the global features to be calibrated includes:

[0092] Two-dimensional global pooling is used to compress the time dimension and frequency dimension of the global features to be calibrated into one dimension, respectively, to obtain the global context channel representation features;

[0093] Global channel dimension-dependent features are obtained by using one-dimensional convolution to represent global context channel features.

[0094] The global channel dimension-dependent features are reweighted using the sigmoid activation function and expanded to the input dimension of the global features to be calibrated. The resulting global learning features are then multiplied by the global features to be calibrated to obtain the global learning features.

[0095] The process of performing feature learning and reweighting on the local features to be calibrated includes:

[0096] The local features to be calibrated are split into convolutional features of different scales through a convolutional transformation containing multiple convolutional branches;

[0097] The convolutional features at different scales are added element-wise to obtain fused features. Global pooling is then performed on the fused features to compress the frequency dimension and time dimension of the fused features into one dimension, thereby obtaining global context channel representation features.

[0098] The channel dimension of the global context channel representation features is compressed to one-quarter of the initial channel dimension by squeezing the excitation structure. The compressed global context channel representation features are then expanded to the initial channel dimension by fully connected layers corresponding to the multiple convolutional branches, resulting in global context channel representation features for different branches.

[0099] The weights of different branch channels are obtained by calculating the same channels of the global context channel representation features of different branches using a normalized exponential function;

[0100] The local learning features are obtained by weighting and summing the different branch channel weights with the different branch global context channel representation features.

[0101] Specifically, the Group Feature Learning (GFL) submodule proposed in this embodiment aims to group and enhance a subset of channel features in the intermediate representation of the network from multiple dimensions. Its overall framework is as follows: Figure 6 As shown.

[0102] For any data in a training batch, let the feature tensor output by its intermediate feature representation layer be... Wherein, C, T, and F represent the channel dimension, time dimension, and frequency dimension, respectively.

[0103] First, the feature tensor is split into two parts along the channel dimension, and then... The features are used as the features to be calibrated, and the remaining features are... The features are preserved, where p = 0.5. Then, the partial features X to be calibrated are... cal Further split along the channel dimension into four feature groups of the same dimension. Feature learning and reweighting are performed independently and in parallel from four perspectives: time domain, frequency domain, global domain, and local domain, resulting in four sets of feature learning outputs. Finally, the grouped and calibrated features are concatenated with the retained features along the channel dimension to obtain an output with the same dimension as the module input. Right now:

[0104]

[0105] The core of the group feature learning submodule is to guide attention modeling from different dimensions, and then reweight the input features after learning the attention weights.

[0106] The first is temporal learning (TL), such as... Figure 5 As shown, firstly, one-dimensional frequency domain pooling is used to compress the frequency dimension of the input features to obtain intermediate features. Then, referring to the squeeze excitation structure, two cascaded one-dimensional convolutions (1*1 Conv) are used to compress and expand the feature channel dimension to learn the temporal context dependency. The first convolution compresses the channel dimension from pC / 4 to pC / 4r, where r = 4, and the second convolution expands the channel dimension from pC / 4r to pC / 4. Finally, the reweighted weights are obtained by applying the sigmoid activation function, expanding the input dimension, and then multiplying by the original input to obtain the feature representation after temporal feature learning.

[0107] The second is frequency domain feature learning (FL), such as... Figure 5 As shown, firstly, one-dimensional time pooling is used to compress the time dimension of the input features to obtain intermediate features. Then, a Graph Attention Network (GAT) is used to model the global dependencies between different frequency bands. Finally, reweighted weights are obtained through sigmoid and expand operations, and multiplied by the original input to obtain the feature representation after frequency domain feature learning.

[0108] Here, the GAT structure in frequency domain feature learning is as follows: Figure 7 As shown, in order to use GAT for frequency band relationship modeling, the representations of different frequency bands are... View the graph node representation set and For ease of display, Figure 7 GAT structure in Figure 6 It is simplified to 5 nodes e1, e2, e3, e4, and e5. The representation dimension of each node e is pC / 4. The entire frequency band is regarded as a complete graph of number F nodes connected by undirected edges. Among them, for complete graphs The attention weight between any two nodes n and v in the network is calculated using the following formula:

[0109] ω n,c =softmax(W att ·tanh(e n ⊙e v ))

[0110] Among them, W att Let be the learnable weight parameters, e be the node representation, and ⊙ denote element-wise multiplication. GAT updates the feature representation of each node by aggregating neighboring nodes through attention weights. For node n, its aggregated feature m n Represented as:

[0111]

[0112] in, Let m be the set of neighboring nodes of node n, which is equivalent to the entire set of graph nodes in the frequency band feature learning task. Finally, the output of each node in GAT is generated by the aggregated feature m. n With the original node representation e n The residual connections are summed to obtain the result. For node n, after one layer of GAT, the output is as follows:

[0113] GAT(e n )=BN(f att (m n )+f res (e n ))

[0114] Where BN represents the batch normalization layer, f att This represents a linear layer that maps aggregated features m to the output dimension, f. res This represents a linear layer that maps the original node representation e to the output dimension.

[0115] The third is Global Learning (GL), such as... Figure 5 As shown, firstly, two-dimensional adaptive global pooling is used to compress the time and frequency dimensions of the input features to obtain the global context channel representation. Then, a one-dimensional pointwise convolution with a 1*1 kernel is used to model the interdependent features of the channel dimensions. Finally, after sigmoid and expand operations to obtain reweighted weights, the dot product of the original input yields the feature representation after global feature learning.

[0116] The fourth is Local Learning (LL), which employs a selective kernel structure, as shown in the diagram. Figure 8 As shown, this structure can adaptively adjust the receptive field size of each neuron, flexibly sensing events with different scales in the time-frequency domain. The selective kernel module is a multi-branch convolution and reweighted fusion structure, including three core operations: splitting, fusion, and selection.

[0117] Splitting, which involves applying several convolutions of different scales to the same feature input, can be represented as: in, For convolution transformation, N is the number of convolution branches. For the output of each convolutional branch, in this embodiment of the invention, N=2, and two convolutional branches with kernels of 3*3 and 5*5 are used.

[0118] Fusion refers to the fusion and compression of multi-resolution features. First, the results of multiple convolutional branches are added element-wise to obtain... Then, the fusion features are analyzed. Perform global pooling to compress the frequency and time axes to one dimension and obtain the global context channel representation.

[0119] Selection involves reweighting and summing the outputs of different convolutional branches along the channel dimension, adaptively allocating contributions to the features of different branches. The calculation of the weights for each branch employs a squeeze excitation (SE) structure, first further compressing... The channel dimension is obtained. Then apply a fully connected layer with an equal number of branches, Extended to the initial channel dimension Finally, different branches The channel weights for different branches are obtained by applying the normalized exponential function softmax activation to the same channel. The feature representation obtained after weighted summation of each branch is the feature representation learned from local features.

[0120] Based on the above embodiments, task-aware activation is applied to the grouped enhanced features to obtain adaptive features, including:

[0121] The grouped enhancement features are split into positive features and negative features. Global pooling is used to compress the frequency dimension and time dimension of the positive features and negative features to one dimension, resulting in compressed positive features and compressed negative features.

[0122] The connection feature is obtained by connecting the compressed positive feature and the compressed negative feature based on the channel dimension. The connection feature is then input into the extrusion excitation structure to obtain the excitation feature.

[0123] The activation features are split according to the channel dimension to obtain positive feature weights and negative feature weights;

[0124] The adaptive feature is obtained by multiplying the positive feature by its weight and then adding the product of the negative feature and its weight.

[0125] Specifically, in this embodiment of the invention, the Task-aware Activation (TA) submodule is located after the group feature learning submodule. This submodule is based on the idea of ​​dynamic activation and performs differentiated processing on the output representation of group feature learning to realize channel-dimensional attention modeling and feature selection.

[0126] It is understandable that task-aware activation is a piecewise linear activation function. Piecewise linear activation functions, represented by the Rectified Linear Unit (ReLU), can be uniformly expressed as:

[0127]

[0128] Where a and b are activation parameters, x is the input representation, and c represents different channels. Task-aware activation affects the features x of different channels. c Using different activation parameters a c Furthermore, the activation parameters are entirely adaptively learned from the input. To reduce the number of parameters and computational complexity, b is always set to 0. Since the two linear functions intersect at zero when b=0, it can be seen that the activation function separately processes the positive and negative parts of the network representation. Therefore, the above equation can be transformed into:

[0129] y = α·max(x,0) + β·min(x,0)

[0130] Where α and β are the activation parameters that the task-aware activation module adaptively learns based on the input x.

[0131] The task-aware activation module is structurally inspired by work related to dynamic activation, and its overall process is as follows: Figure 9 As shown. The structure of this module is similar to that of the selective kernel structure, involving three steps: splitting, merging, and further splitting. For the input feature vector... The first step is to split it into two branches: positive and negative. The positive branch sets the negative portion to 0, and the negative branch sets the positive portion to 0. Then, global pooling is used to compress both the frequency and time dimensions of the two branches to one dimension, resulting in... The second step involves connecting the bi-branch representation along the channel dimension to obtain Y = cincat(X′1, X′2), and then inputting a common excitation structure. get The third step is to decompose W along the channel dimension to obtain the weights of the positive and negative parts of the above formula.

[0132] To reduce computational complexity and enhance the representation of each group, the embodiment of the present invention incorporates connection, splitting, and squeezing excitation structures for the task-aware activation module. It is also designed to be grouped, such as Figure 9 As shown, when the positive and negative bi-branch is connected along the channel dimension, it is connected according to the grouping in the grouped feature learning sub-module. Positive and negative representations from the same group are adjacent connections. The two 1*1 convolutions stacked in the squeeze excitation structure are also designed as grouped convolutions. When the channels of the same group are calculated, they only depend on the feature input of the current group, avoiding interference from irrelevant information from other groups.

[0133] The Group Feature Calibration Module (GFC) proposed in this embodiment is a plug-and-play module. To further illustrate the improvement effect of this module on existing mainstream SED networks, the parameter quantities of different network structures shown in Table 1 are compared. Among them, the DCASE (Challenge on Detection and Classification of Acoustic Scenes and Events) competition is currently the most authoritative competition in the SED field. The three networks used in the experiment are the most commonly used or best-performing SED networks, and are also the networks used in this competition.

[0134] Table 1

[0135] SED network structure describe Parameters Baseline CRNN model One of the network structures used by the first place winner of DCASE 2022 Task 4 4.428M FDY-CRNN model One of the network structures used by the second-place winner of DCASE 2022 Task 4. 11.061M SK-CRNN model One of the network structures used by the second-place winner of DCASE 2022 Task 4. 5.040M GFC module This invention proposes 1.06M

[0136] Table 2 shows the experimental performance of the GFC module on three different mainstream SED backbone networks. The F1 score is the harmonic mean of precision and recall. The Polyphonic Sound Event Detection Scores (PSDS) is an intersection-based AUC (Area Under Curve). This invention calculates PSDS scores for two scenarios: PSDS1 focuses on the accuracy of event boundary localization, while PSDS2 focuses on the accuracy of event classification. For all metrics, higher scores indicate better detection performance. Table 2 shows that the GFC module can improve the performance of different backbone networks across various evaluation metrics, demonstrating that this module can enhance the network representation capability of the SED model. The baseline CRNN+GFC has a total parameter count of 5.488M, approximately half that of FDY-CRNN, but its detection performance surpasses FDY-CRNN in most metrics, proving that the proposed grouped feature calibration module can achieve significant performance improvements with lower complexity. Furthermore, this method performs well on both the DCASE and Urban-SED datasets, demonstrating the good generalization ability of the GFC module across different data distributions. The current state-of-the-art (SOTA) method on the Urban-SED dataset is [work name missing], with an SB-F1 score of 0.647, while the FDY-CRNN model incorporating the GFC module achieves an SB-F1 score of 0.654.

[0137] Table 2

[0138]

[0139] Based on this, Table 3 presents the ablation experiment results of the grouped feature learning submodule. When using a single feature learning branch, all metrics show varying degrees of improvement; when any feature learning branch is removed, all metrics show a slight decrease; when all four feature learning branches are used simultaneously, the model performance reaches its optimal level. The results of this ablation experiment demonstrate that feature learning in different dimensions is effective and complementary in improving model performance. The experimental results show that the frequency domain feature learning branch (FL) has the most significant impact on the PSDS score, reflecting the importance of frequency band information modeling in the SED task.

[0140] Table 3

[0141]

[0142] Figure 10 and Figure 11 The study demonstrates the improvement effects of four feature learning branches on different categories of sound events, among which... Figure 10 A comparison chart of PSDS1 scores is shown. Figure 11 The PSDS2 score comparison chart shows that different feature learning branches exhibit varying performance across different sound event categories. Combining PSDS1 and PSDS2 scores, Local Feature Learning (LL) performs best on cat and shv (electric shaver) events, but falls below baseline on spch (human voice) and vcm (vacuum cleaner) events. Temporal Feature Learning (TL) performs best on spch, but falls below baseline on dsh (plate) events. Global Feature Learning (GL) performs best on bld (bld) events, but falls below baseline on vcm events. Frequency Domain Feature Learning (FL) performs well in most categories, generally exceeding or matching the baseline in all categories, reflecting the effectiveness of the GAT structure in frequency band modeling. These results demonstrate that each of the four feature learning branches is effective for different sound event categories and can be used together to improve model performance.

[0143] Finally, Table 4 presents the comparative experimental results of the GFC module using different activation methods. Only ReLU is a static activation method, while the rest are dynamic activation methods. Considering all indicators, the task-aware activation method proposed in this invention exhibits the best performance, achieving a PSDS score of 1.117, surpassing both the static activation method ReLU and existing dynamic activation methods. This result demonstrates the effectiveness of group-based activation parameter computation. Group computation eliminates interference information in feature learning branches of different dimensions, allowing for more targeted learning of activation parameters for each feature branch's channels.

[0144] Table 4

[0145] GFC Activation Method PSDS1 PSDS2 EB-F1 IB-F1 ReLU 0.425 0.648 0.525 0.737 CG 0.442 0.666 0.537 0.756 APReLU 0.426 0.665 0.514 0.747 DyReLU 0.434 0.654 0.527 0.742 TA 0.443 0.674 0.538 0.752

[0146] The sound event detection system based on group feature calibration provided by the present invention is described below. The sound event detection system based on group feature calibration described below can be referred to in correspondence with the sound event detection method based on group feature calibration described above.

[0147] Figure 12 This is a schematic diagram of the structure of the sound event detection system based on grouped feature calibration provided in an embodiment of the present invention, as shown below. Figure 12 As shown, it includes: an acquisition module 1201, a calibration module 1202, and a detection module 1203, wherein:

[0148] The acquisition module 1201 is used to acquire audio feature data of the sound event to be detected; the calibration module 1202 is used to input the audio feature data into a time-frequency learning network, acquire a time-frequency spectrogram through a convolutional neural network, perform group feature learning on the intermediate representation of the time-frequency spectrogram based on multiple dimensions to obtain grouped enhanced features, and perform task-aware activation on the grouped enhanced features to obtain adaptive features; the detection module 1203 is used to input the adaptive features into a context modeling network, acquire the temporal correlation features of the audio signal, classify the temporal correlation features of the audio signal, and obtain the sound event category detection result.

[0149] Figure 13 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 13 As shown, the electronic device may include a processor 1310, a communications interface 1320, a memory 1330, and a communication bus 1340, wherein the processor 1310, the communications interface 1320, and the memory 1330 communicate with each other via the communication bus 1340. The processor 1310 can call logical instructions in the memory 1330 to execute a sound event detection method based on grouped feature calibration. This method includes: acquiring audio feature data of the sound event to be detected; inputting the audio feature data into a time-frequency learning network, obtaining a time-spectrum image through a convolutional neural network, performing grouped feature learning on the intermediate representations of the time-spectrum image based on multiple dimensions to obtain grouped enhancement features, performing task-aware activation on the grouped enhancement features to obtain adaptive features; inputting the adaptive features into a context modeling network to obtain audio signal temporal correlation features, classifying the audio signal temporal correlation features to obtain a sound event category detection result.

[0150] Furthermore, the logical instructions in the aforementioned memory 1330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the sound event detection method based on grouped feature calibration provided by the above methods. The method includes: acquiring audio feature data of a sound event to be detected; inputting the audio feature data into a time-frequency learning network, acquiring a time-spectrum image through a convolutional neural network, performing grouped feature learning on the intermediate representations of the time-spectrum image based on multiple dimensions to obtain grouped enhancement features, performing task-aware activation on the grouped enhancement features to obtain adaptive features; inputting the adaptive features into a context modeling network to acquire audio signal temporal correlation features, classifying the audio signal temporal correlation features, and obtaining a sound event category detection result.

[0152] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of sound event detection based on packet feature calibration, characterized in that, The method comprises: obtaining audio feature data of a sound event to be detected; inputting the audio feature data into a time-frequency learning network, obtaining a time-frequency spectrum graph through a convolutional neural network, performing grouped feature learning on intermediate representations of the time-frequency spectrum graph based on multiple dimensions to obtain grouped reinforced features, and performing task-aware activation on the grouped reinforced features to obtain adaptive features; inputting the adaptive features into a context modeling network to obtain audio signal time domain correlation features, classifying the audio signal time domain correlation features, and obtaining a sound event category detection result; performing grouped feature learning on intermediate representations of the time-frequency spectrum graph based on multiple dimensions to obtain grouped reinforced features, comprising: obtaining any training data of the time-frequency spectrum graph, determining that a feature tensor output by an intermediate feature representation layer of the any training data comprises a channel dimension, a time dimension and a frequency dimension; based on the channel dimension, dividing the feature tensor into to-be-calibrated features and reserved features according to a preset ratio; based on the channel dimension, dividing the to-be-calibrated features into time domain to-be-calibrated features, frequency domain to-be-calibrated features, global to-be-calibrated features and local to-be-calibrated features; performing feature learning and re-weighting on the time domain to-be-calibrated features, the frequency domain to-be-calibrated features, the global to-be-calibrated features and the local to-be-calibrated features respectively to obtain grouped calibration features; connecting the grouped calibration features and the reserved features to obtain the grouped reinforced features; performing task-aware activation on the grouped reinforced features to obtain adaptive features, comprising: splitting the grouped reinforced features into positive features and negative features, compressing the frequency dimension and the time dimension of the positive features and the negative features to one dimension by global pooling to obtain compressed positive features and compressed negative features; connecting the compressed positive features and the compressed negative features based on the channel dimension to obtain connection features, and inputting the connection features into a squeeze excitation structure to obtain excitation features; splitting the excitation features according to the channel dimension to obtain positive feature weights and negative feature weights; multiplying the positive features by the positive feature weights and adding the product of the negative features and the negative feature weights to obtain the adaptive features.

2. The packet feature based calibration sound event detection method according to claim 1, characterized in that, performing feature learning and re-weighting on the time domain to-be-calibrated features, comprising: adopting one-dimensional frequency domain pooling to compress the frequency dimension of the time domain to-be-calibrated features to one dimension to obtain time domain intermediate features; compressing and expanding the feature channel dimension of the time domain intermediate features through two one-dimensional convolutions connected in series to obtain time context dependency features of the time domain intermediate features; utilizing an activation function sigmoid to re-weight the time context dependency features and expand them to the input dimension of the time domain to-be-calibrated features, and multiplying the time context dependency features by the time domain to-be-calibrated features to obtain time domain learning features.

3. The packet feature based calibration sound event detection method according to claim 1, wherein, performing feature learning and re-weighting on the frequency domain to-be-calibrated features, comprising: adopting one-dimensional time domain pooling to compress the time dimension of the frequency domain to-be-calibrated features to one dimension to obtain frequency domain intermediate features; obtaining a graph node representation set of the frequency domain intermediate features through a graph attention network; The hyperbolic tangent function is calculated after element-wise multiplication of any two original node representations in the set of node representation, and then multiplied by a learnable weight parameter, and the attention weight between any two nodes is calculated using a normalized exponential function; The sum of the product of the attention weight between any two nodes and any original node representation is obtained to obtain the aggregated feature of any node; The residual connection between the aggregated feature of any node and the original node representation is obtained through the batch normalization layer to obtain the graph node output feature; The graph node output feature is reweighted using the activation function sigmoid and expanded to the input dimension of the frequency domain to be calibrated, and then multiplied by the frequency domain to be calibrated to obtain the frequency domain learning feature.

4. The packet feature based calibration sound event detection method according to claim 1, wherein, The feature learning and reweighting of the global to-be-calibrated feature include: The time dimension and the frequency dimension of the global to-be-calibrated feature are compressed into one dimension respectively using two-dimensional global pooling to obtain a global context channel representation feature; The global context channel representation feature is obtained through one-dimensional convolution to obtain a global channel dimension dependent feature; The global channel dimension dependent feature is reweighted using the activation function sigmoid and expanded to the input dimension of the global to-be-calibrated feature, and then multiplied by the global to-be-calibrated feature to obtain a global learning feature.

5. The packet feature based calibration sound event detection method according to claim 1, wherein, The feature learning and reweighting of the local to-be-calibrated feature include: The local to-be-calibrated feature is split into different scale convolution features through convolution transformation including multiple convolution branches; The different scale convolution features are added element by element to obtain a fusion feature, and the frequency dimension and the time dimension of the fusion feature are compressed into one dimension through global pooling to obtain a global context channel representation feature; The channel dimension of the global context channel representation feature is compressed to one fourth of the initial channel dimension through a squeeze excitation structure, and the compressed global context channel representation feature is expanded to the initial channel dimension through a fully connected layer corresponding to the multiple convolution branches to obtain different branch global context channel representation features; The same channels of the different branch global context channel representation features are calculated using a normalized exponential function to obtain different branch channel weights; The different branch channel weights and the different branch global context channel representation features are weighted and summed to obtain a local learning feature.

6. A sound event detection system calibrated based on grouping features, based on the sound event detection method calibrated based on grouping features according to any one of claims 1 to 5, characterized in that, It includes: An acquisition module is configured to acquire audio feature data of a sound event to be detected; A calibration module is configured to input the audio feature data into a time-frequency learning network, acquire a time-frequency spectrogram through a convolutional neural network, perform grouped feature learning on intermediate representations of the time-frequency spectrogram based on multiple dimensions to obtain grouped reinforced features, and perform task-aware activation on the grouped reinforced features to obtain adaptive features; A detection module is configured to input the adaptive features into a context modeling network, acquire audio signal time domain correlation features, and classify the audio signal time domain correlation features to obtain a sound event category detection result.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the sound event detection method based on grouped feature calibration according to any one of claims 1-5. The processor executes the program to implement the sound event detection method based on grouped feature calibration according to any one of claims 1-5.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the sound event detection method based on calibration of packet features as claimed in any one of claims 1 to 5.

Images