Neural Network Systems and Methods for Audio Event Detection

Abstract

This invention discloses a neural network system and method for audio event detection. The system includes a feature extraction layer, a convolutional layer, a recurrent neural network, a feedforward network, and a self-attention module. The system processes the log-Mel spectrum features obtained from the extracted audio to obtain a three-dimensional feature map in the convolutional kernel space. It then calculates the frequency-adaptive attention weights for the three dimensions of the convolutional kernel space using these feature maps. Based on the three-dimensional frequency-adaptive attention weights and the base convolutional kernel, it performs a multiplication operation to determine the frequency-adaptive convolutional kernel. The frequency-adaptive convolutional kernel is used to perform multi-dimensional frequency dynamic convolution processing on the log-Mel spectrum features obtained from the extracted audio to obtain a first output feature. The sequence data of the first output feature is then processed to obtain a second output feature. Finally, strong and weak labels for the audio are obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of audio detection, and in particular to a neural network system and method for audio event detection. BACKGROUND

[0002] The audio event detection task aims to identify specific audio events occurring from a piece of audio signal, so that the model has the ability to identify the audio events emitted by the target sound source from various environments. Audio event detection is an important part of the research on non-speech perception and understanding in the field of acoustics, and is widely used in the fields of audio retrieval, smart home, wearable devices and smart city. By enabling the model to identify sound events, important information is provided for processing audio clips or understanding the environment, thus having high research and application value.

[0003] Audio event detection mainly consists of two steps. The first step is audio feature extraction, which extracts useful features from the input audio clip, and then models and classifies. The current audio event detection method based on deep neural network usually uses log mel spectrum as input feature, which converts one-dimensional waveform signal into two-dimensional time-frequency spectrogram, which is convenient for deep neural network to learn specified acoustic events. The second step is model classification. Early studies usually use machine learning methods such as hidden Markov model, Gaussian mixture model, support vector machine, random forest to model the time-frequency features of audio events. With the development of deep neural networks in computer vision, speech and other fields, audio event detection also began to use neural network methods. The commonly used neural network models for audio event detection currently mainly include convolutional neural network, recurrent neural network, convolutional recurrent neural network, transformer, etc.

[0004] Convolutional neural networks have shown superior performance in processing two-dimensional image data, so they are often used to extract high-dimensional features of two-dimensional time-frequency spectrograms in audio event detection. However, traditional two-dimensional convolution has translational invariance in both dimensions, while the two-dimensional time-frequency spectrogram used in audio event detection is different from the conventional image, which only has translational invariance in the time dimension. This difference leads to poor performance of traditional two-dimensional convolution in extracting two-dimensional time-frequency spectrogram features. On the other hand, a single convolution kernel has certain limitations in feature extraction capability and computational efficiency. Recent studies have shown that learning a dynamic linear combination of multiple convolution kernels and weighting it with input-related attention can significantly improve the feature extraction capability of convolutional neural networks while maintaining good computational efficiency. However, these methods mostly only give dynamic properties to one dimension in the convolution kernel space, ignoring the other multiple dimensions. SUMMARY

[0005] To solve the problems in the prior art, the embodiments of the present application provide a neural network system and method for audio event detection.

[0006] In a first aspect, the present application provides a computer-implemented neural network system for audio event detection, comprising:

[0007] a feature extraction layer configured to extract a log-mel spectrogram feature map of the audio as an input feature;

[0008] a convolution layer configured to process the input feature to obtain a feature map in three dimensions of a convolution kernel space, determine a frequency adaptive attention weight in three dimensions based on calculation of the feature map in three dimensions, perform a multiplication operation based on the frequency adaptive attention weight in three dimensions and a base convolution kernel to determine a frequency adaptive convolution kernel, and perform a multi-dimensional frequency dynamic convolution on the input feature using the frequency adaptive convolution kernel to obtain a first output feature;

[0009] a recurrent neural network configured to process sequence data of the first output feature to obtain a second output feature;

[0010] a feedforward network configured to process the second output feature to obtain a strong label of the audio;

[0011] a self-attention module configured to process the second output feature to obtain a weak label of the audio.

[0012] In some embodiments, the convolution layer comprises:

[0013] a first convolution layer and four second convolution layers;

[0014] The first convolution layer comprises a convolution block and an average pooling layer, wherein the convolution block comprises a traditional convolution layer, a batch normalization layer and an activation function GLU.

[0015] The second convolution layer comprises a multi-dimensional frequency dynamic convolution layer, a batch normalization layer, an activation function GLU and an average pooling layer, wherein the multi-dimensional frequency dynamic convolution layer comprises a three-branch convolution block.

[0016] In some embodiments, the mathematical expression of the frequency adaptive attention weight in three dimensions is:

[0017] alpha wi (f = pi wi (x, f)

[0018] alpha fi (f = pi fi (x, f)

[0019] alpha ci (f = pi ci (x, f)

[0020] wherein, α wi (f is the frequency adaptive attention weight of the i-th convolution kernel, α fi (f is the frequency adaptive attention weight of the output channel dimension, α ci (f is the frequency adaptive attention weight of the input channel dimension, π wi (x,f), π fi (x,f), π ci (x,f) are multi-head attention modules, respectively.

[0021] The mathematical expression of the multi-dimensional frequency dynamic convolution processing is as follows:

[0022]

[0023] wherein, x is the input, y is the output, represents the multiplication operation along the dimension of different kernel spaces, W i is the basic convolution kernel.

[0024] In some embodiments, the second output feature is processed to obtain a strong label of the audio, comprising:

[0025] The second output feature is input into a feedforward network to obtain a feature map with a dimension of TxC;

[0026] The Sigmoid function is input to the feature map with a dimension of TxC to obtain a posterior probability map of the feature map;

[0027] The posterior probability map is thresholded and median filtered to obtain the strong label of the audio;

[0028] wherein, T is the number of audio frames, and C is the number of event categories.

[0029] In some embodiments, the second output feature is processed to obtain a weak label of the audio, comprising:

[0030] The second output feature is input into a self-attention module to obtain a weight in the computation time dimension;

[0031] The posterior probability map is pooled based on the weight in the computation time dimension and input into a Softmax function to obtain a probability map with a dimension of 1xC;

[0032] The probability map with a dimension of 1xC is thresholded to obtain the weak label of the audio.

[0033] In another aspect, the present application provides a computer-implemented audio event detection method, comprising:

[0034] Extracting a log-mel spectrum feature map of the audio as an input feature;

[0035] processing the input feature to obtain a feature map of three dimensions of a convolution kernel space; determining frequency adaptive attention weights of the three dimensions of the convolution kernel space through calculation on the feature map of the three dimensions; performing multiplication operation based on the frequency adaptive attention weights of the three dimensions and a basic convolution kernel to determine a frequency adaptive convolution kernel; and performing multi-dimensional frequency dynamic convolution processing on the input feature using the frequency adaptive convolution kernel to obtain a first output feature;

[0036] inputting the first output feature into a recurrent neural network for processing to obtain a second output feature;

[0037] inputting the second output feature into a feedforward network for processing to obtain a strong label of the audio;

[0038] inputting the second output feature into a self-attention module for processing to obtain a weak label of the audio.

[0039] In some possible embodiments, processing the input feature to obtain a feature map of three dimensions of a convolution kernel space; determining frequency adaptive attention weights of the three dimensions of the convolution kernel space through calculation on the feature map of the three dimensions; performing multiplication operation based on the frequency adaptive attention weights of the three dimensions and a basic convolution kernel to determine a frequency adaptive convolution kernel; and performing multi-dimensional frequency dynamic convolution processing on the input feature using the frequency adaptive convolution kernel to obtain a first output feature, comprising:

[0040] inputting the input feature into an average pooling layer to obtain a feature map with dimensions of F x c in x 1;

[0041] inputting the feature map with dimensions of F x c in x 1 into a convolution block to obtain a feature map with dimensions of F x c in x 1;

[0042] inputting the feature map with dimensions of F x c in x 1 into a three-branch convolution block to obtain a feature map of three dimensions of a convolution kernel space; wherein the convolution block of each branch corresponds to a one-dimensional convolution layer kernel activation function, including a Softmax function of a convolution kernel number dimension branch, a Sigmoid function of an input channel branch and an output channel branch; the feature map of the three dimensions has sizes of F x n x 1, F x c in x 1 and F x c out x 1, respectively;

[0043] determining frequency adaptive attention weights of the three dimensions of the convolution kernel space through calculation on the feature map of the three dimensions;

[0044] The frequency-adaptive convolution kernel is determined by multiplication operations based on three-dimensional frequency-adaptive attention weights and the basic convolution kernel;

[0045] The first output feature is obtained by performing multidimensional frequency dynamic convolution on the input features using a frequency-adaptive convolution kernel.

[0046] On the other hand, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, characterized in that the processor implements the above-described audio event detection method when executing the computer program.

[0047] On the other hand, embodiments of the present invention provide a computer-readable medium having processor-executable non-volatile program code, characterized in that the program code causes the processor to execute the above-described audio event detection method.

[0048] Compared to existing technologies, the neural network system and method for audio event detection provided in this embodiment of the invention replaces the traditional two-dimensional convolution in a deep convolutional recurrent network with multi-dimensional frequency dynamic convolution, giving dynamic characteristics to the input channels, output channels, and number of convolutional kernels in the kernel space. These dynamic characteristics are obtained by calculating the frequency adaptive attention mechanism related to the input two-dimensional time spectrum, which greatly enhances the feature extraction capability of the convolutional kernel for audio events. Attached Figure Description

[0049] Figure 1 This is an architecture diagram of a neural network system used for audio event detection.

[0050] Figure 2 A schematic diagram of a multidimensional frequency dynamic convolution structure;

[0051] Figure 3 Here is a flowchart of a method for audio event detection;

[0052] Figure 4 Flowchart of the method for multidimensional frequency dynamic convolution processing;

[0053] Figure 5 This is a schematic diagram of the audio event detection model structure based on multidimensional frequency dynamic convolution. Detailed Implementation

[0054] To enhance the feature extraction capability of convolutional kernels for audio events, embodiments of this invention disclose a neural network system and method for audio event detection.

[0055] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0056] On the one hand, the present invention provides a neural network system for audio event detection. Figure 1 This is an architecture diagram of a neural network system for audio event detection, which includes:

[0057] S110: Feature Extraction Layer

[0058] The feature extraction layer is used to extract the log-Mel spectrum feature map of the audio.

[0059] First, the logarithmic Mel spectrum of the audio is extracted as the input feature, and the one-dimensional waveform signal is transformed into a two-dimensional time-spectrum graph and fed into the convolutional layer.

[0060] S120: Convolutional layer

[0061] The convolutional layer is used to process the input features to obtain feature maps in three dimensions of the convolutional kernel space; and to determine the frequency-adaptive attention weights in the three dimensions of the convolutional kernel space by calculating the feature maps in the three dimensions; to determine the frequency-adaptive convolutional kernel by multiplying the frequency-adaptive attention weights in the three dimensions and the basic convolutional kernel; and to obtain the first output feature (output feature y) by performing multi-dimensional frequency dynamic convolution processing on the input feature (input feature x) using the frequency-adaptive convolutional kernel.

[0062] The convolutional layer includes one convolutional layer 1 (first convolutional layer) and four convolutional layers 2 (second convolutional layers). Convolutional layer 1 includes convolutional blocks and average pooling layers. The convolutional blocks include traditional convolutional layers, batch normalization layers, and the activation function GLU. Convolutional layer 2 includes multi-dimensional frequency dynamic convolutional layers, batch normalization layers, the activation function GLU, and average pooling layers. The multi-dimensional frequency dynamic convolutional layers include convolutional blocks with three branches.

[0063] Figure 2This is a schematic diagram of a multi-dimensional frequency-dynamic convolutional structure, which is an improvement on the basic two-dimensional convolution. A basic convolution can be represented as y = W* + b, where W and b represent the weights and biases of the convolutional kernel, respectively. For dynamic convolution, each convolution contains multiple kernels, which are dynamically aggregated in parallel based on attention. The attention calculates the weights of each kernel based on the input features, thus generating an adaptive dynamic convolution. The non-linear superposition of multiple kernels using attention weights provides stronger representational power. However, in the kernel space, the number of kernels is only one dimension; the kernel space also has dimensions of the number of input channels, the number of output channels, and the kernel size.

[0064] Traditional dynamic convolution only imparts dynamic features to one dimension while neglecting the other three. Therefore, our invented multi-dimensional frequency dynamic convolution extends this dynamic property to other dimensions in the convolution kernel space.

[0065] The system uses frequency-adaptive attention weights instead of traditional attention weights, meaning that for every 5 inputs, the calculated attention weights differ across different frequency bands. This dynamic characteristic is more in line with...

[0066] Time-frequency pattern of audio event detection task.

[0067] Multidimensional frequency dynamic convolution can be expressed mathematically as:

[0068]

[0069] α wi (f)=π wi (x,f)

[0070] α fi (f)=π fi (x,f)

[0071] α ci (f)=π ci (x,f)

[0072] Where x is the input feature, y is the output feature, and α wi (f) represents the frequency-adaptive attention weights of the i-th convolutional kernel, α fi (f) represents the frequency-adaptive attention weights for the output channel dimension, α ci (f) represents the frequency-adaptive attention weights along the input channel dimension. ⊙ denotes the multiplication operation along the dimensions of different kernel spaces.

[0073] α wi (f), α ci (f), α ci (f) Each is composed of a multi-head attention module πwi (x,f), π fi (x,f), π ci (x,f)

[0074] The calculation yielded the results. It's important to note that we did not use the kernel size dimension because the kernel size involves weights in the time and frequency dimensions of the kernel, and we cannot directly calculate the frequency-adaptive attention weights using this dimension.

[0075] In one possible embodiment, when the input feature is Output features are hour.

[0076] First, the input feature x is fed into the average pooling layer of convolutional layer 1 and pooled along the time dimension to obtain a pool with dimension F×c. in A feature map of ×1;

[0077] Then, the dimension is F×c in A feature map of dimension F×1 is input into a convolutional block. After passing through the convolutional block, the F dimension of the feature map is compressed by a ratio r, resulting in a dimension of F×c. in / r×1 feature map;

[0078] Secondly, the dimension is F×c in The feature map of size / r×1 is input to a three-branch convolutional block, which calculates frequency-adaptive attention weights in the three dimensions of the kernel space. Each branch of the convolutional block corresponds to a one-dimensional convolutional layer kernel activation function. Since the sum of the attention weights of multiple convolutional kernels is limited to 1, but not for the input and output channel dimensions, the number of kernels in the convolutional kernel branch uses the Softmax function, while the input and output channel branches use the Sigmoid function. After passing through the three-branch convolutional block, the feature map yields a three-dimensional feature map in the kernel space, with sizes Fr, ...

[0079] ×n×1、F×c in ×1、F×c out ×1;

[0080] From this point onward, by considering the sizes F×n×1 and F×c respectively... in ×1、F×c out The three-dimensional feature maps are used to calculate and determine the frequency adaptive attention weights α in the three dimensions of the convolution kernel space. wi (f), α ci (f),

[0081] α ci (f);

[0082] Finally, the frequency-adaptive attention weights for each dimension are adjusted by the base convolution kernel W. i Multiplication is performed to obtain the final frequency-adaptive convolution kernel; the frequency-adaptive convolution kernel is used to perform multi-dimensional frequency dynamic convolution on the input feature x to obtain the output feature y.

[0083] In multidimensional frequency dynamic convolution, for each convolution kernel W i α ci (f) give c in Each input channel is assigned frequency-adaptive attention weights, α fi (f) give c out Each output channel is assigned an adaptive frequency weight, α wi (f) Give the entire convolution kernel W i Assign frequency-adaptive weights. Theoretically, these three attention weights are complementary in kernel space, and applying them to the convolutional kernel can greatly enhance the feature extraction capability of the convolutional neural network.

[0084] S130: Recurrent Neural Network

[0085] Recurrent neural networks are used to process the sequence data of the first output feature to obtain the second output feature.

[0086] The recurrent neural network includes two layers of bidirectional gated recurrent units, which are used to input the first output feature into the recurrent neural network to learn contextual information.

[0087] S140: Feedforward network

[0088] The feedforward network is used to process the second output features to obtain a strong label for the audio.

[0089] In one specific embodiment, processing the second output feature to obtain a strong label for the audio includes:

[0090] The second output feature is input into a feedforward network to obtain a feature map of dimension T×C; the feature map of dimension T×C is input into a Sigmoid function to obtain the posterior probability map of the feature map; the posterior probability map is subjected to thresholding and median filtering to obtain the strong label of the audio; where T is the number of audio frames and C is the number of event categories.

[0091] S150: Self-Attention Module

[0092] The self-attention module is used to process the second output features to obtain weak labels for the audio. In one specific embodiment, processing the second output features to obtain weak labels for the audio includes:

[0093] The second output feature is input into the self-attention module to obtain the weights for calculating the time dimension;

[0094] The posterior probability map is pooled based on the weights of the computation time dimension and then input into the Softmax function to obtain a probability map with dimension 1×C.

[0095] Weak labels for the audio are obtained by thresholding a probability graph with dimension 1×C.

[0096] On the other hand, the present invention provides a method for audio event detection. Figure 3 This is a flowchart of a method for audio event detection, which includes:

[0097] S310: Extract the log-Mel spectrum feature map of the audio as the input feature;

[0098] S320: Process the input features to obtain feature maps in three dimensions of the convolution kernel space; determine the frequency-adaptive attention weights in the three dimensions of the convolution kernel space by calculating the feature maps in the three dimensions; determine the frequency-adaptive convolution kernel by multiplying the frequency-adaptive attention weights in the three dimensions and the basic convolution kernel; use the frequency-adaptive convolution kernel to perform multi-dimensional frequency dynamic convolution processing on the input features to obtain the first output feature;

[0099] Figure 4 This is a flowchart of the multidimensional frequency dynamic convolution processing method. Figure 5 This is a schematic diagram of an audio event detection model based on multidimensional frequency dynamic convolution, combined with... Figure 4 , Figure 5 The method includes:

[0100] S321: Input the input features into the average pooling layer to obtain a feature map with dimension F×c_in×1;

[0101] S322: Input the feature map of dimension F×c_in×1 into the convolutional block to obtain a feature map of dimension F×c_in / r×1;

[0102] S323: Input the feature map of dimension F×c_in / r×1 into the three-branch convolution block to obtain a feature map of three dimensions in the convolution kernel space;

[0103] S324: Determine the frequency-adaptive attention weights of the three dimensions of the convolution kernel space by calculating the feature maps of the three dimensions;

[0104] S325: Determine the frequency-adaptive convolution kernel by performing multiplication operations based on three-dimensional frequency-adaptive attention weights and basic convolution kernels;

[0105] S326: The first output feature is obtained by performing multi-dimensional frequency dynamic convolution on the input features using a frequency adaptive convolution kernel;

[0106] S330: Input the first output feature into the recurrent neural network for processing to obtain the second output feature;

[0107] S340: Input the second output feature into the feedforward network for processing to obtain the strong label of the audio;

[0108] S350: Input the second output feature into the self-attention module for processing to obtain the weak label of the audio.

[0109] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0110] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0111] The embodiments provided by this invention disclose a neural network system and method for audio event detection. It replaces the traditional two-dimensional convolution in a deep convolutional recurrent network with multi-dimensional frequency dynamic convolution. By calculating the frequency adaptive attention mechanism related to the input two-dimensional time spectrum, it obtains the dynamic characteristics of the input channel, output channel, and number of convolutional kernels in the convolutional kernel space, which greatly enhances the feature extraction capability of the convolutional neural network.

[0112] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A neural network system for audio event detection executed by a computer, comprising: a feature extraction layer configured to extract a log-mel spectrogram of the audio as input features; a convolutional layer configured to process the input features to obtain a feature map of three dimensions of a convolution kernel space, determine frequency adaptive attention weights of the three dimensions of the convolution kernel space by calculating the feature map of the three dimensions, determine a frequency adaptive convolution kernel based on the frequency adaptive attention weights of the three dimensions and a base convolution kernel, perform multi-dimensional frequency dynamic convolution on the input features using the frequency adaptive convolution kernel to obtain first output features, wherein the three dimensions include a number of convolution kernel dimension, an input channel dimension and an output channel dimension; a recurrent neural network configured to process sequence data of the first output features to obtain second output features; a feedforward network configured to process the second output features to obtain strong labels of the audio; a self-attention module configured to process the second output features to obtain weak labels of the audio.

2. The neural network system for audio event detection of claim 1, wherein, the convolutional layer comprises: a first convolutional layer and four second convolutional layers; the first convolutional layer comprises a convolution block and an average pooling layer, wherein the convolution block comprises a traditional convolutional layer, a batch normalization layer and an activation function GLU; the second convolutional layer comprises a multi-dimensional frequency dynamic convolution layer, a batch normalization layer, an activation function GLU and an average pooling layer, wherein the multi-dimensional frequency dynamic convolution layer comprises a three-branch convolution block.

3. The neural network system for audio event detection of claim 1, wherein, a mathematical expression of the frequency adaptive attention weights of the three dimensions is: a wi (f) = π wi (x,f) a fi (f) = π fi (x,f) a ci (f) = π ci (x,f) wherein, α wi (f) is the frequency adaptive attention weight of the i-th convolution kernel, α fi (f) is the frequency adaptive attention weight of the output channel dimension, α ci (f) is the frequency adaptive attention weight of the input channel dimension, f is the frequency; π wi (x, f), π fi (x, f), π ci (x, f) are three attention heads corresponding to the three dimensions of the multi-head attention module respectively; wi is the convolution kernel number dimension, fi is the input channel dimension, and ci is the output channel dimension. a mathematical expression of the multi-dimensional frequency dynamic convolution is: where x is input, y is output, represents multiplication operation along the dimension of different kernel spaces, W i is the base convolution kernel.

4. The neural network system for audio event detection of claim 1, wherein, processing the second output features to obtain the strong labels of the audio comprises: inputting the second output features into the feedforward network to obtain a feature map with a dimension of TxC; inputting a Sigmoid function to the feature map with the dimension of TxC to obtain a posterior probability map of the feature map; performing threshold processing and median filtering on the posterior probability map to obtain the strong labels of the audio; wherein T is a number of audio frames and C is a number of event categories.

5. The neural network system for audio event detection of claim 4, wherein, processing the second output features to obtain the weak labels of the audio comprises: inputting the second output features into the self-attention module to obtain weights of a calculation time dimension; performing pooling on the posterior probability map based on the weights of the calculation time dimension and inputting a Softmax function to obtain a probability map with a dimension of 1xC; performing threshold processing on the probability map with the dimension of 1xC to obtain the weak labels of the audio.

6. A method for audio event detection executed by a computer, comprising: extracting a log-mel spectrogram of the audio as input features; processing the input features to obtain a feature map of three dimensions of a convolution kernel space; determining frequency adaptive attention weights of the three dimensions of the convolution kernel space by calculating the feature map of the three dimensions, determining a frequency adaptive convolution kernel based on the frequency adaptive attention weights of the three dimensions and a base convolution kernel, performing multi-dimensional frequency dynamic convolution on the input features using the frequency adaptive convolution kernel to obtain first output features, wherein the three dimensions include a number of convolution kernel dimension, an input channel dimension and an output channel dimension; inputting the first output feature into a recurrent neural network to obtain a second output feature; inputting the second output feature into a feedforward network to obtain a strong label of the audio; inputting the second output feature into a self-attention module to obtain a weak label of the audio.

7. The audio event detection method of claim 6, wherein, the processing of the input feature obtains a feature map of three dimensions of the convolution kernel space; and the frequency adaptive attention weight of three dimensions of the convolution kernel space is determined by calculating the feature map of three dimensions; the frequency adaptive convolution kernel is determined by multiplying the frequency adaptive attention weight of three dimensions and the basic convolution kernel; the input feature is processed by using the frequency adaptive convolution kernel to obtain the first output feature, including: The input features are fed into an average pooling layer to obtain a dimension of F×c. in ×1 feature map; c in is the number of input channels; F is the dimension of the feature map after the convolutional block; The feature map with the dimension of F x c in is input to the convolution block to obtain a feature map with the dimension of F x c in r x 1; r is the compression ratio. The feature map with the dimension of Fxc in is input into the three-branch convolution block to obtain a feature map with three dimensions of convolution kernel space; each branch of the convolution block corresponds to a one-dimensional convolution layer, an activation function, a Softmax function of the number of convolution kernels, a Sigmoid function of the input channel branch, and a Sigmoid function of the output channel branch; the sizes of the three dimensions of the feature map are Fxn×1, Fxc in ×1, and Fxc out ×1, respectively. out c is the number of output channels; n is the number of convolution kernels. the frequency adaptive attention weight of three dimensions of the convolution kernel space is determined by calculating the feature map of three dimensions; the frequency adaptive convolution kernel is determined by multiplying the frequency adaptive attention weight of three dimensions and the basic convolution kernel; the input feature is processed by using the frequency adaptive convolution kernel to obtain the first output feature.

8. An electronic device comprising a memory, a processor, the memory having stored thereon a computer program executable on the processor, characterized in that, The processor executes the computer program to realize the method of any one of claims 6-7.

9. A computer readable medium having a non-transitory program code executable by a processor, the program code comprising instructions for: The program code causes the processor to execute the method of any one of claims 6-7.

Images