Multi-layer neural network-based multi-sound event detection method

Abstract

The invention discloses a multi-sound event detection method based on a multilayer neural network, and relates to the field of sound event detection. The invention provides a weak label multi-sound event detection model based on a fusion empirical mode decomposition algorithm, a capsule network, an attention mechanism and a bidirectional gating loop network. In the model, firstly, an AMM-EMD algorithm is used for preprocessing, sound event feature information is enriched, and a parallel convolution structure is introduced to extract advanced feature information; further processing the advanced features by using the strong generalization ability of the capsule network and the ability of capturing global dependence by an attention mechanism; and meanwhile, the BiGRU module is used for acquiring context information. According to the model provided by the invention, contrast ablation experiments are carried out on different data sets, the sound event detection effect is evaluated by utilizing F1 and ER, the model provided by the invention is improved compared with other baseline models, and F1 and ER are improved.

Description

Technical Field

[0001] The invention relates to a multi-sound event detection method based on a multi-layer neural network, and belongs to the field of sound event detection. Background Art

[0002] Sound Event Detection (SED) is a task that accurately marks the start and offset time information and type of each sound event in the sound signal. The ease of collection and information richness of audio data make it possible to process and construct auditory scenes in real time, making this technology extremely widely used. SED has great potential in security monitoring, biological monitoring, equipment abnormality monitoring, wildlife monitoring, urban environmental sound analysis equipment and smart home monitoring equipment.

[0003] SED can be divided into two types: single sound event detection and multiple sound event detection. Single sound event detection generally refers to the occurrence of only one sound event in the audio signal. This single sound event detection model can only detect the more obvious sounds in the audio stream, while other sounds are ignored; multiple sound event detection refers to the occurrence of more than two sound events at the same time in the audio stream. The multiple sound event detection model needs to detect the type of each sound event and its start time and end time. From the detection results, the start time, end time, and degree of overlap of the sound event can be clearly observed. SED has developed rapidly, but it still faces many challenges, especially in complex scenarios, where the performance of multiple sound event detection is not ideal. Therefore, how to further improve the performance of multiple sound event detection in complex scenarios will become an important challenge for multiple sound event detection technology.

[0004] In the past, many works have been done in SED. Classifiers used for SED include Hidden Markov Model (HMM), Non-negative Matrix Factorization (NMF), Support Vector Machine (SVM), Multi-Layer Perceptron (MLP) and Gaussian Mixture Model (GMM). Despite their promising performance, most of these classification methods are developed for the task of single-tone sound event detection and small-scale datasets.

[0005] In recent years, with the increase in the amount of training data, the models used for classification have gradually become more complex. With the development of deep learning technology, deep learning-based models can automatically learn high-level representations from audio without complex feature extraction methods, becoming the mainstream of SED. At first, researchers used this technology to experiment on strongly labeled datasets and proposed a variety of solutions. The models used have evolved from single models: Feedforward Neural Network (FNN), Long Short-Term Memory (LSTM), etc. to hybrid models: Convolution Recurrent Neural Network (CRNN), etc., and the performance of the models has also been greatly improved.

[0006] When applied to actual scenarios, the performance of the SED system is limited by the annotation of the dataset. Strongly annotated datasets need to mark the type, start time, and offset time of each sound event, but they are expensive to obtain and have limited data volume. In contrast, weakly annotated datasets only need to provide the category labels of the sound events, which are easy to obtain and annotate, providing SED with a large amount of training data and alleviating the problem of insufficient data. Building deep learning models for weakly annotated datasets to improve the performance of SED has become an important direction of current research.

[0007] For weakly labeled SED, Multi Instance Learning (MIL) is a popular framework. In weakly labeled SED, an audio segment is divided into many frames, each frame is regarded as an instance, and the entire segment is regarded as a packet. We only know the label of the packet, but not the label of each instance. In this method, a neural network predicts the probability of each sound event occurring in each frame. Then, the frame-level probabilities are aggregated into segment-level probabilities through a pooling function. Methods based on Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and their combination have shown state-of-the-art performance in the weakly labeled SED task. A CRNN with learnable Gated Linear Units (GLUs) is used for weakly labeled SED. With GLUs, the network can focus on sound events and ignore irrelevant sounds. Bidirectional Long Short-Term Memory Network (BLSTM) is also applied to weakly labeled learning. Due to the use of weakly labeled data, accurate error calculation cannot be performed to update the parameters of BLSTM. The authors introduced Connection Temporal Classification (CTC) to calculate the loss. At the same time, adaptive pooling operations have been shown to perform better than commonly used pooling operations (such as maximum pooling or average pooling) in weakly labeled SED tasks. Cakir et al. proposed the CRNN method based on the CNN network architecture and achieved advanced results in SED tasks. CRNN is a hybrid model composed of CNN and recurrent neural network (RNN), in which CNN can extract high-level features and has translation invariance, but cannot obtain long-term context information, while RNN can learn long-term context but cannot capture frequency domain invariance. The two complement each other, and CRNN is often used to complete SED tasks.

[0008] Recently, Capsule Network (CapsNet) has also been introduced into SED to handle the detection of overlapping events. Cakir et al. designed a CapsNet-RNN model that combines CapsNet and RNN to enhance the relationship between the local and the whole. Deep learning models with attention mechanisms have been widely used in many fields. The attention mechanism effectively reduces the computational complexity of the network by extracting more relevant high-level features and ignoring information irrelevant to the event (such as background noise). At the same time, CapsNet can solve the problem of sound overlap through its dynamic routing algorithm and voting mechanism. Summary of the invention

[0009] Aiming at the problems of unstable time-frequency changes of sound and strong background noise, overlap between multiple sound events and scarcity of weakly labeled data sets, the present invention proposes a multiple sound event detection method based on a multi-layer neural network.

[0010] The present invention is a multi-sound event detection method based on a multi-layer neural network, the method comprising:

[0011] S1, using AMM-EMD to pre-process and preliminarily reduce noise on multi-sound signals;

[0012] S2, based on preprocessing, use PCS structure to extract and fuse different feature information to form high-level feature information;

[0013] S3. Based on the formation of high-level feature information using the PCS structure, a ME-AttCapsNet-BiGRU multi-sound event detection model is constructed;

[0014] S4. After the BiGRU module obtains the context information, a feed-forward layer with a sigmoid activation function is used to obtain the probability of event activity in each frame.

[0015] Preferably, the S1 includes:

[0016] S11, using PCS structure to form high-level feature information;

[0017] S12. Use the AttCapsNet-BiGRU network to process high-level feature information.

[0018] Preferably, in S1, the PSC structure consists of three parallelized convolution paths, each path consists of 4 convolution layers, and the convolution kernel size of the same path is the same, and the convolution layers of the three paths have the same parameters except for the different convolution kernel sizes. The convolution kernel sizes of the three paths are 3×3, 5×5, and 7×7, respectively. The convolution kernel size is asymmetric on the frequency axis and the time axis, and a larger size is used on the time axis to obtain more time information. In the convolution layer, using convolution kernels of different sizes can make the original features have different resolutions. Use maximum pooling after the convolution layer to reduce the dimension.

[0019] Preferably, in S1, according to the powerful generalization ability of the CapsNet network and the ability of the attention mechanism to capture global dependencies, the present invention uses a residual network structure to combine the CapsNet network with the attention mechanism to construct an AttCapsule module. The attention mechanism is implemented based on transformer, in which 6 encoders and 6 decoders are set, and each encoder and decoder uses a multi-head attention mechanism, which is set to 8 heads. At the same time, in order to obtain contextual information, the present invention uses BiGRU with a fully connected layer with time distribution as the BiGRU module. These two modules are used to construct the AttCapsNet-BiGRU network of the present invention.

[0020] Preferably, in S2, since the IMF components are obtained by EMD transformation of the multi-sound signal, they are divided into LF group and HF group according to the frequency characteristics, and then the HF group is converted into a high-frequency signal and the LF group is reconstructed into a low-frequency signal after conversion by the AMM algorithm. At the same time, the original signal, the high-frequency signal and the low-frequency signal are respectively converted into a logarithmic Mel spectrum. If the three are simply added together, it will inevitably lead to information redundancy or information distortion of the multi-sound signal characteristics. In summary, the PSC structure is considered to process feature information of different dimensions, thereby effectively solving such problems.

[0021] Preferably, in S3, based on the constructed ACBGRM network, a ME-AttCapsNet-BiGRU network is created according to the advantages of extracting and integrating different feature information according to the PSC structure.

[0022] Preferably, in S4, the multi-sound signal is first transformed by EMD to obtain IMF components, which are divided into LF group and HF group according to frequency characteristics, and then the HF group is converted into a high-frequency signal and the LF group is reconstructed into a low-frequency signal after conversion by AMM algorithm. At the same time, the original signal, high-frequency signal and low-frequency signal are respectively converted into logarithmic Mel spectrograms, and the PSC structure is used to extract and fuse different feature information of the logarithmic Mel spectrogram to form high-level feature information, and a ME-AttCapsNet-BiGRU multi-sound event detection model is constructed.

[0023] The beneficial results of the present invention are as follows: in order to solve the problems of unstable time-frequency changes of sound and strong background noise, overlap between multiple sound events and data scarcity of weakly labeled data sets, the present invention proposes an Adaptive Mean Median-Empirical Mode Decomposition-Attention Capsule Bidirectional Gated Recurrent Multilayer Convolutional Neural Network (ME-AttCapsNet-BiGRU) weakly labeled multiple sound event detection model based on the empirical mode decomposition algorithm (EMD), capsule network (Capsule Network, CapsNet), attention mechanism (AttentionMechanism) and bidirectional gated recurrent unit (BiGRU). The model includes three modules: high-level feature extraction module, attention capsule network (Attention Capsule Network, AttCapsNet) module and BiGRU module. Specifically: (1) In view of the problems of unstable sound time-frequency changes and strong background noise in multi-sound event detection, in the high-level feature extraction module, the present invention uses the AMM-EMD algorithm to pre-process the sound signal to enrich the sound event feature information, and introduces an early fusion-based parallel convolutional structure (Parallel ConvolutionalStructure, PCS) with a recurrent layer to extract high-level feature information. (2) In view of the two problems of insufficient data set size in weakly labeled multi-sound event detection, which leads to the inability to fully utilize deep and complex neural networks and the overlap of multiple sound events, the AttCapsule module is proposed. The AttCapsule module utilizes the powerful generalization ability of the capsule network and the ability of the attention mechanism to capture global dependencies, and applies it to weakly labeled data sets, which helps to solve the data scarcity in model training, the problem of overlapping multiple sound events and more effectively utilize the sound event feature information. (3) In order to obtain context information, this paper adopts BiGRU with a time-distributed fully connected layer as the BiGRU module. The model proposed in this paper conducts ablation experiments and comparative experiments on the dataset, uses F1 and ER to evaluate the sound event detection effect, and conducts a large number of experiments on the public TUT SoundEvents2017 dataset.Comparative studies show that the models proposed by this method are improved over other baseline models, with F1 and ER increased, respectively, achieving competitive results. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The overall structure diagram of the ME-AttCapsNet-BiGRU multi-sound event detection model;

[0025] Figure 2 AttCapsNet structure diagram;

[0026] Figure 3 Working principle diagram of dynamic routing algorithm;

[0027] Figure 4 PSC structure diagram;

[0028] Figure 5 AttCapsNet-BiGRU structure diagram;

[0029] Figure 6 BiGRU module structure diagram; DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0031] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0032] The present invention will be further described below in conjunction with the accompanying drawings and specific examples, but it is not intended to be a limitation of the present invention. The multi-sound event detection method based on a multi-layer neural network in this embodiment is first, the multi-sound signal is subjected to EMD transformation to obtain the IMF component, and is divided into a low-frequency signal LF group and a high-frequency signal HF group according to the frequency characteristics. Secondly, the HF group is converted into a high-frequency signal by the AMM algorithm and the Mel spectrum graph is extracted. At the same time, the LF group is reconstructed into a low-frequency signal and the Mel spectrum graphs of the original signal and the low-frequency signal are respectively extracted, and the early fusion-based PCS structure with a loop layer is used to extract high-level feature information. At the same time, on the basis of using the PCS structure to form high-level feature information, a ME-AttCapsNet-BiGRU multi-sound event detection model is constructed, and after the BiGRU module obtains the context information, a feedforward layer with a sigmoid activation function is used to obtain the probability of event activity in each frame. The overall implementation structure block diagram is shown as follows. Figure 1This embodiment implements a method for detecting multiple sound events based on a multi-layer neural network, including:

[0033] S1. Use AMM-EMD to pre-process and preliminarily reduce noise on multiple sound signals, including:

[0034] S11, using PCS structure to form high-level feature information;

[0035] S12. Use the AttCapsNet-BiGRU network to process high-level feature information.

[0036] Based on the characteristics of EMD algorithm in both denoising and decomposing different frequency components, this implementation divides all IMFs after EMD decomposition into low-frequency signal LF group and high-frequency signal HF group, and uses the adaptive average median threshold method to denoise all components in the HF group with a soft threshold function and reconstruct them into high-frequency signals, and extract the Mel spectrum. At the same time, the LF group is reconstructed into a low-frequency signal and the Mel spectrum is extracted for the original signal and the low-frequency signal respectively, and the PCS structure is used to extract high-level feature information. Then, it is sent to the neural network for learning.

[0037] In view of the problem of overlapping sound events in multiple sound events and the problem of insufficient dataset size due to weak annotation, more and more scholars are using CapsNet networks to handle the detection of overlapping events in SED. The concept of capsule network was proposed by Hinton et al. in 2017. It was proposed to overcome some limitations of CNN, especially the information loss caused by maximum pooling for translation invariance. Deep learning models with attention mechanism have been widely used in many fields. The attention mechanism effectively reduces the computational complexity of the network by extracting more relevant high-level features and ignoring information irrelevant to the event (such as background noise). At the same time, CapsNet can solve the problem of sound overlap through its dynamic routing algorithm and voting mechanism.

[0038] Based on the powerful generalization ability of CapsNet network and the ability of attention mechanism to capture global dependencies, the present invention uses residual network structure to combine CapsNet network with attention mechanism to construct AttCapsule module. In AttCapsNet module, capsule layer and attention layer are introduced. The attention layer uses residual network connection structure in parallel with the second capsule layer. Its structure is as follows Figure 2 shown.

[0039] In the capsule layer, the first capsule layer is a convolutional layer with 32 channels, each channel consists of an 8-dimensional capsule. In the second capsule layer, the prediction vector of the high-level capsule is first calculated by multiplying the output of the low-level capsule by the weight matrix. The prediction vectors are then weighted and summed to obtain the output vector, whose weights are determined by the so-called dynamic routing, the purpose of which is to allow the low-level capsules to autonomously choose the best path to pass information to the high-level capsules. Finally, the output vector is nonlinearly mapped to obtain the final output vector. The working principle of the dynamic routing algorithm is as follows Figure 3 As shown:

[0040] Taking a two-layer capsule network as an example, assume that each basic capsule i in layer l has an activity vector The jth capsule in layer l+1 will receive u i And calculate its corresponding transformation matrix W ij , the resulting vector That is, it is the prediction of capsule i at layer l for the entity represented by capsule j at layer l+1, which can also be understood as the contribution of basic capsule i to high-level capsule j. For high-level capsule j, its output v j It can be calculated as follows:

[0041]

[0042] Where c ij is the coupling coefficient, s j is the weighted sum of all base capsule predictions of high-level capsule j, v j For the network to capsule vector u i The prediction vector v j All prediction vectors are composed of a set of coupling coefficients c ij The weighted and compressed by the nonlinear squashing function (3) to represent the probability of the target existence. The coupling coefficient c in (2) ij Determined by the dynamic routing algorithm. The updated c ij The formula is as follows:

[0043]

[0044] where b ij is the initial variable of weight, and v j The higher the similarity between them (measured by the inner product), the ij The more increments there are. In each forward propagation, b ij Initialized to 0, the coupling coefficient c is calculated from equation (4) ij The initial value of , and then calculate v from the forward propagation of the network j Update b according to (5) ij The value of c is updated using (5) ijThe value of s is further modified by forward propagation j The value of to change the output vector v j Finally, a set of optimal coupling coefficients is obtained.

[0045] The attention layer is implemented based on transformer, in which 6 encoders and 6 decoders are set up. A multi-head attention mechanism is used in each encoder and decoder, and the multi-head attention mechanism is set to 8 heads.

[0046] The input of the attention layer is the number of capsules representing audio frames at each time frame, and attention weights are given based on these capsules, allowing the network to find important frames related to the sound event class. The attention weight att(t) for each time frame is expressed as:

[0047] att(t)=σ(w*u(t)+b) (6)

[0048] Where u(t) is the input feature, w is the weight, b is the bias, and σ is the sigmoid function. The output of the AttCapsNet module is obtained by multiplying att(t) and the output of the second capsule layer.

[0049] At the same time, in order to obtain context information, the present invention uses BiGRU with a time-distributed fully connected layer as the BiGRU module. The AttCapsNet-BiGRU network of the present invention is constructed using these two modules.

[0050] S2. Based on preprocessing, PCS structure is used to extract and fuse different feature information to form advanced feature information:

[0051] In this implementation, since the multi-sound signal is transformed by EMD to obtain the IMF component, it is divided into LF group and HF group according to the frequency characteristics, and then the HF group is converted into a high-frequency signal and the LF group is reconstructed into a low-frequency signal after conversion by the AMM algorithm. At the same time, the original signal, high-frequency signal and low-frequency signal are respectively converted into logarithmic Mel spectrum diagrams. If the three are simply added together, it is bound to cause information redundancy or information distortion of the multi-sound signal characteristics. In summary, the PSC structure is considered to process feature information of different dimensions, thereby effectively solving such problems. The PSC structure consists of three parallelized convolution paths, each path consists of 4 convolution layers, and the convolution kernel size of the same path is the same. The convolution layers of the three paths have the same parameters except for the different convolution kernel sizes. The convolution kernel sizes of the three paths are 3×3, 5×5, and 7×7 respectively. The convolution kernel size is asymmetric on the frequency axis and the time axis, and a larger size is used on the time axis to obtain more time information. In the convolution layer, using convolution kernels of different sizes can make the original features have different resolutions. Max pooling is used after the convolutional layer to reduce the dimension. The PSC structure is as follows Figure 4shown.

[0052] S3. Based on the PCS structure to form high-level feature information, the ME-AttCapsNet-BiGRU multi-sound event detection model is constructed:

[0053] Based on the powerful generalization ability of CapsNet network and the ability of attention mechanism to capture global dependencies, this implementation adopts a residual network structure to combine CapsNet network with attention mechanism to construct AttCapsule module. At the same time, in order to obtain context information, the present invention adopts BiGRU with time-distributed fully connected layer as BiGRU module. These two modules are used to construct the AttCapsNet-BiGRU network of the present invention. Its structure is as follows Figure 5 shown.

[0054] S4. After the BiGRU module obtains the context information, a feed-forward layer with a sigmoid activation function is used to obtain the probability of event activity per frame:

[0055] The multi-sound signals are transformed by EMD to obtain IMF components, which are divided into LF group and HF group according to the frequency characteristics. Then the HF group is converted into a high-frequency signal and the LF group is reconstructed into a low-frequency signal after conversion by the AMM algorithm. At the same time, the original signal, high-frequency signal and low-frequency signal are converted into logarithmic Mel spectrograms respectively, and the PSC structure is used to extract and fuse different feature information of the logarithmic Mel spectrogram to form high-level feature information, which is then input into the constructed ME-AttCapsNet-BiGRU multi-sound event detection model. After the BiGRU module obtains the context information, a feedforward layer with a sigmoid activation function is used to obtain the probability of event activity in each frame. Its structure is as follows Figure 6 shown.

[0056] experiment

[0057] 1. Experimental environment

[0058] The experimental environment is carried out under the Pytorch framework. The experimental equipment uses the processor Inter(R)Core(TM)i7-9700 CPU@3.00GHz, installed memory 32G, operating system 64-bit Windows10, GPU model GEFORCE RTX 2080Ti, and the experiment is run in GPU mode.

[0059] 2. Sound Dataset

[0060] The present invention is evaluated using the weakly labeled dataset provided by DCASE 2017 Task 4, which is a subset of the largest corpus AudioSet. This dataset is a subset of AudioSet and consists of 17 sound events divided into two categories: "warning" and "vehicle". The maximum duration of each audio clip is 10 seconds and may correspond to multiple potentially overlapping sound events. Two tasks are evaluated on this dataset: audio annotation and sound event detection. For the audio annotation subtask, the micro-average of precision, recall, and F1 score is used to evaluate the performance of the model. For SED, a segmentation-based error rate with a resolution of 1 second is calculated.

[0061] The present invention uses the logarithmic Mel spectrum as input features. Before extracting features, each audio clip is resampled to 16kHz. The logarithmic Mel spectrum is calculated using a 64ms frame length, 20ms overlap, and 64 Mel frequency units per frame. For every 10 seconds of sample, a feature vector will be generated.

[0062] To reduce overfitting and speed up the convergence of the model, batch normalization is used after each convolutional layer and base capsule layer. For the number of routing iterations r, this paper sets it to 4. The Adam optimizer is used for training, and the learning rate is set to 0.001 and decays by 0.9 every two rounds. For the loss function, binary cross entropy is used and the gradient is calculated with a mini-batch size of 44. A total of 30 rounds are trained.

[0063] The present invention averages the obtained model predictions to obtain the final result. In the present invention, the detection thresholds are set to τ1 = 0.3 and τ2 = 0.6. For SED, the dilation and erosion sizes are set to 10 and 5, respectively.

[0064] 3. Experimental results

[0065] The method proposed in this invention is based on a weakly labeled dataset, where weakly labeled data only provides the event type in the audio without any time information. In order to verify the performance of the EM-AttCapsNet-BiGRU method proposed in this invention, comparative experiments and ablation experiments were conducted, and the experimental results are shown in Table 1, Table 2 and Table 3:

[0066] Table 1 Comparison of different kernel sizes

[0067]

[0068] The present invention tests the effect of different kernel sizes on model performance. As shown in Table 1, the model performs best when the kernel size is 7×3. The increase in kernel size on the time axis leads to better performance on the sound event detection subtask. However, further increasing the convolution size on the time axis increases the complexity of the model. Increasing the convolution kernel size on the frequency axis does not seem to improve performance. The main reason is that the pooling operation is performed along the frequency axis. If the convolution kernel size is large, the extracted features may come from the extended area instead of the local details of the signal.

[0069] Table 2 Performance results of audio annotation subtask

[0070]

[0071] Table 3 Performance results of the sound event detection subtask

[0072]

[0073] It can be seen from Tables 2 and 3 that for audio annotation, the ME-AttCapsNet-BiGRU proposed in the present invention performs best, with an F1 score of 0.615. For SED, ME-AttCapsNet-BiGRU achieved the highest F1 score and the lowest ER, with an F1 score of 0.587 and an ER of 0.615. The introduction of the attention layer enhances the ability of the AttCapsNet-BiGRU model to capture global dependencies and ignore information irrelevant to the event (such as background noise), effectively reducing the computational complexity of the network. Because its score is significantly higher than that of CapsNet-BiGRU, and the error rate is relatively low. It can be seen that the model proposed in the present invention has a significant improvement in the detection effect of multiple sound events.

Claims

1. A method for detecting multiple sound events based on a multi-layer neural network, characterized in that: include: S1, using AMM-EMD to pre-process and preliminarily reduce noise on multi-sound signals; S2, based on preprocessing, use PCS structure to extract and fuse different feature information to form high-level feature information; S3. Based on the formation of high-level feature information using the PCS structure, a ME-AttCapsNet-BiGRU multi-sound event detection model is constructed; S4. After the BiGRU module obtains the context information, a feed-forward layer with a sigmoid activation function is used to obtain the probability of event activity in each frame.

2. The method for detecting multiple sound events based on a multi-layer neural network according to claim 1, characterized in that: The S1 includes: S11, using PCS structure to form high-level feature information; S12. Use the AttCapsNet-BiGRU network to process high-level feature information.

3. The method for detecting multiple sound events based on a multi-layer neural network according to claim 2, characterized in that: In S11: The PSC structure consists of three parallelized convolution paths, each path consists of 4 convolution layers, and the convolution kernel size of the same path is the same. The convolution layers of the three paths have the same parameters except for the different convolution kernel sizes. The convolution kernel sizes of the three paths are 3×3, 5×5, and 7×7, respectively. The convolution kernel size is asymmetric on the frequency axis and the time axis, and a larger size is used on the time axis to obtain more time information. In the convolution layer, using convolution kernels of different sizes can make the original features have different resolutions. Use maximum pooling after the convolution layer to reduce the dimension.

4. The method for detecting multiple sound events based on a multi-layer neural network according to claim 2, characterized in that: In S12: According to the powerful generalization ability of the CapsNet network and the ability of the attention mechanism to capture global dependencies, the present invention uses a residual network structure to combine the CapsNet network with the attention mechanism to construct an AttCapsule module. The first capsule layer in the CapsNet network is a convolutional layer with 32 channels, and each channel consists of an 8-dimensional capsule. The attention mechanism is implemented based on the transformer, in which 6 encoders and 6 decoders are set up, and a multi-head attention mechanism is used in each encoder and decoder, and the multi-head attention mechanism is set to 8 heads. At the same time, in order to obtain contextual information, the present invention uses a BiGRU with a time-distributed fully connected layer as a BiGRU module. These two modules are used to construct the AttCapsNet-BiGRU network of the present invention.

5. The method for detecting multiple sound events based on a multi-layer neural network according to claim 1, characterized in that: In S2: Since the multi-sound signal is transformed into IMF components by EMD, it is divided into LF group and HF group according to the frequency characteristics, and then the HF group is converted into a high-frequency signal by AMM algorithm and the LF group is reconstructed into a low-frequency signal. At the same time, the original signal, high-frequency signal and low-frequency signal are respectively converted into logarithmic Mel spectrum graphs. If the three are simply added together, it will inevitably lead to information redundancy or information distortion of the multi-sound signal characteristics. In summary, the PSC structure is considered to process feature information of different dimensions, so as to effectively solve this kind of problem.

6. The method for detecting multiple sound events based on a multi-layer neural network according to claim 1, characterized in that: In S3: Based on the constructed AttCapsNet-BiGRU network, the ME-AttCapsNet-BiGRU network is created according to the advantages of extracting and integrating different feature information according to the PSC structure.

7. The method for detecting multiple sound events based on a multi-layer neural network according to claim 1, characterized in that: In S4: first, the multi-sound signal is transformed by EMD to obtain the IMF component, and it is divided into LF group and HF group according to the frequency characteristics, and then the HF group is converted into a high-frequency signal and the LF group is reconstructed into a low-frequency signal after conversion by the AMM algorithm. At the same time, the original signal, the high-frequency signal and the low-frequency signal are respectively converted into a logarithmic Mel spectrum graph, and the PSC structure is used to extract and fuse different feature information of the logarithmic Mel spectrum graph to form high-level feature information, and a ME-AttCapsNet-BiGRU multi-sound event detection model is constructed. After the BiGRU module obtains the context information, a feedforward layer with a sigmoid activation function is used to obtain the probability of event activity in each frame.

Images