A method of sound event detection

Abstract

The application provides a sound event detection method, belonging to the technical field of sound event detection, and comprising the following steps: obtaining an audio signal to be detected; pre-processing the audio signal to be detected; performing data enhancement on the pre-processed audio signal; performing feature extraction on the data-enhanced audio signal; extracting context information from the extracted features to obtain first branch features; inputting the data-enhanced audio signal into a pre-trained audio teacher-student model based on Transfomer to obtain second branch features; and inputting the first branch features and the second branch features into a classifier after splicing to obtain a sound event detection result of the audio signal to be detected. The application can obtain sufficient background information corresponding to different sound events, and improves the multi-scale feature capturing capability of the model.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of sound event detection, and particularly relates to a sound event detection method. BACKGROUND

[0002] Sound event detection (SED) is an important sub-discipline of machine hearing and one of the main tasks in acoustic scene analysis, aiming to determine the categories of sound events in the input audio and their corresponding start and offset times (time stamps) in the audio segment. Currently, SED related technologies have been widely applied in smart home, equipment fault monitoring, autonomous driving and other fields. Sound event detection helps to monitor the environment in real time and discover abnormal sounds in a timely manner, which is particularly important for intelligent transportation, industrial fault monitoring, smart home and other scenarios.

[0003] The use of convolutional recurrent neural network (CRNN) to realize the SED task is very extensive, and it is the mainstream model at present. Its structure is composed of cascading convolutional neural networks (CNN) and recurrent neural networks (RNN), in which CNN is responsible for feature extraction and RNN is responsible for obtaining context information. Guo Fanghong (patent application number: CN116798448A) and others based on CRNN network use hyperparameter complex convolutional layer for feature extraction, which reduces the model parameters and computational complexity and improves the performance. Xie Zongxia (patent number application number: CN114881212A) uses a double-branch feature extraction and fusion method to improve the discrimination between sound event classes and improve the accuracy of sound event detection. However, the current technology ignores the unique prior knowledge of sound events in acoustic scenes, and the lack of spatial information leads to insufficient feature extraction, which limits the performance of sound event detection in multi-sound source scenes.

[0004] From the research status at home and abroad, it can be seen that the sound event detection technology has a certain research foundation. However, there are still problems to be improved in terms of feature extraction, low detection accuracy in complex scenes with overlapping multiple sound sources, etc. The main points are as follows:

[0005] (1) The translational invariance of CNN and its fixed receptive field make it difficult to capture the subtle relationship between different frequency dimensions in the spectrogram, resulting in limited detection accuracy.

[0006] (2) In existing sound event detection networks, the unique prior knowledge of sound events in acoustic scenes is ignored, leading to missed detection and false detection problems in overlapping sound event detection.

[0007] The existing model usually needs to convert the audio into an image when extracting the feature of the audio signal by using the CNN, and extract the time-frequency feature of the signal on the spectrum graph. However, unlike the image, even if the high-frequency part and the low-frequency part on the audio spectrum graph present the same feature, it may not be the same acoustic event. This is the biggest difference between the spectrum graph and the natural image. The traditional CNN does not consider this point. In order to solve this problem, in recent years, various attention mechanisms combined with convolution blocks have been proposed, such as SENET (Squeeze-and-Excitation Networks), CBAM (Convolutional Block Attention Module) and the like. However, since the spectrum graph of the audio signal is not static, the attention weight is required to be constant, and these network architectures generally obtain dynamic attention weights by changing the image, which leads to limited feature precision and affects the accuracy of SED. Therefore, a frequency dynamic convolution proposed by a researcher generates a convolution kernel that adaptively inputs the frequency component through frequency dimension weighting, which achieves good results. However, the receptive field range of the adaptive convolution kernel is still fixed, ignoring the strong and valuable prior knowledge of the sound event in the acoustic scene, resulting in the loss of feature information. The reason is that the duration and feature complexity of different sound events are very different. For example, alarm and bell are usually single and explicit sound events, and their features are relatively obvious, and they require relatively less context information for detection. However, the sound of cat and dog needs to consider their species, activity state and environment, so the feature is relatively complex. Since different breeds of dogs have different barking or calling methods, and may be affected by the surrounding environment and emotional state, the feature changes complexly with time, and the recognition of such events depends on extensive context information, because the surrounding environment can provide valuable clues about their location, duration, environmental conditions and the like. The fixed size of the convolution receptive field will cause the network to only focus on the local features of the sound event, and when there are complex or overlapping sound events in the scene, the detection performance will be reduced. Therefore, in order to improve the accuracy and robustness of SED in complex scenes, a large receptive field range and adaptive convolution kernel need to be studied.

[0008] (3) The pre-trained model BEATs commonly used in existing research is patch-wise organized (block level) input, and the extracted features are more similar to the CNN, and lack attention to global features, so when the two are integrated, the performance of sound event detection is limited. SUMMARY

[0009] To overcome the shortcomings of the prior art, the present application provides a sound event detection method, which enables the model to obtain sufficient background information for different sound events, thereby improving the multi-scale feature capturing capability of the model. In addition, the ATST is a frame-wise input, which compensates for the shortcomings of CNN in extracting local features and enhances the global characteristics, thereby providing higher benefits from CNN. The method of the present application has better compatibility and significantly improved detection effect.

[0010] In a first aspect, the present application provides a sound event detection method, comprising:

[0011] obtaining an audio signal to be detected;

[0012] preprocessing the audio signal to be detected;

[0013] performing data enhancement on the preprocessed audio signal;

[0014] performing feature extraction on the data-enhanced audio signal;

[0015] extracting context information from the extracted features to obtain first branch features;

[0016] inputting the data-enhanced audio signal into a pre-trained audio teacher-student model based on Transfomer to obtain second branch features;

[0017] inputting the first branch features and the second branch features into a classifier after splicing to obtain a sound event detection result of the audio signal to be detected.

[0018] The preprocessing of the audio signal to be detected comprises:

[0019] resampling the audio signal to be detected;

[0020] performing Fourier transform on the resampled signal to obtain a two-dimensional frequency spectrum;

[0021] The data enhancement on the preprocessed audio signal comprises:

[0022] randomly moving the preprocessed audio signal along the time axis by a time frame to obtain a first enhancement result;

[0023] randomly mixing the preprocessed audio signal using a mixed parameter to obtain a second enhancement result;

[0024] selecting a time window to be masked according to a mask, shielding the selected time window by consecutive time frames, so that the corresponding part of the preprocessed audio signal is empty, to obtain a third enhancement result;

[0025] The first enhancement result, the second enhancement result, and the third enhancement result are taken as a result of data enhancement of the original audio signal.

[0026] The data-enhanced audio signal is subjected to feature extraction, including:

[0027] The data-enhanced audio signal is input into a CNN module for preliminary feature extraction to obtain initial features.

[0028] The initial features are input into a plurality of frequency dynamic convolution modules of attention mechanisms for in-depth feature extraction to obtain in-depth feature extraction results.

[0029] The data-enhanced audio signal is input into a CNN module for preliminary feature extraction to obtain initial features, including:

[0030] The data-preprocessed audio signal is input into a CNN layer for feature extraction.

[0031] The output of the CNN layer is input into a first normalization layer for normalization processing.

[0032] The normalized result is input into a first GLU layer for linear processing.

[0033] The linearly processed result is input into a first pooling layer for pooling processing.

[0034] The pooled result is regularized to obtain a regularization result, and the regularization result is taken as the initial features.

[0035] The initial features are input into a plurality of frequency dynamic convolution modules of attention mechanisms for in-depth feature extraction to obtain in-depth feature extraction results, including:

[0036] Step S4.2.1: The initial features are subjected to frequency dynamic convolution.

[0037] Step S4.2.2: The frequency dynamic convolution result is input into a second normalization layer for normalization processing, and the normalized result is taken as the first features.

[0038] Step S4.2.3: The normalized result is input into a first convolution kernel and a second convolution kernel respectively to obtain a first convolution kernel output and a second convolution kernel output respectively.

[0039] Step S4.2.4: The first convolution kernel output is input into a first convolution layer to obtain a first convolution result.

[0040] Step S4.2.5: The second convolution kernel output is input into a second convolution layer to obtain a second convolution result.

[0041] Step S4.2.6: The first convolution result is spliced with the second convolution result to obtain a first splicing result;

[0042] Step S4.2.7: The first splicing result is input into a second pooling layer for pooling processing;

[0043] Step S4.2.8: The pooling processing result is input into a third convolution layer;

[0044] Step S4.2.9: The third convolution layer output is input into a Sigmoid activation function to obtain a first attention weight and a second attention weight;

[0045] Step S4.2.10: The first convolution result is multiplied by the first attention weight to obtain a first multiplication result;

[0046] Step S4.2.11: The second convolution result is multiplied by the second attention weight to obtain a second multiplication result;

[0047] Step S4.2.12: The first multiplication result is spliced with the second multiplication result to obtain a second splicing result;

[0048] Step S4.2.13: The second splicing result is multiplied by the first feature to obtain a dynamic convolution structure extraction feature result;

[0049] Step S4.2.14: The dynamic feature structure extraction feature result is input into a second GLU layer to obtain a linear processing result;

[0050] Step S4.2.15: The linear processing result is subjected to pooling processing by a third pooling layer to obtain a first deep feature extraction result;

[0051] Step S4.2.16: The first deep feature extraction result is taken as an initial feature, and the step S4.2.1 is returned until a preset repetition number is reached, and the last deep feature extraction result is taken as a final deep feature extraction result.

[0052] The frequency dynamic convolution is used for frequency dimension segmentation and weighting of the initial feature, and the calculation formula is as follows:

[0053]

[0054] G k =π k (f,x)

[0055] y k (t,f)=W k *X(t,f)+B

[0056] wherein Y(t,f,x) is a frequency dynamic convolution result, t is time, f is frequency, k represents the kth base kernel, N represents the total number of base kernels, x is input data, and π k is an attention function of the kth base kernel, y k (t,f) is an output of the kth base kernel, G k is an attention weight of the kth base kernel, W k is a weight matrix of input, B is a bias matrix, and X(t,f) is an initial feature.

[0057] The extracted feature is subjected to context information extraction to obtain a first branch feature, including:

[0058] The extracted feature is input into a BGRU module.

[0059] The output of the BGRU module is input into an attention module, and the output of the attention module is taken as the first branch feature.

[0060] In a second aspect, the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method for sound event detection.

[0061] Beneficial effects:

[0062] The present application provides a method for sound event detection, in order to capture the deep features of different sound events, to make the model adaptively adjust the receptive field, to fully utilize the spatial information, and to propose a frequency dynamic convolution module of attention mechanism (i.e. LSKFDY-CNN convolution structure). On the one hand, more frequency information of the sound event is retained, and on the other hand, a wide and adaptive receptive range is provided for the model. By dynamically adjusting the receptive field, the model can obtain sufficient background information corresponding to different sound events, thereby improving the multi-scale feature capturing capability of the model. The defects that the existing convolution is prone to lose the relationship between features and lose unique context background information when extracting features by the convolutional neural network due to the translational invariance and fixed receptive field are overcome.

[0063] In order to effectively utilize the pre-trained model and further improve the sound event detection performance, the pre-trained Transfomer-based audio teacher-student model (ATST model) is integrated with the frequency dynamic convolution module of attention mechanism in the present application, so as to obtain more discriminative features and realize efficient feature extraction under limited strong label data. The ATST model is very good at generating frame-level audio representation, and the features are more fine and have obvious linear separability. The LSKFDY-CNN convolution structure can greatly benefit from it, which compensates for the deficiency of CNN in extracting local features and enhances the global characteristics. Therefore, the combination of the features extracted by the two improves the detection effect. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 A method flow chart of sound event detection according to an embodiment of the present application;

[0065] Figure 2 A flowchart of a sound event detection method based on a dynamic convolution structure according to an embodiment of the present application;

[0066] Figure 3 A structural diagram of an LSKFDY-CNN module according to an embodiment of the present application;

[0067] Figure 4 Model structure stability analysis according to an embodiment of the present application, wherein (a) is a PSDS1 diagram, and (b) is a PSDS2 diagram;

[0068] Figure 5 Actual scene model prediction result visualization comparison according to an embodiment of the present application, wherein (a) is a single sound scene prediction result, and (b) is a multi-sound scene prediction result. DETAILED DESCRIPTION

[0069] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0070] The present application proposes a sound event detection method, i.e., a sound event detection method based on a large selective kernel frequency dynamic convolution (Large Selective Kernel Frequency Dynamic Convolutional Neural Network, LSKFDY-CNN), which aims to improve the sound event detection accuracy and enhance the robustness of the sound event detection model in different complexity acoustic scenes. The present application builds a new convolution structure, which extracts features through the combination of FDY (Frequency Dynamic) convolution and large selective kernel (Large Selective Kernel, LSK) attention mechanism. The FDY convolution block is responsible for obtaining the feature differences of different frequency band spectrum graphs, and the LSK module dynamically adjusts the receptive field of the feature extraction backbone, which can capture the spatial information of different events and different duration events, and adaptively extract context information of different scales.

[0071] On this basis, the ATST model (ATST: Audio Representation Learning with Teacher-Student Transformer) is introduced to extract embedding features, realize efficient feature extraction in limited data, and finally effectively improve the performance of sound detection, overcoming the defects of existing networks that cannot effectively utilize the strong and valuable prior knowledge of different events.

[0072] Embodiment 1

[0073] The embodiment provides a sound event detection method, as shown in the following formula (1) : Figure 1

[0074] Step S1: acquiring an audio signal to be detected;

[0075] Step S2: preprocessing the audio signal to be detected;

[0076] Step S3: data enhancement is performed on the audio signal after data preprocessing

[0077] Step S4: feature extraction is performed on the audio signal after data enhancement;

[0078] Step S5: context information extraction is performed on the extracted features to obtain first branch features;

[0079] Step S6: inputting the data enhanced audio signal into a pre-trained audio teacher-student model based on Transfomer to obtain second branch features;

[0080] Step S7: inputting the first branch features and the second branch features after splicing into a classifier to obtain a sound event detection result of the audio signal to be detected.

[0081] In step S2, the preprocessing of the audio signal to be detected includes:

[0082] Step S2.1: resampling the audio signal to be detected;

[0083] Step S2.2: performing Fourier transform on the resampled signal to obtain a two-dimensional spectrum graph.

[0084] ​In this embodiment, the audio signal to be detected of the input system is preprocessed, and converted from one-dimensional audio signal to two-dimensional spectrogram. The specific preprocessing process is: the 44.1kHz audio signal is resampled to 16kHz, the audio signal is divided into frames each having 2,048 samples, and the skip length has 256 samples. Each time frame is subjected to 2048-point fast Fourier transform (FFT), and then subjected to 128-dimensional mel filter bank analysis, and finally 128-dimensional Log-Mel spectrogram is obtained.

[0085] In step S3, the preprocessed audio signal is subjected to data enhancement, including:

[0086] Step S3.2: data enhancement is performed on the preprocessed original audio signal, and the enhanced result is taken as a training sample, including:

[0087] Step S3.2.1: the preprocessed audio signal is randomly moved along the time axis to obtain a first enhanced result;

[0088] Step S3.2.2: the preprocessed audio signal is randomly mixed by using a mixed parameter to obtain a second enhanced result;

[0089] Step S3.2.3: according to a mask, a time window to be masked is selected, and the selected time window is shielded in a continuous time frame, so that the corresponding part of the preprocessed audio signal is empty, to obtain a third enhanced result;

[0090] Step S3.2.4: the first enhanced result, the second enhanced result and the third enhanced result are taken as the data enhancement result of the historical audio signal.

[0091] In this embodiment, the data after preprocessing is subjected to data enhancement. In the present application, three different data enhancement techniques, frame shift, data mixing and time masking, are used to realize training data expansion. Among them, frame shift is to randomly move the time frame along the time axis to generate new training samples; the mixing method is to randomly mix the selected data samples with a mixing parameter to obtain new samples; time masking is to first select the time window to be masked according to the strong mask, and then shield the selected time window in a continuous time frame, so that the corresponding part of the spectrogram is empty, to obtain new training data. Using data enhancement technology can enhance the diversity of training data, to a certain extent, overcome the problem of small amount of data, and at the same time, help the model to better identify the characteristics of different sound events and their subtle differences, and enhance the robustness and detection accuracy of the model.

[0092] In step S4, the data enhanced audio signal is subjected to feature extraction, as shown in Figure 2 , including:

[0093] Step S4.1: input the data-enhanced audio signal into the CNN module for preliminary feature extraction to obtain initial features, including:

[0094] Step S4.1.1: input the data-preprocessed audio signal into the CNN layer for feature extraction;

[0095] Step S4.1.2: input the output of the CNN layer into the first normalization layer for normalization processing;

[0096] Step S4.1.3: input the result after normalization processing into the first GLU layer for linear processing;

[0097] Step S4.1.4: input the result after linear processing into the first pooling layer for pooling processing;

[0098] Step S4.1.5: regularize the result after pooling to obtain a regularization result, and take the regularization result as the initial feature.

[0099] In this embodiment, after the data-preprocessed audio signal is extracted by the CNN to obtain shallow features, the initial features are obtained through the normalization layer, the first GLU layer and the first pooling layer.

[0100] Step S4.2: input the initial features into a plurality of frequency dynamic convolution modules of attention mechanisms for in-depth feature extraction to obtain in-depth feature extraction results, as shown in Figure 3 , including:

[0101] Step S4.2.1: frequency dynamic convolution is performed on the initial features;

[0102] Step S4.2.2: input the frequency dynamic convolution result into the second normalization layer for normalization processing, and take the result after normalization processing as the first feature;

[0103] Step S4.2.3: input the result after normalization processing into the first convolution kernel and the second convolution kernel respectively to obtain the first convolution kernel output and the second convolution kernel output respectively;

[0104] Step S4.2.4: input the first convolution kernel output into the first convolution layer to obtain a first convolution result;

[0105] Step S4.2.5: input the second convolution kernel output into the second convolution layer to obtain a second convolution result;

[0106] Step S4.2.6: splice the first convolution result and the second convolution result to obtain a first splicing result;

[0107] Step S4.2.7: input the first splicing result into the second pooling layer for pooling processing;

[0108] Step S4.2.8: input the pooling processing result into the third convolutional layer;

[0109] Step S4.2.9: input the third convolutional layer output into the Sigmoid activation function to obtain the first attention weight and the second attention weight;

[0110] Step S4.2.10: multiply the first convolutional result with the first attention weight to obtain the first multiplication result;

[0111] Step S4.2.11: multiply the second convolutional result with the second attention weight to obtain the second multiplication result;

[0112] Step S4.2.12: splice the first multiplication result with the second multiplication result to obtain the second splicing result;

[0113] Step S4.2.13: multiply the second splicing result with the first feature to obtain the dynamic convolutional structure extraction feature result;

[0114] Step S4.2.14: input the dynamic feature structure extraction feature result into the second GLU layer to obtain the linear processing result;

[0115] Step S4.2.15: perform the pooling processing on the linear processing result through the third pooling layer to obtain the first deep feature extraction result; Step S4.2.16: take the first deep feature extraction result as the initial feature, and return to step S4.2.1 until the preset repetition number is reached, and take the last deep feature extraction result as the final deep feature extraction result.

[0116] In this embodiment, the model architecture is as follows Figure 3The features of the spectrogram are extracted using LSKFDY-CNN, including frequency features, spatial context information and high-level semantic information. LSKFDY-CNN is the core module proposed in the application, which is composed of an FDY convolution block and an LSK module. The FDY convolution block generates adaptive frequency convolution kernels by splitting and weighting the frequency dimension of the feature, which weakens the translation invariance of CNN on the frequency axis, obtains sufficient frequency information, and makes the convolution operation more reasonable for audio features. The LSK module uses its adaptive receptive field to enhance and fuse the feature maps according to the differences of different sound event features, and pays more attention to spatial information to provide reasonable context information for different event types. The adaptive selection mechanism of LSKFDY can effectively weight the features processed by a series of large kernels and then combine them in space. The weights of these kernels are dynamically determined based on the input, allowing the model to adaptively use different large kernels (i.e., the first and second convolution kernels) and adjust the receptive field of each target in space as needed. The first convolution kernel k1 and the second convolution kernel k2 are two large convolution kernels of different sizes, which are decomposed from a larger large kernel. The decomposed large convolution kernel adopts the method of dilated convolution, for example, replacing a (5x5) convolution with a (3x3) dilated kernel with a (2x2) dilated rate. The advantage of this is that it relieves the quadratic increase in computational cost caused by the large size of the ordinary convolution kernel, and the dilated convolution quickly expands the receptive field through the dilated factor, which can improve the speed and efficiency of the model. After the first convolution kernel k1 and the second convolution kernel k2, a series of operations such as dimension reduction, pooling, activation function and feature map splicing are performed on the different scale features extracted by the two large kernels to assign corresponding weights to them, i.e., complete the selection mechanism. The detailed principle and process are as follows:

[0117] The FDYCNN splits the frequency dimension of the spectrogram feature, calculates the weight of each base kernel, and obtains the output of each base kernel:

[0118] y k (t,f)=W k *X(t,f)+B

[0119] Where t and f represent time and frequency, k represents the kth base kernel, W represents the weight matrix, and B represents the bias matrix. Then, the attention weights of the k base kernels are generated adaptively:

[0120] G k =π k (f,x)

[0121] Finally, the weighted sum of the output results is calculated:

[0122]

[0123] where Y(t, f, x) is the frequency dynamic convolution result, i.e., the output of the frequency dynamic layer after aggregating multiple base kernels, t is time, f is frequency, k represents the kth base kernel, N represents the total number of base kernels, x is input data, and k is the attention function of the kth base kernel, y k (t, f) is the output of the kth base kernel, G k is the attention weight of the kth base kernel, W k is the weight matrix of the input, B is the bias matrix, and X(t, f) is the initial feature.

[0124] The detailed workflow of the LSK module is as follows: two branches of large convolution kernels are used for feature extraction, and the sizes of the two selected large kernels are 5x5 (i.e., the first convolution kernel k1) and 25x25 (i.e., the second convolution kernel k2) in the experiment. Then, the size of the feature map is reduced to 1 dimension, i.e., X1, X2∈R 1×F×F , by (1x1) convolution. The features of two different scales are spliced to obtain X. In order to allow information interaction between different spatial descriptors, the spatial maximum pooling and average features are connected, and after spatial pooling, the following is obtained:

[0125] Agg = (P max (X) + P avg (X))

[0126] where P max and P avg represent the maximum pooling and average pooling operations, respectively, and Agg represents the feature after spatial pooling.

[0127] Subsequently, the sigmoid function is used to process the vector, mapping its value to between 0 and 1, to obtain the attention weights Wi of the two branch feature maps:

[0128] W i = sigmoid(F 2→c (Agg))

[0129] where W i represents the attention weights of the feature maps of different branches, including the first attention weight W1 and the second attention weight W2.

[0130] Finally, the attention weights are multiplied by the original feature map to obtain the weighted feature map, and the final output F(X) is obtained after feature fusion.

[0131]

[0132] where, is the i-th convolution result, including: the first convolution result and the second convolution result In this embodiment, N is 2, and X' is the first feature.

[0133] In step S5, the extracted features are subjected to context information extraction to obtain first branch features, including:

[0134] Step S5.1: input the extracted features into the BGRU module;

[0135] Step S5.2: input the output of the BGRU module into the attention module, and take the input of the attention module as the first branch feature.

[0136] In this embodiment, the extracted features are input into the BGRU module to fit the context information, and then a single LSK module is used to enhance the feature map.

[0137] Step S6: the pre-trained Transfomer-based audio teacher-student model extracts features from the data-enhanced samples.

[0138] The pre-trained Transfomer-based audio teacher-student model (i.e., the pre-trained model ATST) is integrated with the backbone network as another branch to extract more discriminative features in the audio, helping the original model to obtain high-level features containing rich semantic information under limited data volume. The audio feature dimension extracted by ATST is (250x1x768), and the audio feature dimension extracted by LSKFDY-CNN is (250x1x256). After splicing, the feature with a channel number of 1024 is obtained.

[0139] In step S7, the first branch feature and the second branch feature are spliced and input into the classifier to obtain the sound event detection result of the audio signal to be detected.

[0140] In this embodiment, the features extracted by the two branches are spliced and input into the extracted feature input classifier, a time-distributed fully connected layer (FC) and an activation function sigmoid are applied, the existence probability of the sound event along the time axis is predicted, and finally the predicted probability is aggregated to output the sound event category and timestamp information.

[0141] The embodiment uses the indoor sound event detection dataset (DESED), which is the most mainstream dataset at present. The embodiment calculates the complex sound event detection score (PSDS) and the F1 score for evaluating the performance of the SED system. The F1 score is the harmonic mean of the precision (P) and the recall (R), and the PSDS index is obtained by the normalized area under the ROC curve formed by a series of coordinates. A higher PSDS value indicates better model performance, and it is more robust to label subjectivity and can better understand data bias and classification stability across sound classes. In the present application, PSDS1 and PSDS2 are calculated respectively. The former focuses on the rapid detection of the starting time in the timestamp information of the sound event, and the latter focuses on the size of the model's ability to distinguish sound classes.

[0142] (1) The sound event detection in the multi-source acoustic scene is accurately detected. The present application is simple to implement, has high detection accuracy, is highly compatible with other sound processing tasks, and meets the application requirements. In order to verify the superiority of the proposed structure, the mainstream model is compared in the presence and absence of a pre-trained model, and the results are shown in Tables 1 and 2.

[0143] Table 1 Comparison of performance of baseline and different dynamic convolution structure models

[0144]

[0145] Table 2 Comparison of effects of pre-trained model ATST feature embedding model

[0146]

[0147] As can be seen from Tables 1 and 2, the four performance indicators of the LSKFDY-CNN proposed in the present application are all higher than those of other systems of the same type, which shows the effectiveness of the LSKFDY convolution. At the same time, the network performance is further improved after adding the pre-trained model ATST in Table 2. This further shows the effectiveness of the LSKFDY structure. In addition, since ATST is better than BEATs at generating frame-level audio representations, the features are more fine-grained and have obvious linear separability. Therefore, the LSKFDY structure proposed in the present application is more compatible and complementary with the pre-trained model ATST. ATST compensates for the characteristics of CNN in extracting local features and enhances the global characteristics. The LSKFDY-CNN structure proposed in the present application can obtain sufficient information of itself and from the environment for different feature complexity sound events. The large receptive field of the convolution can enhance the resolution capability of long and short time events, and the selective mechanism can focus attention on important features related to the corresponding event. Through selective attention, the model can identify and emphasize the key features related to the event, thereby improving the detection accuracy.

[0148] (2) The model has better stability and better performance. In order to test the dynamic structural stability of the LSKFDY-CNN network proposed in this paper, we recorded the PSDS index of 12 experiments respectively, and compared it with FDYCNN, as shown in the following table. Figure 4

[0149] As can be seen from the evaluation experiment results, the stability of the model PSDS1 proposed by us has been obviously enhanced relative to FDYCNN; the stability of the PSDS2 index has not been significantly improved, but the effect of each experiment is better than that of FDYCNN, and the performance is still greatly improved. The reason is that the events of different time-frequency mode complexity obtain their corresponding prior information, the model is less affected by the diversity of data and the small amount of data, so the accuracy is improved and the detection speed is accelerated, making the PSDS1 index more stable.

[0150] (3) High detection accuracy in multi-sound overlapping scenarios. The model can effectively model overlapping sound events, provide accurate classification and timestamp detection, and the test results in actual scenarios are as shown in the following table. Figure 5

[0151] Figure 5 (a) and (b) directly show the output sound class and event activity range results of different models on the test set in single-sound and multi-sound acoustic scenarios, where the true label shows the true sound class and event activity range of each time frame, providing a standard for evaluating the prediction accuracy of other models. As can be seen from the results, compared with the existing mainstream model structure, the prediction results of the model structure proposed by us are closest to the true label, and the classification characteristics and timestamp prediction also achieve the best detection effect. The sound class can be accurately predicted in most time frames, the misjudgment rate is low, and our model is more sensitive in detecting the start and end of the audio event, and the overlapping sound detection effect is the best. In the single-sound acoustic scenario, it almost perfectly matches the true label throughout the time sequence, and in the multi-sound overlapping complex acoustic scenario, it can also successfully detect the class and timestamp information, which shows that the overlapping sound events are effectively modeled, and even in the complex environment of multiple sound sources, a high detection accuracy can be achieved.

[0152] Embodiment 2:

[0153] The embodiment proposes a computer program product, including a computer program or instructions, which are executed by a processor to realize the method of sound event detection.

[0154] Based on such understanding, the technical solution of the present application, in essence or the part that contributes to the prior art, or part of the technical solution can be embodied in the form of a computer program product.

[0155] ​​The various embodiments in the present application are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments.

[0156] The scope of the present application is not limited to the above-described embodiments. It is apparent that a person skilled in the art can make various modifications and changes to the present disclosure without departing from the scope and spirit of the present disclosure. If these modifications and changes belong to the scope of the claims of the present disclosure and equivalent technologies thereof, the present disclosure intends to include these modifications and changes.

Claims

1. A method for detecting a sound event, characterized in that: include: Obtaining the audio signal to be detected; Preprocessing the audio signal to be detected; Perform data enhancement on the audio signal after data preprocessing; Perform feature extraction on the audio signal after data enhancement; Context information is extracted from the extracted features to obtain the first branch features; The data-augmented audio signal is input into the pre-trained Transformer-based audio teacher-student model to obtain the second branch features; The first branch feature and the second branch feature are spliced ​​and input into the classifier to obtain the sound event detection result of the audio signal to be detected; The performing data enhancement on the audio signal after data preprocessing includes: Randomly shifting the time frame of the audio signal after data preprocessing along the time axis to obtain a first enhancement result; Randomly mixing the preprocessed audio signal using a mixing parameter to obtain a second enhancement result; Selecting a time window to be masked according to the mask, shielding the selected time window for consecutive time frames so that a corresponding portion of the preprocessed audio signal is empty, thereby obtaining a third enhancement result; The first enhancement result, the second enhancement result, and the third enhancement result are used as the data enhancement results of the original audio signal; The feature extraction of the audio signal after data enhancement includes: The audio signal after data enhancement is input into the CNN module for preliminary feature extraction to obtain the initial features; The initial features are input into the frequency dynamic convolution module of multiple attention mechanisms for in-depth feature extraction to obtain in-depth feature extraction results; The audio signal after data enhancement is input into the CNN module for preliminary feature extraction to obtain initial features, including: The audio signal after data preprocessing is input into the CNN layer for feature extraction; Input the output of the CNN layer into the first normalization layer for normalization; The normalized result is input into the first GLU layer for linear processing; The result after linear processing is input to the first pooling layer for pooling processing; Regularize the pooled result to obtain the regularized result, and use the regularized result as the initial feature; The initial features are input into the frequency dynamic convolution module of multiple attention mechanisms for in-depth feature extraction, and in-depth feature extraction results are obtained, including: Step S4.2.1: performing frequency dynamic convolution on the initial features; Step S4.2.2: Input the frequency dynamic convolution result into the second normalization layer for normalization, and use the normalization result as the first feature; Step S4.2.3: Input the normalized results into the first convolution kernel and the second convolution kernel, respectively, to obtain the first convolution kernel output and the second convolution kernel output; Step S4.2.4: Input the output of the first convolution kernel into the first convolution layer to obtain the first convolution result; Step S4.2.5: Input the second convolution kernel output into the second convolution layer to obtain the second convolution result; Step S4.2.6: Concatenate the first convolution result and the second convolution result to obtain a first concatenated result; Step S4.2.7: Input the first splicing result into the second pooling layer for pooling processing; Step S4.2.8: Input the pooling result into the third convolutional layer; Step S4.2.9: Input the output of the third convolutional layer into the Sigmoid activation function to obtain the first attention weight and the second attention weight; Step S4.2.10: Multiply the first convolution result by the first attention weight to obtain a first multiplication result; Step S4.2.11: Multiply the second convolution result by the second attention weight to obtain a second multiplication result; Step S4.2.12: Concatenate the first multiplication result and the second multiplication result to obtain a second concatenated result; Step S4.2.13: Multiply the second concatenation result by the first feature to obtain a dynamic convolution structure extraction feature result; Step S4.2.14: Input the dynamic feature structure extraction feature result into the second GLU layer to obtain the linear processing result; Step S4.2.15: The linear processing result is pooled through the third pooling layer to obtain the first in-depth feature extraction result; Step S4.2.16: Use the first in-depth feature extraction result as the initial feature, return to step S4.2.1, and repeat until the preset number of times is reached, and use the last in-depth feature extraction result as the final in-depth feature extraction result; The frequency dynamic convolution is used to perform frequency dimension segmentation and weighting on the initial features. The calculation formula is as follows: G k =π k (f,x); y k (t,f)=W k *X(t,f)+B; Among them, Y(t,f,x) is the frequency dynamic convolution result, t is time, f is frequency, k is the kth base kernel, N is the total number of base kernels, x is the input data, π k is the attention function of the k-th base kernel, y k (t,f) is the output of the k-th base kernel, G k is the attention weight of the k-th base kernel, W k is the input weight matrix, B is the bias matrix, and X(t,f) is the initial feature.

2. The method for detecting a sound event according to claim 1, wherein: The preprocessing of the audio signal to be detected includes: resampling the audio signal to be detected; The resampled signal is Fourier transformed to obtain a two-dimensional spectrum.

3. The method for detecting a sound event according to claim 1, wherein: The extracting context information from the extracted features to obtain first branch features includes: Input the extracted features into the BGRU module; The output of the BGRU module is input into the attention module, and the output of the attention module is used as the first branch feature.

4. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method for detecting a sound event according to any one of claims 1 to 3 is implemented.

Images