A method and apparatus for sound event detection based on audio cue learning

Abstract

This invention relates to the field of sound event detection technology, and discloses a sound event detection method and apparatus based on audio cue learning. The method introduces a learnable audio cue vector into the input of each layer of the Swin Transformer Block in the Swin Transformer model. During model training, the backbone network parameters are maintained, and only the audio cue vector and the network structure related to downstream tasks are fine-tuned. Strong and weak labels are added to jointly optimize model training during the fine-tuning process, which reduces the number of training parameters, making the model more accurate in frame-level localization and obtaining a first sound event detection model with better detection performance. The sound to be detected is converted into a first Mel spectrogram, which is input into the first sound event detection model, and the event frame-level distribution is output, thereby obtaining the sound event category and its activity time in the sound to be detected, which can improve the accuracy and efficiency of sound event detection.

Description

Technical Field

[0001] This invention relates to the field of sound event detection technology, and in particular to a sound event detection method and apparatus based on audio cue learning. Background Technology

[0002] Sound event detection involves detecting the presence of target sound events within an audio segment and labeling the event type and start / end times to achieve audio scene parsing and analysis. It falls under the category of audio pattern recognition. Most existing sound event detection technologies are derived by improving traditional audio classification tasks and modifying the output structure. While numerous pre-trained models based on audio classification exist, applying these models to downstream sound event detection tasks remains a significant challenge.

[0003] Currently, existing sound event detection technologies involve fine-tuning all parameters of a pre-trained model. However, this fine-tuning can affect the feature representation capabilities of the pre-trained model, requires longer training time, a larger number of training parameters, and sometimes necessitates a larger dataset, resulting in significant costs and budget. Existing training methods constrain frame-level loss using frame-level labels, but in real-world scenarios, sufficient frame-level label data is unavailable, and labeling it requires substantial resources. Therefore, this training method lacks versatility. Summary of the Invention

[0004] This invention provides a sound event detection method and apparatus based on audio cue learning, which can reduce model training time and storage space, save training costs, and improve the accuracy and efficiency of sound event detection.

[0005] To address the aforementioned technical problems, this invention provides a sound event detection method based on audio cue learning, comprising:

[0006] Feature extraction is performed on the sound to be detected to obtain the first Mel spectrogram corresponding to the sound to be detected;

[0007] The first Mel spectrogram is input into a preset first sound event detection model, the first Mel spectrogram is encoded to form a first sequence, and the first sequence is transformed into a second sequence through a preset number of Swing Transformer Blocks. Then, the second sequence is structurally modified using Token-Semantic CNN to output the event frame-level distribution of the sound to be detected.

[0008] Based on the event frame-level distribution of the sound to be detected, the sound event category and sound event activity time of the sound to be detected are determined.

[0009] This invention extracts features from the sound to be detected and converts them into a first Mel spectrogram. The first Mel spectrogram is then input into a first sound event detection model. Data processing, such as encoding and structural modification, is performed on the first Mel spectrogram to output an event frame-level distribution. Based on the event frame-level distribution output by the model, the sound event categories and their event activity times in the sound to be detected can be determined, thereby improving the accuracy and efficiency of sound event detection.

[0010] Further, the step of extracting features from the sound to be detected to obtain the first Mel spectrogram corresponding to the sound to be detected specifically involves:

[0011] Acquire the first time-domain signal of the sound to be detected;

[0012] Perform a short-time Fourier transform on the first time-domain signal to obtain the first short-time Fourier transform amplitude spectrum;

[0013] Using a preset Mel filter bank, the first short-time Fourier transform amplitude spectrum is converted into the first Mel short-time Fourier transform amplitude spectrum;

[0014] The first Mel short-time Fourier transform amplitude spectrum is obtained by taking the logarithm of the first Mel spectrum.

[0015] Further, the encoding of the first Mel spectrum to form a first sequence specifically involves:

[0016] The first Mel spectrum is split into several patches;

[0017] The patches are concatenated into a first matrix so that the size of the first matrix is ​​the same as the input size of the first sound event detection model.

[0018] The first matrix is ​​divided into several patch tokens using the patch-embed CNN in the first sound event detection model.

[0019] Several patch tokens are arranged according to a preset order to form the first sequence.

[0020] This invention inputs the first Mel spectrum into the model, adjusts the size of the first Mel spectrum to match the input size of the model, encodes the first Mel spectrum, completes data preprocessing, and forms a first sequence to adapt to the processing form of the model and avoid data mismatch.

[0021] Furthermore, the structural modification of the second sequence using Token-Semantic CNN to obtain the event frame-level distribution of the sound to be detected is specifically as follows:

[0022] Before modifying the structure of the second sequence, the second sequence is subjected to dimensionality increase processing to form a third sequence, which includes frequency information, time information and category information.

[0023] Using Token-Semantic CNN, the frequency information of the third sequence is aggregated, and the temporal information of the third sequence is interpolated to obtain the event frame-level distribution of the sound to be detected.

[0024] This invention performs dimensionality-upgrading on the second sequence to form a third sequence; it aggregates the frequency information contained in the third sequence, eliminating the frequency information dimension and leaving only the time and category information dimensions. By integrating the data information before model output, the model output information becomes more concise and clear, improving the efficiency of sound event detection.

[0025] Furthermore, before outputting the event frame-level distribution of the sound to be detected, the method further includes:

[0026] Determining whether the event frame-level distribution of the sound to be detected is valid involves the following steps:

[0027] Determine whether the prediction of the first event is greater than a preset first threshold; wherein, the prediction of the first event is output by the first sound event detection model after processing the sound to be detected;

[0028] When the first event prediction is greater than the first threshold, the event frame-level distribution of the sound to be detected is determined to be valid, and the event frame-level distribution of the sound to be detected is output from the first sound event detection model.

[0029] Before the model outputs the event frame-level distribution, this invention first judges the validity of the event frame-level distribution based on the event prediction corresponding to the sound to be detected. That is, it uses a threshold method to detect whether the event prediction is greater than a preset first threshold. If it is greater, the event frame-level release can be judged to be valid. After concluding that the event frame-level release is valid, the event frame-level release is then output from the model to improve the accuracy of sound event detection.

[0030] Furthermore, the first sound event detection model is specifically as follows:

[0031] Select the Swing Transformer model, and introduce an audio cue vector into the input of each Swing Transformer Block in the Swing Transformer model;

[0032] Lock the backbone network parameters of the Swin Transformer model and determine that the objects of model training are the audio cue vector and the Token-Semantic CNN in the Swin Transformer model;

[0033] The Swin Transformer model is trained and fine-tuned using both strongly labeled and weakly labeled datasets with strong and weak supervision.

[0034] The trained Swing Transformer model was selected as the first sound event detection model.

[0035] This invention introduces a learnable audio cue vector into the input of each layer of the Swin Transformer Block in the pre-trained Swin Transformer model. During model training, the backbone network parameters in the model are maintained, and only the audio cue vector and the network structure related to downstream tasks are fine-tuned. This can reduce the number of training parameters, shorten the training time, and better maintain the original model's strong representational ability during the training process.

[0036] Furthermore, the training process for the first sound event detection model is as follows:

[0037] Obtain the source sound; all sound events of the source sound are of the same category;

[0038] Create several background sounds based on the source sound;

[0039] Data augmentation is performed on the aforementioned background sounds to form a training dataset;

[0040] Based on the training dataset, construct a strongly labeled dataset and a weakly labeled dataset respectively;

[0041] The training dataset is used to extract features and convert them into training Mel spectrograms.

[0042] The trained Mel spectrogram is input into the Swin Transformer model so that the Swin Transformer model outputs training event predictions and training frame-level predictions.

[0043] The Sigmoid function is used to activate the prediction of the training events and the prediction of the training frames.

[0044] A loss function is formed by combining binary cross-entropy and smoothed absolute error; wherein the binary cross-entropy is used to calculate the loss for training event prediction, and the smoothed absolute error is used to calculate the loss for training frame-level prediction.

[0045] The Swing Transformer model is updated based on the loss function.

[0046] This invention constructs a strongly labeled dataset, trains a model to output training event predictions and training frame-level predictions, activates the two predictions, and then calculates the loss using different loss functions to update the model. By incorporating joint optimization of model training with strong and weak labels during model fine-tuning, the model can achieve more accurate frame-level localization and obtain better detection performance.

[0047] Furthermore, the step of performing data augmentation on the plurality of background sounds to form a training dataset specifically involves:

[0048] The aforementioned sound backgrounds are time-shifted, and the time-shifted sound backgrounds are used to form a training dataset;

[0049] The specific formula for time-shifting the plurality of background sounds is as follows:

[0050] x sift (n) = x(n+l)

[0051] Wherein, the x sift The background sound is time-shifted, and x is the normalized audio signal. sift (n) represents the nth point of the background sound after time shift, and l represents the number of points shifted.

[0052] Furthermore, the construction of a strong-label dataset and a weak-label dataset based on the training dataset specifically involves:

[0053] The specific formula for constructing a weakly labeled dataset is as follows:

[0054] y weak (i) = [0,0,…,1,…,0] T i∈[1,class_num]

[0055] Among them, y weak (i) is the weak label of the i-th labeled sound event in the training dataset; class_num is the number of sound event categories; i is the label index;

[0056] The specific formula for constructing a strongly labeled dataset is as follows:

[0057]

[0058] i is the tag index; j is the frame index; class_num is the number of sound event categories; frame_size is the number of frames.

[0059] This invention provides a sound event detection method based on audio cue learning. In the Swin Transformer model, a learnable audio cue vector is introduced as input to each Swin Transformer Block layer. During model training, the backbone network parameters are maintained, and only the audio cue vector and the network structure related to downstream tasks are fine-tuned. Strong and weak labels are added to jointly optimize model training during the fine-tuning process, which reduces the number of training parameters and makes the model more accurate in frame-level localization, resulting in a first sound event detection model with better detection performance. The sound to be detected is converted into a first Mel spectrogram, which is then input into the first sound event detection model to output the event frame-level distribution, thereby obtaining the sound event category and its activity time in the sound to be detected, which can improve the accuracy and efficiency of sound event detection.

[0060] Accordingly, the present invention provides a sound event detection device based on audio prompt learning, comprising: an acquisition module, a detection module, and a determination module;

[0061] The acquisition module is used to extract features from the sound to be detected and obtain the first Mel spectrogram corresponding to the sound to be detected.

[0062] The detection module is used to input the first Mel spectrogram into a preset first sound event detection model, encode the first Mel spectrogram to form a first sequence, and convert the first sequence into a second sequence through a preset number of Swing Transformer Blocks. Then, the second sequence is structurally modified using Token-Semantic CNN to output the event frame-level distribution of the sound to be detected.

[0063] The determining module is used to determine the sound event category and sound event activity time of the sound to be detected based on the event frame-level distribution of the sound to be detected.

[0064] This invention provides a sound event detection device based on audio cue learning. Based on the organic combination of modules, it can reduce model training time and storage space, save training costs, and improve the accuracy and efficiency of sound event detection. Attached Figure Description

[0065] Figure 1 This is a flowchart illustrating an embodiment of the sound event detection method based on audio cue learning provided by the present invention.

[0066] Figure 2 A flowchart illustrating an embodiment of the Mel spectrogram encoding method provided by the present invention;

[0067] Figure 3A flowchart illustrating an embodiment of the post-processing method provided by the present invention;

[0068] Figure 4 A flowchart illustrating an embodiment of the first sound event detection model training method provided by the present invention;

[0069] Figure 5 This is a flowchart illustrating another embodiment of the sound event detection method based on audio cue learning provided by the present invention.

[0070] Figure 6 This is a schematic diagram of one embodiment of the sound event detection device based on audio prompt learning provided by the present invention. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] Example 1

[0073] See Figure 1 This is a flowchart illustrating an embodiment of the sound event detection method based on audio cue learning provided by the present invention. The method includes steps 101 to 103, each of which is detailed below:

[0074] Step 101: Perform feature extraction on the sound to be detected to obtain the first Mel spectrogram corresponding to the sound to be detected.

[0075] Furthermore, in the first embodiment of the present invention, feature extraction is performed on the sound to be detected to obtain the first Mel spectrogram corresponding to the sound to be detected, specifically as follows:

[0076] Acquire the first time-domain signal of the sound to be detected;

[0077] Perform a short-time Fourier transform on the first time-domain signal to obtain the first short-time Fourier transform amplitude spectrum;

[0078] Using a preset Mel filter bank, the first short-time Fourier transform amplitude spectrum is converted into the first Mel short-time Fourier transform amplitude spectrum;

[0079] The first Mel short-time Fourier transform amplitude spectrum is obtained by taking the logarithm of the first Mel spectrum.

[0080] In the first embodiment of the present invention, in order to facilitate the use of the model for sound event detection of the sound to be detected, feature extraction of the sound to be detected is required first. First, the sound file to be detected is read to obtain the first time-domain signal of the sound to be detected; then, the input first time-domain signal is subjected to a short-time Fourier transform to obtain the first short-time Fourier transform amplitude spectrum; the first short-time Fourier transform amplitude spectrum is passed through a preset Mel filter bank to obtain the first Mel short-time Fourier transform amplitude spectrum; finally, the first logarithmic Mel short-time Fourier transform amplitude spectrum, i.e., the first Mel spectrum, is obtained by taking the logarithm; the above process can be expressed by the following formula:

[0081]

[0082] Where M(f,t) is the value of the Mel spectrum at frequency f and time t; |H i (f)| is the response value of the i-th Mel filter at frequency f; |X i (t)| is the amplitude of the frequency domain signal corresponding to the i-th filter at time t.

[0083] Step 102: Input the first Mel spectrogram into the preset first sound event detection model, encode the first Mel spectrogram to form a first sequence, and convert the first sequence into a second sequence through a preset number of Swing Transformer Blocks. Then, use Token-Semantic CNN to modify the structure of the second sequence and output the event frame-level distribution of the sound to be detected.

[0084] Furthermore, in the first embodiment of the present invention, the first Mel spectrum is encoded to form a first sequence, specifically as follows:

[0085] The first Mel spectrum is split into several patches;

[0086] The patches are concatenated into a first matrix so that the size of the first matrix is ​​the same as the input size of the first sound event detection model.

[0087] The first matrix is ​​divided into several patch tokens using the patch-embed CNN in the first sound event detection model.

[0088] Several patch tokens are arranged according to a preset order to form the first sequence.

[0089] In the first embodiment of the present invention, see Figure 2This is a flowchart illustrating an embodiment of the Mel spectrogram encoding method provided by the present invention. Typically, the time dimension is longer than the frequency dimension; therefore, to better capture the relationship between frequency bins within the same time frame, the Mel spectrogram size of the audio needs to be adjusted. For example... Figure 2 As shown, by splitting the data into N patches and then concatenating them into a new matrix, the size of the Mel spectrogram can be made consistent with the input size of the first sound event detection model. This spectrogram is then segmented into different patch tokens using a patch-embed CNN with a kernel size of (p×p). Finally, following the order of time → frequency → window, patches of different frequency bins within the same time frame are sequentially arranged in the input sequence to obtain the first sequence.

[0090] Furthermore, in the first embodiment of the present invention, the second sequence is structurally modified using Token-Semantic CNN to obtain the event frame-level distribution of the sound to be detected, specifically as follows:

[0091] Before modifying the structure of the second sequence, the second sequence is subjected to dimensionality increase processing to form a third sequence, which includes frequency information, time information and category information.

[0092] Using Token-Semantic CNN, the frequency information of the third sequence is aggregated, and the temporal information of the third sequence is interpolated to obtain the event frame-level distribution of the sound to be detected.

[0093] As an example of the first embodiment of the present invention, assuming that the first sound event detection model has 4 layers of Swing Transformer Blocks, the first sequence E0 will form a second sequence after passing through the 4 layers of Swing Transformer Blocks. Because the input and output of a CNN network are three-dimensional vectors, the second sequence needs to be processed before passing through the Token-Semantic CNN. Restore to the third sequence The second sequence E4 is transformed from a two-dimensional vector into a three-dimensional vector, and then the third sequence is input into the Token-Semantic CNN to modify the output structure. The Token-Semantic CNN can be configured with different kernel sizes. A CNN network with padding size (1,0) is used to compress the frequency dimension of the third sequence to 1, aggregate the frequency information, that is, eliminate the frequency dimension, leaving only the time and category dimensions; after interpolation, the event frame-level distribution of the sound to be detected is obtained.

[0094] Furthermore, in the first embodiment of the present invention, before outputting the event frame-level distribution of the sound to be detected, the method further includes:

[0095] Determining whether the event frame-level distribution of the sound to be detected is valid involves the following steps:

[0096] Determine whether the prediction of the first event is greater than a preset first threshold; wherein, the prediction of the first event is output by the first sound event detection model after processing the sound to be detected;

[0097] When the first event prediction is greater than the first threshold, the event frame-level distribution of the sound to be detected is determined to be valid, and the event frame-level distribution of the sound to be detected is output from the first sound event detection model.

[0098] In the first embodiment of the present invention, before the model outputs the event frame-level distribution, the validity of the event frame-level distribution is first judged based on the event prediction corresponding to the sound to be detected. That is, the threshold method is used to detect whether the event prediction is greater than a preset first threshold. If it is greater, the event frame-level release can be judged to be valid. After the conclusion that the event frame-level release is valid is reached, the event frame-level release is output from the model to improve the accuracy of sound event detection.

[0099] Step 103: Determine the sound event category and sound event activity time of the sound to be detected based on the event frame-level distribution of the sound to be detected.

[0100] In the first embodiment of this invention, after the model outputs a valid event frame-level distribution, the sound event category of the sound to be detected can be obtained through mean pooling, and the activity time of the sound event can be obtained through post-processing. See also Figure 3 This is a flowchart illustrating an embodiment of the post-processing method provided by the present invention; wherein, y clip It is event prediction, y frame This involves frame-level prediction, where the event frame distribution of the sound to be detected includes frame-level predictions. A threshold method can be used to determine the category of sound events, and event predictions can be used to infer whether the frame-level predictions are valid or invalid. Then, the event activity time is determined based on the frame-level predictions that are determined to be valid. For example, a frame-level threshold is set, and the frame-level prediction for each frame in the sound to be detected is compared with the frame-level threshold. If the frame-level prediction is greater than the frame-level threshold, then an event has occurred in that frame. The frames with events are counted, thus determining the sound event activity time.

[0101] Furthermore, in the first embodiment of the present invention, the first sound event detection model is specifically:

[0102] Select the Swing Transformer model, and introduce an audio cue vector into the input of each Swing Transformer Block in the Swing Transformer model;

[0103] Lock the backbone network parameters of the Swin Transformer model and determine that the objects of model training are the audio cue vector and the Token-Semantic CNN in the Swin Transformer model;

[0104] The Swin Transformer model is trained and fine-tuned using both strongly labeled and weakly labeled datasets with strong and weak supervision.

[0105] The trained Swing Transformer model was selected as the first sound event detection model.

[0106] In the first embodiment of this invention, by introducing audio cue vectors into the Swing Transformer model, the backbone network parameters in the model are maintained during training, and only the audio cue vectors and the network structure related to downstream tasks are fine-tuned. Specifically, in the pre-trained Swing Transformer model, a set of learnable audio cue vectors of length n is introduced into the input of each Swing Transformer Block after Patch-Embed; this method is called deep cueing. The formula for the deep cueing Swing Transformer can be expressed as:

[0107] [_,E i ] = L i ([P i-1 E i-1 ]), i = 1, 2, ..., N

[0108] Among them, E i P represents the feature vector of the i-th layer. i-1 L represents the audio cue vector of the i-th layer, and L represents the Swing Transformer Block of the i-th layer.

[0109] As an example of the first embodiment of the present invention, see Figure 4This is a flowchart illustrating an embodiment of the training method for the first sound event detection model provided by the present invention. E0 is the feature vector of the original input, P0 is the audio cue vector of the original input, and the 4-layer Swin Transformer Block is the backbone network in the first sound event detection model. Token-Semantic CNN is the network structure related to downstream tasks. Therefore, when training the first sound event detection model, the backbone network is frozen, and only the audio cue vector and the network structure related to downstream tasks are fine-tuned; that is, the backbone network is not changed. Figure 4 The parameters of the four Swing Transformer Blocks and the feature vectors of each Swing Transformer Block are fine-tuned only for the Token-Semantic CNN and the audio cue vectors of each Swing Transformer Block. Compared to full parameter fine-tuning, the model fine-tuning method of this invention can reduce the number of training parameters, shorten the training time, and better maintain the strong representational ability of the original model during training.

[0110] Furthermore, in the first embodiment of the present invention, the training process of the first sound event detection model is specifically as follows:

[0111] Obtain the source sound; all sound events of the source sound are of the same category;

[0112] Create several background sounds based on the source sound;

[0113] Data augmentation is performed on the aforementioned background sounds to form a training dataset;

[0114] Based on the training dataset, construct a strongly labeled dataset and a weakly labeled dataset respectively;

[0115] The training dataset is used to extract features and convert them into training Mel spectrograms.

[0116] The trained Mel spectrogram is input into the Swin Transformer model so that the Swin Transformer model outputs training event predictions and training frame-level predictions.

[0117] The Sigmoid function is used to activate the prediction of the training events and the prediction of the training frames.

[0118] A loss function is formed by combining binary cross-entropy and smoothed absolute error; wherein the binary cross-entropy is used to calculate the loss for training event prediction, and the smoothed absolute error is used to calculate the loss for training frame-level prediction.

[0119] The Swing Transformer model is updated based on the loss function.

[0120] Furthermore, in the first embodiment of the present invention, data augmentation is performed on the plurality of background sounds to form a training dataset, specifically as follows:

[0121] The aforementioned sound backgrounds are time-shifted, and the time-shifted sound backgrounds are used to form a training dataset;

[0122] The specific formula for time-shifting the plurality of background sounds is as follows:

[0123] x sift (n) = x(n+l)

[0124] Wherein, the x sift The background sound is time-shifted, and x is the normalized audio signal. sift (n) represents the nth point of the background sound after time shift, and l represents the number of points shifted.

[0125] Furthermore, in the first embodiment of the present invention, a strongly labeled dataset and a weakly labeled dataset are constructed based on the training dataset, specifically as follows:

[0126] The specific formula for constructing a weakly labeled dataset is as follows:

[0127] y weak (i) = [0,0,…,1,…,0] T i∈[1,class_num]

[0128] Among them, y weak (i) is the weak label of the i-th labeled sound event in the training dataset; class_num is the number of sound event categories; i is the label index;

[0129] Wherein, the y weak For a one-hot vector, when y weak When (i) = 1, it indicates that the i-th labeled sound event exists in the training dataset. weak When (i) = 0, it means that there is no i-th label sound event in the training dataset.

[0130] The specific formula for constructing a strongly labeled dataset is as follows:

[0131]

[0132] i is the tag index; j is the frame index; class_num is the number of sound event categories; frame_size is the number of frames;

[0133] Among them, y strong For a one-hot vector, when y strongWhen (j,i)=1, it means that the j-th frame in the training dataset contains the i-th labeled sound event; strong When (j,i)=0, it means that there is no i-th label sound event in the j-th frame of the training dataset.

[0134] In the first embodiment of this invention, a strong and weak label dataset can be constructed using mixing and data augmentation methods. Specifically, mixing involves acquiring source audio of a single sound event category and combining or transforming it in different ways, such as using the Scaper open-source library, to create an infinite number of different soundscapes from the same source audio. To improve the robustness of the training model, time-shifting can be used for data augmentation, taking into account the actual soundscape conditions. The augmented sound background is then used as the training dataset, and strong-label and weak-label datasets are constructed separately. The model is then trained to output training event predictions and training frame-level predictions. After the model outputs training event predictions and training frame-level predictions, a sigmoid activation function is used to activate both predictions. Then, binary cross-entropy is used to calculate the loss for training event predictions, and smoothed absolute error is used to calculate the loss for training frame-level predictions, thereby updating the model. The global loss function is shown below:

[0135] L(X) = BCE(y) clip ,z clip )+α*SmoothL1loss(y frame ,z frame )

[0136] Where L(X) is the global loss function, BCE is the binary cross-entropy loss, SmoothL1 loss is the smoothing absolute error loss, and y clip To train event prediction, z clip To train event labels, y frame To train frame-level predictions, z frame α is a frame-level label and α is a hyperparameter.

[0137] As an example of the first embodiment of the present invention, the present invention trains the first sound event detection model by using the Adam optimizer, setting 100 epochs (loops), batch_size (number of training samples per iteration) to 64, and learning rate to 0.001. The first 5 epochs adopt a warm-up training strategy, and the learning rate is decayed every 10 epochs thereafter. When the learning rate decays to less than 0.0001, no further decay is performed.

[0138] As an example of the first embodiment of the present invention, see Figure 5This is a flowchart illustrating another embodiment of the audio cue learning-based sound event detection method provided by the present invention. A trained first sound event detection model is used to detect sound events. In this model, a set of learnable audio cue vectors is introduced into the input of each Swing Transformer Block after Patch-Embed. Features of the sound to be detected are extracted and converted into a first Mel spectrogram. This first Mel spectrogram is input into the first sound event detection model, encoded, and passed through a Patch-Embed CNN network to form a first sequence E0. This sequence then passes through four Swing Transformer Blocks to form a second sequence E4. The second sequence is then upscaled to form a third sequence. A Token-Semantic CNN is used to modify the output structure of the third sequence. Finally, the model outputs an event frame-level distribution. Mean pooling and post-processing are performed based on the event frame-level distribution output by the model to determine the sound event category and its activity time in the sound to be detected, thus improving the accuracy and efficiency of sound event detection.

[0139] In summary, the first embodiment of this invention provides a sound event detection method based on audio cue learning. A learnable audio cue vector is introduced as input to each layer of the Swin Transformer Block in the Swin Transformer model. During model training, the backbone network parameters are maintained, and only the audio cue vector and the network structure related to downstream tasks are fine-tuned. Strong and weak labels are added to jointly optimize model training during the fine-tuning process, which reduces the number of training parameters, making the model more accurate in frame-level localization and obtaining a first sound event detection model with better detection performance. The sound to be detected is converted into a first Mel spectrogram, which is then input into the first sound event detection model to output the event frame-level distribution, thereby obtaining the sound event category and its activity time in the sound to be detected, which can improve the accuracy and efficiency of sound event detection.

[0140] Example 2

[0141] See Figure 6 This is a schematic diagram of an embodiment of the sound event detection device based on audio prompt learning provided by the present invention. The device includes an acquisition module 201, a detection module 202 and a determination module 203.

[0142] The acquisition module 201 is used to extract features from the sound to be detected and acquire the first Mel spectrogram corresponding to the sound to be detected.

[0143] The detection module 202 is used to input the first Mel spectrogram into a preset first sound event detection model, encode the first Mel spectrogram to form a first sequence, and convert the first sequence into a second sequence through a preset number of Swing Transformer Blocks. Then, the second sequence is structurally modified using Token-Semantic CNN to output the event frame-level distribution of the sound to be detected.

[0144] The determining module 203 is used to determine the sound event category and sound event activity time of the sound to be detected based on the event frame-level distribution of the sound to be detected.

[0145] Furthermore, in the second embodiment of the present invention, the acquisition module 201 includes an acquisition unit, a first conversion unit, a second conversion unit, and a third conversion unit;

[0146] The acquisition unit is used to acquire the first time-domain signal of the sound to be detected;

[0147] The first conversion unit is used to perform a short-time Fourier transform on the first time-domain signal to obtain the first short-time Fourier transform amplitude spectrum;

[0148] The second conversion unit is used to convert the first short-time Fourier transform amplitude spectrum into the first Mel short-time Fourier transform amplitude spectrum using a preset Mel filter bank.

[0149] The third conversion unit is used to take the logarithm of the amplitude spectrum of the first Mel short-time Fourier transform to obtain the first Mel spectrum.

[0150] Furthermore, in the second embodiment of the present invention, the detection module 202 includes a splitting unit, a splicing unit, a cutting unit, and an arranging unit;

[0151] The splitting unit is used to split the first Mel spectrum into several patches;

[0152] The patches are concatenated into a first matrix so that the size of the first matrix is ​​the same as the input size of the first sound event detection model.

[0153] The first matrix is ​​divided into several patch tokens using the patch-embed CNN in the first sound event detection model.

[0154] Several patch tokens are arranged according to a preset order to form the first sequence.

[0155] Furthermore, in the second embodiment of the present invention, the detection module 202 further includes a dimension-upgrading unit and a structure modification unit;

[0156] The dimension-upgrading unit is used to perform dimension-upgrading processing on the second sequence before structural modification; to form a third sequence, which includes frequency information, time information and category information;

[0157] The structural modification unit is used to aggregate the frequency information of the third sequence and interpolate the temporal information of the third sequence using Token-Semantic CNN to obtain the event frame-level distribution of the sound to be detected.

[0158] Furthermore, in the second embodiment of the present invention, the detection module 202 further includes a judgment unit;

[0159] The judgment unit is used to determine whether the event frame-level distribution of the sound to be detected is valid, specifically:

[0160] Determine whether the prediction of the first event is greater than a preset first threshold; wherein, the prediction of the first event is output by the first sound event detection model after processing the sound to be detected;

[0161] When the first event prediction is greater than the first threshold, the event frame-level distribution of the sound to be detected is determined to be valid, and the event frame-level distribution of the sound to be detected is output from the first sound event detection model.

[0162] Furthermore, in the second embodiment of the present invention, the first sound event detection model is specifically as follows:

[0163] Select the Swing Transformer model, and introduce an audio cue vector into the input of each Swing Transformer Block in the Swing Transformer model;

[0164] Lock the backbone network parameters of the Swin Transformer model and determine that the objects of model training are the audio cue vector and the Token-Semantic CNN in the Swin Transformer model;

[0165] The Swin Transformer model is trained and fine-tuned using both strongly labeled and weakly labeled datasets with strong and weak supervision.

[0166] The trained Swing Transformer model was selected as the first sound event detection model.

[0167] Furthermore, in the second embodiment of the present invention, the training process of the first sound event detection model is specifically as follows:

[0168] Obtain the source sound; all sound events of the source sound are of the same category;

[0169] Create several background sounds based on the source sound;

[0170] Data augmentation is performed on the aforementioned background sounds to form a training dataset;

[0171] Based on the training dataset, construct a strongly labeled dataset and a weakly labeled dataset respectively;

[0172] The training dataset is used to extract features and convert them into training Mel spectrograms.

[0173] The trained Mel spectrogram is input into the Swin Transformer model so that the Swin Transformer model outputs training event predictions and training frame-level predictions.

[0174] The Sigmoid function is used to activate the prediction of the training events and the prediction of the training frames.

[0175] A loss function is formed by combining binary cross-entropy and smoothed absolute error; wherein the binary cross-entropy is used to calculate the loss for training event prediction, and the smoothed absolute error is used to calculate the loss for training frame-level prediction.

[0176] The Swing Transformer model is updated based on the loss function.

[0177] Furthermore, in the second embodiment of the present invention, data augmentation is performed on the plurality of background sounds to form a training dataset, specifically as follows:

[0178] The aforementioned sound backgrounds are time-shifted, and the time-shifted sound backgrounds are used to form a training dataset;

[0179] The specific formula for time-shifting the plurality of background sounds is as follows:

[0180] x sift (n) = x(n+l)

[0181] Wherein, the x sift The background sound is time-shifted, and x is the normalized audio signal. sift (n) represents the nth point of the background sound after time shift, and l represents the number of points shifted.

[0182] Furthermore, in the second embodiment of the present invention, a strongly labeled dataset and a weakly labeled dataset are constructed based on the training dataset, specifically as follows:

[0183] The specific formula for constructing a weakly labeled dataset is as follows:

[0184] y weak (i) = [0,0,…,1,…,0]T i∈[1,class_num]

[0185] Among them, y weak (i) is the weak label of the i-th labeled sound event in the training dataset; class_num is the number of sound event categories; i is the label index;

[0186] The specific formula for constructing a strongly labeled dataset is as follows:

[0187]

[0188] i is the tag index; j is the frame index; class_num is the number of sound event categories; frame_size is the number of frames.

[0189] In summary, the second embodiment of this invention provides a sound event detection device based on audio cue learning. Based on the organic integration of modules, a learnable audio cue vector is introduced into the input of each layer of the Swin Transformer Block in the Swin Transformer model. During model training, the backbone network parameters are maintained, and only the audio cue vector and the network structure related to downstream tasks are fine-tuned. Strong and weak labels are added to jointly optimize model training during the fine-tuning process, which reduces the number of training parameters, making the model more accurate in frame-level localization and obtaining a first sound event detection model with better detection performance. The sound to be detected is converted into a first Mel spectrogram, which is input into the first sound event detection model, outputting the event frame-level distribution, thereby obtaining the sound event category and its activity time in the sound to be detected, which can improve the accuracy and efficiency of sound event detection.

[0190] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A sound event detection method based on audio cue learning, characterized in that, include: Feature extraction is performed on the sound to be detected to obtain the first Mel spectrogram corresponding to the sound to be detected; The first Mel spectrogram is input into a preset first sound event detection model, the first Mel spectrogram is encoded to form a first sequence, and the first sequence is transformed into a second sequence through a preset number of Swing Transformer Blocks. Then, the second sequence is structurally modified using Token-Semantic CNN to output the event frame-level distribution of the sound to be detected. Based on the event frame-level distribution of the sound to be detected, determine the sound event category and sound event activity time of the sound to be detected; Specifically, the first sound event detection model is as follows: Select the Swing Transformer model, and introduce an audio cue vector into the input of each Swing TransformerBlock layer in the Swing Transformer model; Lock the backbone network parameters of the Swin Transformer model and determine that the objects of model training are the audio cue vector and the Token-Semantic CNN in the Swin Transformer model; The Swin Transformer model is trained and fine-tuned using both strongly labeled and weakly labeled datasets with strong and weak supervision. The trained Swing Transformer model was selected as the first sound event detection model.

2. The sound event detection method based on audio cue learning according to claim 1, characterized in that, The step of extracting features from the sound to be detected to obtain the first Mel-spectrum corresponding to the sound to be detected is specifically as follows: Obtain the first time-domain signal of the sound to be detected; Perform a short-time Fourier transform on the first time-domain signal to obtain the first short-time Fourier transform amplitude spectrum; Using a preset Mel filter bank, the first short-time Fourier transform amplitude spectrum is converted into the first Mel short-time Fourier transform amplitude spectrum; The first Mel short-time Fourier transform amplitude spectrum is obtained by taking the logarithm of the first Mel spectrum.

3. The sound event detection method based on audio cue learning according to claim 1, characterized in that, The encoding of the first Mel spectrum to form the first sequence specifically involves: The first Mel spectrum is split into several patches; The patches are concatenated into a first matrix so that the size of the first matrix is ​​the same as the input size of the first sound event detection model. The first matrix is ​​divided into several patch tokens using the patch-embed CNN in the first sound event detection model. Several patch tokens are arranged according to a preset order to form the first sequence.

4. The sound event detection method based on audio cue learning according to claim 1, characterized in that, The step of using Token-Semantic CNN to modify the structure of the second sequence and outputting the event frame-level distribution of the sound to be detected is specifically as follows: Before modifying the structure of the second sequence, the second sequence is subjected to dimensionality increase processing to form a third sequence, which includes frequency information, time information and category information. Using Token-Semantic CNN, the frequency information of the third sequence is aggregated, and the temporal information of the third sequence is interpolated to obtain the event frame-level distribution of the sound to be detected.

5. The sound event detection method based on audio cue learning according to claim 1, characterized in that, Before outputting the event frame-level distribution of the sound to be detected, the method further includes: Determining whether the event frame-level distribution of the sound to be detected is valid involves the following steps: Determine whether the prediction of the first event is greater than a preset first threshold; wherein, the prediction of the first event is output by the first sound event detection model after processing the sound to be detected; When the first event prediction is greater than the first threshold, the event frame-level distribution of the sound to be detected is determined to be valid, and the event frame-level distribution of the sound to be detected is output from the first sound event detection model.

6. The sound event detection method based on audio cue learning according to claim 1, characterized in that, The training process for the first sound event detection model is as follows: Obtain the source sound; all sound events of the source sound are of the same category; Create several background sounds based on the source sound; Data augmentation is performed on the aforementioned background sounds to form a training dataset; Based on the training dataset, construct a strongly labeled dataset and a weakly labeled dataset respectively; The training dataset is used to extract features and convert them into training Mel spectrograms. The trained Mel spectrogram is input into the Swin Transformer model so that the Swin Transformer model outputs training event predictions and training frame-level predictions. The Sigmoid function is used to activate the prediction of the training events and the prediction of the training frames. A loss function is formed by combining binary cross-entropy and smoothed absolute error; wherein the binary cross-entropy is used to calculate the loss for training event prediction, and the smoothed absolute error is used to calculate the loss for training frame-level prediction; The Swing Transformer model is updated based on the loss function.

7. The sound event detection method based on audio cue learning according to claim 6, characterized in that, The step of performing data augmentation on the plurality of background sounds to form a training dataset specifically involves: The aforementioned sound backgrounds are time-shifted, and the time-shifted sound backgrounds are used to form a training dataset; The specific formula for time-shifting the plurality of background sounds is as follows: Among them, the The background sound after time shifting. For the standardized audio signal, For the nth point of the background sound after time shift, The number of points in time shift.

8. The sound event detection method based on audio cue learning according to claim 6, characterized in that, The construction of a strong-label dataset and a weak-label dataset based on the training dataset specifically involves: The specific formula for constructing a weakly labeled dataset is as follows: in, It is the weak label of the i-th labeled sound event in the training dataset; Number of sound event categories; For tag indexing; The specific formula for constructing a strongly labeled dataset is as follows: in, To obtain strong labels for the i-th labeled sound event in the j-th frame of the training dataset; For tag indexing; For frame number index, Number of sound event categories; The number of frames.

9. A sound event detection device based on audio prompt learning, characterized in that, include: Acquisition module, detection module, and determination module; The acquisition module is used to extract features from the sound to be detected and obtain the first Mel spectrogram corresponding to the sound to be detected. The detection module is used to input the first Mel spectrogram into a preset first sound event detection model, encode the first Mel spectrogram to form a first sequence, and convert the first sequence into a second sequence through a preset number of Swing Transformer Blocks. Then, the second sequence is structurally modified using Token-Semantic CNN to output the event frame-level distribution of the sound to be detected. The determining module is used to determine the sound event category and sound event activity time of the sound to be detected based on the event frame-level distribution of the sound to be detected; Specifically, the first sound event detection model is as follows: Select the Swing Transformer model, and introduce an audio cue vector into the input of each Swing TransformerBlock layer in the Swing Transformer model; Lock the backbone network parameters of the Swin Transformer model and determine that the objects of model training are the audio cue vector and the Token-Semantic CNN in the Swin Transformer model; The Swin Transformer model is trained and fine-tuned using both strongly labeled and weakly labeled datasets with strong and weak supervision. The trained Swing Transformer model was selected as the first sound event detection model.

Images