Sound event detection method and storage medium

By introducing MSSC residual convolution blocks with multi-scale banding convolution and residual structures into the sound event detection model, combining two-dimensional convolution and bidirectional gating loop units, the problem of short-term sound event feature capture accuracy is solved. , enhances detection capabilities and reduces computational complexity, and realizes efficient detection of short-term sound events.

CN119068911APending Publication Date: 2024-12-03JIANGSU ZHIHENG INFORMATION TECH SERVICES CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411148910.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art lacks accuracy when capturing the characteristics of short-term sound events, especially when background sounds occupy most areas of the spectrum map, traditional network convolution methods will introduce too much background sound information, affecting the accurate capture of features.

Method used

MSSC residual convolution blocks using multi-scale strip convolution and residual structures, match sound events of different scales through multiple depth strip convolution channels, and combine two-dimensional convolution blocks, global average pooling layer and bidirectional The gated loop unit extracts multi-scale linear features and information features in the time dimension to enhance the detection ability of short-term sound events.

Benefits of technology

Effectively capture different context information of short-term sound events, enhance the detection ability of short-term sound events, reduce the parameter quantity and calculation complexity of the model, reduce the power consumption of IoT devices, and realize energy saving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068911A_ABST
    Figure CN119068911A_ABST
Patent Text Reader

Abstract

The invention provides a sound event detection method. The sound event detection method comprises the following steps: acquiring logarithmic Mel spectrum features of an audio to be detected; inputting the logarithmic Mel spectrum features into a trained sound event detection model, and performing multi-scale feature extraction on the logarithmic Mel spectrum features through a feature extraction module in the model to obtain multi-scale linear features; extracting information features of the multi-scale linear features in the time dimension through a time positioning module in the model; and classifying the information features of the multi-scale linear features in the time dimension through an event decision module in the model to obtain a detection tag corresponding to the audio to be detected. Wherein an MSSC residual error convolution block in the feature extraction module uses a plurality of channel structures which respectively comprise two depth stripe convolutions to identify information of different scales of the audio data to be detected, so that the method has remarkable advantages in the aspect of short-time sound event detection, and the overall performance is effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a sound event detection method, a device and a storage medium, and belongs to the field of sound event detection. Background Art

[0002] Sound Event Detection (SED) is a key audio processing technology that aims to identify and analyze specific sound events in audio clips, such as alarms, animal calls, or machine operation sounds, and determine the start and end time of these sound events. This technology has broad application prospects in many fields such as surveillance, smart home, and human-computer interaction.

[0003] In recent years, with the rapid development of deep learning technology, the field of sound event detection has made significant progress. The general process is to first extract acoustic features from audio, then input these features into the neural network to learn higher-level abstract features, and finally use these abstract features to classify and detect sound events. The application of neural networks in sound event detection began with deep neural networks (DNN). The introduction of deep neural networks has greatly improved the detection ability of sound events. In order to more accurately capture local features in audio, researchers began to try to apply convolutional neural networks (CNN). CNN can use limited audio context information to make decisions for each frame of audio clips, but due to the fixed size of its convolution kernel, it can only process context information of a fixed time period, which limits its ability to detect complex sound events. In order to overcome this limitation, recurrent neural networks (RNN) were introduced into sound event detection. RNN can effectively use the context information of audio to make frame-by-frame decisions through its internal loop structure. In order to combine the advantages of CNN and RNN, researchers proposed Convolution Recurrent Neural Networks (CRNN). CRNN combines the excellent feature extraction ability of CNN and the powerful temporal modeling ability of RNN, thus achieving significant performance improvement in sound event detection. However, different types of sound events have different characteristics, including duration, frequency scale, etc., and it is still difficult to effectively capture different types of sound event characteristics using fixed-scale convolution kernels.

[0004] To solve this problem, researchers have proposed a variety of multi-scale feature extraction methods. For example, by integrating low-level and high-level features of CNN through up- and down-sampling and residual skip connections; applying the concept of feature pyramid to CRNN, and realizing multi-scale feature extraction through the fusion of high-level and low-level features; and replacing RNN with Transformer encoder to better capture the temporal dependencies of audio signals. Although these methods have enhanced the overall performance of the model to a certain extent, the accurate extraction of features of short-term sound events is still a challenge due to their short duration and possible sparse distribution on the spectrogram. In particular, when most of the area of ​​the spectrogram is occupied by background sound, traditional network convolution methods seem to be unable to cope with this situation, because they tend to introduce too much background sound information, thus affecting the accurate capture of the features of short-term sound events. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a sound event detection method to accurately capture the characteristics of short-term sound events and enhance the detection capability of short-term sound events.

[0006] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0007] In a first aspect, the present invention provides a method for detecting a sound event, comprising:

[0008] Get the logarithmic Mel spectrum features of the audio to be detected;

[0009] Inputting the logarithmic Mel spectrum feature into a trained sound event detection model, wherein the sound event detection model comprises: a feature extraction module, a time positioning module and an event decision module;

[0010] Performing multi-scale feature extraction on the several-mel frequency spectrum features by the feature extraction module to obtain multi-scale linear features;

[0011] Extracting information features of the multi-scale linear features in the time dimension through the time positioning module;

[0012] The event decision module is used to classify the information features of the multi-scale linear features in the time dimension to obtain a detection label corresponding to the audio to be detected.

[0013] Furthermore, the feature extraction module includes a two-dimensional convolution block, a global average pooling layer and a plurality of MSSC residual convolution blocks; the MSSC residual convolution block includes: a plurality of depth strip convolution channels;

[0014] The step of performing multi-scale feature extraction on the several-mel frequency spectrum features by the feature extraction module to obtain multi-scale linear features includes:

[0015] Aggregating local information of the logarithmic Mel spectrum feature through a two-dimensional convolution block to obtain a first feature;

[0016] Extracting the first features through each MSSC residual convolution block to obtain features extracted by each MSSC residual convolution block;

[0017] The features extracted by each MSSC residual convolution block are input into the global average pooling layer for pooling calculation to obtain multi-scale linear features.

[0018] Further, the extracting the first features through each MSSC residual convolution block respectively to obtain the features extracted by each MSSC residual convolution block includes:

[0019] Enter the first feature into the preset In the depth strip convolution channel, the The depth strip convolution channel uses the convolution kernel and The first feature is extracted by two deep strip convolutions; the The output features of the depth strip convolution channels, where ;

[0020] The output features of all depth strip convolution channels are fused with the first feature, and then used Convolution models the relationship between the deep strip convolution channels; and re-weights the first feature using the relationship between the deep strip convolution channels as an attention weight;

[0021] use The convolution performs channel dimension upgrading on the re-weighted first feature and the logarithmic Mel spectrum feature to obtain the output feature of the MSSC residual convolution block.

[0022] Furthermore, the expression of the feature extraction module is:

[0023]

[0024] In the formula, represents the input features of the feature extraction module, Represents the first feature, is the output feature of the feature extraction module; Indicates The median scale of the depth strip convolution channel is The depth strip convolution pair of ; express convolution; Used for channel dimension upgrading; represents average pooling.

[0025] Furthermore, the time positioning module includes: 2 layers of bidirectional gated recurrent units.

[0026] Furthermore, the event decision module includes: a first fully connected layer, a second fully connected layer and a sigmoid activation function;

[0027] The step of classifying the information features of the multi-scale linear features in the time dimension by the event decision module to obtain a detection label corresponding to the audio to be detected includes:

[0028] The information features of the multi-scale linear features in the time dimension are transformed by the sigmoid activation function after passing through the first fully connected layer and the second fully connected layer in sequence to obtain the frame-level detection label of the sound event type, and then the frame-level detection label is converted into a fragment-level detection label through the global average pooling layer.

[0029] Furthermore, the training process of the sound event detection model includes:

[0030] Collect historical audio samples to build a training set;

[0031] Extracting the logarithmic Mel-spectrogram features of the audio samples in the training set, and encoding the labels on the time axis of the audio samples to convert them into a real label matrix;

[0032] Inputting the extracted logarithmic Mel-spectrogram features into the sound event detection model to obtain a detection label corresponding to the audio sample;

[0033] Encode the detection labels corresponding to the audio samples and transform them into a detection label matrix;

[0034] Calculating the loss function of the sound event detection model based on the true label matrix and the detection label matrix of the audio sample;

[0035] Determine whether the loss function has reached the convergence condition. If so, the training of the sound event detection model is completed. Otherwise, adjust the parameters of the feature extraction module, the time positioning module, and the event decision module, and re-execute the training steps until the maximum number of iterations is reached or the convergence condition is reached.

[0036] Among them, the horizontal axis of the real label matrix and the detection label matrix represents the frame number, the vertical axis represents the preset sound event type, and the element value in the matrix represents whether a sound event of the corresponding type occurs in the corresponding frame.

[0037] Furthermore, the loss function of the sound event detection model is a binary cross entropy loss, which is expressed as:

[0038]

[0039] in, represents the number of samples, Indicates The true label matrix of audio samples, Indicates The detection label matrix of audio samples.

[0040] Furthermore, the training set includes: unlabeled audio data, weakly labeled audio data that only marks the sound event type, and strongly labeled audio data that marks the sound event type and the specific time point of occurrence.

[0041] In a second aspect, the present invention provides a computer-readable storage medium having program instructions stored thereon, wherein the program instructions, when executed by a processor, execute the steps of the sound event detection method described in the first aspect.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] (1) The sound event detection model provided by the present invention is provided with an MSSC residual convolution block including a multi-scale strip convolution and a residual structure. The MSSC residual convolution block matches sound events of different scales through multiple deep strip convolution channels and captures different contextual information of short-time sound events. In each deep strip convolution channel, two deep strip convolutions are used instead of traditional two-dimensional convolutions, so that the sound event detection model of the present invention can better match the linear features presented by short-time sound events after multiple pooling, thereby enhancing the detection capability of short-time sound events;

[0044] (2) The sound event detection model provided by the present invention significantly reduces the number of parameters and computational complexity, thereby reducing the power consumption of the IoT devices involved in the sound detection process, thereby achieving energy saving effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of the feature extraction module in Example 1 of the present invention. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0047] Example 1

[0048] This embodiment provides a sound event detection method, including:

[0049] S1: Obtain the logarithmic Mel spectrum features of the audio to be detected, including:

[0050] S11: Pre-emphasis: read audio samples and use digital filters for pre-emphasis;

[0051] S12: Framing and windowing: The input audio samples are framed, with a frame length of 0.02s and a frame shift of 0.01s to obtain each frame signal. The window function is a Hamming window. Each frame signal is multiplied by the Hamming window to obtain a windowed frame signal.

[0052] S13: Extracting logarithmic spectrum features: performing discrete Fourier transform on the windowed frame signal to obtain a linear spectrum, then passing the linear spectrum through a Mel frequency filter bank to obtain a Mel spectrum, and finally performing a logarithmic operation to obtain a logarithmic spectrum.

[0053] S14: Finally, the logarithmic spectrum of each frame is arranged into a feature matrix according to the order of the frames, that is, the logarithmic Mel spectrum feature; wherein the number of rows of the feature matrix is ​​the order of the frames, and the number of columns of the feature matrix is ​​the feature dimension.

[0054] S2: Input the logarithmic Mel-spectrogram feature into the trained sound event detection model.

[0055] Specifically, the sound event detection model includes: a feature extraction module, a time positioning module and an event decision module.

[0056] S3: Perform multi-scale feature extraction on the logarithmic Mel spectrum feature through the feature extraction module to obtain multi-scale linear features.

[0057] like Figure 1 As shown, in this embodiment, the feature extraction module includes a two-dimensional convolution block, a global average pooling layer and an MSSC residual convolution block; the MSSC residual convolution block includes: multiple depth strip convolution channels.

[0058] Specifically, performing multi-scale feature extraction on the logarithmic Mel spectrum feature by the feature extraction module includes:

[0059] S31: Aggregate the local information of the logarithmic Mel spectrum feature through two two-dimensional convolution blocks to obtain a first feature;

[0060] S32: extracting the first feature through five MSSC residual convolution blocks respectively to obtain features extracted by each MSSC residual convolution block;

[0061] S33: Input the features extracted by the 5 MSSC residual convolution blocks into the global average pooling layer for pooling calculation to obtain multi-scale linear features.

[0062] Further, the first features are respectively extracted through 5 MSSC residual convolution blocks, including:

[0063] S321: Input the first features into the preset In the depth strip convolution channel, the The depth strip convolution channel uses the convolution kernel and The first feature is extracted by two deep strip convolutions; the The output features of the depth strip convolution channels, where ;

[0064] S322: Fuse the output features of all depth strip convolution channels with the first feature, and then use Convolution models the relationship between the deep strip convolution channels; and re-weights the first feature using the relationship between the deep strip convolution channels as an attention weight;

[0065] S323: Use The convolution performs channel dimension upgrading on the re-weighted first feature and the logarithmic Mel spectrum feature to obtain the output feature of the MSSC residual convolution block.

[0066] Furthermore, the expression of the feature extraction module is:

[0067]

[0068] In the formula, represents the input features of the feature extraction module, Represents the first feature, is the output feature of the feature extraction module; Indicates The median scale of the depth strip convolution channel is The depth strip convolution pair of ; express convolution; Used for channel dimension upgrading; represents average pooling.

[0069] S4: extracting information features of the multi-scale linear features in the time dimension through the time positioning module.

[0070] Specifically, the time positioning module includes: 2 layers of bidirectional gated recurrent units.

[0071] S5: Classifying the information features of the multi-scale linear features in the time dimension through the event decision module to obtain a detection label corresponding to the audio to be detected. The event decision module includes: a first fully connected layer, a second fully connected layer and a sigmoid activation function.

[0072] Specifically, the information features of the multi-scale linear features in the time dimension are transformed by the sigmoid activation function after passing through the first fully connected layer and the second fully connected layer in sequence to obtain the frame-level detection label of the sound event type, and then the frame-level detection label is converted into a fragment-level detection label through the global average pooling layer.

[0073] Example 2

[0074] This embodiment provides a training process of a sound event detection model based on Embodiment 1, including:

[0075] Collect historical audio samples to build a training set;

[0076] Specifically, the training set comes from the DESED dataset provided by Task 4 of the DCASE2022 Challenge. The training set includes: unlabeled audio data, weakly labeled audio data that only marks the sound event type, and strongly labeled audio data that marks the sound event type and the specific time of occurrence.

[0077] Extracting the logarithmic Mel-spectrogram features of the audio samples in the training set, and encoding the labels on the time axis of the audio samples to convert them into a real label matrix;

[0078] Inputting the extracted logarithmic Mel-spectrogram features into the sound event detection model to obtain a detection label corresponding to the audio sample;

[0079] Encode the detection labels corresponding to the audio samples and transform them into a detection label matrix;

[0080] Calculating the loss function of the sound event detection model based on the true label matrix and the detection label matrix of the audio sample;

[0081] Specifically, the loss function of the sound event detection model is the binary cross entropy loss:

[0082]

[0083] in, represents the number of samples, Indicates The true label matrix of audio samples, Indicates The detection label matrix of audio samples.

[0084] Determine whether the loss function has reached the convergence condition. If so, the training of the sound event detection model is completed. Otherwise, adjust the parameters of the feature extraction module, the time positioning module, and the event decision module, and re-execute the training steps until the maximum number of iterations is reached or the convergence condition is reached.

[0085] It should be noted that the horizontal axis of the real label matrix and the detection label matrix represents the frame number, the vertical axis represents the preset sound event type, and the element value in the matrix represents whether a sound event of the corresponding type occurs in the corresponding frame.

[0086] Example 3

[0087] According to the second aspect of the present invention, a computer-readable storage medium is provided in this embodiment, on which program instructions are stored, and when the program instructions are executed by the processor, the sound event detection method described in Example 1 or Example 2 is implemented. The storage medium may include, for example, a storage component of a tablet computer, a hard disk of a computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer storage medium may be any combination of one or more computer storage media.

[0088] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A sound event detection method, characterized in that: include: Get the logarithmic Mel spectrum features of the audio to be detected; Inputting the logarithmic Mel spectrum feature into a trained sound event detection model, wherein the sound event detection model comprises: a feature extraction module, a time positioning module and an event decision module; Performing multi-scale feature extraction on the several-mel frequency spectrum features by the feature extraction module to obtain multi-scale linear features; Extracting information features of the multi-scale linear features in the time dimension through the time positioning module; The event decision module is used to classify the information features of the multi-scale linear features in the time dimension to obtain a detection label corresponding to the audio to be detected.

2. The sound event detection method according to claim 1, characterized in that: The feature extraction module includes a two-dimensional convolution block, a global average pooling layer and a plurality of MSSC residual convolution blocks; the MSSC residual convolution block includes: a plurality of depth strip convolution channels; The step of performing multi-scale feature extraction on the several-mel frequency spectrum features by the feature extraction module to obtain multi-scale linear features includes: Aggregating local information of the logarithmic Mel spectrum feature through a two-dimensional convolution block to obtain a first feature; Extracting the first features through each MSSC residual convolution block to obtain features extracted by each MSSC residual convolution block; The features extracted by each MSSC residual convolution block are input into the global average pooling layer for pooling calculation to obtain multi-scale linear features.

3. The sound event detection method according to claim 2, characterized in that: The extracting the first features respectively through each MSSC residual convolution block to obtain the features extracted by each MSSC residual convolution block includes: Enter the first feature into the preset In the depth strip convolution channel, the The depth strip convolution channel uses the convolution kernel and The first feature is extracted by two deep strip convolutions; the The output features of the depth strip convolution channels, where ; The output features of all depth strip convolution channels are fused with the first feature, and then used Convolution models the relationship between the deep strip convolution channels; and reweights the first feature using the relationship between the deep strip convolution channels as an attention weight; use The convolution performs channel dimension upgrading on the re-weighted first feature and the logarithmic Mel spectrum feature to obtain the output feature of the MSSC residual convolution block.

4. The sound event detection method according to claim 3, characterized in that: The expression of the feature extraction module is: ; In the formula, represents the input features of the feature extraction module, Represents the first feature, is the output feature of the feature extraction module; Indicates The median scale of the depth strip convolution channel is The depth strip convolution pair of ; express convolution; Used for channel dimension upgrading; represents average pooling.

5. The sound event detection method according to claim 1, characterized in that: The time positioning module includes: 2 layers of bidirectional gated cyclic units.

6. The sound event detection method according to claim 5, characterized in that: The event decision module includes: a first fully connected layer, a second fully connected layer and a sigmoid activation function; The step of classifying the information features of the multi-scale linear features in the time dimension by the event decision module to obtain a detection label corresponding to the audio to be detected includes: The information features of the multi-scale linear features in the time dimension are transformed by the sigmoid activation function after passing through the first fully connected layer and the second fully connected layer in sequence to obtain the frame-level detection label of the sound event type, and then the frame-level detection label is converted into a fragment-level detection label through the global average pooling layer.

7. The sound event detection method according to claim 6, characterized in that: The training process of the sound event detection model includes: Collect historical audio samples to build a training set; Extracting the logarithmic Mel-spectrogram features of the audio samples in the training set, and encoding the labels on the time axis of the audio samples to convert them into a real label matrix; Inputting the extracted logarithmic Mel-spectrogram features into the sound event detection model to obtain a detection label corresponding to the audio sample; Encode the detection labels corresponding to the audio samples and transform them into a detection label matrix; Calculating the loss function of the sound event detection model based on the true label matrix and the detection label matrix of the audio sample; Determine whether the loss function has reached the convergence condition. If so, the training of the sound event detection model is completed. Otherwise, adjust the parameters of the feature extraction module, the time positioning module, and the event decision module, and re-execute the training steps until the maximum number of iterations is reached or the convergence condition is reached. Among them, the horizontal axis of the real label matrix and the detection label matrix represents the frame number, the vertical axis represents the preset sound event type, and the element value in the matrix represents whether a sound event of the corresponding type occurs in the corresponding frame.

8. The sound event detection method according to claim 7, characterized in that: The loss function of the sound event detection model is binary cross entropy loss, expressed as: ; in, represents the number of samples, Indicates The true label matrix of audio samples, Indicates The detection label matrix of audio samples.

9. The sound event detection method according to claim 8, characterized in that: The training set includes: unlabeled audio data, weakly labeled audio data that only marks the sound event type, and strongly labeled audio data that marks the sound event type and the specific time point of occurrence.

10. A computer-readable storage medium, characterized in that: Program instructions are stored thereon, wherein the program instructions, when executed by a processor, implement the sound event detection method as described in any one of claims 1 to 9.