Audio sound event detection method and device, readable storage medium and equipment

Abstract

The invention discloses an audio sound event detection method and device, a readable storage medium and equipment, and belongs to the technical field of audio detection.The audio sound event detection method comprises the steps that a first audio feature map of target audio is acquired; inputting the first audio feature map into a pre-trained sound event detection model based on a cross attention mechanism, and determining a prediction time label of the target audio; and determining a specific sound event in the target audio according to the predicted time label of the target audio. According to the method, the relation between different time frequency points can be effectively established through a cross attention mechanism, the feature representation capability is enhanced, subtle changes and key features in a sound event can be accurately recognized, and the problems that information of a short-time sound event is lost and the recognized content is incomplete due to a current sound event recognition method are solved.

Description

Technical Field

[0001] The present invention relates to an audio sound event detection method, device, readable storage medium and equipment, belonging to the technical field of audio detection. Background Art

[0002] Sound Event Detection (SED) technology is an important audio processing technology that aims to automatically detect and identify specific types of sound events from complex audio signals, such as car horns, dog barking, human voices, etc.; this technology is widely used in many fields, such as smart homes, security monitoring, environmental monitoring, medical diagnosis, etc.

[0003] The field of sound event detection includes a series of advanced signal processing and machine learning technologies. First, the sound signal is collected and converted into digital form; then it is subjected to noise reduction, filtering and feature extraction through preprocessing steps to improve the effect of subsequent processing; In sound event detection, the attention mechanism plays a key role in feature extraction. It dynamically adjusts the feature weights of different positions or channels, allowing the model to pay more attention to important information and suppress unimportant parts of the task; However, if the attention mechanism is used improperly, it may also lead to the loss of features of short-term sound events, thus affecting the accurate detection of these events; many of the widely used attention mechanisms currently use global pooling or compression to convert feature values ​​within a certain time period or frequency band into a single value. Such processing may lead to the loss of information of short-term sound events, especially those with short duration and fast spectrum changes in the time-frequency diagram. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide an audio sound event detection method, apparatus, readable storage medium and device. Through the cross-attention mechanism, the connection between different time-frequency points can be effectively established, the feature characterization capability can be enhanced, and the subtle changes and key features in the sound events can be accurately identified, thus solving the problem that the current sound event recognition method may cause the loss of information of short-term sound events and incomplete recognition content.

[0005] In order to achieve the above purpose / solve the above technical problems, the present invention is implemented by adopting the following technical solutions: The present invention provides an audio sound event detection method, comprising: Obtaining a first audio feature map of the target audio; Inputting the first audio feature map into a sound event detection model pre-trained based on a criss-cross attention mechanism to determine a predicted time label of the target audio; The pre-trained sound event detection model includes a feature extraction network and an event recognition network; the input of the feature extraction network is a first audio feature map, and the output is a second audio feature map generated based on the first audio feature map; the input of the event recognition network is the second audio feature map, and the output is a predicted time label; Determine a specific sound event in the target audio according to the predicted time label of the target audio; The training of the pre-trained sound event detection model includes: Obtain a first audio feature graph of a training audio sample and a real time label corresponding to the first audio feature graph; Inputting the first audio feature map into a target feature extraction network to obtain a second audio feature map; Inputting the second audio feature graph into the target event recognition network to obtain a predicted time label corresponding to the second audio feature graph; Training a target feature extraction network through real time labels and predicted time labels until the target feature extraction network generates a second audio feature map that meets the requirements; Get the trained sound event detection model.

[0006] Furthermore, the feature extraction network includes a first two-dimensional convolution block, a second two-dimensional convolution block, a cross attention mechanism network layer and a plurality of multi-scale strip convolution residual modules connected in sequence; The multi-scale strip convolution residual module includes a multi-scale strip convolution layer and a residual structure layer connected to the multi-scale strip convolution layer; The event recognition network includes a first bidirectional gated recurrent unit, a second bidirectional gated recurrent unit, a first fully connected layer, and a second fully connected layer which are sequentially connected.

[0007] Furthermore, the step of obtaining a first audio feature graph of a training audio sample and a real time label corresponding to the first audio feature graph includes: Calculate the energy spectrum of the training audio samples; Performing logarithmic Mel feature extraction on the energy spectrum to obtain a first audio feature map; The time-continuous labels in the first audio feature map are encoded to obtain real time labels corresponding to the first audio feature map.

[0008] Furthermore, the step of inputting the first audio feature map into a target feature extraction network to obtain a second audio feature map includes: Performing convolution processing on the first audio feature map to obtain a first spatial dimension tensor, a second spatial dimension tensor, and a third spatial dimension tensor; Obtaining an intermediate audio feature map based on the first spatial dimension tensor and the second spatial dimension tensor; Based on the third spatial dimension tensor and the intermediate audio feature map, a second audio feature map is obtained.

[0009] Furthermore, obtaining the intermediate audio feature map based on the first spatial dimension tensor and the second spatial dimension tensor includes: The similarity between the first spatial dimension tensor and the second spatial dimension tensor is calculated using the following formula: in: Represents the first spatial dimension tensor A vector corresponding to each position u in the spatial dimension; Represents the position extracted in the second spatial dimension tensor u Eigenvectors in the same row and column; express The i elements, , F represents the number of channels in the first audio feature map, and C represents the number of categories in the feature map; , Represents the similarity between the first spatial dimension tensor and the second spatial dimension tensor, express and similarity; represents transpose; Softmax is used to normalize the similarity between the first spatial dimension tensor and the second spatial dimension tensor to obtain the intermediate audio feature map.

[0010] Furthermore, the obtaining of the second audio feature map based on the third spatial dimension tensor and the intermediate audio feature map includes: The third spatial dimension tensor and the intermediate audio feature map are aggregated using the following formula to obtain the second audio feature map: in: Indicates the position of the second audio feature map u The eigenvector at ; Represents the first audio feature map Location u The eigenvector at ; F Represents the first audio feature map The number of channels in represents the time step; Represents the position extracted from the third spatial dimension tensor u The eigenvectors in the same row and column are express The i elements; A means Intermediate audio feature map, Indicates the position of the intermediate audio feature map uNo. i Scalar value on each channel.

[0011] Furthermore, the target feature extraction network is trained by using the real time label and the predicted time label until the target feature extraction network generates a second audio feature map that meets the requirements, including: Use binary cross entropy loss as the loss function for training the target feature extraction network Loss : in: represents the training audio samples, Indicates i The real time label of the first audio feature map, Indicates i The predicted time label of the first audio feature map.

[0012] A second aspect of the present invention provides an audio sound event detection device, comprising: An acquisition module, used to acquire a first audio feature map of the target audio; A detection module, configured to input the first audio feature map into a sound event detection model pre-trained based on a cross-attention mechanism to determine a predicted time label of the target audio; The processing module is used to determine a specific sound event in the target audio according to the predicted time tag of the target audio.

[0013] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned audio sound event detection method.

[0014] The present invention also provides a computer device, comprising: A memory for storing instructions; The processor is used to execute the instructions so that the device implements the above-mentioned audio sound event detection method.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention uses a pre-trained sound time detection model to perform feature extraction based on the first audio feature map of the target audio to generate a second audio feature map, obtains a predicted time label based on the second audio feature map, and determines a specific sound event in the target audio based on the predicted time label. It can accurately identify subtle changes and key features in sound events, and solves the problem that current sound event recognition methods may cause information loss of short-term sound events and incomplete recognition content.

[0016] 2. The present invention trains a sound event detection model through a cross-attention mechanism, and obtains a first spatial dimension tensor, a second spatial dimension tensor and a third spatial dimension tensor based on a first audio feature map. Based on the spatial dimension tensor, it can effectively aggregate long-range context information in the horizontal and vertical directions, and establish connections between different time-frequency points, so that the feature representation ability of the time-frequency points is enhanced, so that the model can better capture subtle changes and key features in sound events. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of an audio sound event detection method according to an embodiment of the audio sound event detection method, device, readable storage medium and equipment of the present invention; Figure 2 It is a training flow chart of a sound event detection model of an audio sound event detection method of an embodiment of the audio sound event detection method, device, readable storage medium and equipment of the present invention; Figure 3 It is a structural schematic diagram of an audio sound event detection system according to an embodiment of the audio sound event detection method, apparatus, readable storage medium and device of the present invention; Figure 4 It is a schematic diagram of the working principle of a feature extraction network in an audio sound event detection method according to an embodiment of the audio sound event detection method, device, readable storage medium and equipment of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention. Example

[0019] like Figure 1 As shown, an audio sound event detection method includes: Obtaining a first audio feature map of the target audio; Inputting the first audio feature map into a sound event detection model pre-trained based on a criss-cross attention mechanism to determine a predicted time label of the target audio; Determine a specific sound event in the target audio according to the predicted time label of the target audio; The pre-trained sound event detection model includes a feature extraction network and an event recognition network; The input of the feature extraction network is a first audio feature map, and the output is a second audio feature map generated based on the first audio feature map, wherein the feature extraction network includes a first two-dimensional convolution block, a second two-dimensional convolution block, a cross attention mechanism network layer, and a plurality of multi-scale strip convolution residual modules connected in sequence; In this embodiment, the number of multi-scale strip convolution residual modules is 5; The multi-scale strip convolution residual module includes a multi-scale strip convolution layer and a residual structure layer connected to the multi-scale strip convolution layer; The input of the event recognition network is the second audio feature map, and the output is the predicted time label, wherein the event recognition network includes a first bidirectional gated recurrent unit, a second bidirectional gated recurrent unit, a first fully connected layer, and a second fully connected layer connected in sequence.

[0020] like Figure 2 As shown, the training of the pre-trained sound event detection model includes: Obtain a first audio feature graph of a training audio sample and a real time label corresponding to the first audio feature graph, specifically: Calculate the energy spectrum of the training audio samples; Performing logarithmic Mel feature extraction on the energy spectrum to obtain a first audio feature map; The time-continuous labels in the first audio feature graph are encoded to obtain real time labels corresponding to the first audio feature graph. In this embodiment, the encoding process mainly includes the following steps: Assume that the time-continuous labels in the first audio feature graph are: labels = ['happy', 'sad', 'angry', 'happy', 'neutral']; Convert the labels to a one-hot encoding matrix, including: Determine unique tags: unique_labels = list(set(labels)) # ['happy', 'sad', 'angry', 'neutral']; Assign each tag a unique index: label_to_index = {label: idx for idx, label in enumerate(unique_labels)} # {'happy': 0, 'sad': 1, 'angry': 2, 'neutral': 3}; Create a zero matrix with the number of rows equal to the number of samples and the number of columns equal to the number of categories: one_hot_matrix = np.zeros((len(labels), len(unique_labels))) # 5x4 matrix; Fill the matrix according to the labels: for i, label in enumerate(labels): one_hot_matrix[i, label_to_index[label]] = 1; Get the real time label corresponding to the first audio feature map: .

[0021] The first audio feature map is input into the target feature extraction network to obtain the second audio feature map. Specifically: Performing convolution processing on the first audio feature map to obtain a first spatial dimension tensor, a second spatial dimension tensor, and a third spatial dimension tensor; like Figure 4 As shown, based on the first spatial dimension tensor and the second spatial dimension tensor, an intermediate audio feature map is obtained, including: Use similar operations to calculate the first spatial dimension tensor Q With the second spatial dimension tensor K Similarity: in: Represents the first spatial dimension tensor Q At every position in the spatial dimension u A corresponding vector; Represents a tensor in the second spatial dimension K Extracted from and position u Eigenvectors in the same row and column; express The i elements, , F represents the number of channels in the first audio feature map, and C represents the number of categories in the feature map; , Represents the similarity between the first spatial dimension tensor and the second spatial dimension tensor, express and similarity; represents transpose; Softmax is used to normalize the similarity between the first spatial dimension tensor and the second spatial dimension tensor to obtain an intermediate audio feature map; Based on the third spatial dimension tensor V , an intermediate audio feature map, and obtaining a second audio feature map, comprising: The third spatial dimension tensor and the intermediate audio feature map are aggregated using the following formula to obtain the second audio feature map: in: Represents the second audio feature map Locationu The eigenvector at ; Represents the first audio feature map Location u The eigenvector at ; F Represents the first audio feature map The number of channels in represents the time step; Represents the position extracted from the third spatial dimension tensor u The eigenvectors in the same row and column are express The i elements; Represents the intermediate audio feature map A Location u No. i Scalar value on each channel.

[0022] Inputting the second audio feature graph into the target event recognition network to obtain a predicted time label corresponding to the second audio feature graph; The target feature extraction network is trained by using the real time label and the predicted time label until the target feature extraction network generates a second audio feature map that meets the requirements, including: Use binary cross entropy loss as the loss function for training the target feature extraction network : in: represents the training audio samples, Indicates i The real time label of the first audio feature map, Indicates i a predicted time label of a first audio feature map; Get the trained sound event detection model.

[0023] Example 2 like Figure 3 As shown, an audio sound event detection device comprises: An acquisition module, used to acquire a first audio feature map of the target audio; A detection module, configured to input the first audio feature map into a sound event detection model pre-trained based on a cross-attention mechanism to determine a predicted time label of the target audio; The processing module is used to determine a specific sound event in the target audio according to the predicted time tag of the target audio.

[0024] Example 3 A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the following method: Obtaining a first audio feature map of the target audio; Inputting the first audio feature map into a sound event detection model pre-trained based on a criss-cross attention mechanism to determine a predicted time label of the target audio; The pre-trained sound event detection model includes a feature extraction network and an event recognition network; the input of the feature extraction network is a first audio feature map, and the output is a second audio feature map generated based on the first audio feature map; the input of the event recognition network is the second audio feature map, and the output is a predicted time label; Determine a specific sound event in the target audio according to the predicted time label of the target audio; The training of the pre-trained sound event detection model includes: Obtain a first audio feature graph of a training audio sample and a real time label corresponding to the first audio feature graph; Inputting the first audio feature map into a target feature extraction network to obtain a second audio feature map; Inputting the second audio feature graph into the target event recognition network to obtain a predicted time label corresponding to the second audio feature graph; Training a target feature extraction network through real time labels and predicted time labels until the target feature extraction network generates a second audio feature map that meets the requirements; Get the trained sound event detection model.

[0025] Example 4 A computer device comprising: A memory for storing instructions; The processor is used to execute instructions so that the device performs the steps of implementing the following method: Obtaining a first audio feature map of the target audio; Inputting the first audio feature map into a sound event detection model pre-trained based on a criss-cross attention mechanism to determine a predicted time label of the target audio; The pre-trained sound event detection model includes a feature extraction network and an event recognition network; the input of the feature extraction network is a first audio feature map, and the output is a second audio feature map generated based on the first audio feature map; the input of the event recognition network is the second audio feature map, and the output is a predicted time label; Determine a specific sound event in the target audio according to the predicted time label of the target audio; The training of the pre-trained sound event detection model includes: Obtain a first audio feature graph of a training audio sample and a real time label corresponding to the first audio feature graph; Inputting the first audio feature map into a target feature extraction network to obtain a second audio feature map; Inputting the second audio feature graph into the target event recognition network to obtain a predicted time label corresponding to the second audio feature graph; Training a target feature extraction network through real time labels and predicted time labels until the target feature extraction network generates a second audio feature map that meets the requirements; Get the trained sound event detection model.

[0026] In the description of the present invention, it is to be understood that the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the feature. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0027] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0028] The present application is described with reference to the flowcharts of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0029] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A function specified in a process or multiple processes.

[0030] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 The steps of a specified function in a process or multiple processes.

[0031] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A method for detecting an audio sound event, characterized in that: include: Obtaining a first audio feature map of the target audio; Inputting the first audio feature map into a sound event detection model pre-trained based on a criss-cross attention mechanism to determine a predicted time label of the target audio; The pre-trained sound event detection model includes a feature extraction network and an event recognition network; the input of the feature extraction network is a first audio feature map, and the output is a second audio feature map generated based on the first audio feature map; the input of the event recognition network is the second audio feature map, and the output is a predicted time label; Determine a specific sound event in the target audio according to the predicted time label of the target audio; The training of the pre-trained sound event detection model includes: Obtain a first audio feature graph of a training audio sample and a real time label corresponding to the first audio feature graph; Inputting the first audio feature map into a target feature extraction network to obtain a second audio feature map; Inputting the second audio feature graph into the target event recognition network to obtain a predicted time label corresponding to the second audio feature graph; Training a target feature extraction network through real time labels and predicted time labels until the target feature extraction network generates a second audio feature map that meets the requirements; Get the trained sound event detection model.

2. The audio sound event detection method according to claim 1, characterized in that: The feature extraction network includes a first two-dimensional convolution block, a second two-dimensional convolution block, a cross attention mechanism network layer and a plurality of multi-scale strip convolution residual modules connected in sequence; The multi-scale strip convolution residual module includes a multi-scale strip convolution layer and a residual structure layer connected to the multi-scale strip convolution layer; The event recognition network includes a first bidirectional gated recurrent unit, a second bidirectional gated recurrent unit, a first fully connected layer, and a second fully connected layer which are sequentially connected.

3. The audio sound event detection method according to claim 1, characterized in that: The step of obtaining a first audio feature graph of a training audio sample and a real time label corresponding to the first audio feature graph includes: Calculate the energy spectrum of the training audio samples; Performing logarithmic Mel feature extraction on the energy spectrum to obtain a first audio feature map; The time-continuous labels in the first audio feature map are encoded to obtain real time labels corresponding to the first audio feature map.

4. The audio sound event detection method according to claim 1, characterized in that: The step of inputting the first audio feature graph into a target feature extraction network to obtain a second audio feature graph includes: Performing convolution processing on the first audio feature map to obtain a first spatial dimension tensor, a second spatial dimension tensor, and a third spatial dimension tensor; Obtaining an intermediate audio feature map based on the first spatial dimension tensor and the second spatial dimension tensor; Based on the third spatial dimension tensor and the intermediate audio feature map, a second audio feature map is obtained.

5. The audio sound event detection method according to claim 4, characterized in that: The step of obtaining an intermediate audio feature map based on the first spatial dimension tensor and the second spatial dimension tensor includes: The similarity between the first spatial dimension tensor and the second spatial dimension tensor is calculated using the following formula: in: Represents the first spatial dimension tensor At every position in the spatial dimension A corresponding vector; Represents the position extracted in the second spatial dimension tensor Feature vectors in the same row and column The i elements, , F represents the number of channels in the first audio feature map, and C represents the number of categories in the feature map; , Represents the similarity between the first spatial dimension tensor and the second spatial dimension tensor, express and similarity; represents transpose; Softmax is used to normalize the similarity between the first spatial dimension tensor and the second spatial dimension tensor to obtain the intermediate audio feature map.

6. The audio sound event detection method according to claim 4, characterized in that: The obtaining of the second audio feature map based on the third spatial dimension tensor and the intermediate audio feature map includes: The third spatial dimension tensor and the intermediate audio feature map are aggregated using the following formula to obtain the second audio feature map: in: Represents the second audio feature map Location u The eigenvector at ; Represents the first audio feature map Location u The eigenvector at ; F represents the number of channels in the first audio feature map, represents the time step; Represents the position extracted from the third spatial dimension tensor u Feature vectors in the same row and column The i elements; Represents the intermediate audio feature map A Location u No. i Scalar value on each channel.

7. The audio sound event detection method according to claim 2, characterized in that: The target feature extraction network is trained by using the real time label and the predicted time label until the target feature extraction network generates a second audio feature map that meets the requirements, including: Use binary cross entropy loss as the loss function for training the target feature extraction network Loss : in: represents the training audio samples, Indicates i The real time label of the first audio feature map, Indicates i The predicted time label of the first audio feature map.

8. An audio sound event detection device, characterized in that: include: An acquisition module, used to acquire a first audio feature map of the target audio; A detection module, configured to input the first audio feature map into a sound event detection model pre-trained based on a cross-attention mechanism to determine a predicted time label of the target audio; The processing module is used to determine a specific sound event in the target audio according to the predicted time tag of the target audio.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the audio sound event detection method according to any one of claims 1 to 7 is implemented.

10. A computer device, characterized in that: include: A memory for storing instructions; The processor is used to execute the instruction so that the device implements the audio sound event detection method as described in any one of claims 1 to 7.

Images