A method and system for audio event detection based on multi-task learning
By combining multi-task learning with audio event detection and event type classification, the problem of increased manual annotation workload in existing technologies is solved, achieving efficient audio event detection, improving detection accuracy and reducing model parameters.
Patent Information
- Application Number
- CN202111153760.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-09-29
AI Technical Summary
Existing audio event detection methods require additional audio scene recognition labels, which increases the workload of manual labeling and affects the detection accuracy.
A multi-task learning approach is adopted to combine audio event detection with audio event type classification. A single model can simultaneously achieve both audio event detection and audio event type classification. The model is trained using a loss function constructed with cross-entropy and sum of squared differences, without the need for additional annotation information.
It significantly improves the accuracy of audio event detection, reduces the number of model parameters, improves computational efficiency, and avoids additional manual annotation work.
Smart Images

Figure CN113921039B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field related to audio event detection, and in particular to an audio event detection method and system based on multi-task learning. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] Audio event detection refers to detecting the type and start and end time of audio events in an audio stream. Audio event detection has broad application prospects and can be used in audio security monitoring, bioacoustic research, urban sound analysis, etc. Therefore, the research on audio event detection has received widespread attention in the academic community. Different types of audio events in an audio stream often overlap in time, and the audio stream is often mixed with various noises, which brings great difficulties and challenges to the accurate detection of audio events.
[0004] In recent years, some studies have proposed using multi-task learning to improve the accuracy of audio event detection, that is, adding an auxiliary task to the audio event detection task, completing two tasks at the same time by building a model, and using the auxiliary task to help improve the accuracy of audio event detection. Some studies have added the auxiliary task of audio scene recognition to the audio event detection task, but the audio scene recognition auxiliary task requires additional scene labels for audio documents, which increases the workload of manual labeling. Summary of the invention
[0005] In order to solve the problem that the above-mentioned auxiliary tasks require additional annotation, which increases the workload of manual annotation, the present invention provides an audio event detection method and system based on multi-task learning. The method adds an auxiliary task of classifying the number of audio event types to the audio event detection task, and simultaneously realizes audio event detection and audio event type classification in one model to improve the accuracy of audio event detection. At the same time, the workload of manual annotation is not increased on the basis of improving the accuracy.
[0006] In order to achieve the above object, the present invention adopts the following technical solution:
[0007] A first aspect of the present invention provides an audio event detection method based on multi-task learning.
[0008] An audio event detection method based on multi-task learning, comprising:
[0009] Based on the feature matrix of the audio segment to be detected, the trained audio event detection model is used to obtain the audio events in the audio segment to be detected;
[0010] The process of training the audio event detection model includes:
[0011] According to each audio frame in the audio clip, output values of all frames in the audio clip at the audio event detection output layer and output values of the audio event type number classification output layer are obtained, a cross entropy sum is obtained based on the output values at the audio event detection output layer, and a difference square sum is obtained based on the output values at the audio event type number classification output layer;
[0012] The loss function is constructed by using the cross entropy sum and the difference square sum to train the audio event detection model, thereby obtaining a trained audio event detection model.
[0013] Furthermore, before obtaining the cross entropy sum and the difference square sum, the method includes: obtaining audio clips, extracting a feature matrix of each audio clip, adding an audio event label and an audio event type number label to each audio frame in the audio clip, and constructing an audio feature training set.
[0014] Furthermore, the process of extracting the feature matrix of each audio segment includes: dividing the audio segment into frames, extracting the audio features of each frame, and the audio features of all frames in each audio segment constitute the feature matrix of the audio segment.
[0015] Furthermore, the process of adding an audio event label and a label of the number of audio event types to each audio frame in the audio clip includes: assuming that there are m different types of audio events in the training set, the audio event label is an m-dimensional vector, and the elements in the vector take values 0 or 1. If the audio frame contains an audio event of a certain type, the element value at the label vector position corresponding to the audio event of this type is 1, otherwise it is 0; adding a label of the number of audio event types to each audio frame in the audio clip, and the number of audio event types is obtained by counting the number of element 1s in the audio event label.
[0016] Furthermore, the process of obtaining the output values of all frames in the audio clip at the audio event detection output layer and obtaining the cross entropy sum based on the output values includes: calculating the output value of each audio frame at the audio event detection output layer according to each audio frame in the audio clip, using the output value and the audio event label of the audio frame to obtain the cross entropy, and summing the cross entropies of all frames in the audio clip to obtain the cross entropy sum.
[0017] Furthermore, the process of obtaining the output values of the audio event type number classification output layer of all frames in the audio clip and obtaining the sum of square differences based on the output values includes: calculating the output value of each audio frame in the audio event type number classification output layer according to each audio frame in the audio clip, using the output value and the audio event type number label of the audio frame to obtain the square difference value, and summing the square difference values of all frames in the audio clip to obtain the sum of square differences.
[0018] Furthermore, the process of obtaining the audio events in the audio segment to be detected includes: performing frame processing on the audio segment to be detected using the same method as in the training stage, extracting the audio features of each frame, and the audio features of all frames in the audio segment constitute a feature matrix of the audio segment to be identified. The feature matrix of the audio segment to be identified is used as the input of the trained audio event detection model, and the probability value of each audio frame in the audio segment to be detected containing each audio event is obtained at the audio event detection output layer of the trained audio event detection model, and a threshold is set. The audio events with probability values greater than the set threshold are the audio events contained in the audio frame.
[0019] A second aspect of the present invention provides an audio event detection system based on multi-task learning.
[0020] An audio event detection system based on multi-task learning, comprising:
[0021] A detection module is configured to: obtain audio events in the audio segment to be detected by using a trained audio event detection model based on a feature matrix of the audio segment to be detected;
[0022] The model training module is configured as follows: the process of training the audio event detection model includes: according to each audio frame in the audio clip, obtaining the output value of all frames in the audio clip at the audio event detection output layer and the output value at the audio event type number classification output layer, obtaining the cross entropy sum based on the output value at the audio event detection output layer, and obtaining the difference square sum based on the output value at the audio event type number classification output layer; using the cross entropy sum and the difference square sum to construct a loss function, training the audio event detection model, and obtaining a trained audio event detection model.
[0023] A third aspect of the present invention provides a computer-readable storage medium.
[0024] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the audio event detection method based on multi-task learning as described in the first aspect above.
[0025] A fourth aspect of the present invention provides a computer device.
[0026] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the audio event detection method based on multi-task learning as described in the first aspect above are implemented.
[0027] Compared with the prior art, the present invention has the following beneficial effects:
[0028] (1) The present invention combines audio event detection and audio event type number classification through multi-task learning, and simultaneously realizes audio event detection and audio event type number classification through one model. Compared with the method of performing audio event detection alone without multi-task learning, the present invention can significantly improve the accuracy of audio event detection.
[0029] (2) The audio event type number classification auxiliary task designed by the present invention does not require additional annotation information and will not increase the workload of manual annotation, so that the multi-task learning method proposed by the present invention can be carried out on the basis of the existing database.
[0030] (3) Compared with modeling for audio event detection alone, under the same detection performance, the multi-task learning model designed by the present invention can significantly reduce the number of model parameters and improve computational efficiency.
[0031] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0033] Figure 1 is a schematic diagram of an audio event detection model based on multi-task learning of the present invention;
[0034] Figure 2 yes Figure 1 Schematic diagram of the structure of the convolutional neural network 1 in;
[0035] Figure 3 yes Figure 1 Schematic diagram of the structure of the convolutional neural network 2 in;
[0036] Figure 4 yes Figure 1 Schematic diagram of the structure of the convolutional neural network 3 in. DETAILED DESCRIPTION
[0037] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0038] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0039] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0040] Embodiment 1
[0041] like Figure 1 As shown, this embodiment provides an audio event detection method based on multi-task learning.
[0042] An audio event detection method based on multi-task learning, comprising:
[0043] Based on the feature matrix of the audio segment to be detected, the trained audio event detection model is used to obtain the audio events in the audio segment to be detected;
[0044] The process of training the audio event detection model includes:
[0045] According to each audio frame in the audio clip, output values of all frames in the audio clip at the audio event detection output layer and output values of the audio event type number classification output layer are obtained, a cross entropy sum is obtained based on the output values at the audio event detection output layer, and a difference square sum is obtained based on the output values at the audio event type number classification output layer;
[0046] The loss function is constructed by using the cross entropy sum and the difference square sum to train the audio event detection model, thereby obtaining a trained audio event detection model.
[0047] Specifically, the technical solution of this embodiment can be implemented by the following steps:
[0048] Step (1): Obtain an audio database and construct an audio feature training set.
[0049] Wherein step (1) comprises:
[0050] Step (1-1): perform frame processing on each audio clip in the database, and extract audio features commonly used in the audio field from each frame to obtain a feature matrix for each audio clip;
[0051] Step (1-2): add an audio event label to each audio frame in the audio clip. The audio event label is in the form of a multi-hot vector. Assuming that there are m different types of audio events in the training set, the audio event label is an m-dimensional vector. The elements in the vector are 0 or 1. If the audio frame contains a certain type of audio event, the element value at the label vector position corresponding to the type of audio event is 1, otherwise it is 0.
[0052] Step (1-3): Add a label indicating the number of audio event types to each audio frame in the audio segment. The number of audio event types is obtained by counting the number of element 1s in the audio event label.
[0053] Step (1-4): obtain an audio feature training set with audio event labels and audio event type number labels.
[0054] Step (2): Follow Figure 1 Construct an audio event detection model based on multi-task learning, take the feature matrix of the audio clip as the input of the model, and calculate the cross entropy sum S1 and the difference square sum S2.
[0055] Wherein step (2) comprises:
[0056] Step (2-1): select an audio segment from the audio feature training set in sequence, and use the feature matrix of the audio segment as the input of the model;
[0057] Step (2-2): for each audio frame in the audio clip, the cross entropy is calculated based on its audio event label and the output of the audio event detection output layer, and the cross entropies of all frames in the audio clip are summed to obtain the cross entropy sum S1;
[0058] Step (2-3): for each audio frame in the audio clip, calculate the square value of the difference between its audio event type number label and the output value of the audio event type number classification output layer, and sum the square values of the difference of all frames in the audio clip to obtain the sum of squares of the difference S2;
[0059] Step (3): Construct a loss function based on the cross entropy and S1 and the difference square sum S2. Figure 1 The audio event detection model in is trained to obtain a trained audio event detection model;
[0060] Wherein step (3) comprises:
[0061] Step (3-1): Let S = a × S1 + (1-a) × S2, with S as Figure 1 The loss function of the entire network model is used to train the entire network model, where a is a constant factor in the range of [0,1], and its value is determined by model training;
[0062] Step (3-2): Repeat step (2) to step (3-1) until the loss function value S is less than a preset threshold;
[0063] Step (3-3): Output the trained audio event detection model.
[0064] Step (4): The audio segment to be detected is divided into frames using the same method as in step (1), and the audio features of each frame are extracted. The audio features of all frames in the audio segment constitute the feature matrix of the audio segment to be identified. The feature matrix of the audio segment to be identified is input into the trained audio event detection model, and the probability value of each audio frame in the audio segment to be detected containing each audio event is obtained at the audio event detection output layer. A threshold is set, and the audio event with a probability value greater than the threshold is the audio event contained in the audio frame.
[0065] Figure 1 The output layer of audio event detection is a fully connected layer. The number of nodes in the output layer is the total number of audio event types included in the training set. The sigmoid function is used as the activation function, and its output value represents the probability value of each different audio event contained in the audio frame.
[0066] Figure 1 The output layer for classifying the number of audio event types is a fully connected layer. The number of output layer nodes is 1. The ReLU function is used as the activation function, and its output value represents the number of different audio events contained in the audio frame.
[0067] Figure 2 yes Figure 1 The structural diagram of the convolutional neural network 1 in , including a convolutional layer, a maximum pooling layer, a convolutional layer, and a maximum pooling layer connected in sequence. The convolution stride parameter of the convolutional layer is set to 1, and the size of the output of the convolutional layer in the frame dimension is made the same as the size of the input of the input end of the convolutional neural network 2 in the frame dimension through the padding method. Parameters such as the number of nodes in each convolutional layer and the size of the convolution kernel function window are obtained through debugging according to the data and training model. The stride parameter of the maximum pooling layer is set to 1, and the window size of the maximum pooling layer is obtained through debugging according to the data and training model.
[0068] Figure 3 yes Figure 1The schematic diagram of the structure of the convolutional neural network 2 in FIG. 1 includes a convolutional layer and a maximum pooling layer connected in succession. The convolution stride parameter of the convolutional layer is set to 1, and the size of the output of the convolutional layer in the frame dimension is made the same as the size of the input of the input end of the convolutional neural network 2 in the frame dimension through the padding method. Parameters such as the number of convolutional layer nodes and the size of the convolution kernel function window are obtained through debugging according to the data and training model. The stride parameter of the maximum pooling layer is set to 1, and the window size of the maximum pooling layer is obtained through debugging according to the data and training model.
[0069] Figure 4 yes Figure 1 The schematic diagram of the structure of the convolutional neural network 3 in FIG. 1 includes a convolutional layer, a maximum pooling layer, and a fully connected layer connected in sequence. The convolution stride parameter of the convolutional layer is set to 1, and the size of the output of the convolutional layer in the frame dimension is made the same as the size of the input of the input end of the convolutional neural network 3 in the frame dimension through the padding method. Parameters such as the number of convolutional layer nodes and the size of the convolution kernel function window are obtained through debugging according to the data and training model. The stride parameter of the maximum pooling layer is set to 1, and the window size of the maximum pooling layer is obtained through debugging according to the data and training model.
[0070] Embodiment 2
[0071] This embodiment provides an audio event detection system based on multi-task learning.
[0072] An audio event detection system based on multi-task learning, comprising:
[0073] A detection module is configured to: obtain audio events in the audio segment to be detected based on a feature matrix of the audio segment to be detected and using a trained audio event detection model;
[0074] The model training module is configured as follows: the process of training the audio event detection model includes: according to each audio frame in the audio clip, obtaining the output value of all frames in the audio clip at the audio event detection output layer and the output value at the audio event type number classification output layer, obtaining the cross entropy sum based on the output value at the audio event detection output layer, and obtaining the difference square sum based on the output value at the audio event type number classification output layer; using the cross entropy sum and the difference square sum to construct a loss function, training the audio event detection model, and obtaining a trained audio event detection model.
[0075] Embodiment 3
[0076] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the audio event detection method based on multi-task learning as described in the first embodiment are implemented.
[0077] Embodiment 4
[0078] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the audio event detection method based on multi-task learning as described in the first embodiment are implemented.
[0079] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.
[0080] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0081] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0083] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for audio event detection based on multi-task learning, characterized in that: include: Based on the feature matrix of the audio segment to be detected, the trained audio event detection model is used to obtain the audio events in the audio segment to be detected; Before obtaining the cross entropy sum and the difference square sum, the process includes: obtaining an audio clip, extracting a feature matrix of each audio clip, adding an audio event label and a label of the number of audio event types to each audio frame in the audio clip, and constructing an audio feature training set; the process of training the audio event detection model includes: According to each audio frame in the audio clip, the output value of all frames in the audio clip at the audio event detection output layer and the output value of the audio event type number classification output layer are obtained, the cross entropy sum is obtained based on the output value at the audio event detection output layer, and the difference square sum is obtained based on the output value at the audio event type number classification output layer; the process of obtaining the output value of all frames in the audio clip at the audio event type number classification output layer and obtaining the difference square sum based on the output value comprises: according to each audio frame in the audio clip, calculating the output value of each audio frame at the audio event type number classification output layer, obtaining the difference square value by using the output value and the audio event type number label of the audio frame, and summing the difference square values of all frames in the audio clip to obtain the difference square sum; The process of adding an audio event label and an audio event type number label to each audio frame in the audio clip includes: assuming that there are m different types of audio events in the training set, the audio event label is an m-dimensional vector, and the element value in the vector is 0 or 1. If the audio frame contains an audio event of a certain type, the element value at the label vector position corresponding to the audio event of the type is 1, otherwise it is 0; adding an audio event type number label to each audio frame in the audio clip, and the number of audio event types is obtained by counting the number of element 1s in the audio event label; The loss function is constructed by using the cross entropy sum and the difference square sum to train the audio event detection model, thereby obtaining a trained audio event detection model.
2. The audio event detection method based on multi-task learning according to claim 1, characterized in that: The process of extracting the feature matrix of each audio segment includes: dividing the audio segment into frames, extracting the audio features of each frame, and the audio features of all frames in each audio segment constitute the feature matrix of the audio segment.
3. The audio event detection method based on multi-task learning according to claim 1, characterized in that: The process of obtaining the output values of all frames in the audio clip at the audio event detection output layer and obtaining the cross entropy sum based on the output values includes: calculating the output value of each audio frame at the audio event detection output layer according to each audio frame in the audio clip, obtaining the cross entropy by using the output value and the audio event label of the audio frame, and summing the cross entropies of all frames in the audio clip to obtain the cross entropy sum.
4. The audio event detection method based on multi-task learning according to claim 1 is characterized in that: The process of obtaining the audio events in the audio segment to be detected includes: performing frame processing on the audio segment to be detected using the same method as in the training stage, extracting the audio features of each frame, and the audio features of all frames in the audio segment constitute a feature matrix of the audio segment to be identified; using the feature matrix of the audio segment to be identified as the input of the trained audio event detection model, obtaining the probability value of each audio frame in the audio segment to be detected containing each audio event at the audio event detection output layer of the trained audio event detection model, setting a threshold, and the audio events with probability values greater than the set threshold are the audio events contained in the audio frame.
5. An audio event detection system based on multi-task learning, characterized in that: include: A detection module is configured to: obtain audio events in the audio segment to be detected by using a trained audio event detection model based on a feature matrix of the audio segment to be detected; The model training module is configured as follows: the process of training the audio event detection model includes: obtaining, according to each audio frame in the audio segment, the output values of all frames in the audio segment at the audio event detection output layer and the output values of the audio event type number classification output layer, obtaining the cross entropy sum based on the output values at the audio event detection output layer, and obtaining the difference square sum based on the output values at the audio event type number classification output layer; constructing a loss function using the cross entropy sum and the difference square sum, training the audio event detection model, and obtaining a trained audio event detection model; Before obtaining the cross entropy sum and the difference square sum, the method includes: obtaining an audio segment, extracting a feature matrix of each audio segment, adding an audio event label and an audio event type number label to each audio frame in the audio segment, and constructing an audio feature training set; The process of obtaining the output values of the audio event type number classification output layer of all frames in the audio clip and obtaining the sum of squared differences based on the output values comprises: calculating the output value of each audio frame in the audio event type number classification output layer according to each audio frame in the audio clip, obtaining the squared difference value by using the output value and the audio event type number label of the audio frame, and summing the squared difference values of all frames in the audio clip to obtain the sum of squared differences; The process of adding an audio event label and a label of the number of audio event types to each audio frame in the audio clip includes: assuming that there are m different types of audio events in the training set, the audio event label is an m-dimensional vector, and the elements in the vector are 0 or 1. If the audio frame contains an audio event of a certain type, the element value at the label vector position corresponding to the audio event of this type is 1, otherwise it is 0; adding a label of the number of audio event types to each audio frame in the audio clip, and the number of audio event types is obtained by counting the number of elements 1 in the audio event label.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the audio event detection method based on multi-task learning as described in any one of claims 1 to 4 are implemented.
7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the audio event detection method based on multi-task learning as described in any one of claims 1 to 4 are implemented.
Citation Information
Patent Citations
Sound scene recognizing method based on label amplification and multi-spectrum fusion
CN108932950A
Audio scene recognition method combining deep neural network and topic model and system thereof
CN109448703A
Specific sound event retrieval and positioning method based on sequence classification
CN111161715A
Voice event detection method and device, electronic equipment and computer storage medium
CN112447189A
Audio scene recognition method and device based on multi-task learning
CN113220932A
