Sound event detection method, apparatus, device, and storage medium

Abstract

The application relates to the technical field of sound recognition, and provides a low-resource sound event detection method, device and equipment and a storage medium, wherein the method comprises the following steps: acquiring sound to be detected; inputting the sound to be detected into a trained sound group category neural network model to obtain sound group category information; inputting the sound to be detected into an encoder of a pre-trained sound event category judgment model to obtain fine-grained feature information, and splicing and fusing the group category information and the fine-grained feature information to obtain fusion feature information; inputting the fusion feature information into a decoder of the sound event category judgment model, and decoding the fusion feature information based on an attention mechanism and in combination with a pre-obtained group category representation matrix to obtain a sound event judgment result. The application forms large-class group sound group category information containing rich information by using the sound to be detected, and realizes auxiliary discrimination based on large-class sound event results through a self-attention mechanism.

Description

Technical Field

[0001] This application relates to the field of sound recognition technology, and in particular to a sound event detection method, apparatus, device, and storage medium. Background Technology

[0002] Sound event detection aims to distinguish different sound signals from a continuous stream of sound signals and to label sound events with symbols. Unlike speech processing tasks such as speech recognition, voiceprint recognition, and emotion recognition, which focus only on the content of human speech signals, sound event detection has a broader scope, encompassing various sounds from nature, machinery, plants and animals, and all kinds of sounds we encounter in our daily lives.

[0003] Sound event detection, as a fundamental task for identifying sound signals, has been applied in various fields, such as urban traffic and security monitoring, factory noise and equipment anomaly sound detection, and natural environmental sound detection. In recent years, with the development of deep learning technology, research on sound event detection based on different neural network structures (such as CNN / CRNN / Transformer / Conformer) has emerged in large numbers. However, as a data-driven approach, deep learning technology is prone to overfitting with low-resource data, especially when the amount of sound data for a certain category of events is limited or difficult to label effectively. In such cases, neural network models struggle to effectively model these sound events. Transfer learning, as a technique to mitigate the resource sparsity problem, relies on a base model that may not match the target sound events, leading to the inability to effectively model some sound events. Summary of the Invention

[0004] This application provides a sound event detection method, apparatus, device, and storage medium to solve the problem that some sound events cannot be effectively classified in the existing system.

[0005] To solve the above-mentioned technical problems, one technical solution adopted in this application is: to provide a sound event detection method, including: acquiring the sound to be detected;

[0006] The sound to be detected is input into a trained sound group category neural network model to obtain sound group category information;

[0007] The sound to be detected is input into the encoder of a pre-trained sound event category judgment model to obtain fine-grained feature information, and the group category information is concatenated and fused with the fine-grained feature information to obtain fused feature information;

[0008] The fused feature information is input into the decoder of the sound event category judgment model, and the fused feature information is decoded based on the attention mechanism and combined with the pre-obtained group category representation matrix to obtain the sound event judgment result.

[0009] As a further improvement to this application, the pre-training of the sound group category neural network model includes:

[0010] Acquire sound sample data and construct group category labels based on the sound sample data. The group category labels include the sound group category corresponding to each sound sample data.

[0011] The sound sample data is input into the sound group category neural network model to be trained to obtain the sample predicted group category information;

[0012] The sound group category neural network model is updated in reverse based on the sound group category corresponding to each sound sample data and the sample predicted group category information, so as to train the sound group category neural network model.

[0013] As a further improvement to this application, a group category representation matrix is ​​pre-constructed, including:

[0014] The sound sample data is input into a trained sound group category neural network model to obtain a group category feature vector representation corresponding to each sound sample data.

[0015] The group category feature vector representations corresponding to sound sample data belonging to the same group category are sampled and averaged to obtain the group category vector for each sound group category;

[0016] By concatenating all group category vectors, the group category representation matrix is ​​obtained.

[0017] As a further improvement to this application, the sound event category judgment model is pre-trained, including:

[0018] The sound sample data is input into the sound event category judgment model to be trained, and the encoder of the sound event category judgment model extracts fine-grained features from the sound sample data.

[0019] The fine-grained features of the samples are concatenated and fused with the sample predicted group category information output by the sound group category neural network model to obtain sample fusion feature information;

[0020] The sample fusion feature information is input into the decoder of the sound group category neural network model, and the sample fusion feature information is decoded based on the attention mechanism and the group category representation matrix to obtain the sample sound event judgment result.

[0021] The sound event category judgment model is updated in reverse based on the real sound events corresponding to the sound sample data and the judgment results of the sample sound events.

[0022] As a further improvement to this application, the step of constructing group category labels based on the sound sample data includes:

[0023] Feature extraction is performed on the sound sample data to obtain the basic sound features;

[0024] Frame-level spectral averaging is performed on the basic sound features to obtain coarse sound feature information;

[0025] The coarse feature information is clustered to obtain at least one cluster, and each cluster corresponds to a sound group category;

[0026] The group category label is constructed based on the group category corresponding to each cluster.

[0027] As a further improvement to this application, before performing feature extraction on the sound sample data to obtain the basic sound features, the method further includes:

[0028] Perform data augmentation on each of the aforementioned sound sample data.

[0029] As a further improvement to this application, the data augmentation operation includes at least one of time-domain warping, time-domain masking, and frequency-domain masking.

[0030] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a sound event detection device, including: an acquisition module for acquiring the sound to be detected;

[0031] The detection module inputs the sound to be detected into a trained sound group category neural network model to obtain sound group category information;

[0032] The fusion module inputs the sound to be detected into the encoder of a pre-trained sound event category judgment model to obtain fine-grained feature information, and then concatenates and fuses the group category information with the fine-grained feature information to obtain fused feature information;

[0033] The discrimination module inputs the fused feature information into the decoder of the sound event category judgment model, and decodes the fused feature information based on the attention mechanism and the pre-obtained group category representation matrix to obtain the sound event judgment result.

[0034] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a computer device, the computer device including a processor and a memory coupled to the processor, the memory storing program instructions, and when the program instructions are executed by the processor, causing the processor to perform the steps of any of the above-mentioned sound event detection methods.

[0035] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a storage medium that stores program instructions, which are executed by a processor to implement any of the above-mentioned sound event detection methods.

[0036] The beneficial effects of this application are as follows: The low-resource sound event detection method of this application includes: acquiring the sound to be detected; inputting the sound to be detected into a trained sound group category neural network model to obtain sound group category information; inputting the sound to be detected into the encoder of a pre-trained sound event category judgment model to obtain fine-grained feature information, and concatenating and fusing the group category information and the fine-grained feature information to obtain fused feature information; inputting the fused feature information into the decoder of the sound event category judgment model, and decoding the fused feature information based on an attention mechanism and a pre-obtained group category representation matrix to obtain the sound event judgment result. This invention forms the sound to be detected into large-class sound group category information containing rich information, and achieves auxiliary discrimination based on the large-class sound event result through a self-attention mechanism. Attached Figure Description

[0037] Figure 1 This is a schematic flowchart of the sound event detection method according to an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the model structure of the sound group category neural network model and the sound event category judgment model of the sound event detection method according to an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of the functional modules of the sound event detection device according to an embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0041] Figure 5 This is a schematic diagram of the structure of the storage medium according to an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0043] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0044] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0045] Figure 1 This is a flowchart illustrating the sound event detection method according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the model structure of the sound group category neural network model and the sound event category judgment model of the sound event detection method according to an embodiment of the present invention. It should be noted that if substantially the same result is obtained, the method of this application does not necessarily require further clarification. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, the method includes:

[0046] Step S1: Obtain the sound to be detected.

[0047] Specifically, the research scope of sound event detection includes acquiring sounds from nature, mechanical equipment, plants and animals, and various sounds we encounter in our daily lives.

[0048] Step S2: Input the sound to be detected into the trained sound group category neural network model to obtain sound group category information.

[0049] Specifically, the research scope of sound event detection includes all kinds of sounds from nature, mechanical equipment, plants and animals, and the sounds we encounter in our daily lives. The sound to be detected is input into a trained sound group category neural network model to obtain sound group category information.

[0050] Furthermore, the neural network model for the sound group categories is pre-trained, including:

[0051] Step S21: Obtain sound sample data and construct group category labels based on the sound sample data. The group category labels include the sound group category corresponding to each sound sample data.

[0052] Specifically, sound sample data is acquired, and group category labels are constructed based on the sound sample data. The group category labels include the sound group category corresponding to each sound sample data.

[0053] Step S22: Input the sound sample data into the sound group category neural network model to be trained to obtain the sample prediction group category information.

[0054] Specifically, sound sample data is input into the sound group category neural network model to be trained, so as to achieve stable training of the subsequent sound group category neural network model and obtain sample prediction group category information.

[0055] Step S23: Update the sound group category neural network model in reverse according to the sound group category corresponding to each sound sample data and the sample predicted group category information, so as to train the sound group category neural network model.

[0056] Specifically, the sound group category neural network model adopts an N-layer transformer structure, followed by a global pooling layer, and finally outputs group category information through a feedforward neural network. The output of this neural network serves as the intermediate layer input of the subsequent detection encoder. A fully connected output layer is added to the improved model based on the number of sound group categories and predicted group categories required for each sound sample data. The number of fully connected layers added corresponds to the number of categories. The improved model is the main body, with the fully connected output layers added for the sound group categories and predicted group categories for each sound sample data serving as two training branches. The model is trained alternately using the sound group categories and predicted group categories for each sound sample data. The sound group categories for each sound sample data are trained using the cross-entropy loss function, while the predicted group category information is trained using the class-aware loss function. Based on the magnitude of the forward propagation loss value, the sound group category neural network model is updated backward. This backpropagation iteratively updates the weights of each previous layer until the model's loss value converges. At this point, training stops, and the added output layers are removed.

[0057] Further, group category labels are constructed based on the sound sample data, including:

[0058] Step S211: Extract features from the sound sample data to obtain basic sound features.

[0059] Specifically, the sound sample data refers to continuously changing sound data collected over time, used for model training. This sound sample data includes clean sound data and clean noise data. The ratio of clean sound data to clean noise data in the sound sample data is 1:1, meaning that obtaining an equal proportion of clean sound data and clean noise data effectively prevents overfitting during model training, resulting in more accurate recognition performance from the model trained using the sound sample data. In this case, the sound sample data can be in WAV, MP3, or other formats.

[0060] Step S212: Perform frame-level spectral averaging on the basic sound features to obtain coarse sound feature information.

[0061] Specifically, frame-level spectral averaging is performed on the basic sound features to obtain coarse sound feature information. Among them, filter-bank (Fbank) features are commonly used sound features in the sound recognition process. Since the commonly used Mel features undergo dimensionality reduction during model training or recognition, resulting in the loss of some information, filter features are used in this case instead of commonly used Mel features to avoid the above problems, which can help improve the accuracy of subsequent model recognition.

[0062] The Mel filter bank refers to passing the energy spectrum of the Fast Fourier Transform (FFT) output (i.e., the spectrum of the sound data under test) through a set of Mel-scale triangular filters to smooth the spectrum and eliminate filtering effects. This can highlight the formant characteristics of the sound and reduce the computational load. Then, the logarithmic energy of the output of each triangular filter in the Mel filter bank is calculated, and this logarithmic energy is the sound characteristic of the filter under test.

[0063] Step S213: Cluster the coarse feature information to obtain at least one cluster, and each cluster corresponds to a sound group category.

[0064] Specifically, frame-level spectral averaging is performed on the basic sound features to obtain coarse sound feature information. This coarse feature information is then clustered to obtain at least one cluster, with each cluster corresponding to a sound group category. It should be noted that, based on the spectrum, similar spectral sound events are clustered to obtain at least one cluster.

[0065] Assume that the sentence-level feature information of a certain sound event is X. i =(x i0 ,x i1 …x in ), where x i,j For the Fbank features of the j-th frame of sample i, the sentence-level features are obtained by averaging the frame-level features of the current audio sample data. i w j Based on the basic features, define the similarity metric function d(w) in spectral clustering. i ,w j The formula is as follows:

[0066] Meanwhile, the affine matrix A is defined as:

[0067] in,

[0068] The corresponding degree matrix D is defined as follows:

[0069]

[0070] in

[0071] Construct the Laplacian matrix L,

[0072]

[0073] By performing eigenvalue decomposition on L, the number of K largest eigenvalues ​​is selected as the final number of clusters. The eigenvectors v1, v2, ..., v corresponding to the K largest eigenvalues ​​are then used to form the eigenvectors. k Standardized to e by row i =[v1,v2,…,v k Then, the k-means algorithm is used to cluster all features to obtain the cluster labels for each sound sample data. Assuming k categories are obtained, for each sound sample data, the distance between it and the center of each category is calculated, and the category with the closest distance is assigned to the current sound sample data. This category is then used to label the sound sample data.

[0074] Step S214: Construct the group category label based on the group category corresponding to each cluster.

[0075] Specifically, group category labels are constructed based on the group category corresponding to each cluster. This addresses the sparsity problem of some low-resource categories and enables stable training of the subsequent sound group category neural network model. The group category labels for each sound event sample are generated in 0-1 one-hot format, and their values ​​are compared with the predicted output value of the neural network after normalization using the softmax function. The cross-entropy function is defined as follows:

[0076]

[0077] Where N is the total number of samples, C is the total number of group categories, and y i c represents the group class label of sample i. j Represents a group category label, p(c j |X i ) represents the corresponding input feature X i c j Posterior probability predictions for the group class.

[0078] Furthermore, before extracting features from the sound sample data to obtain the basic sound features, the process also includes:

[0079] Perform data augmentation on each of the aforementioned sound sample data.

[0080] Specifically, due to the limited amount of low-resource audio data, the lack of diverse samples under different conditions in the audio training samples makes the audio group category neural network model prone to overfitting. To improve the modeling performance of the audio group category neural network model, we need to enrich the sample diversity, enabling the model to have a certain generalization ability for information loss, time-frequency shifts, and new categories. This is mainly achieved through data augmentation based on audio spectrograms, performing data augmentation operations on each audio sample.

[0081] Furthermore, the data augmentation operation includes at least one of time-domain warping, time-domain masking, and frequency-domain masking.

[0082] Specifically, the data augmentation operation includes at least one of time-domain bending, time-domain masking, and frequency-domain masking. It should be noted that time-domain bending involves randomly compressing and stretching each spectral segment while keeping the pitch unchanged, with a bending scale of up to 5%, which means changing the speed of the original sound by 0.95-1.05 times.

[0083] Temporal masking: For continuous frame spectra, a masking operation is performed, setting all spectral information to zero, conforming to a uniform distribution of (where (the time mask parameter)). The temporal masking operation is performed at 2-second intervals based on the length of the sound event.

[0084] Frequency domain mask: The frequency domain mask is applied to the frequency domain axis and zeros out consecutive frequency domain channels, where the uniform distribution of (where is the frequency domain mask parameter) is selected, where is the number of Mel filters.

[0085] Step S3: Input the sound to be detected into the encoder of the pre-trained sound event category judgment model to obtain fine-grained feature information, and then concatenate and fuse the group category information with the fine-grained feature information to obtain fused feature information.

[0086] Specifically, the sound to be detected is input into the encoder of a pre-trained sound event category judgment model to obtain fine-grained feature information. Then, the sound group category information is concatenated and fused with the fine-grained feature information to obtain fused feature information. Assume the output of the sound group category neural network model is g. i The encoder output value of the sound event category judgment model is e. i =(e i0 ,e i1 …e in The two features are merged using a splicing method for output.

[0087] h t =Concat(e it ,g i )

[0088] Where Concat represents the concatenation operation, h t This represents the hidden layer feature of the current sample in frame t. For details, please refer to [link to relevant documentation]. Figure 2 , Figure 2 This diagram illustrates the model structure of the sound group category neural network model and the sound event category judgment model in this embodiment. The sound group category neural network model, as shown in Stage-1 (Class Encoder), comprises an encoder implemented with N Transformer layers, a global pooling layer, and a feedforward neural network, all connected in sequence. The sound event category judgment model consists of an encoder (Detection Encoder) and a decoder (Detection Decoder). The encoder comprises an encoder implemented with N Transformer layers and an intermediate layer (Fusion), while the decoder comprises a decoder implemented with N Transformer layers, a global pooling layer, and a feedforward neural network, all connected in sequence. During sound event detection, the output of the feedforward neural network of the sound group category neural network model is used as the input to the intermediate layer of the subsequent sound event category judgment model encoder (Detection Encoder). This input is repeatedly concatenated and fused with the fine-grained feature information extracted by the encoder (implemented with N Transformer layers) of the sound event category judgment model before sound event category judgment.

[0089] Step S4: Input the fused feature information into the decoder of the sound event category judgment model, and decode the fused feature information based on the attention mechanism and the pre-obtained group category representation matrix to obtain the sound event judgment result.

[0090] Specifically, the sound fusion feature information is input into the decoder of the sound event category judgment model. Based on an attention mechanism, the sound fusion feature information is decoded in conjunction with a pre-obtained group category representation matrix to obtain the sound event judgment result. This judgment result information refers to the specific category of the sound event; for example, in a bird song category judgment task, it could refer to specific species such as "oriole" or "crow". To introduce more group category information to assist in the judgment of the current sound event sample, a group category representation matrix is ​​first constructed through a group category feature extraction network. Then, an attention method is used to achieve fast information matching of the current sample feature information on the group category representation matrix.

[0091] α l,t =dot(W q u l W k h t )

[0092]

[0093]

[0094] Where u l W represents the feature representation of the corresponding group category l. q and W k These are trainable parameters. l,t m represents the weight value of the feature corresponding to category l at the current time. l This is the weighted feature output. Finally, after global pooling, a feedforward neural network layer, and a softmax layer, class determination is performed, using the cross-entropy function as the loss function. Where N is the total number of samples, C is the total number of group categories, and y i c represents the group class label of sample i. j Represents a group category label, p(c j |X i ) represents the corresponding input feature X i c j Posterior probability predictions for the group class.

[0095] Furthermore, a group category representation matrix is ​​pre-constructed, including:

[0096] Step S41: Input the sound sample data into the trained sound group category neural network model to obtain the group category feature vector representation corresponding to each sound sample data.

[0097] Specifically, sound sample data is input into a trained sound group category neural network model to obtain the group category feature vector representation corresponding to each sound sample data. Based on the intermediate features of the latest sound group category neural network model, after the sound group category neural network model has completed training, the output of the feedforward neural network of the sound group category neural network model is directly used as the group category feature vector representation. Assuming the output of the sound group category neural network model is g... i .

[0098] Step S42: Perform a sampling average operation on the group category feature vector representations corresponding to the sound sample data belonging to the same group category to obtain the group category vector for each sound group category.

[0099] Specifically, sound sample data is input into a trained sound group category neural network model to obtain the group category feature vector representation corresponding to each sound sample data. Based on the intermediate features of the latest sound group category neural network model, after the sound group category neural network model has completed training, the output of the feedforward neural network of the sound group category neural network model is directly used as the group category feature vector representation. The group category feature vector representations corresponding to the sound sample data belonging to the same group category are sampled and averaged to obtain the group category vector of each sound group category.

[0100] Step S43: Concatenate all group category vectors to obtain the group category representation matrix.

[0101] Specifically, sound sample data is input into a trained sound group category neural network model to obtain the group category feature vector representation corresponding to each sound sample data. Based on the intermediate features of the latest sound group category neural network model, after the sound group category neural network model has completed training, the output of the feedforward neural network of the sound group category neural network model is directly used as the group category feature vector representation. The group category feature vector representations corresponding to the sound sample data belonging to the same group category are sampled and averaged to obtain the group category vector of each sound group category. All group category vectors are concatenated to obtain the group category representation matrix.

[0102] Furthermore, the sound event category judgment model is pre-trained, including:

[0103] Step S411: Input the sound sample data into the sound event category judgment model to be trained, and the encoder of the sound event category judgment model extracts the fine-grained features of the sample from the sound sample data.

[0104] Specifically, sound sample data is input into the sound event category judgment model to be trained. The encoder of the sound event category judgment model extracts fine-grained features from the sound sample data. The fine-grained features refer to the specific category of the sound event. For example, in the bird song category judgment task, it can refer to specific species such as "oriole" and "crow".

[0105] Step S412: The fine-grained features of the sample are concatenated and fused with the sample predicted group category information output by the sound group category neural network model to obtain sample fusion feature information.

[0106] Specifically, the fine-grained features of the samples are concatenated and fused with the sample predicted group category information output by the sound group category neural network model to obtain sample fused feature information. It is assumed that the output of the sound group category neural network model is g. i The encoder output value of the sound event category judgment model is e. i =(ei0 ,e i1 …e in The two features are merged using a splicing method for output.

[0107] h t =Concat(e it ,g i )

[0108] Where Concat represents the concatenation operation, h t This represents the hidden layer feature of the current sample in frame t.

[0109] Step S413: Input the sample fusion feature information into the decoder of the sound group category neural network model, and decode the sample fusion feature information based on the attention mechanism and the group category representation matrix to obtain the sample sound event judgment result.

[0110] Specifically, the fine-grained features of the samples are concatenated and fused with the sample prediction group category information output by the sound group category neural network model to obtain sample fusion feature information. The sample fusion feature information is then input into the decoder of the sound group category neural network model, and the sample fusion feature information is decoded based on the attention mechanism and combined with the group category representation matrix to obtain the sample sound event judgment result.

[0111] Step S414: Update the sound event category judgment model in reverse based on the real sound events corresponding to the sound sample data and the judgment results of the sample sound events.

[0112] Specifically, the sound event category judgment model is updated in reverse based on the real sound events corresponding to the sound sample data and the judgment results of the sample sound events. The model is trained alternately using the real sound events corresponding to the sound sample data and the judgment results of the sample sound events. The real sound events corresponding to each sound sample data are trained using the cross-entropy loss function, and the judgment results of the sample sound events are trained using the class-aware loss function. The sound event category judgment model is updated in reverse based on the magnitude of the forward propagation loss value. This process is repeated to train the sound event category judgment model and iteratively update the weights of each previous layer until the model's loss value tends to converge, at which point the training of the model stops.

[0113] The sound event detection method of this invention acquires the sound to be detected, inputs the sound to be detected into a trained sound group category neural network model to obtain sound group category information, inputs the sound to be detected into the encoder of a pre-trained sound event category judgment model to obtain fine-grained feature information, and concatenates and fuses the group category information and fine-grained feature information to obtain fused feature information. The fused feature information is input into the decoder of the sound event category judgment model, and decoded based on an attention mechanism and a pre-obtained group category representation matrix to obtain the sound event judgment result. This invention forms the sound to be detected into large-category sound group category information containing rich information, and uses a self-attention mechanism to achieve auxiliary discrimination based on the results of large-category sound events. Clustering is used to aggregate and enhance similar sound events, forming large-category groups containing rich information, and data augmentation is used to improve the noise robustness of single-category sound events.

[0114] Figure 3 This is a functional module diagram of the sound event detection device according to an embodiment of this application. Figure 3 As shown, the sound event detection device 2 includes an acquisition module 21, a detection module 22, a fusion module 23, and a discrimination module 24.

[0115] Acquisition module 21 is used to acquire the sound to be detected;

[0116] Detection module 22 is used to input the sound to be detected into a trained sound group category neural network model to obtain sound group category information;

[0117] The fusion module 23 is used to input the sound to be detected into the encoder of a pre-trained sound event category judgment model to obtain fine-grained feature information, and to concatenate and fuse the group category information with the fine-grained feature information to obtain fused feature information;

[0118] The discrimination module 24 is used to input the fused feature information into the decoder of the sound event category judgment model, and decode the fused feature information based on the attention mechanism and in combination with the pre-obtained group category representation matrix to obtain the sound event judgment result.

[0119] Optionally, the neural network model for the sound group categories is pre-trained, including:

[0120] Acquire sound sample data and construct group category labels based on the sound sample data. The group category labels include the sound group category corresponding to each sound sample data.

[0121] The sound sample data is input into the sound group category neural network model to be trained to obtain the sample predicted group category information;

[0122] The sound group category neural network model is updated in reverse based on the sound group category corresponding to each sound sample data and the sample predicted group category information, so as to train the sound group category neural network model.

[0123] Optionally, a group category representation matrix is ​​pre-constructed, including:

[0124] The sound sample data is input into a trained sound group category neural network model to obtain a group category feature vector representation corresponding to each sound sample data.

[0125] The group category feature vector representations corresponding to sound sample data belonging to the same group category are sampled and averaged to obtain the group category vector for each sound group category;

[0126] By concatenating all group category vectors, the group category representation matrix is ​​obtained.

[0127] Optionally, the sound event category judgment model is pre-trained, including:

[0128] The sound sample data is input into the sound event category judgment model to be trained, and the encoder of the sound event category judgment model extracts fine-grained features from the sound sample data.

[0129] The fine-grained features of the samples are concatenated and fused with the sample predicted group category information output by the sound group category neural network model to obtain sample fusion feature information;

[0130] The sample fusion feature information is input into the decoder of the sound group category neural network model, and the sample fusion feature information is decoded based on the attention mechanism and the group category representation matrix to obtain the sample sound event judgment result.

[0131] The sound event category judgment model is updated in reverse based on the real sound events corresponding to the sound sample data and the judgment results of the sample sound events.

[0132] Optionally, constructing group category labels based on the sound sample data includes:

[0133] Feature extraction is performed on the sound sample data to obtain the basic sound features;

[0134] Frame-level spectral averaging is performed on the basic sound features to obtain coarse sound feature information;

[0135] The coarse feature information is clustered to obtain at least one cluster, and each cluster corresponds to a sound group category;

[0136] The group category label is constructed based on the group category corresponding to each cluster.

[0137] Optionally, before performing feature extraction on the sound sample data to obtain the basic sound features, the method further includes:

[0138] Perform data augmentation on each of the aforementioned sound sample data.

[0139] Optionally, the data augmentation operation includes at least one of time-domain warping, time-domain masking, and frequency-domain masking.

[0140] For further details regarding the implementation techniques of each module in the sound event detection device of the above embodiments, please refer to the description of the sound event detection method in the above embodiments, which will not be repeated here.

[0141] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system-type embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0142] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of this application. Figure 4 As shown, the computer device 30 includes a processor 31 and a memory 32 coupled to the processor 31.

[0143] The memory 32 stores program instructions, which, when executed by the processor 31, cause the processor 31 to perform the steps of the sound event detection method in the above embodiment.

[0144] The processor 31 can also be referred to as a CPU (Central Processing Unit). The processor 31 may be an integrated circuit chip with signal processing capabilities. The processor 31 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0145] See Figure 5 , Figure 5This is a schematic diagram of the structure of the storage medium in an embodiment of this application. The storage medium in this embodiment stores program instructions 41, which are executed by processor 31 to implement the sound event detection method described above. The computer program 41 can be stored in the storage medium as a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or computer devices such as computers, servers, mobile phones, and tablets. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0147] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method of sound event detection, characterized by, The method comprises the following steps: acquiring a sound to be detected; inputting the sound to be detected into a trained sound group category neural network model to obtain sound group category information; inputting the sound to be detected into an encoder of a pre-trained sound event category judgment model to obtain fine-grained feature information, and splicing and fusing the group category information and the fine-grained feature information to obtain fused feature information; inputting the fused feature information into a decoder of the sound event category judgment model, and decoding the fused feature information based on an attention mechanism and in combination with a pre-obtained group category representation matrix to obtain a sound event judgment result; pre-training the sound group category neural network model, comprising: acquiring sound sample data, and constructing a group category label according to the sound sample data, wherein the group category label comprises a sound group category corresponding to each sound sample data; inputting the sound sample data into a sound group category neural network model to be trained to obtain sample predicted group category information; updating the sound group category neural network model in a reverse direction according to the sound group category corresponding to each sound sample data and the sample predicted group category information, so as to train the sound group category neural network model; pre-training the sound event category judgment model, comprising: inputting the sound sample data into a sound event category judgment model to be trained, and extracting sample fine-grained features from the sound sample data by an encoder of the sound event category judgment model; splicing and fusing the sample fine-grained features and sample predicted group category information output by the sound group category neural network model to obtain sample fused feature information; inputting the sample fused feature information into a decoder of the sound group category neural network model, and decoding the sample fused feature information based on an attention mechanism and in combination with the group category representation matrix to obtain a sample sound event judgment result; updating the sound event category judgment model in a reverse direction according to a real sound event corresponding to the sound sample data and the sample sound event judgment result; the method further comprises the following steps before the step of acquiring the sound to be detected: performing a data augmentation operation on each sound sample data. ​ ​ ​ 2. The voice event detection method of claim 1, wherein, ​ ​ ​ ​ 3. The voice event detection method of claim 1, wherein, ​ ​ 4. The sound event detection method of claim 3, wherein, The data augmentation operation includes at least one of time domain bending, time domain masking, and frequency domain masking.

5. A sound event detection apparatus for implementing the sound event detection method according to any one of claims 1 to 4, characterized by It comprises: An acquisition module acquires a sound to be detected; A detection module inputs the sound to be detected into a trained sound group category neural network model to obtain sound group category information; A fusion module inputs the sound to be detected into an encoder of a pre-trained sound event category judgment model to obtain fine-grained feature information, and splices and fuses the group category information and the fine-grained feature information to obtain fusion feature information; A discrimination module inputs the fusion feature information into a decoder of the sound event category judgment model, and decodes the fusion feature information based on an attention mechanism and in combination with a pre-obtained group category representation matrix to obtain a sound event judgment result.

6. A computer device, comprising: The computer device comprises a processor, a memory coupled to the processor, and the memory stores program instructions, which are executed by the processor to cause the processor to perform the steps of the sound event detection method according to any one of claims 1-4.

7. A storage medium, characterized by The memory stores program instructions capable of implementing the sound event detection method according to any one of claims 1-4.

Images