Acoustic event detection model training and detection method based on multi-stage knowledge distillation and pseudo-label dynamic correction

Abstract

The invention discloses an acoustic event detection model training and detection method based on multi-stage knowledge distillation and pseudo-label dynamic correction, and belongs to the technical field of acoustic event detection. The objective of the invention is to solve the problem of poor detection accuracy and robustness of model-based detection in acoustic event detection with a very small number of acoustic event samples. In the first stage, supervised learning and self-supervised learning are performed on the teacher network; setting the parameter of the expert network as the index moving average of the parameter of the teacher network, and performing self-supervision loss reverse fine tuning on the teacher network between the output of the expert network and the output of the teacher network; in the second stage, unsupervised learning based on pseudo labels is carried out on the student network, and the expert network is utilized to generate the pseudo labels to train the student network; and in the third stage, the teacher network performs unsupervised learning, and the student network is utilized to generate a pseudo tag to finely adjust the teacher network, so that the final teacher network trained based on the knowledge distillation mode is obtained. The teacher network is used for actual acoustic event detection.

Description

Technical Field

[0001] This invention belongs to the field of acoustic event detection technology, and relates to a training method for an acoustic event detection model and its detection method. Background Technology

[0002] Sound is one of the most common and natural information carriers in human daily life, permeating all aspects of society and the environment, carrying rich emotional and semantic information. Sound not only helps humans understand their environment and perceive the world, but also plays an irreplaceable role in communication, interaction, and cognition. With the rapid development of information technology, research simulating human auditory cognitive mechanisms has gradually emerged and quickly become an important direction in the international field of artificial intelligence and cognitive computing. Furthermore, the State Council's "New Generation Artificial Intelligence Development Plan" explicitly lists auditory perception and computation in acoustic scenarios as a key future research direction, demonstrating the strategic importance and development potential of this field.

[0003] The ultimate goal of sound perception and understanding is to enable computers to possess auditory cognitive abilities similar to humans, allowing them to identify and analyze diverse sound sources in the environment, and even overcome the limitations of human hearing to achieve deeper semantic understanding. Generally speaking, sound can be divided into two main categories: speech and non-speech. Compared to the rapid development in speech recognition and understanding, research on non-speech sound perception is still in its relatively early stages. However, since the beginning of the 21st century, non-speech sound analysis has gradually become a frontier in auditory computing research, with two core directions—Sound / Acoustic / Audio Event Detection (SED) and Acoustic Scene Classification (ASC)—receiving widespread attention. The former focuses on identifying specific sound source activities in audio, while the latter focuses on the acoustic characteristics and categories of the overall environment. In comparison, acoustic event detection can provide richer, finer-grained information at the temporal and frequency levels, thus occupying a more crucial position in sound understanding research.

[0004] Based on the overlap of sound sources, acoustic event detection can be divided into monophonic SED and polyphonic SED. Given the prevalence of multiple sound sources in real life, polyphonic SED more closely reflects the acoustic characteristics of the real world, but it also faces greater technical challenges. The overlap and interference of multiple acoustic events make event recognition and boundary localization more complex. The research aims to analyze auditory scenes with interwoven multiple sound sources, uncover the implicit structures and patterns between sound signals, and thus improve the robustness and generalization ability of computer auditory systems.

[0005] At the task level, overlapping acoustic event detection typically includes two core objectives: first, identifying multiple types of acoustic events coexisting in audio segments (audio tagging); and second, accurately predicting the start and end times of each event (boundary detection). Traditional boundary detection relies on large amounts of precisely labeled audio data, which must clearly indicate the event type and its temporal boundaries. However, acquiring this data is costly and heavily influenced by subjective human factors. In contrast, weakly labeled and softly labeled data can be obtained at a lower cost, but their temporal and category information is often incomplete or biased. In complex scenarios with multiple overlapping sound sources, scarce annotations, and even noisy annotations, in-depth research into acoustic event detection is particularly necessary. Such research helps to explore the potential structure and semantic relationships of sound signals from the perspective of acoustics, providing new theoretical basis and learning paradigms for machine hearing. This not only has profound academic significance but also demonstrates enormous development potential and application value in practical applications such as security monitoring, smart cities, and autonomous driving.

[0006] In acoustic event detection with a very small number of acoustic event samples, the limited sample size is insufficient for training an acoustic event detection model. Models trained with a small number of samples exhibit poor detection accuracy and robustness in actual acoustic event detection. Even when using generative models to generate samples for training, the samples generated by these models cannot be accurately labeled, making it difficult to obtain acoustic event detection models with both high accuracy and robustness. Summary of the Invention

[0007] This invention addresses the problem that model-based detection suffers from poor accuracy and robustness in acoustic event detection when the number of acoustic event samples is very small.

[0008] A training method for an acoustic event detection model with multi-stage knowledge distillation and dynamic pseudo-label correction includes:

[0009] In the first stage, supervised learning and self-supervised learning are carried out on the teacher network; the parameters of the expert network are set as the exponential moving average of the teacher network parameters, i.e., EMA; the expert network is input with strong and weak labeled data and obtains the corresponding output; then the teacher network is finely adjusted by self-supervised loss between the output of the expert network and the output of the teacher network.

[0010] In the second stage, unsupervised learning based on pseudo-labels is carried out on the student network, and pseudo-labels generated by the expert network are used to train the student network.

[0011] In the third stage, the teacher network undergoes unsupervised learning, and pseudo-labels generated by the student network are used to fine-tune the teacher network, resulting in the final teacher network trained based on knowledge distillation.

[0012] The teacher network, expert network, and student network are all neural network models with the same structure.

[0013] Furthermore, the specific training process in the first phase includes:

[0014] Supervised and self-supervised learning are performed on the teacher network. The parameters of the expert network are frozen. Strongly labeled and weakly labeled data are input into both the teacher and expert networks. The teacher network outputs frame-level and fragment-level features. The teacher network updates its parameters based on the supervised loss between these two outputs and the labels. Then, the parameters of the expert network are set to the exponential moving average (EMA) of the teacher network parameters. Since the expert network has also previously received strongly labeled and weakly labeled data, it outputs corresponding frame-level and fragment-level features. Finally, the teacher network is fine-tuned by applying a self-supervised loss between the two outputs of the expert network and the two outputs of the teacher network.

[0015] Furthermore, the process by which the teacher network obtains frame-level feature output and fragment-level feature output includes:

[0016] A1. The three models, namely Teacher Network, Student Network, and Expert Network, all use the CRNN structure, which processes the frame-level temporal feature sequence v by the CRNN. t Perform linear feature transformation to obtain feature sequence s t ;

[0017] A2, s t Aggregate into a fragment-level feature vector u:

[0018] (8)

[0019] in The symbol represents element-wise multiplication, T is the number of time steps in the partition, and the weight matrix is... It is obtained by applying a linear feature transformation and a softmax function to the output of the RNN network.

[0020] Furthermore, in the first stage, the loss function for supervised and self-supervised learning of the teacher network includes the supervised loss of the teacher network under strong and weak labels, the self-supervised loss of the teacher network under strong and weak labels, and the time-order regularization term loss; the calculation process of the time-order regularization term loss is as follows:

[0021] First, the fragment-level feature vector u and the frame-level feature vector v are... t Perform feature dimension alignment; then output each frame of the expert network's frame-level features. With the next frame The difference between the segments is calculated, and then the similarity is calculated by multiplying the segment-level feature output u of the teacher network by a dot product. Finally, the summation is performed to obtain the temporal regularization term. .

[0022] Furthermore, the supervision loss of the teacher network under strong and weak labels adopts cross-entropy loss, and the self-supervision loss of the teacher network under strong and weak labels adopts mean squared error loss.

[0023] Furthermore, the specific training process in the second phase includes:

[0024] The student network undergoes unsupervised learning based on pseudo-labels, while the parameters of the teacher network are frozen. Before the start of the second phase of training, the first phase of training yielded a trained teacher network and an expert network. The expert network is used to guide the student network, with unlabeled data input into both the student and expert networks. The output of the expert network is used as the pseudo-label for training the student network. The pseudo-labels generated by the expert network and the output of the student network are used to calculate the unsupervised learning loss based on pseudo-labels, and the parameters of the student network are updated using the objective function.

[0025] Furthermore, in the second stage, the objective function for the student network to perform unsupervised learning based on pseudo-labels includes the loss of pseudo-labels generated by the student network under strong and weak labels, with cross-entropy loss being used.

[0026] Furthermore, the specific training process of the third stage includes:

[0027] The teacher network parameters are unfrozen, while the student network parameters are frozen. The teacher network undergoes unsupervised learning based on pseudo-labels. Unlabeled data is simultaneously input into both the teacher and student networks. The output of the student network is used as the pseudo-label for training the teacher network. The pseudo-labels generated by the student network and the output of the teacher network are used to perform unsupervised learning loss based on pseudo-labels, thereby fine-tuning the teacher network.

[0028] Furthermore, in the third stage, the objective function of the teacher network's unsupervised learning based on pseudo-labels includes the loss of pseudo-labels generated by the teacher network under strong and weak labels, with cross-entropy loss being used.

[0029] An acoustic event detection method based on multi-stage knowledge distillation and dynamic pseudo-label correction is proposed. The method uses the aforementioned multi-stage knowledge distillation and dynamic pseudo-label correction acoustic event detection model training method to obtain a teacher network after a third stage, and then uses the obtained teacher network for acoustic event detection.

[0030] Beneficial effects:

[0031] This invention is suitable for acoustic event detection with very few samples, and it can also effectively train and optimize the model, thereby improving the model's detection accuracy and robustness. After adding a multi-stage distillation architecture and a temporal correlation aggregation network, the PSDS1 score of the teacher model is 0.4845, and the PSDS1(sed eval) score of the teacher model is 0.5008. Experiments show that this method improves both PSDS1 and PSDS1(sed eval) evaluation metrics, proving that this method can effectively improve the system's performance. Attached Figure Description

[0032] Figure 1 This is a schematic diagram illustrating the training process of an acoustic event detection model with multi-stage knowledge distillation and dynamic correction of pseudo-labels.

[0033] Figure 2 This is a schematic diagram illustrating the construction process of a time-series-dependent aggregation structure.

[0034] Figure 3 The corresponding model effect diagram of the method described in this invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] In this embodiment of the invention, the teacher model is trained using strongly labeled data in the first stage of model training. Then, the parameters of the expert model are set to the EMA of the trained teacher model parameters. In the second stage, the student model is trained using unlabeled data, while pseudo-labels are generated using the already trained expert model. In the third stage, unlabeled data is also used to fine-tune the teacher and expert models; that is, pseudo-labels are generated using the student model trained in the second stage to fine-tune the teacher and expert models. In all three models, a temporal correlation aggregation network is used to aggregate frame-level feature vectors into fragment-level feature vectors.

[0037] Specific implementation method one: Combining Figure 1 This implementation method is described below.

[0038] The acoustic event detection method with multi-stage knowledge distillation and dynamic correction of pseudo-labels described in this embodiment includes the construction process of the temporal correlation aggregation structure and the training process (training method) of the knowledge distillation architecture, as well as the model processing process; wherein the teacher model after three stages of training is reflected in the processing process and detection method in actual use.

[0039] The construction of the time-series correlation aggregation structure includes the following steps:

[0040] A1. The frame-level temporal feature sequence v t Perform linear feature transformation to obtain s t ;

[0041] A2, [The following is a description of feature s] t Finally, the feature vectors are aggregated into fragment-level feature vectors u. The first two steps construct the temporal representation aggregation network.

[0042] A3. Using u and v t The inner product is performed between them, and then the summation is performed at the frame level to obtain the temporal regularization term. The temporal representation aggregation network and the temporal regularization term together construct the temporal correlation aggregation structure.

[0043] The training of the knowledge distillation architecture includes the following steps:

[0044] S1. In the first stage, the teacher model performs supervised learning and self-supervised learning, and sets the parameters of the expert model to the exponential moving average (EMA) of the teacher model parameters.

[0045] S2. In the second stage, the student model performs unsupervised learning based on pseudo-labels, using expert models to generate pseudo-labels to train the student model.

[0046] In the third stage (S3), the teacher model undergoes unsupervised learning, and pseudo-labels generated by the student model are used to fine-tune the teacher model, ultimately completing the construction of the knowledge distillation architecture.

[0047] More specifically, the embodiments of the present invention employ three networks: teacher, expert, and student. All three networks adopt the CRNN network structure. The specific implementation of the knowledge distillation architecture and temporal correlation set structure of the present invention is as follows.

[0048] like Figure 1 The knowledge distillation architecture shown includes the following steps:

[0049] S1. In the first stage, the teacher model performs supervised and self-supervised learning. The parameters of the expert model are frozen. Strongly labeled and weakly labeled data (in the field of acoustic event detection, they are divided into strongly labeled, weakly labeled, and unlabeled, etc.) are input into the teacher model and the expert model. The teacher model obtains frame-level feature output and fragment-level feature output. Then, the teacher model updates its parameters by performing supervised loss based on these two outputs and the labels. Then, the parameters of the expert model are set to the exponential moving average (EMA) of the teacher model parameters. Since the expert model (which does not participate in training itself) also inputs strongly labeled and weakly labeled data, it obtains corresponding frame-level feature output and fragment-level feature output. Then, the teacher model is fine-tuned by performing self-supervised loss between the two outputs of the expert model and the two outputs of the teacher model (self-supervised learning). During the first stage of training, since the performance of both the teacher model and the expert model is poor in the early stage of training, supervised learning plays a dominant role and self-supervised learning plays a secondary role. Here, we set a weight w for supervised learning and self-supervised learning. The weight ranges from 0 to 1 and increases with the number of training rounds.

[0050] The objective function for teacher-supervised learning is as follows:

[0051] (1)

[0052] In the formula The supervisory loss of teachers in online networks under strong labeling, For the supervisory loss of teacher network T under weak labeling, For the self-monitoring loss of teacher network T under strong labels, For the self-monitoring loss of teacher network T under weak labels, The time regularization term is implemented at A3. The input of the time regularization term is the fragment-level feature output of the teacher model and the frame-level feature output of the expert model. In addition, the specific calculation formulas of each loss function (formula (4) and formula (5)) will be explained later.

[0053] S2. In the second stage, the student model performs unsupervised learning based on pseudo-labels, while the parameters of the teacher model are frozen. Before the start of the second stage training, the first stage training yielded two well-performing models (the teacher model and the expert model). Here, we use the expert model to guide the student model. Unlabeled data (the dataset contains strongly labeled data, weakly labeled data, and unlabeled data) is input into the student model and the expert model. The output of the expert model is used as the pseudo-label for training the student model (the expert model also has two outputs, namely, frame-level feature output and fragment-level feature output mentioned above). The pseudo-labels generated by the expert model and the outputs obtained by the student model are used to perform unsupervised learning loss based on pseudo-labels. The parameters of the student model are updated using the following objective function.

[0054] The objective function for unsupervised learning of the student model is as follows:

[0055] (2)

[0056] In the formula The loss of pseudo-labels generated by student network S under strong labeling. The loss of pseudo-labels generated by the student network under weak labeling is calculated in formula (4).

[0057] In the third stage (S3), the parameters of the teacher model are unfrozen, while the parameters of the student model are frozen. The teacher model performs unsupervised learning based on pseudo-labels. Unlabeled data is simultaneously input into both the teacher model and the student model. The output of the student model is used as the pseudo-label for training the teacher model (the student model also has two outputs, namely the frame-level feature output and the fragment-level feature output mentioned above). The pseudo-labels generated by the student model and the output of the teacher model are used to perform unsupervised learning loss based on pseudo-labels, thereby achieving the effect of fine-tuning the teacher model.

[0058] The objective function for unsupervised learning of the teacher model based on pseudo-labels is shown below:

[0059] (3)

[0060] In the formula The loss of pseudo-labels generated by the teacher network T under strong labels. The loss is the pseudo-label generated by the teacher network T under weak labeling. The specific calculation of the two losses is in formula (4).

[0061] Regarding the objective function, it can be divided into several categories based on the learning strategy: supervised learning loss based on strong and weak labels ( and ), self-supervised learning loss based on strong and weak labels ( and Unsupervised learning loss based on pseudo-labels () , , and These losses, although applied in different scenarios, all fall under the umbrella of self-supervised learning losses based on strong and weak labels ( and All others can be represented using a unified mathematical paradigm, namely, by measuring the difference between the model's predictions and the true labels (or pseudo-labels):

[0062] (4)

[0063] This is the cross-entropy loss. For different types of labels, q(x) represents the probability that x predicts a positive sample (the probability that the neural network predicts a certain class). When the label is strong / weak, x is a frame-level / fragment-level feature vector (x is the input to the loss function, i.e., the output of the neural network. The neural network has two outputs: one is a frame-level feature output with temporal features, and the other is a fragment-level feature output after aggregation. When the input label is strong-labeled data, the frame-level feature vector is used as the input for loss, i.e., x is a frame-level feature vector; when the input label is weak-labeled data, the fragment-level feature vector is used as the input for loss, i.e., x is a fragment-level feature vector), p(x). {0,1} are the labels for x; when the labels are pseudo-labels, p(x) {0,1} is the pseudo-label for x. This mathematical representation can constrain the similarity between the prediction results and the labels, as well as between the prediction results at different stages, so that the training and optimization of the model are carried out under a unified mathematical system.

[0064] Self-supervised learning loss based on strong and weak labels ( and The formula represents the mean squared error loss. Specifically, it is expressed as:

[0065] (5)

[0066] Where N is the total number of elements in the sample, The actual value (label). This is the model's predicted value (model output).

[0067] During the experiment involving the first three steps, we found that the teacher model trained in the third stage performed the best. Therefore, in practical applications, we should use the teacher model trained in the third stage.

[0068] Steps S1, S2, and S3 improve the model training process, while A1, A2, and A3 improve the model structure. The loss in the S1-S3 training process is obtained by processing the output of the CRNN model through A1-A3 (only S1 undergoes the time regularization term A3). The following section will combine... Figure 2 Explain the construction process of the time-series correlation aggregation structure, including the following steps:

[0069] A1. The teacher model, student model, and expert model all use the CRNN structure, which processes the frame-level temporal feature sequence v using the CRNN. t Perform linear feature transformation to obtain s t ;where v t The shape is [B,T,2*RNN], where B is the batch size, T is the number of time steps, and the last dimension is twice the number of RNN units;

[0070] Get s t The process: The feature sequence v output by the CRNN t Multiplying by a linear eigenvalue transformation matrix, the result is then shifted by b spaces to finally obtain s. t s t The shape is [B,T,C], where C is the number of categories.

[0071] The formula for linear feature transformation is as follows:

[0072] (6)

[0073] v t This represents the output of the CRNN network. W is the linear feature transformation matrix, and b is the bias vector used to shift the output space. The purpose of this feature transformation is to map the last dimension of the CRNN output to the number of categories.

[0074] A2, [The following is a description of feature s] t The features are aggregated into fragment-level feature vectors u, and A1 and A2 construct a temporal representation aggregation network.

[0075] The feature sequence s obtained from A1 t With weight matrix Performing dot product, where the weight matrix... It is obtained by applying a linear feature transformation and a softmax function to the output of the RNN network. The specific calculation formula is:

[0076] (7)

[0077] Then, summation is performed over the time dimension to obtain an aggregated fragment-level feature vector. To achieve a normalized weighted average, the expected value representing each frame is divided by the weight sum to obtain the fragment-level feature vector u.

[0078] The feature aggregation formula is as follows:

[0079] (8)

[0080] in The symbol represents element-wise multiplication, and T represents the number of time steps. This represents the attention weight, s t The eigenvector is the feature vector obtained after linear feature transformation in step A1.

[0081] A3. Using u and v t The inner product is performed between the components, and then summed at the frame level to obtain the temporal regularization term. The temporal representation aggregation network and the temporal regularization term together construct the temporal correlation aggregation structure. A3 is only used in the first stage of training.

[0082] The input to the temporal regularization algorithm is the segment-level feature vector u and the frame-level feature vector v. t (For example, in the first stage of this invention, u represents the fragment-level feature output of the teacher model, and v...) t It is the frame-level feature output of the expert model. Specifically, each frame of the expert model's frame-level feature output is... With the next frame The similarity is calculated by performing a difference between u and v, and then multiplying the result with the fragment-level feature output u of the teacher model (only positive values ​​are taken). Finally, all similarities are summed to obtain the temporal regularization loss term. In conventional frame-by-frame differencing, each frame requires calculation with neighboring frames, resulting in high time complexity and increased training costs. To address this issue, this invention employs the following implementation process: First, to ensure that u and v... t Aligning the feature dimensions, reshaping the shape of vector u, so that the shape of u becomes [B,1,C], and then... t Transpose the second and third dimensions, v t The shape changes from [B,T,C] to [B,C,T], where B is the batch number, T is the number of time steps, and C is the number of channels. Then, in v... t The last time step is removed from the last dimension T to obtain a new frame-level feature vector embeddings1 with shape [B, C, T-1]. Then v tThe last dimension is truncated by the first time step to obtain a new frame-level feature vector, embeddings2, with the shape [B, C, T-1]. In other words, embeddings1 truncates the last frame, and embeddings2 truncates the first frame. This is done to prepare for the difference between subsequent adjacent frames. Subtracting embeddings2 from embeddings1 yields the adjacent feature difference embeddings3, with the shape [B, C, T-1]. The second and third dimensions of embeddings3 are then transposed, denoted as... Then, the similarity is calculated by multiplying the fragment-level feature output u, and finally, the summation is performed on the last dimension to obtain the temporal regularization term, i.e.:

[0083] (9)

[0084] Where u is the segment-level output of the neural network, and t is the frame-level output of the neural network. In the formula... It involves performing differential operations between adjacent frames, from the first frame to the T-1 frame.

[0085] Example

[0086] This embodiment provides an acoustic event detection method based on multi-stage distillation and dynamic pseudo-label correction. The method consists of three stages during training. The first stage trains a teacher model using strong and weak labels, then sets the parameters of the expert model to the EMA of the teacher model's parameters. The second stage inputs unlabeled data into the expert model, using it to generate pseudo-labels to train a student model. The third stage inputs unlabeled data into the student model, using the trained student model to generate pseudo-labels to fine-tune the teacher model. In this invention, a small amount of strongly and weakly labeled data and a large amount of unlabeled data are used during model training because strongly and weakly labeled data are extremely difficult to obtain in the field of acoustic event detection. To address this difficulty, we decided to first train the model using a small amount of strongly and weakly labeled data, and then use the already trained model to generate pseudo-labels for unlabeled data to train other models, thus achieving the goal of solving the aforementioned problem. Furthermore, a temporal correlation aggregation network is added after the CRNN network, which performs feature transformation, aggregation, and adds temporal regularization terms to the RNN output. The baseline performance is as follows: the PSDS1 score for the teacher model is 0.4391, and the PSDS1(sed eval) score is 0.4690, as shown in Table 1.

[0087] Table 1

[0088]

[0089] After incorporating a multi-stage distillation architecture and a temporal correlation aggregation network, the teacher model achieved a PSDS1 score of 0.4845 and a PSDS1(sed eval) score of 0.5008. Experiments show that this method improves both PSDS1 and PSDS1(sed eval) evaluation metrics, demonstrating its effectiveness in enhancing system performance. Furthermore, when testing the trained model, predicted events are categorized by confidence level, generating files with different confidence levels. Each confidence level file presents the event's start time, end time, and category in a table format. The model performance corresponding to the method described in this invention is as follows: Figure 3 As shown.

[0090] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A training method for an acoustic event detection model with multi-stage knowledge distillation and dynamic pseudo-label correction, characterized in that, include: In the first phase, teachers will undergo supervised learning and self-supervised learning on the network. The parameters of the expert network are set to the exponential moving average (EMA) of the teacher network parameters; the expert network is input with strong and weak labeled data and obtains the corresponding output; then the teacher network is fine-tuned by applying self-supervised loss between the output of the expert network and the output of the teacher network. In the second stage, unsupervised learning based on pseudo-labels is carried out on the student network, and pseudo-labels generated by the expert network are used to train the student network. In the third stage, the teacher network undergoes unsupervised learning, and pseudo-labels generated by the student network are used to fine-tune the teacher network, resulting in the final teacher network trained based on knowledge distillation. The teacher network, expert network, and student network are all neural network models with the same structure.

2. The method for training an acoustic event detection model with multi-stage knowledge distillation and dynamic pseudo-label correction according to claim 1, characterized in that, The specific training process in the first phase includes: Supervised and self-supervised learning are performed on the teacher network. The parameters of the expert network are frozen. Strongly labeled and weakly labeled data are input into both the teacher and expert networks. The teacher network outputs frame-level and fragment-level features. The teacher network updates its parameters based on the supervised loss between these two outputs and the labels. Then, the parameters of the expert network are set to the exponential moving average (EMA) of the teacher network parameters. Since the expert network has also previously received strongly labeled and weakly labeled data, it outputs corresponding frame-level and fragment-level features. Finally, the teacher network is fine-tuned by applying a self-supervised loss between the two outputs of the expert network and the two outputs of the teacher network.

3. The method for training an acoustic event detection model with multi-stage knowledge distillation and dynamic pseudo-label correction according to claim 2, characterized in that, The process by which the teacher network obtains frame-level and fragment-level feature outputs includes: A1. The three models, namely Teacher Network, Student Network, and Expert Network, all use the CRNN structure, which processes the frame-level temporal feature sequence v by the CRNN. t Perform linear feature transformation to obtain feature sequence s t ; A2, s t Aggregate into a fragment-level feature vector u: （8） in The symbol represents element-wise multiplication, T is the number of time steps in the partition, and the weight matrix is... It is obtained by applying a linear feature transformation and a softmax function to the output of the RNN network.

4. The method for training an acoustic event detection model with multi-stage knowledge distillation and dynamic pseudo-label correction according to claim 3, characterized in that, In the first stage, the loss function for supervised and self-supervised learning of the teacher network includes the supervised loss and self-supervised loss of the teacher network under strong and weak labels, as well as the time-order regularization loss; the calculation process of the time-order regularization loss is as follows: First, the fragment-level feature vector u and the frame-level feature vector v are... t Perform feature dimension alignment; then output each frame of the expert network's frame-level features. With the next frame The difference between the segments is calculated, and then the similarity is calculated by multiplying the segment-level feature output u of the teacher network by a dot product. Finally, the summation is performed to obtain the temporal regularization term. .

5. The method for training an acoustic event detection model with multi-stage knowledge distillation and dynamic pseudo-label correction according to claim 4, characterized in that, The supervised loss of the teacher network under strong and weak labels is the cross-entropy loss, and the self-supervised loss of the teacher network under strong and weak labels is the mean squared error loss.

6. The method for training an acoustic event detection model with multi-stage knowledge distillation and dynamic pseudo-label correction according to claim 5, characterized in that, The specific training process in the second phase includes: The student network undergoes unsupervised learning based on pseudo-labels, while the parameters of the teacher network are frozen. Before the start of the second phase of training, the first phase of training yielded a trained teacher network and an expert network. The expert network is used to guide the student network, with unlabeled data input into both the student and expert networks. The output of the expert network is used as the pseudo-label for training the student network. The pseudo-labels generated by the expert network and the output of the student network are used to calculate the unsupervised learning loss based on pseudo-labels, and the parameters of the student network are updated using the objective function.

7. The method for training an acoustic event detection model with multi-stage knowledge distillation and dynamic pseudo-label correction according to claim 6, characterized in that, In the second stage, the objective function for the student network to perform unsupervised learning based on pseudo-labels includes the loss of pseudo-labels generated by the student network under strong and weak labels, and the loss adopts cross-entropy loss.

8. The method for training an acoustic event detection model with multi-stage knowledge distillation and dynamic pseudo-label correction according to any one of claims 5 to 7, characterized in that, The specific training process in the third phase includes: The teacher network parameters are unfrozen, while the student network parameters are frozen. The teacher network undergoes unsupervised learning based on pseudo-labels. Unlabeled data is simultaneously input into both the teacher and student networks. The output of the student network is used as the pseudo-label for training the teacher network. The pseudo-labels generated by the student network and the output of the teacher network are used to perform unsupervised learning loss based on pseudo-labels, thereby fine-tuning the teacher network.

9. The method for training an acoustic event detection model with multi-stage knowledge distillation and dynamic pseudo-label correction according to claim 8, characterized in that, In the third stage, the objective function of the teacher network's unsupervised learning based on pseudo-labels includes the loss of pseudo-labels generated by the teacher network under strong and weak labels, and the loss adopts cross-entropy loss.

10. An acoustic event detection method with multi-stage knowledge distillation and dynamic correction of pseudo-labels, characterized in that, The acoustic event detection model training method of multi-stage knowledge distillation and pseudo-label dynamic correction as described in any one of claims 1 to 9 is used to obtain a teacher network after the third stage, and the obtained teacher network is used for acoustic event detection.

Images