An audio and video parsing method based on noise label learning

Abstract

The application belongs to the technical field of deep learning, and relates to an audio and video parsing method based on noise label learning, which comprises the following steps: preprocessing original audio and video to obtain a segment-level input sequence; constructing a mutual learning noise-resistant double-flow network; training the mutual learning noise-resistant double-flow network according to a training set; comparing the validation set indicators of two sub-networks in the trained mutual learning noise-resistant double-flow network, and taking the sub-network with the larger validation set indicator as an audio and video parsing model; and parsing through the audio and video parsing model according to a test set to obtain a video prediction result. The mutual learning noise-resistant double-flow network is composed of two sub-networks with the same structure but different initializations, a cross filtering mechanism is executed according to the clean masks generated by the two sub-networks during the training of the mutual learning noise-resistant double-flow network, and the dynamic confidence ratio is gradually reduced through a cosine strategy, so that the problems of high pseudo-label noise rate and easy overfitting noise in the existing audio and video parsing task are solved.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and specifically discloses an audio and video parsing method based on noise label learning. Background Technology

[0002] Audio and Video Verification (AVVP) is a weakly supervised multimodal learning task that aims to locate and classify purely audio, purely visual, and combined audio and visual events in videos within seconds, using only video-level labels. In recent years, researchers have frequently used large-scale pre-trained models (such as CLIP and CLAP) to generate pseudo-labels for video clips, transforming the weakly supervised problem into a noisy, fully supervised one. However, due to the inconsistency between the pre-training distribution of large-scale pre-trained models and the class label distribution of downstream tasks, the generated pseudo-labels contain significant noise, with large differences in noise rates across different categories. For example, the noise rate for the category "dog barking" is only 8%, while the noise rate for the category "printer" is as high as 37%.

[0003] Currently, existing methods in AVVP tasks, such as VALOR, typically use pseudo-labels directly as supervision signals for training without effectively handling noise during the training process. While the model can learn clean patterns in the early stages of training, it gradually overfits to noisy samples as training epochs increase, leading to performance degradation. Furthermore, noise handling for pseudo-labels is usually done through pre-processing (washing). Although pre-processing can reduce the noise rate to some extent, it relies on manually setting the washing threshold, which cannot avoid information loss due to under-washing or over-removal. Moreover, the pre-processing step is independent of model training; once residual noise enters the model, the model will still be forced to remember incorrect labels. Therefore, pre-processing cannot adapt to dynamic scenarios with unknown noise rates and is prone to information loss.

[0004] Therefore, how to adaptively identify and suppress pseudo-label noise during training without increasing inference overhead has become the key to improving the performance of the AVVP task. Summary of the Invention

[0005] The purpose of this invention is to provide an audio and video parsing method based on noise label learning, which solves the problems of high pseudo-label noise rate and easy overfitting noise in existing audio and video parsing tasks.

[0006] The specific solution of the present invention is as follows:

[0007] An audio / video parsing method based on noise label learning, comprising:

[0008] The original audio and video are preprocessed to obtain segment-level input sequences, which are then divided into training, validation, and test sets.

[0009] Construct a mutual learning, noise-resistant dual-stream network;

[0010] The mutual learning noise-resistant dual-stream network is trained based on the training set.

[0011] The trained mutual-learning noise-resistant dual-stream network is validated using a validation set. The validation set metrics of two sub-networks in the trained mutual-learning noise-resistant dual-stream network are compared, and the sub-network with the higher validation set metric than the other sub-network is selected as the audio / video parsing model. Based on the test set, the audio / video parsing model is used to parse the video and obtain the video prediction results.

[0012] Preferably, preprocessing the original audio and video to obtain segment-level input sequences includes:

[0013] Multiple segments are obtained by uniformly dividing the original audio and video without overlap by setting a window;

[0014] Based on multiple segments, 768-dimensional visual and audio features are extracted using the CLIP image encoder and the CLAP audio encoder, respectively, to generate segment-level input sequences.

[0015] Preferred, mutually learning, noise-resistant dual-stream networks include:

[0016] Two sub-networks with the same structure but different initializations are used. The sub-networks include a multimodal feature fusion network layer, a fully connected projector head, and a multimodal multi-instance decoding network layer. The multimodal feature fusion network layer is connected to the fully connected projector head and the multimodal multi-instance decoding network layer, respectively.

[0017] Preferably, the mutual learning noise-resistant two-stream network further includes: the initial values ​​of the weights of the two sub-networks are different.

[0018] Preferably, the mutual learning noise-resistant dual-stream network is trained based on the training set, including:

[0019] Mutual learning noise-resistant two-stream network consists of subnetwork A and subnetwork B;

[0020] S1. Based on the training set, obtain the video-level prediction results of subnetwork A and subnetwork B;

[0021] S2. Based on the video-level prediction results of sub-network A and sub-network B, clean masks for sub-network A and sub-network B are generated respectively using the binary classification cross-entropy loss function and dynamic confidence ratio, and the dynamic confidence ratio is updated using the cosine strategy.

[0022] S3. Based on the clean mask of sub-network A and the clean mask of sub-network B, the gradient of the mutual learning noise-resistant dual-stream network is updated through a cross-filtering mechanism.

[0023] S4. Repeat S1-S3 until the validation set metrics of subnetwork A and subnetwork B show no improvement for five consecutive rounds, then end the training.

[0024] Preferably, based on the training set, the video-level prediction results of subnetwork A and subnetwork B are obtained, including:

[0025] Based on the training set, the enhanced features of sub-network A and sub-network B are obtained respectively through the multimodal feature fusion network layer of sub-network A and the multimodal feature fusion network layer of sub-network B.

[0026] Based on the enhancement features of subnetwork A and subnetwork B, the video prediction results and audio prediction results of subnetwork A, as well as the video prediction results and audio prediction results of subnetwork B, are obtained through the fully connected projection head of subnetwork A and the fully connected projection head of subnetwork B, respectively.

[0027] Based on the video and audio prediction results of subnetwork A and subnetwork B, the video-level prediction results of subnetwork A and subnetwork B are obtained by passing the multimodal multi-instance decoding network layer of subnetwork A and subnetwork B, respectively.

[0028] Preferably, generating clean masks for sub-network A and sub-network B respectively includes:

[0029] Obtain video tags. Based on the video tags, the video-level prediction results of sub-network A and sub-network B, obtain the loss values ​​of sub-network A and sub-network B respectively through the binary classification cross-entropy loss function.

[0030] After sorting the loss values ​​of subnetwork A and subnetwork B in ascending order, the segments before the dynamic confidence ratio are selected to obtain the clean mask of subnetwork A and subnetwork B.

[0031] Preferably, the expression for the binary classification cross-entropy loss function is:

[0032] ,

[0033] in, This is the loss value; This is a video-level prediction result; For video tags.

[0034] The preferred formula for calculating the dynamic confidence ratio update is:

[0035] ,

[0036] in, For dynamic confidence ratios, This represents the current training round number. This is the preset total number of training rounds.

[0037] Preferably, the formula for calculating the validation set metric is:

[0038] ,

[0039] in, Here, P represents precision and R represents recall, which are the validation set metrics.

[0040] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0041] This invention utilizes a collaborative training mechanism between two sub-networks with identical structures but different initializations to form a noise-resistant dual-stream network. This enables autonomous identification and suppression of pseudo-label noise, preventing overfitting and significantly improving the accuracy and robustness of segment-level event localization and classification in audio and video. A cross-filtering mechanism is implemented based on the clean masks generated by the two sub-networks, mutually eliminating potentially misclassified noise samples. This allows for autonomous identification and suppression of pseudo-label noise during training, improving fitting speed and reducing overfitting risk. A cosine strategy is used to gradually reduce the dynamic confidence ratio, progressively decreasing the proportion of clean samples participating in gradient updates during training and suppressing overfitting to residual noise. After training, only a single sub-network is retained for audio and video parsing, achieving automatic identification and suppression of pseudo-label noise without increasing inference overhead. Attached Figure Description

[0042] Figure 1 This is a flowchart of an audio / video parsing method based on noise label learning in an embodiment of the present invention.

[0043] Figure 2 This is a block diagram of a mutual learning noise-resistant dual-stream network in an embodiment of the present invention.

[0044] Figure 3 This is a flowchart of the training process for a mutual learning, noise-resistant dual-stream network in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0046] An audio / video parsing method based on noise label learning, such as Figure 1As shown, it includes the following steps:

[0047] S1. Preprocess the original audio and video to obtain segment-level input sequences, and divide the segment-level input sequences into training set, validation set and test set;

[0048] The original audio and video are acquired, and then uniformly segmented into multiple segments without overlap using a set window (which can be a 1-second window). Based on these segments, visual features are extracted using a frozen CLIP image encoder and a CLAP audio encoder, respectively. and audio features Generate fragment-level input sequences, i.e., datasets; visual features and audio features All belong to 768-dimensional features.

[0049] Divide the dataset into training, validation, and test sets according to a preset ratio. For example, the ratio of training set:validation set:test set is 7:2:1.

[0050] S2. Construct a mutual learning, noise-resistant dual-stream network;

[0051] Mutual learning noise-resistant dual-stream networks are DNN deep neural networks, such as... Figure 2 As shown, it includes two sub-networks with the same structure but different initializations. The two sub-networks are sub-network A and sub-network B. Sub-network A and sub-network B both include a multimodal feature fusion network layer, a fully connected projector head, and a multimodal multi-instance decoding network layer. The multimodal feature fusion network layer is connected to the fully connected projector head and the multimodal multi-instance decoding network layer, respectively.

[0052] The multimodal feature fusion network layer is used for feature fusion computation, including feature self-attention computation and cross-modal feature attention computation.

[0053] The fully connected projection head is a multilayer perceptron network layer used to map features to classification prediction confidence, complete the classification task, that is, classify and predict video events and audio events, and obtain visual prediction results and audio prediction results.

[0054] The multimodal multi-instance decoding network layer is used to map visual prediction results and audio prediction results to video-level prediction confidence through a learnable network layer, thus obtaining video-level prediction results.

[0055] Neural network parameters include weights and biases. The initialization of subnetworks A and B differs because the initial values ​​of the weights in their neural network parameters are set differently. In completing the AVVP task, the noise resistance of two identical subnetworks with different weights differs because the different weights lead to different optimization processes, causing subnetworks A and B to converge to different optimization surfaces. Therefore, it can be considered that both subnetworks have the ability to complete the classification task, but their noise resistance is slightly different. Subsequently, the clean sample results selected by subnetwork A are given to subnetwork B, and the clean sample results selected by subnetwork B are given to subnetwork A. Through this cross-training, the two subnetworks, with their different denoising abilities, can supervise each other and improve the noise resistance in the training of the mutually learning noise-resistant dual-stream network.

[0056] For example, the initial values ​​of the weights of subnetwork A and subnetwork B can be randomized according to a normal distribution, ensuring that the initial values ​​of the weights of subnetwork A and subnetwork B are different.

[0057] S3. Train the mutual learning noise-resistant dual-stream network based on the training set;

[0058] like Figure 3 As shown, the training of the mutual learning noise-resistant dual-stream network includes:

[0059] S31. Based on the training set, obtain the video-level prediction results of subnetwork A and subnetwork B;

[0060] The training set is input into the mutual learning noise-resistant dual-stream network. The two sub-networks A and B with the same structure but different initializations receive the training set in parallel.

[0061] Based on the training set, the enhanced features of sub-network A and sub-network B are obtained respectively through the multimodal feature fusion network layer of sub-network A and the multimodal feature fusion network layer of sub-network B.

[0062] Based on the enhancement features of subnetwork A and subnetwork B, the video prediction results of subnetwork A are obtained through the fully connected projection heads of subnetwork A and subnetwork B, respectively. and audio prediction results And the video prediction results of the B sub-network. and audio prediction results ;

[0063] Based on the video and audio prediction results of subnetwork A and subnetwork B, multi-instance pooling operations are performed on the multimodal multi-instance decoding network layers of subnetwork A and subnetwork B respectively to obtain the video-level prediction result of subnetwork A. Video-level prediction results of the B sub-network .

[0064] , ,

[0065] in, The video-level prediction results for subnetwork A. The video prediction results for subnetwork A. Here are the audio prediction results for subnetwork A and the video prediction results for subnetwork B. The video prediction results for subnetwork B. This represents the audio prediction results for the B subnetwork.

[0066] Because video and audio may be out of sync, the video prediction result is not necessarily equal to the audio prediction result. For example, the video prediction result of subnetwork A may differ from the audio prediction result. It is not necessarily equal to the audio prediction result of subnetwork A. To comprehensively consider both audio and video modalities, a video-level prediction result is determined to exist in the video if the category appears in any modality, such as the video-level prediction result of subnetwork A. ,Right now .

[0067] S32. Based on the video-level prediction results of sub-network A and sub-network B, clean masks for sub-network A and sub-network B are generated respectively using the binary classification cross-entropy loss function and dynamic confidence ratio, and the dynamic confidence ratio is updated using the cosine strategy.

[0068] Obtain video tags, calculate the binary cross-entropy loss between the video-level prediction results of sub-network A and the video tags using the binary cross-entropy loss function, and obtain the loss value of sub-network A; sort the loss values ​​of sub-network A in ascending order, and then apply the dynamic confidence ratio. Select the segment before the dynamic confidence ratio as the clean mask to generate the clean mask for subnetwork A. ;

[0069] Similarly, using the binary cross-entropy loss function, the binary cross-entropy loss between the video-level prediction result of sub-network B and the video label is calculated to obtain the loss value of sub-network B; after sorting the loss values ​​of sub-network B in ascending order, the dynamic confidence ratio is used... The segment before the dynamic confidence ratio is selected as the clean mask to generate the clean mask for the B subnetwork. ;

[0070] The expression for the binary classification cross-entropy loss function is:

[0071] ,

[0072] in, This is the loss value; This is a video-level prediction result; The video tags are derived from clean, labeled data in the dataset.

[0073] The confidence ratio is updated dynamically using a cosine strategy.

[0074] In the training of the mutual learning noise-resistant dual-stream network, the loss values ​​of the sample labels are sorted in ascending order from low to high. It is assumed that the higher the loss value, the higher the noise, and vice versa. The sample labels whose loss values ​​are ranked before the dynamic confidence ratio are considered clean samples and are taken out to participate in the loss gradient backpropagation. The remaining sample labels are considered noise samples and do not participate in the loss gradient backpropagation.

[0075] By sorting the loss values ​​in ascending order and taking the fragments before the dynamic confidence ratio as clean samples, information loss caused by traditional cleaning preprocessing can be avoided.

[0076] The initial value of the dynamic confidence ratio can be preset to 0.7. The dynamic confidence ratio can be gradually reduced to 0.3 through the cosine strategy, which can take into account both early convergence and later generalization. That is, it ensures sufficient gradient signals in the early stage and only fine-tunes the high-confidence subset in the later stage, thereby improving the validation set index by 1.2%.

[0077] The formula for calculating the dynamic confidence ratio update is:

[0078] ,

[0079] in, For dynamic confidence ratios, This represents the current training round number. This is the preset total number of training rounds.

[0080] As the mutual learning noise-resistant dual-stream network is trained, its fitting ability improves, and its fit to noise also increases. Therefore, by reducing the screening ratio, i.e., reducing the dynamic confidence ratio, we can obtain relatively cleaner sample labels for training, which can suppress overfitting to residual noise and improve the accuracy and robustness of the model.

[0081] S33. Based on the clean mask of subnetwork A and the clean mask of the B subnet Gradient updates are performed on the mutual learning noise-resistant dual-stream network through a cross-filtering mechanism;

[0082] Based on the clean mask of subnetwork A Gradient updates are performed on the B sub-network based on the clean mask of the B sub-network. Perform gradient updates on subnetwork A.

[0083] By using a cross-filtering mechanism, the clean mask of sub-network A is used to update the gradient of sub-network B, thus eliminating misclassified noise in sub-network B; similarly, the clean mask of sub-network B is used to update the gradient of sub-network A, thus eliminating misclassified noise in sub-network A. This mutual elimination of potentially misclassified noise samples from each other can improve the fitting speed of the mutually learning noise-resistant dual-stream network by 18%, reduce the risk of overfitting, and thus enable the autonomous identification and suppression of pseudo-label noise during training, significantly improving the accuracy and robustness of the model.

[0084] S34. Repeat S31-S33 until the validation set metrics of subnetwork A and subnetwork B show no improvement for five consecutive rounds, then end the training.

[0085] The formula for calculating the validation set metric is:

[0086] ,

[0087] in, Here, P represents precision and R represents recall, which are the validation set metrics.

[0088] Training with a mutual-learning noise-resistant dual-stream network can improve the validation set metrics for segment-level audio events, visual events, and audio-visual joint event classification from 68.1, 68.4, and 61.9 to 68.9, 69.8, and 63.4, respectively. Due to the different initialization of the mutual-learning noise-resistant dual-stream network, there will be about 12% discrepancy in the loss ranking of the same sample in the early stage of training, which precisely covers the boundary offset and modal misalignment cases with the highest misclassification probability in pseudo-labels.

[0089] S4. Validate the trained mutual learning noise-resistant dual-stream network based on the validation set. Compare the validation set metrics of the two sub-networks in the trained mutual learning noise-resistant dual-stream network, and select the sub-network with the higher validation set metric as the audio / video parsing model. Based on the test set, parse the video using the audio / video parsing model to obtain the video prediction results.

[0090] The validation set metrics of subnetwork A and subnetwork B in the selected trained mutual learning noise-resistant dual-stream network are compared. If the validation set metrics of subnetwork A are greater than those of subnetwork B, then subnetwork A is selected as the audio and video parsing model.

[0091] If the validation set metric of subnetwork A is less than that of subnetwork B, then subnetwork B is selected as the audio / video parsing model.

[0092] By retaining only one sub-network for audio and video parsing, we can achieve autonomous identification and suppression of false label noise without increasing inference overhead, thereby reducing the risk of noise memory by 23%. This effectively implements collaborative denoising during training and zero burden during inference.

[0093] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An audio / video parsing method based on noise label learning, characterized in that, include: The original audio and video are preprocessed to obtain segment-level input sequences, which are then divided into training, validation, and test sets. Construct a mutual learning, noise-resistant dual-stream network; The mutual learning noise-resistant dual-stream network is trained based on the training set. The training of the mutual learning noise-resistant dual-stream network based on the training set includes: The mutual learning noise-resistant dual-stream network includes sub-network A and sub-network B; S1. Based on the training set, obtain the video-level prediction results of subnetwork A and subnetwork B; S2. Based on the video-level prediction results of sub-network A and sub-network B, clean masks for sub-network A and sub-network B are generated respectively using the binary classification cross-entropy loss function and dynamic confidence ratio, and the dynamic confidence ratio is updated using the cosine strategy. S3. Based on the clean mask of sub-network A and the clean mask of sub-network B, the gradient of the mutual learning noise-resistant dual-stream network is updated through a cross-filtering mechanism. S4. Repeat S1-S3 until the validation set metrics of subnetwork A and subnetwork B show no improvement for five consecutive rounds, then end the training. The trained mutual-learning noise-resistant dual-stream network is validated using a validation set. The validation set metrics of two sub-networks in the trained mutual-learning noise-resistant dual-stream network are compared, and the sub-network with the higher validation set metric than the other sub-network is selected as the audio / video parsing model. Based on the test set, the audio / video parsing model is used to parse the video and obtain the video prediction results.

2. The audio and video parsing method based on noise label learning according to claim 1, characterized in that, The preprocessing of the original audio and video to obtain the segment-level input sequence includes: Multiple segments are obtained by uniformly dividing the original audio and video without overlap by setting a window; Based on multiple segments, visual and audio features are extracted using the CLIP image encoder and the CLAP audio encoder, respectively, to generate segment-level input sequences.

3. The audio and video parsing method based on noise label learning according to claim 1, characterized in that, The mutual learning noise-resistant dual-stream network includes two sub-networks with the same structure but different initialization. Each sub-network includes a multimodal feature fusion network layer, a fully connected projector head, and a multimodal multi-instance decoding network layer. The multimodal feature fusion network layer is connected to the fully connected projector head and the multimodal multi-instance decoding network layer, respectively.

4. The audio and video parsing method based on noise label learning according to claim 3, characterized in that, The mutual learning noise-resistant two-stream network also includes the following: the initial values ​​of the weights of the two sub-networks are different.

5. The audio and video parsing method based on noise label learning according to claim 1, characterized in that, The process of obtaining the video-level prediction results of subnetwork A and subnetwork B based on the training set includes: Based on the training set, the enhanced features of sub-network A and sub-network B are obtained respectively through the multimodal feature fusion network layer of sub-network A and the multimodal feature fusion network layer of sub-network B. Based on the enhancement features of subnetwork A and subnetwork B, the video prediction results and audio prediction results of subnetwork A, as well as the video prediction results and audio prediction results of subnetwork B, are obtained through the fully connected projection head of subnetwork A and the fully connected projection head of subnetwork B, respectively. Based on the video and audio prediction results of subnetwork A and subnetwork B, the video-level prediction results of subnetwork A and subnetwork B are obtained by passing the multimodal multi-instance decoding network layer of subnetwork A and subnetwork B, respectively.

6. The audio and video parsing method based on noise label learning according to claim 1, characterized in that, The process of generating clean masks for sub-network A and sub-network B, respectively, includes: Obtain video tags. Based on the video tags, the video-level prediction results of sub-network A and sub-network B, obtain the loss values ​​of sub-network A and sub-network B respectively through the binary classification cross-entropy loss function. After sorting the loss values ​​of subnetwork A and subnetwork B in ascending order, the segments before the dynamic confidence ratio are selected to obtain the clean mask of subnetwork A and subnetwork B.

7. The audio and video parsing method based on noise label learning according to claim 6, characterized in that, The expression for the binary classification cross-entropy loss function is: ， in, This is the loss value; This is a video-level prediction result; For video tags.

8. The audio and video parsing method based on noise label learning according to claim 1, characterized in that, The formula for calculating the dynamic confidence ratio update is as follows: ， in, For dynamic confidence ratios, This represents the current training round number. This is the preset total number of training rounds.

9. The audio and video parsing method based on noise label learning according to claim 1, characterized in that, The formula for calculating the validation set metric is as follows: ， in, Here, P represents precision and R represents recall, which are the validation set metrics.

Images