Weakly supervised audio-video parsing system and method based on decoupled multi-level contrastive learning
By decoupling multi-level contrastive learning, the problems of blurred boundaries and low positioning accuracy in weakly supervised audio and video parsing are solved. Through foreground-background decoupling and multi-level anchor point-guided contrastive learning, more accurate audio and video alignment and clearer event boundaries are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UNIV OF SCI & TECH BEIJING
- Filing Date
- 2026-02-11
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies struggle to effectively handle fine-grained segment structures and temporal correspondences within and outside modalities in weakly supervised audio and video parsing, leading to blurred or missing boundaries, especially resulting in low positioning accuracy when dealing with sparse, weak, or overlapping events.
A system and method based on decoupled multi-level contrastive learning are adopted. Through a foreground-background decoupling fusion module and a multi-level anchor point guided contrastive learning module, audio and video features are aligned and learned at the segment, global and video levels, respectively. Feature separation and weighting are performed using a preset projection function, multi-head attention mechanism and dynamic gating mechanism. Positive and negative samples are compared by combining adaptive boundary and global objective function.
It improves the positioning accuracy and modal consistency of audio and video parsing, reduces the impact of background noise, and achieves clearer event boundaries and more robust audio and video alignment.
Smart Images
Figure CN122157091A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of audio and video parsing technology, and in particular to a weakly supervised audio and video parsing system and method based on decoupled multi-level contrastive learning. Background Technology
[0002] Perceiving events through sound and vision is central to video understanding, prompting the development of audiovisual learning (AVL) tasks to investigate cross-modal correspondences, such as temporal localization (AVE), spatial localization (AVSSL), and pixel-level segmentation (AVS). Many of these tasks implicitly assume synchronization between audio and video signals, which often fails in real-world videos. Recent research has applied two main strategies to address the lack of temporal annotation in weakly supervised audio-video parsing: label refinement and model architecture design. Regarding label refinement, representative examples include using contrastive learning to refine video-level labels and constructing fragment-level pseudo-labels based on semantic knowledge from pre-trained models such as CLIP / CLAP. In terms of model architecture, most work focuses on video-level interaction modeling, temporal modeling modules, and robust training objectives under noisy, weak supervision.
[0003] Despite the progress made by the aforementioned methods, weak supervision at the video level still faces numerous challenges. Many methods operate primarily at the video level, emphasizing global features, temporal interactions, or weakly labeled targets, while paying insufficient attention to fine-grained segment structures, including intramodal dynamics and cross-modal temporal correspondences. Furthermore, other methods (such as segment pseudo-labels) are often constructed and optimized independently for each segment, making it difficult to ensure coherent relationships between adjacent segments over time. Moreover, frequent and dominant background content can influence video-level aggregation and pseudo-label construction, leading to blurred or missing boundaries and reduced localization accuracy, especially when dealing with sparse, weak, or overlapping events. Summary of the Invention
[0004] To address the aforementioned technical problems in existing technologies, this invention provides a weakly supervised audio and video parsing system and method based on decoupled multi-level contrastive learning. The technical solution is as follows: On the one hand, a weakly supervised audio and video parsing system based on decoupled multi-level contrastive learning is provided, including: a foreground-background decoupling fusion module and a multi-level anchor-guided contrastive learning module; wherein, the foreground-background decoupling fusion module is used to: project the features of the segment to be parsed onto a dimensionality-reduced decoupled latent space based on a preset projection function to obtain a foreground matrix and a background vector; the features of the segment to be parsed include features of the audio segment to be parsed or features of the time-frequency segment to be parsed; integrate the foreground matrix and the background vector based on a multi-head attention mechanism to obtain a context-aware foreground representation matrix and a background representation vector; and adaptively weight the background contribution of each category based on a dynamic gating mechanism to obtain a foreground weighted feature based on the gating vector and an adaptive foreground weighted feature. The feature should be fused; the multi-level anchor-guided contrastive learning module is used to: project the foreground weighted features onto a shared space to generate aligned fragment features; determine the individual anchor box fragments corresponding to the fragment features of each category based on the trained encoder and the preset modal text prototype; construct positive and negative sample pairs based on adaptive boundaries using the individual anchor box fragments corresponding to the fragment features of each category as references; perform fragment-level contrastive learning on the individual anchor box fragments based on the positive and negative sample pairs and the fragment-level contrastive learning loss function; perform global contrastive learning on the modality of each category based on the preset modal sequence and the global objective function; and perform video-level contrastive learning on the video to be parsed based on the video-level contrastive objective function and the positive and negative sample pairs constructed by label overlap.
[0005] Optionally, the preset projection function includes:
[0006] In the formula, Mapping to a multilayer perceptron; The foreground matrix and the background vector include:
[0007] In the formula, and Let represent the foreground matrix and the background vector, respectively, and t represent the time step. The features of the segment to be parsed; The decoupling loss function for the foreground matrix and the background vector includes:
[0008] In the formula, T is the total time step of the features of the segment to be parsed, and C is the total number of the foreground matrices.
[0009] Optionally, the gating vector includes:
[0010] The foreground-weighted features include: ; The adaptive fusion features include: ; The gating loss function of the dynamic gating mechanism includes:
[0011] In the formula, These are the foreground representation matrix and the background representation vector, respectively, and Y is the video-level label.
[0012] Optionally, the adaptive boundary includes:
[0013] In the formula, and α represents the mean and standard deviation of the similarity between a single anchor frame fragment and the preset modal text prototype, respectively, and α is a control variable.
[0014] Optionally, the fragment-level contrastive learning loss function includes:
[0015] In the formula, sim represents the similarity. This represents a single anchor box segment, where p is the positive sample in the positive-negative sample pair, and q is the sample in the positive-negative sample pair. This refers to the temperature parameter.
[0016] Optionally, the global objective function includes:
[0017] In the formula, This represents the set of positive samples in a positive-negative sample pair. The preset mode sequence.
[0018] Optionally, the video-level contrast objective function includes:
[0019] In the formula, This represents a set of videos in a small batch. Videos defined by tag overlap The main collection, To represent different segment features.
[0020] On the other hand, a weakly supervised audio and video parsing method based on decoupled multi-level contrastive learning is also provided, which is used in the weakly supervised audio and video parsing system based on decoupled multi-level contrastive learning provided in the embodiments of the present invention. The method includes: decoupling the foreground and background based on the features of the segment to be parsed by a foreground-background decoupling fusion module, and optimizing it based on a multi-head attention mechanism with adaptive gating to obtain foreground weighted features and adaptive fusion features; and performing segment-level contrastive learning, global contrastive learning and video-level contrastive learning on the foreground weighted features based on a multi-level anchor-guided contrastive learning module.
[0021] Optionally, the foreground and background decoupling and fusion module decouples the features of the segment to be parsed from the background and optimizes them based on a multi-head attention mechanism with adaptive gating to obtain foreground weighted features and adaptive fusion features. This includes: projecting the features of the segment to be parsed onto a dimensionality-reduced decoupled latent space based on a preset projection function to obtain a foreground matrix and a background vector; the features of the segment to be parsed include audio segment features or time-frequency segment features to be parsed; integrating the foreground matrix and the background vector with contextual information based on the multi-head attention mechanism to obtain a context-aware foreground representation matrix and a background representation vector; and adaptively weighting the background contribution of each category based on a dynamic gating mechanism to obtain foreground weighted features and adaptive fusion features based on the gating vector.
[0022] Optionally, based on the multi-level anchor-guided contrastive learning module, segment-level contrastive learning, global contrastive learning, and video-level contrastive learning are performed on the foreground weighted features, including: projecting the foreground weighted features onto a shared space to generate aligned segment features; determining individual anchor box segments corresponding to the segment features of each category based on a trained encoder and a preset modal text prototype; constructing positive and negative sample pairs based on adaptive boundaries using the individual anchor box segments corresponding to the segment features of each category as references; performing segment-level contrastive learning on the individual anchor box segments based on the positive and negative sample pairs and a multi-positive contrastive objective function; performing global contrastive learning on the modality of each category based on a preset modal sequence and a global objective function; and performing video-level contrastive learning on the video to be parsed based on a video-level contrastive objective function and positive and negative sample pairs constructed by label overlap.
[0023] This invention provides a weakly supervised audio and video parsing system and method based on decoupled multi-level contrastive learning. It is constructed around foreground-background decoupling and fine-grained temporal relationship learning to address the problem of inaccurate boundary localization. Through a foreground-background decoupling fusion module and a multi-level anchor-guided contrastive learning mechanism, it separates category-specific backgrounds from shared backgrounds and forces alignment at the segment, global, and video levels, thereby improving intramodal structure and intermodal consistency under weak labeling. This alleviates the technical problems of blurred or missing boundaries and low localization accuracy in existing technologies. Attached Figure Description
[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0025] Figure 1 This is a schematic diagram of the framework of a weakly supervised audio and video parsing system based on decoupled multi-level contrastive learning provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the framework of a foreground / background decoupling and fusion module provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the framework of a multi-level anchor-guided contrastive learning module provided in an embodiment of the present invention; Figure 4 This is a flowchart of a weakly supervised audio and video parsing method based on decoupled multi-level contrastive learning provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the time segmentation results of LSLD, VALOR, and the method of the present invention provided in the embodiments of the present invention. Detailed Implementation
[0026] The technical solution of the present invention will now be described with reference to the accompanying drawings.
[0027] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.
[0028] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.
[0029] Figure 1 This is a schematic diagram of the framework of a weakly supervised audio and video parsing system based on decoupled multi-level contrastive learning, according to an embodiment of the present invention. Figure 1 As shown, the system includes: a foreground / background decoupling and fusion module (i.e., FBDF in the figure) and a multi-level anchor-guided contrastive learning module (i.e., MACL in the figure); wherein, The foreground / background decoupling and fusion module is used for: The features of the segment to be parsed are projected onto the dimension-reduced decoupled latent space based on a preset projection function to obtain the foreground matrix and background vector; the features of the segment to be parsed include audio segment features or time-frequency segment features. The foreground matrix and background vector are integrated with contextual information based on the multi-head attention mechanism to obtain a context-aware foreground representation matrix and background representation vector; Based on the dynamic gating mechanism, the background contribution of each category is adaptively weighted to obtain foreground weighted features and adaptive fusion features based on gating vectors; The multi-level anchor-guided comparative learning module is used for: Foreground-weighted features are projected onto a shared space to generate aligned fragment features; Based on the trained encoder and the pre-defined modal text prototype, determine the single anchor box fragment corresponding to the fragment features of each category; Using the individual anchor box fragments corresponding to the fragment features of each category as a reference, positive and negative sample pairs are constructed based on adaptive boundaries; Fragment-level contrastive learning is performed on a single anchor frame fragment based on positive and negative sample pairs and a fragment-level contrastive learning loss function; Based on a preset modality sequence and a global objective function, global comparative learning is performed on the modality of each category; Based on the video-level contrast objective function and positive and negative sample pairs constructed by label overlap, video-level contrast learning is performed on the video to be analyzed.
[0030] Specifically, in this embodiment of the invention, given an unconstrained video V, it is divided into T non-overlapping time segments, and segment-level features of the audio and video streams are extracted using a pre-trained encoder. Representing the D-dimensional audio and visual features of the t-th segment, the stacked features form The goal is to perform segmented multi-label classification of C event categories, where each segment may contain multiple events or no events.
[0031] During testing, the fragment-level target was a one-hot tag. , representing audible and visible events respectively, and this invention further defines the audio-video symbiotic tag as The evaluation tuple is obtained. However, during training, there was only one modality-insensitive video-level label. It is available. This indicates that event c occurs somewhere in the video (in the audio, in the visual, or both), without specifying its time segment or modality. Furthermore, segment-level pseudo-tags for each modality can be used, represented as... .
[0032] In weakly supervised audio and video parsing, most methods encode each segment as a whole embedding, which often obfuscates event cues with background context. When using only video-level tags, segments with more background may be overweighted, making it difficult to separate category features and increasing semantic confusion. To mitigate this problem, this invention introduces a category-aware module that can explicitly separate event foreground from background. For simplicity, the embodiments of this invention only describe the visual flow; the audio design is the same.
[0033] Figure 2 This is a schematic diagram of the framework of a foreground / background decoupling and fusion module according to an embodiment of the present invention. Figure 2 As shown, for visual streams, this embodiment of the invention applies a set of class-specific nonlinear mappings to project the fragment features at time step t into a dimension-reduced, decoupled latent space. Specifically, the preset projection function includes:
[0034] Used to separate the foreground from the background of an event. Each of these is... Each is an independent multilayer perceptron mapping The decomposition formula is as follows: (1) here, and These represent the decoupled foreground matrix and background vector, respectively. Specifically, The Travel Generate, and represent category-specific. Potential characteristics. To facilitate interaction, embodiments of the present invention conceptually extend the context to... .
[0035] To decouple the semantics of specific categories under weak supervision, embodiments of this invention introduce a bidirectional cross-attention mechanism. Specifically, a multi-head attention mechanism (MHA) is used, where the foreground pays attention to the background to integrate contextual information, and vice versa: (2) in, and These represent the context-aware foreground representation matrix and background representation vector, respectively.
[0036] This invention employs a dynamic gating mechanism to adaptively weight the background contribution of each category. The gating vector... The calculation is as follows: (3) Adaptive fusion features The following is obtained by adaptively fusing and connecting the components: (4) Equation (4) ensures that the fused representation recovers the original feature dimensions while selectively incorporating the relevant context. .
[0037] To ensure the effectiveness of decoupling, this invention imposes two complementary constraints. First, this invention introduces a decoupling separation loss function. The foreground features of the forced aggregation are orthogonal to the background prototype. Specifically, the decoupling and separation loss function includes minimizing their cosine similarity: (5) Secondly, because there are only video-level tags Available, this invention utilizes them as gating vectors Coarse supervision is provided to encourage activation of category-related regions. The corresponding gating loss function and overall background objective (summed over both modalities) are formulated as follows: (6) To better utilize weakly supervised information and alleviate the lack of reliable segment-level relationships in weakly supervised audio and video parsing, this invention introduces a Multi-Level Anchor Guided Contrast Learning (MACL) module. Based on the foreground representation matrix generated by the foreground-background decoupling and fusion module, the MMACL module aims to organize audio and video segments into a shared embedding space, thereby learning meaningful intra- and inter-module relationships even under noisy supervision.
[0038] Specifically, this invention first aggregates foreground weighted features (calculated as...). ) and pass it through the modality-specific head Projecting onto a shared space generates aligned fragment features. This process emphasizes category-related cues while suppressing background noise. To facilitate subsequent anchor box selection, this invention uses pre-trained CLAP (audio) and CLIP (visual) encoders, while simultaneously constructing a pre-defined modal text prototype. For each video and each category The optimal modality is first determined by searching for the fragment that best matches the category prototype. and timestamp : (7) Then, the feature vectors at these indices are designated as single anchor box fragments, denoted as... .
[0039] With a single anchor frame fragment For reference, a positive and negative sample set is constructed. Specifically, this invention does not use a fixed global threshold, but instead employs an instance-adaptive strategy that adjusts according to the distribution of each video. Specifically, this invention calculates the similarity between a single anchor frame fragment and a preset modal text prototype. their mean and standard deviation And the adaptive boundary is defined as: (8) in, Control variables that govern the strictness of the selection. Using these adaptive boundaries, this invention partitions line segments into an initial positive set. and negative set : (9) Since anchor-based similarity alone may still introduce noisy samples, this invention further optimizes these datasets using a simple quality-aware filtering step. For each candidate sample, this invention measures its consistency with its corresponding group and retains only samples with higher consistency: (10) in, This represents the average similarity between positive samples and the remaining positive samples, in order to promote sample compactness; By measuring the similarity between negative samples and positive sample groups, this invention is able to eliminate fuzzy negative samples that are too close to the anchor sample.
[0040] Figure 3 This is a schematic diagram of the framework of a multi-level anchor-guided contrastive learning module provided by an embodiment of the present invention. Figure 3 As shown, specifically, the goal of the multi-level anchor-guided contrastive learning module provided in this embodiment of the invention is to learn fine-grained inter-segment relationships in each video under weak supervision. For a given video and category... This invention uses a single anchor segment. To organize the comparison pairs. For each target mode As mentioned above, a refined set of positive and negative samples is extracted from the same video. and Then, the present invention optimizes a multi-positive contrast objective function, which will... Pull towards the refined positive set and away from the refined negative set: (11) in Values are derived from the refined set. ,and Values are taken from the union of sets. Finally, the fragment-level contrastive learning loss function is obtained as follows: (12) To improve cross-batch stability and extend beyond intra-video segment comparison, this invention introduces a global objective function with a FIFO memory bank. For each modality... This invention maintains a capacity of Modal-specific queues In this memory, the features of each category are stored independently, denoted as sub-queues. After anchor box selection and positive sample refinement, this invention pushes the refined positive samples into the corresponding queue; when the queue is full, the oldest entry is dequeued. This memory provides a lightweight repository for storing the latest high-confidence features across mini-batches. Specifically, the global objective function includes: (13) in, Representing modes Medium category The refined positive fragment features. This invention will... Defined as and The average similarity among all entries. It is the sigmoid function. It is a temperature parameter.
[0041] The global objective function encourages refined positive sample segments to remain close to historical high-confidence representations of the same class, thereby improving consistency across batches and videos without introducing additional negative samples. Unlike local objective functions that are limited to the current batch, the global objective function utilizes historical class features to calibrate the representation space and ensures robust generalization.
[0042] While segment-level contrastive learning loss functions and global objective functions focus on fine-grained relationships, weakly supervised audio-visual parsing is only available at the video level. Therefore, this invention introduces a video-level contrastive objective function, utilizing weak video tags as coarse semantic constraints to encourage videos with overlapping events to share similar representations.
[0043] Specifically, for each video, the present invention obtains a video representation by aggregating and fusing segment features. Given video-level tags This invention constructs positive and negative sample pairs based on label overlap. Specifically, for the target video... ,if If two videos share at least one event tag, then the other video in the same mini-batch will be included. If a sample is considered positive, it is considered negative; otherwise, it is considered negative. Based on this construction, the present invention defines the video-level contrast objective function as: (14) in, This represents a set of videos in a small batch. Videos defined by tag overlap The main collection, It is a temperature parameter.
[0044] The ultimate goal is: (15) Fusion features As input to the backbone network, segmented predictions are generated. and This invention utilizes... We derive audiovisual prediction and apply attention-based MMIL pooling to obtain video-level scores. According to standard protocols, this invention uses a basic loss optimization network: (16) The first item applies to all orbits. Summation, with the second term targeting a single-peaked orbit. Perform pseudo-tag alignment.
[0045] The ultimate goal provided by this invention combines basic loss, contrast alignment, and background regularization: (17) Figure 4 This is a flowchart illustrating a weakly supervised audio / video parsing method based on decoupled multi-level contrastive learning, provided by an embodiment of the present invention. It is used in a weakly supervised audio / video parsing system based on decoupled multi-level contrastive learning provided by an embodiment of the present invention. Figure 5 As shown, the method specifically includes the following steps: Step S402: Based on the foreground-background decoupling and fusion module, the foreground and background of the segment to be parsed are decoupled, and optimized based on the multi-head attention mechanism with adaptive gating to obtain foreground weighted features and adaptive fusion features.
[0046] Step S404: Based on the multi-level anchor point guided contrast learning module, segment-level contrast learning, global contrast learning, and video-level contrast learning are performed on the foreground weighted features respectively.
[0047] Specifically, step S402 further includes the following steps: The features of the segment to be parsed are projected onto the dimension-reduced decoupled latent space based on a preset projection function to obtain the foreground matrix and background vector; the features of the segment to be parsed include audio segment features or time-frequency segment features. The foreground matrix and background vector are integrated with contextual information based on the multi-head attention mechanism to obtain a context-aware foreground representation matrix and background representation vector; By adaptively weighting the background contribution of each category based on a dynamic gating mechanism, we obtain foreground weighted features and adaptive fusion features based on gating vectors.
[0048] Specifically, step S404 further includes the following steps: Foreground-weighted features are projected onto a shared space to generate aligned fragment features; Based on the trained encoder and the pre-defined modal text prototype, determine the single anchor box fragment corresponding to the fragment features of each category; Using the individual anchor box fragments corresponding to the fragment features of each category as a reference, positive and negative sample pairs are constructed based on adaptive boundaries; Fragment-level contrastive learning is performed on individual anchor frame segments based on positive and negative sample pairs and a multi-positive contrastive objective function; Based on a preset modality sequence and a global objective function, global comparative learning is performed on the modality of each category; Based on the video-level contrast objective function and positive and negative sample pairs constructed by label overlap, video-level contrast learning is performed on the video to be analyzed.
[0049] The embodiments of the present invention verify the effectiveness of a weakly supervised audio and video parsing system and method based on decoupled multi-level contrastive learning provided by the present invention through specific experimental settings: (1) Dataset: This invention uses the LLP dataset to evaluate its method. This dataset contains 11,849 unconstrained YouTube videos (10 seconds each) covering 25 event categories. Following standard procedures, this invention uses 10,000 videos with weak video-level labels for training, while the remaining 1,849 videos with precise segment-level annotations are used for validation and testing.
[0050] (2) Evaluation indicators: This invention reports F1 scores (IoU threshold of 0.5) for audio, visual, and audiovisual events at both the segment and event levels. To evaluate overall parsing performance, this invention also provides two aggregate metrics: Type@AV (average F1 scores for audio, visual, and audiovisual events) and Event@AV (global F1 score for all events).
[0051] (3) Implementation details: This invention uses CLIP (ViT-B / 16) and CLAP features, projection dimensions And directly use their pseudo-labels for training. This invention uses the Adam optimizer to... The initial learning rate was set, and training was conducted for 20 epochs with a batch size of 128. A step scheduler reduced the learning rate by 0.25 every 9 epochs and set a linear warm-up period of 3 epochs. For contrastive learning, the temperature, memory size per modality, and momentum were set to 0.07, 1000, and 0.9, respectively. This invention further applies a dynamic thresholding method, with thresholds of... and To optimize positive and negative sample pairs; the standard deviation factor is 1.0. The loss weights of the overall objective function. The values were set to 0.1 and 1.0 respectively. All experiments were run on an RTX 3090 GPU.
[0052] (4) Comparison with the SOTA method: This invention reports quantitative results on the LLP dataset in Table 1. The table is divided into two parts based on the feature backbone network used. The upper part lists conventional methods (e.g., HAN, MM-Pyr), which rely on ResNet-152 and R(2+1)D-18 to extract 2D and 3D visual features and use VGGish to extract audio features. While these methods establish important baselines, their performance is typically limited to around 61% due to the limited representational power of the backbone network. The lower part (labeled as...) This includes state-of-the-art methods such as LSLD and VAPLAN, which leverage more robust pre-trained features from CLAP and CLIP. As expected, the introduction of these large-scale representations results in comprehensive performance improvements.
[0053] The DMCL proposed in this invention outperforms these major competitors. Combining CLAP / CLIP features and pseudo-labels, DMCL achieved an average score of 65.5% under the same experimental settings, reaching the state-of-the-art level. This result is superior to... It is approximately 0.9% higher and outperforms several other excellent baseline methods, such as and .
[0054] This invention demonstrates a significant advantage at the "segment-level," with DMCL outperforming 68.7% in both audio (70.2% vs. 68.7%) and events (69.3% vs. 68.2%). This is 1.5% and 1.1% higher. This indicates that while powerful features improve the performance of all methods, the multi-level anchoring mechanism of this invention effectively optimizes these features, thereby enabling the capture of clearer event boundaries.
[0055] Table 1 compares the methods with state-of-the-art approaches on the test set of the LLP dataset. Evaluation metrics include F1 scores for audio (A), visual (V), audiovisual (AV) events, Type@AV (Type), and Event@AV (Event). The methods above the horizontal line follow standard LLP feature settings, where 2D and 3D visual features are extracted using ResNet-152 and R(2+1)D-18, and audio features are extracted using VGGish. This indicates the method of using CLAP and CLIP features.
[0056] Table 1
[0057] (5) Ablation experiment: For all ablation experiments, results were reported using a weighted average of segmental and event-level metrics to ensure consistent and fair comparisons.
[0058] Effectiveness of the core modules: To examine the role of each component, an ablation study was conducted in Table 2. Starting from the baseline, adding FBDF improved the average F1 score to 64.9% and improved the event score (64.6% → 65.2%), indicating that foreground-background decoupling helps focus on category-related cues. Using MCL alone achieved an average F1 score of 65.0% and a significant improvement in AV (58.4% → 59.9%), consistent with its goal of enhancing audio-visual consistency through relation learning. Combining FBDF and MCL yielded the best performance (65.5%), indicating that clearer foreground features make anchor selection and pairing construction in MCL more reliable, thereby improving the effectiveness of cross-modal learning.
[0059] Table 2 shows the ablation effectiveness of the FBDF and MACL modules.
[0060]
[0061] MACL Granularity Analysis: Table 3 analyzes the impact of different granularity constraints. Implementing them individually yields only a small improvement (64.4% average), indicating that fine-grained alignment is sensitive to label noise when used alone. In contrast, applying global or video-level constraints separately provides stronger regularization (64.6% average), achieved through stabilizing the feature space. Notably, peak performance (65.0% average) is reached when these levels are combined. This confirms that coarse-grained semantic guidance is indispensable for correcting noisy segmented anchors, ensuring robust alignment at all granularities.
[0062] Table 3 shows the ablation studies of different loss components.
[0063]
[0064] Hyperparameter sensitivity: This invention investigated the effects of the parameters in Table 4 (defined in equation (8)). Even with a small threshold ( =0.1, average 64.7), the model still performs strongly, indicating its good robustness. When At a moderate threshold (0.5-0.7), performance drops slightly, likely because the threshold removes some useful positive samples while still retaining some noise. The best results are achieved in... The value is obtained when it is 1.0, at which point the boundary value is... This produces more reliable anchor points and provides clearer pairings for contrast alignment.
[0065] Table 4 shows the sensitivity analysis of the threshold hyperparameter α of LLP.
[0066]
[0067] (6) Qualitative analysis
[0068] Figure 5 This is a schematic diagram illustrating the time segmentation results of LSLD, VALOR, and the method of the present invention according to embodiments of the present invention. Figure 5 As shown, for the visual trajectory, the real-world scenario reveals a vehicle event almost throughout the entire segment. VALOR predicts no event, while LSLD produces fragmented vehicle segments with missing intervals. In contrast, the method of this invention generates a temporally consistent vehicle prediction that better matches the real-world scenario. For the audio trajectory, the real-world scenario includes both speech and vehicles. LSLD exhibits discontinuous detection and inaccurate breakpoints, while VALOR fails to detect vehicle events while providing incomplete speech segments. The method of this invention recovers both speech and vehicles with clearer boundaries and fewer missed areas, demonstrating improved cross-modal consistency in temporal localization.
[0069] As described above, this invention provides a weakly supervised audio-visual parsing system and method based on decoupled multi-level contrastive learning. The foreground-background decoupling fusion module generates a clean, class-aware foreground representation matrix embedding, while the multi-level anchor-guided contrastive learning module refines intra-modal and inter-modal relationships through cue-guided anchors, memory, and video-level contrast, while capturing temporal dependencies. Extensive experiments demonstrate that the method provided in this invention consistently outperforms state-of-the-art methods.
[0070] Compared with the prior art, the present invention has the following technical effects: (1) This invention assigns a separate branch to each event category, as well as a dedicated background branch. This design ensures that the final representation focuses on relevant category information, effectively filtering out background noise. Then, to fuse audio and video information while maintaining category relevance, label-guided cross-attention is used to selectively aggregate information from the foreground and background components of each modality. This design is introduced to reduce background-dominated responses and emphasize category-related associations so that subsequent parsing and relational modeling operations are performed on a more category-aware representation, rather than in an entangled context.
[0071] (2) Based on the category foreground representation generated by the foreground / background decoupling fusion module, this invention employs multi-level anchor-guided contrastive learning. Most existing work on weakly supervised audio-visual parsing focuses on video-level interactions, largely neglecting intra-modal and inter-modal structural relationships. Meanwhile, other methods treat fragment pseudo-labels as independent instances, ignoring their natural dependencies. Therefore, the multi-level anchor-guided contrastive learning module aims to fill this gap by using cross-modal multi-level anchor-guided contrastive learning. The multi-level anchor-guided contrastive learning module captures intra-modal and inter-modal relationships at both the fragment and global levels, while utilizing video-level aggregation to bridge the gap with weak supervision.
[0072] (3) In the fragment-level model, this invention performs local contrastive learning, which uses fixed text anchor embeddings computed from class label cues using CLIP / CLAP to generate fragment representations and uses the fragment with the highest matching degree in each class as the anchor. Based on the similarity between anchors and fragments in the video, the multi-level anchor-guided contrastive learning module uses two lightweight rule generators to create positive and negative clusters around the anchors. Local contrastive loss helps improve intramodal compactness and establish temporal relationships between fragments of the same event. At the global level, the multi-level anchor-guided contrastive learning module maintains a FIFO memory for each modality to store high-confidence fragment features from the positive sample set guided by the anchor boxes, which are distributed across different videos and mini-batches. Each refined positive sample fragment is encouraged to be close to the FIFO memory of its class, thereby achieving stability across videos and batches. At the video level, the multi-level anchor-guided contrastive learning module further utilizes video-level labels by applying contrastive learning to the temporally aggregated video representation from fragment features. Videos with shared overlapping event labels are brought closer together, while videos with disjoint labels are pushed apart, thus providing a coarse but reliable cross-modal semantic alignment.
[0073] In summary, the multi-level anchor-guided contrastive learning module integrates segment-level, global-level, and video-level contrast objectives, addressing the issues of local differentiation, cross-video consistency, and global semantic alignment, respectively. Combined with the foreground / background decoupling fusion module, this design mitigates the impact of background noise and achieves more accurate temporal localization and more robust audio / video alignment in weakly supervised audio / video parsing.
[0074] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0075] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0076] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0077] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0078] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0079] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0080] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A weakly supervised audio-visual parsing system based on decoupled multi-level contrastive learning, characterized in that, include: The module includes a foreground / background decoupling and fusion module and a multi-level anchor-guided comparative learning module; among them... The foreground / background decoupling and fusion module is used for: The features of the segment to be parsed are projected onto the dimension-reduced decoupled latent space based on a preset projection function to obtain the foreground matrix and background vector; the features of the segment to be parsed include audio segment features or time-frequency segment features. The foreground matrix and background vector are integrated with contextual information based on a multi-head attention mechanism to obtain a context-aware foreground representation matrix and background representation vector. Based on the dynamic gating mechanism, the background contribution of each category is adaptively weighted to obtain foreground weighted features and adaptive fusion features based on gating vectors; The multi-level anchor-guided contrastive learning module is used for: The foreground-weighted features are projected onto a shared space to generate aligned fragment features; Based on the trained encoder and the pre-defined modal text prototype, determine the single anchor box fragment corresponding to the fragment features of each category; Using the individual anchor box fragments corresponding to the fragment features of each category as a reference, positive and negative sample pairs are constructed based on adaptive boundaries; Based on the positive and negative sample pairs and the fragment-level contrastive learning loss function, fragment-level contrastive learning is performed on the single anchor box fragment; Based on a preset modality sequence and a global objective function, global comparative learning is performed on the modality of each category; Based on the video-level contrast objective function and positive and negative sample pairs constructed by label overlap, video-level contrast learning is performed on the video to be analyzed.
2. The system of claim 1, wherein, The preset projection function includes: In the formula, is a multi-layer perceptron mapping; The foreground matrix and the background vector include: wherein and denote the foreground matrix and the background vector, respectively, t denotes a time step, is the feature of the segment to be analyzed. The decoupling loss function for the foreground matrix and the background vector includes: In the formula, T is the total time step of the features of the segment to be parsed, and C is the total number of the foreground matrices.
3. The system of claim 2, wherein, The gate vector includes: The foreground-weighted features include: ; The adaptive fusion features include: ; The gating loss function of the dynamic gating mechanism includes: In the formula, These are the foreground representation matrix and the background representation vector, respectively, and Y is the video-level label.
4. The method according to claim 1, characterized in that, The adaptive boundary includes: In the formula, and α represents the mean and standard deviation of the similarity between a single anchor frame fragment and the preset modal text prototype, respectively, and α is a control variable.
5. The method according to claim 4, characterized in that, The fragment-level contrastive learning loss function includes: In the formula, sim represents the similarity. This represents a single anchor box segment, where p is the positive sample in the positive-negative sample pair, and q is the sample in the positive-negative sample pair. This refers to the temperature parameter.
6. The method according to claim 5, characterized in that, The global objective function includes: In the formula, This represents the set of positive samples in a positive-negative sample pair. The preset mode sequence.
7. The method according to claim 5, characterized in that, The video-level contrast objective function includes: In the formula, This represents a set of videos in a small batch. Videos defined by tag overlap The main collection, To represent different segment features.
8. A weakly supervised audio and video parsing method based on decoupled multi-level contrastive learning, characterized in that, A weakly supervised audio and video parsing system based on decoupled multi-level contrastive learning as described in any one of claims 1-7; the method includes: The foreground and background decoupling and fusion module decouples the foreground and background of the fragment to be parsed, and optimizes it based on a multi-head attention mechanism with adaptive gating to obtain foreground weighted features and adaptive fusion features. Based on the multi-level anchor-guided contrast learning module, the foreground weighted features are subjected to segment-level contrast learning, global contrast learning, and video-level contrast learning, respectively.
9. The method according to claim 8, characterized in that, The foreground-background decoupling and fusion module decouples the foreground and background of the features to be parsed, and optimizes them using a multi-head attention mechanism with adaptive gating, resulting in foreground-weighted features and adaptive fused features, including: The features of the segment to be parsed are projected onto the dimension-reduced decoupled latent space based on a preset projection function to obtain the foreground matrix and background vector; the features of the segment to be parsed include audio segment features or time-frequency segment features. The foreground matrix and background vector are integrated with contextual information based on a multi-head attention mechanism to obtain a context-aware foreground representation matrix and background representation vector. By adaptively weighting the background contribution of each category based on a dynamic gating mechanism, we obtain foreground weighted features and adaptive fusion features based on gating vectors.
10. The method according to claim 9, characterized in that, Based on a multi-level anchor-guided contrastive learning module, segment-level contrastive learning, global contrastive learning, and video-level contrastive learning are performed on the foreground weighted features, including: The foreground-weighted features are projected onto a shared space to generate aligned fragment features; Based on the trained encoder and the pre-defined modal text prototype, determine the single anchor box fragment corresponding to the fragment features of each category; Using the individual anchor box fragments corresponding to the fragment features of each category as a reference, positive and negative sample pairs are constructed based on adaptive boundaries; Based on the positive and negative sample pairs and the multi-positive contrast objective function, fragment-level contrast learning is performed on the single anchor frame fragment; Based on a preset modality sequence and a global objective function, global comparative learning is performed on the modality of each category; Based on the video-level contrast objective function and positive and negative sample pairs constructed by label overlap, video-level contrast learning is performed on the video to be analyzed.