Cross-view audio-visual enhancement unsupervised time sequence action positioning method based on CLIP assistance

Abstract

The invention discloses an unsupervised time sequence action positioning method for cross-view audio-visual enhancement based on CLIP assistance. The method comprises the following steps: extracting audio features, classification pre-training (CBP) visual features and visual language pre-training (VLP) features from an input video; performing multi-stage iterative optimization on the audio and CBP features through an audio and video cross attention fusion module, and generating audio-visual fusion features retaining modal features in combination with cross attention weight calculation and dense jump connection; executing a multi-modal attention mechanism on the CBP and VLP features by using a cross-view cooperation normal form, and generating enhanced double-view features; a self-supervised learning mechanism is further introduced, feature decorrelation constraint is applied to the fused features, cross-view instance distinguishing constraint is applied to the enhanced view features so as to enhance the representation discrimination force, and the model is jointly optimized; and finally, outputting a high-precision action time sequence positioning result. According to the method, multi-modal information is fused and self-supervised learning is cooperated, so that the action boundary positioning integrity and the recognition robustness of audio-related actions are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multimedia information processing, and in particular to an unsupervised temporal action localization (UTAL) method based on contrastive language-image pretraining (CLIP) assisted by cross-view audio-visual enhancement (AE). Background Technology

[0002] With the rapid growth of social media video data, video retrieval technology has been a hot research area in information retrieval for decades, but it faces significant challenges. Traditional methods mainly rely on text queries to filter semantic matching results from massive video libraries. However, in real-world applications, users often need to accurately locate the temporal boundaries of specific actions in unedited videos and identify their semantic categories—this need has spurred the rapid development of Temporal Action Localization (TAL) technology.

[0003] Previous methods primarily relied on fully supervised temporal action localization (TAL), such as Figure 1 As shown in the left and middle figures, these methods require time-consuming and error-prone temporal action boundary annotations during model training. Furthermore, the highly subjective annotation results can also affect overall localization performance. Although some methods attempt to reduce the cost of boundary annotation in weakly supervised learning environments, action category annotations are still required, which is also a labor-intensive task. The spatiotemporal complexity of video content (such as ambiguous action start points and concurrent multi-action events) and the weak interpretability of cross-modal semantic associations make it difficult for traditional retrieval methods to meet the needs of fine-grained action analysis. The problems with existing technologies are summarized as follows:

[0004] 1. Reliance on labeled data: Traditional temporal action localization (TAL) methods rely on fully supervised or weakly supervised labeled data (such as time boundaries and action categories), which are costly and subjective, limiting the model's generalization ability.

[0005] 2. Feature limitations: Existing unsupervised methods (UTAL) are mostly based on visual features from classification pre-training (CBP). These features overemphasize highly discriminative regions (such as action keyframes), resulting in incomplete action range localization.

[0006] 3. Modal singularity: Existing methods rely solely on visual modalities, ignoring the supplementary role of audio information in contextual boundaries. For example, actions strongly related to audio, such as blowing a trumpet or singing, are difficult to locate accurately.

[0007] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0008] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide an unsupervised temporal action localization method based on CLIP-assisted cross-view audiovisual enhancement.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] An unsupervised temporal action localization method based on CLIP-assisted cross-view audiovisual enhancement includes the following steps:

[0011] S1. Multimodal feature extraction: Audio features, CBP features, and VLP features are extracted using a pre-trained audio feature extractor, a classification pre-trained (CBP) feature extractor, and a visual language pre-trained (VLP) feature extractor, respectively.

[0012] S2, Audio-visual cross-attention fusion: The audio features and CBP features are input into a multi-stage static cross-attention fusion module. Through iterative cross-attention weight calculation and dense jump connections, audio-visual fusion features that retain modality-specific information are generated.

[0013] S3. Enhanced cross-view collaboration: The CBP features and VLP features are input into the cross-view collaboration paradigm, and enhanced CBP view features and enhanced VLP view features are generated through a multimodal attention mechanism.

[0014] S4. Self-supervised cross-view learning: Perform feature decorrelation constraints on the audiovisual fusion features, and perform cross-view instance discrimination constraints on the enhanced CBP view features and enhanced VLP view features, and jointly optimize the model;

[0015] S5. Action localization output: Output action temporal localization results based on optimized features.

[0016] Furthermore, the audio feature extractor in step S1 adopts a VGG-like network structure, and the VLP feature extractor adopts the CLIP-ViT model.

[0017] Furthermore, step S1 also includes:

[0018] The extracted raw audio features, CBP features, and VLP features are encoded into dimension-aligned feature representations by using fully connected layers that are independently configured for each modality.

[0019] Furthermore, the implementation of the multi-stage static cross-attention fusion module in step S2 includes:

[0020] Modality-specific encoding is performed on both audio features and CBP features;

[0021] Calculate the cross-correlation matrix between the two modes and generate cross-attention weights;

[0022] Attention-weighted features are iteratively optimized through multi-stage dense skip connections to preserve intramodal information and encode intermodal correlations.

[0023] Furthermore, the multi-stage dense skip connection is implemented in the following way:

[0024] In each iteration, the features from the historical stages are summed and fused with the current attention-weighted features, and the optimized features are output through a non-linear activation function.

[0025] Furthermore, the implementation of the cross-view collaboration paradigm described in step S3 includes:

[0026] Segmenting CBP view features with VLP view features;

[0027] Multimodal attention computation is performed on the CBP view and VLP view respectively to generate enhanced cross-view collaborative features.

[0028] Furthermore, the feature decorrelation constraint described in step S4 is implemented as follows:

[0029] Minimize the difference between the autocorrelation matrix and the identity matrix of each dimension of the audiovisual fusion feature, thereby forcing the feature representation to be orthogonal.

[0030] Furthermore, the implementation method of the cross-view instance differentiation constraint in step S4 is as follows:

[0031] Average pooling is applied to both enhanced CBP view features and enhanced VLP view features. The instance discrimination loss function is used to narrow the distance between cross-view features of the same sample and widen the distance between features of different samples.

[0032] Furthermore, the joint optimization in step S4 employs a combined loss function, which is composed of the loss term corresponding to the feature decorrelation constraint and the loss term corresponding to the cross-view instance differentiation constraint.

[0033] A computer program product includes a computer program that, when executed by a processor, implements the CLIP-assisted cross-view audiovisual enhancement unsupervised temporal action localization method.

[0034] The present invention has the following beneficial effects:

[0035] This invention proposes an unsupervised temporal action localization method based on multimodal fusion. By adopting an unsupervised paradigm, it completely eliminates the reliance on time-consuming and highly subjective manual annotation (temporal boundaries and action categories). It overcomes the dual limitations of traditional methods by utilizing a cross-view audiovisual enhancement mechanism—addressing the problem of incomplete localization caused by excessive focus on keyframes in classification pre-training features by introducing a collaborative enhancement mechanism of Visual Language Pre-training (VLP) and Classification Pre-training (CBP), fully capturing the complete action range through cross-view collaboration. Addressing the deficiency of single visual modalities in determining contextual boundaries, it innovatively integrates audio perception information. Through a multi-stage static cross-attention fusion module, it preserves modal characteristics while strengthening audio-visual correlation, significantly improving the localization accuracy of audio-related actions (such as playing horns, singing, etc.). Furthermore, by combining a self-supervised cross-view learning paradigm (feature decorrelation and cross-view instance differentiation), multi-view perception enhancement can be achieved without additional annotation. This invention validates its robustness and generalization ability, surpassing existing state-of-the-art methods, on public datasets, providing an efficient solution for fine-grained action analysis in complex video scenes.

[0036] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the timing action localization at different levels of supervision, showing a comparison of the annotation requirements for fully supervised, weakly supervised, and unsupervised TAL.

[0038] Figure 2 This is the overall flowchart of the unsupervised temporal motion localization method for cross-view audiovisual enhancement AE based on CLIP-assisted method of the present invention.

[0039] Figure 3 This is an algorithm framework diagram of CLIP-AE according to an embodiment of the present invention. Detailed Implementation

[0040] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0041] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0042] Achieving high-precision and robust temporal action localization without manual annotation has become a key issue in promoting the implementation of intelligent video understanding. However, the currently popular unsupervised temporal action localization (UTAL) methods face two major challenges: 1) overemphasis on highly discriminative regions based on classification pre-training features; 2) difficulty in determining context boundaries relying solely on visual modality information. To address these issues, this invention proposes a CLIP-assisted cross-view audiovisual enhancement UAL method, introducing a collaborative enhancement mechanism of visual language pre-training (VLP) and classification pre-training to avoid overemphasis on highly discriminative regions; simultaneously, it integrates audio perception information to provide richer context boundary information. This invention also proposes a self-supervised cross-view learning paradigm that achieves multi-view perception enhancement without additional annotation. Extensive experiments on two public datasets demonstrate (which will be further detailed later) that the model of this invention outperforms several existing state-of-the-art methods.

[0043] See Figure 2 This invention provides an unsupervised temporal action localization method based on CLIP-assisted cross-view audiovisual enhancement, comprising the following steps:

[0044] Step S1, Multimodal Feature Extraction: Audio features, CBP features, and VLP features are extracted using a pre-trained audio feature extractor, a classification pre-trained (CBP) feature extractor, and a visual language pre-trained (VLP) feature extractor, respectively.

[0045] In some embodiments, the audio feature extractor in step S1 adopts a VGG-like network structure, and the VLP feature extractor adopts a CLIP-ViT model.

[0046] In some embodiments, step S1 further includes: encoding the extracted original audio features, CBP features, and VLP features into dimension-aligned feature representations through fully connected layers independently configured for each modality.

[0047] Step S2, Audio-video cross-attention fusion: Input the audio features and CBP features into the multi-stage static cross-attention fusion module, and generate audio-visual fusion features that retain modality-specific information through iterative cross-attention weight calculation and dense jump connections.

[0048] In some embodiments, the implementation of the multi-stage static cross-attention fusion module in step S2 includes: performing modality-specific encoding on audio features and CBP features respectively; calculating the cross-correlation matrix between the two modalities and generating cross-attention weights; iteratively optimizing the attention-weighted features through multi-stage dense skip connections, preserving intramodal information and encoding intermodal correlations.

[0049] In some embodiments, the multi-stage dense skip connection is implemented by summing the features of the historical stages and fusing them with the current attention-weighted features in each stage iteration, and then outputting optimized features through a non-linear activation function.

[0050] Step S3, Cross-view collaboration enhancement: Input the CBP features and VLP features into the cross-view collaboration paradigm, and generate enhanced CBP view features and enhanced VLP view features through a multimodal attention mechanism.

[0051] In some embodiments, the implementation of the cross-view collaboration paradigm in step S3 includes: splicing CBP view features and VLP view features; performing multimodal attention calculations on the CBP view and VLP view respectively to generate enhanced cross-view collaboration features.

[0052] Step S4, Self-supervised cross-view learning: Perform feature decorrelation constraints on the audiovisual fusion features, perform cross-view instance discrimination constraints on the enhanced CBP view features and the enhanced VLP view features, and jointly optimize the model;

[0053] In some embodiments, the feature decorrelation constraint in step S4 is implemented by minimizing the difference between the autocorrelation matrix and the identity matrix of each dimension of the audiovisual fusion feature, thereby forcing the feature representation to be orthogonal.

[0054] In some embodiments, the cross-view instance discrimination constraint in step S4 is implemented by performing average pooling on the enhanced CBP view features and the enhanced VLP view features respectively, and using the instance discrimination loss function to bring the same cross-view features closer together and push the distance between different sample features further apart.

[0055] In some embodiments, the joint optimization in step S4 employs a combined loss function, which is composed of the loss term corresponding to the feature decorrelation constraint and the loss term corresponding to the cross-view instance differentiation constraint.

[0056] Step S5, Action Localization Output: Output action temporal localization results based on the optimized features.

[0057] The following further describes specific embodiments of the present invention, algorithm examples, and experimental verification.

[0058] like Figure 3 As shown, this embodiment of the invention designs a CLIP-assisted cross-view audiovisual enhancement framework, referred to as the CLIP-AE framework, which includes an audio feature extraction module (VGG-like), a CBP feature extraction module (I3D), a VLP feature extraction module (CLIP-ViT), a cross-attention fusion module (CAF), a cross-view collaboration module (CVC), and a self-supervised learning module.

[0059] First, using pre-trained audio, visual language (VLP), and classification pre-trained (CBP) feature extractors, features for each view are extracted: audio feature F audio CBP features F CBP and VLP features F VLP Next, F audio and F CBP The input is fed into the Cross-View Collaboration Paradigm to construct audiovisual perception fusion features. Then, F... CBP and F VLP The inputs are fed into the Cross-View Collaboration Paradigm to generate enhanced visual features. Finally, these enhanced features are fed into a self-supervised Cross-View Learning module, where representation learning is achieved under self-supervised conditions through two proxy tasks.

[0060] 1. Audio-Visual Cross-Attention Fusion Module

[0061] While multimodal fusion can provide more comprehensive information, it may lead to the loss of modality-specific information during the fusion process. To address this, an audio-video cross-attention fusion module based on a multi-stage dense cross-attention mechanism was designed. Each modality independently learns features while being constrained by other modalities, in order to encode intermodal correlations while preserving intramodal information.

[0062] First, the audio F is connected through a modality-specific fully connected layer. audio and visual features F CBP They are respectively encoded as X audio and X CBP ,in and Then, the cross-correlation matrix of these two features is calculated to assess the correlation between the modes. To reduce the heterogeneity gap between the two modes, a learnable weight matrix is ​​introduced, and the cross-correlation is calculated using the following formula:

[0063]

[0064] in, X audio and X CBP It is L2 normalized before calculating cross-correlation.

[0065] In the cross-correlation matrix, a higher correlation coefficient indicates a high correlation between audio and visual features. Therefore, the l-th column of Λ corresponds to... The correlation with the features of L audio segments. Subsequently, the correlation between Λ and Λ... T Perform a softmax operation on the columns to generate cross-attention weights A. audio and A CBP These attention weights are used to reweight fragment features to enhance their discriminative power in other modalities.

[0066] Specifically, the attention-weighted features are obtained using the following formula:

[0067]

[0068] Each modality guides another modality through attention weights to ensure that meaningful intra-modal information is preserved when cross-attention is applied. Cross-attention is applied multiple times to delve deeper into cross-modal information. To prevent suppression of original modality-specific features during multi-stage cross-attention, dense skip connections are introduced. Specifically, in stage t, dense skip connections are implemented as follows:

[0069]

[0070] in It is X audio tanh represents the hyperbolic tangent activation function. The same operation is applied to generate perceptual features for visual modalities.

[0071] Cross-attention is applied to bring the audio and visual embeddings closer together, while dense skip connections emphasize modality-specific information. By alternating between cross-attention and dense skip connections across multiple stages, the optimal embedding for fusion is progressively learned. The goal is to achieve proper compatibility between the two embeddings while preserving modality-specific information, thus optimizing the training objective.

[0072] 2. Cross-View Collaboration Paradigm

[0073] The features of the VLP view are designed to comprehensively capture temporal action intervals. By establishing collaborative learning between the VLP and CBP views, the problem of incomplete localization caused by classification pre-training can be alleviated. Cross-view collaboration mainly includes the following steps: First, extractors based on classification pre-training and visual language pre-training are used to obtain CBP data respectively. Features of VLP views Then combine the two into M:

[0074] M = [X] CBP;X VLP ]

[0075] Cross-view collaboration CBP view features are calculated using a multimodal attention mechanism:

[0076]

[0077] Similarly, the VLP view features for cross-view collaboration can be obtained:

[0078]

[0079] 3. Self-supervised Cross-View Learning

[0080] Audio-Visual feature decorrelation: Representation learning for audio-video cross-attention fusion is achieved through feature decorrelation, aiming to orthogonalize each dimension of the feature representation. The decorrelation process is as follows:

[0081]

[0082] Cross-View Instance Discrimination: Representation learning for cross-view collaboration paradigms is based on instance discrimination. For cross-view collaboration feature Z... VLP and Z CBP First, average pooling is performed to obtain z. VLP , z CBP Then, the instance differentiation process is performed:

[0083]

[0084] Where τ is the temperature hyperparameter (set to 1.0), and It is the direct counterpart of the CBP view and the VLP view. It is a memory bank that maintains the momentum mechanism for each view.

[0085] The final self-supervised loss function is defined as a combination of these sub-losses:

[0086]

[0087] Experimental results

[0088] Table 1 compares the experimental results of this invention with state-of-the-art methods on the THUMOS'14 dataset, with CLIP-AE representing the experimental results of this method. As shown in Table 1, under weakly supervised settings, the method of this invention achieves comparable performance to state-of-the-art methods, obtaining an average mAP of 48.0%. Under unsupervised settings, the method of this invention significantly outperforms the previous state-of-the-art method UGCT, obtaining an average mAP of 41.7%, which fully verifies the effectiveness of the proposed method. Table 2 compares the experimental results of this invention with state-of-the-art methods on the ActivityNet v1.2 dataset, with CLIP-AE representing the experimental results of this method. As shown in Table 2, under weakly supervised settings, the method of this invention surpasses the previous state-of-the-art method APSL, obtaining an average mAP of 30.5%. Under unsupervised settings, the method of this invention significantly outperforms the previous state-of-the-art method APSL, obtaining an average mAP of 28.9%, which fully verifies the effectiveness of the proposed method in larger-scale action scenarios.

[0089] Table 1

[0090]

[0091] Table 1 shows a performance comparison with existing state-of-the-art (SOTA) methods on the THUMOS'14 dataset. 'A' represents a reimplementation of UTAL in UGCT.

[0092] Table 2

[0093]

[0094] Table 2. Performance comparison with existing state-of-the-art (SOTA) methods on the ActivityNet v1.2 dataset. ∩ represents the reimplementation of UTAL in UGCT.

[0095] In summary, this invention proposes an unsupervised temporal action localization method based on CLIP-assisted cross-view audiovisual enhancement. Compared with existing technologies, this invention effectively addresses the shortcomings of existing technologies, such as reliance on labeled data, feature limitations, and modality singularity. First, audio features, classification pre-training (CBP) visual features, and visual language pre-training (VLP) features are extracted from the input video. Then, the audio and CBP features are iteratively optimized in multiple stages through an audio-video cross-attention fusion module. Combining cross-attention weight calculation and dense skip connections, audiovisual fusion features that retain modal characteristics are generated. Simultaneously, a multimodal attention mechanism is applied to the CBP and VLP features using a cross-view collaboration paradigm to generate enhanced dual-view features. Furthermore, a self-supervised learning mechanism is introduced to apply feature decorrelation constraints to the fused features to improve orthogonality, and cross-view instance discrimination constraints are applied to the enhanced view features to strengthen representational discriminative power, jointly optimizing the model. Finally, high-precision action temporal localization results are output. This invention integrates multimodal information and collaborative self-supervised learning, significantly improving the integrity of action boundary localization and the robustness of audio-related action recognition. Experimental results show that it outperforms existing state-of-the-art methods.

[0096] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0097] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0098] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0099] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc or CD-ROM; magnetic surface memory can be disk storage or magnetic tape storage. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0100] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0101] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0102] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0103] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0105] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0106] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0107] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0108] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A CLIP-assisted unsupervised temporal action localization method for cross-view audiovisual enhancement, characterized in that, Includes the following steps: S1. Multimodal feature extraction: Audio features, CBP features, and VLP features are extracted using a pre-trained audio feature extractor, a classification pre-trained (CBP) feature extractor, and a visual language pre-trained (VLP) feature extractor, respectively. S2, Audio-visual cross-attention fusion: The audio features and CBP features are input into a multi-stage static cross-attention fusion module. Through iterative cross-attention weight calculation and dense jump connections, audio-visual fusion features that retain modality-specific information are generated. S3. Enhanced cross-view collaboration: The CBP features and VLP features are input into the cross-view collaboration paradigm, and enhanced CBP view features and enhanced VLP view features are generated through a multimodal attention mechanism. S4. Self-supervised cross-view learning: Perform feature decorrelation constraints on the audiovisual fusion features, and perform cross-view instance discrimination constraints on the enhanced CBP view features and enhanced VLP view features, and jointly optimize the model; S5. Action localization output: Output action temporal localization results based on optimized features.

2. The method as described in claim 1, characterized in that, The audio feature extractor in step S1 adopts a VGG-like network structure, and the VLP feature extractor adopts the CLIP-ViT model.

3. The method as described in claim 1 or 2, characterized in that, Step S1 also includes: The extracted raw audio features, CBP features, and VLP features are encoded into dimension-aligned feature representations by using fully connected layers that are independently configured for each modality.

4. The method according to any one of claims 1 to 3, characterized in that, The implementation of the multi-stage static cross-attention fusion module in step S2 includes: Modality-specific encoding is performed on both audio features and CBP features; Calculate the cross-correlation matrix between the two modes and generate cross-attention weights; Attention-weighted features are iteratively optimized through multi-stage dense skip connections to preserve intramodal information and encode intermodal correlations.

5. The method as described in claim 4, characterized in that, The multi-stage dense skip connection is implemented in the following way: In each iteration, the features from the historical stages are summed and fused with the current attention-weighted features, and the optimized features are output through a non-linear activation function.

6. The method according to any one of claims 1 to 5, characterized in that, The implementation of the cross-view collaboration paradigm in step S3 includes: Segmenting CBP view features with VLP view features; Multimodal attention computation is performed on the CBP view and VLP view respectively to generate enhanced cross-view collaborative features.

7. The method according to any one of claims 1 to 6, characterized in that, The feature decorrelation constraint described in step S4 is implemented as follows: Minimize the difference between the autocorrelation matrix and the identity matrix of each dimension of the audiovisual fusion feature, thereby forcing the feature representation to be orthogonal.

8. The method according to any one of claims 1 to 7, characterized in that, The implementation method of the cross-view instance differentiation constraint in step S4 is as follows: Average pooling is applied to both enhanced CBP view features and enhanced VLP view features. The instance discrimination loss function is used to narrow the distance between cross-view features of the same sample and widen the distance between features of different samples.

9. The method according to any one of claims 1 to 8, characterized in that, The joint optimization in step S4 employs a combined loss function, which is composed of the loss term corresponding to the feature decorrelation constraint and the loss term corresponding to the cross-view instance differentiation constraint.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the CLIP-assisted unsupervised temporal action localization method for cross-view audiovisual enhancement as described in any one of claims 1 to 9.

Images