A multimodal information bottleneck-based video highlight detection method and system

Abstract

This invention discloses a video specular highlight detection method and system with multimodal information bottlenecks. By utilizing an environmental pseudo-label through a causal attention perception module, the input features are decoupled into causal and environmental branches, thus removing spurious correlations from environmental confounding variables at the feature level and enhancing the model's generalization ability on difficult samples. Simultaneously, a single-modal information bottleneck layer constructed based on slice Gromov-Wasserstein distance replaces the traditional relative entropy constraint with a topological rate distortion metric. This preserves the high-dimensional geometric manifold and temporal causal logic of specular highlight actions during compression and dimensionality reduction, avoiding excessive feature smoothing. While reducing feature redundancy, this significantly improves the generalization robustness and localization accuracy of video specular highlight detection.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence multimedia data processing and content analysis technology, and in particular to a video highlight detection method and system for multimodal information bottlenecks. Background Technology

[0002] In multimodal video highlight detection tasks, visual and audio features are typically extracted from the video, and highlight moments are located through feature fusion and temporal modeling. However, real video streams often contain environmental confounding variables. For example, there is a spurious correlation between cheers from the audience and the action in a sports event, causing the model to over-rely on high-energy audio as the basis for highlight detection. This results in a significant drop in generalization ability when faced with counterintuitive samples such as silent highlights in away games.

[0003] Meanwhile, existing information bottleneck methods typically use relative entropy as a constraint metric when compressing multimodal features to remove redundant information. The constraint method that only focuses on the alignment of statistical distributions is prone to destroying the temporal geometric manifold of specular action in the original high-dimensional space, resulting in over-smoothing of features and reduced detection accuracy. Summary of the Invention

[0004] In view of this, the purpose of this invention is to propose a video specular detection method and system for multimodal information bottlenecks.

[0005] To achieve the above-mentioned technical objectives, the technical solution adopted by this invention is as follows: In a first aspect, this application provides a video specular detection method for multimodal information bottlenecks, comprising: Obtain the original visual and audio features of the video to be detected; The original visual features and original audio features are input into the causal attention perception module. The causal attention perception module uses the environment pseudo-labels extracted from the audio signal to decouple the input features into mutually orthogonal causal branch features and environment branch features in the feature space. After removing the environmental bias interference, the output is an environment-unchanged causal feature. The environmentally invariant causal features are input into a single-modal information bottleneck layer constructed based on the slice Gromov-Wasserstein distance. The single-modal information bottleneck layer generates a topology-aware single-modal bottleneck embedding by constraining the geometric manifold preservation during the compression process through a topology rate distortion metric. The topology-aware unimodal bottleneck embedding is input into the multimodal information bottleneck layer. The multimodal information bottleneck layer collaboratively learns the loss and variational constraints, and performs cross-modal semantic refinement on the semantically aligned visual and audio bottleneck embeddings to generate a multimodal bottleneck embedding with minimal redundancy. The multimodal bottleneck is embedded into the highlight scorer, which outputs the highlight detection score.

[0006] In some embodiments, the original visual features and original audio features are respectively input to the causal attention perception module. The causal attention perception module uses environmental pseudo-labels extracted from the audio signal to decouple the input features in the feature space into mutually orthogonal causal branch features and environmental branch features. After removing environmental bias interference, it outputs environmentally invariant causal features, including: The root mean square energy of the original audio features is extracted. The root mean square energy is divided into high-energy environment priors and low-energy environment priors using a binarized step function. Discrete environment pseudo-labels are generated, which are used to explicitly distinguish the source of environmental confusion signals in subsequent steps. The original visual features and original audio features are respectively input into the causal attention perception module. The causal attention perception module has a parallel splitting architecture built inside. The parallel splitting architecture includes an environment confusion network and a causal feature network. The environment confusion network and the causal feature network share the same input features but are optimized independently. The original visual or audio features are input into the environment confusion network. The binary cross-entropy loss is used to fit the environment pseudo-labels, so that the environment confusion network learns to capture the environment bias components in the features that are highly correlated with the environment pseudo-labels and outputs the environment branch features. The original visual or audio features are input into the causal feature network. A gradient inversion layer is set before the input layer of the causal feature network. The adversarial loss is used to forcibly weaken the statistical dependence between the output features of the causal feature network and the environmental pseudo-labels, blocking the penetration of environmental bias components into the causal branch and outputting causal branch features. By using an orthogonal loss function to constrain environmental branch features and causal branch features to orthogonalize in geometric space, and by calculating the ratio of the inner product of the two sets of branch features to the product of their respective moduli, the two parts of features are forced to decouple in the feature space, and the output causal features with unchanged environment are obtained.

[0007] In some embodiments, raw visual or raw audio features are input into a causal feature network. A gradient inversion layer is set before the input layer of the causal feature network. Adversarial loss is used to forcibly weaken the statistical dependence between the output features of the causal feature network and the environmental pseudo-labels, blocking the penetration of environmental bias components into the causal branch, and outputting causal branch features, including: The system receives raw visual or audio features as input features to a causal feature network, which learns causal invariant representations that are independent of environmental pseudo-label statistics. A gradient reversal layer is inserted between the input layer and the feature extraction layer of the causal feature network. The gradient reversal layer keeps the input features unchanged during forward propagation and reverses the gradient direction from the adversarial loss during backward propagation. Construct an adversarial loss function, which is used to measure the statistical dependence between the output features of the causal feature network and the pseudo-labels of the environment. The lower the adversarial loss value, the weaker the statistical dependence. During backpropagation, the gradient inversion layer reverses the gradient direction obtained from the adversarial loss calculation and passes it to the feature extraction layer of the causal feature network, causing the causal feature network to update its parameters in the direction of maximizing the adversarial loss. In other words, it forces the causal feature network to learn feature representations that are independent of the pseudo-label statistics of the environment. Through the synergistic effect of gradient inversion layer and adversarial loss, the mutual information between the causal branch features output by the causal feature network and the environmental pseudo-label is minimized, the environmental bias component is blocked outside the causal branch, and the output is a causal branch feature stripped of environmental bias.

[0008] In some embodiments, an orthogonal loss function is used to constrain environmental branch features and causal branch features to orthogonalize in geometric space. By calculating the ratio of the inner product of the two sets of branch features to the product of their respective moduli, the two parts of features are forced to decouple in the feature space, outputting environmentally invariant causal features, including: The environmental branch features and causal features output by the environmental confusion network are received. The environmental branch features are rich in environmental bias components that are highly correlated with the environmental pseudo-labels. The statistical dependence of the causal branch features on the environmental pseudo-labels has been weakened by the gradient inversion layer and adversarial loss. Construct an orthogonal loss function, which is used to calculate the cosine of the angle between the environmental branch features and the causal branch features in geometric space. The cosine of the angle is obtained by calculating the ratio of the inner product of the two sets of branch features to the product of their respective moduli. The cosine of the included angle is added as an orthogonal constraint term to the training objective. During the optimization process, the orthogonal constraint term drives the included angle between the environmental branch features and the causal branch features in the geometric space to tend to 90 degrees, so that the two sets of features become orthogonal in the feature space. Through the synergistic effect of orthogonal constraint terms and adversarial loss, adversarial loss weakens the dependency between causal branch features and environmental pseudo-labels at the statistical level, while orthogonal constraint terms force environmental branch features and causal branch features to occupy mutually independent feature subspaces at the geometric level, forming a dual decoupling mechanism. Driven by the dual decoupling mechanism, environmental branch features and causal branch features are completely separated in the feature space. Environmental bias components are completely isolated in the environmental branch, while only environmentally invariant semantic information related to the specular detection task is retained in the causal branch, outputting environmentally invariant causal features.

[0009] In some embodiments, environmentally invariant causal features are input into a single-modal information bottleneck layer constructed based on slice Gromov-Wasserstein distance. The single-modal information bottleneck layer generates a topology-aware single-modal bottleneck embedding by constraining geometric manifold preservation during the compression process through a topology rate distortion metric, including: The received environment-invariant visual causal features and environment-invariant audio causal features are used as unimodal input features. The unimodal input features contain the temporal geometric manifold structure of specular action in the original high-dimensional space. By removing the relative entropy constraint from the standard variational information bottleneck framework, the measure of information compression is reconstructed into a topological rate-distortion metric based on the slice Gromov-Wasserstein distance. The topological rate-distortion metric is used to measure the geometric structural difference between the original high-dimensional feature distribution and the compressed low-dimensional embedding distribution. The single-modal input features are projected into a low-dimensional bottleneck embedding space. During the projection process, the slice Gromov-Wasserstein distance between the feature distributions before and after projection is calculated using the topological rate distortion metric. The slice Gromov-Wasserstein distance is then added to the training objective as a geometric constraint regularization loss. By using geometric constraint regularization loss to guide the unimodal information bottleneck layer to retain the original features of the high-dimensional geometric manifold structure and action temporal causal logic during dimensionality reduction and compression, the problem of excessive feature smoothing caused by relative entropy constraints in traditional variational information bottlenecks is avoided, and a topology-aware unimodal bottleneck embedding is generated.

[0010] In some embodiments, the relative entropy constraint is removed from the standard variational information bottleneck framework, and the measure of information compression is reconstructed as a topological rate-distortion metric based on the slice Gromov-Wasserstein distance. This topological rate-distortion metric measures the geometric structural difference between the original high-dimensional feature distribution and the compressed low-dimensional embedding distribution, including: The relative entropy constraint term in the standard variational information bottleneck framework is identified. The relative entropy constraint term drives the compression process by measuring the probability density difference between the original feature distribution and the compressed embedding distribution, but only focuses on the alignment at the statistical distribution level, ignoring the geometric manifold structure and temporal causal logic of the data in high-dimensional space. The relative entropy constraint term is removed from the standard variational information bottleneck framework to eliminate the smoothing effect of the relative entropy constraint term on the feature geometric manifold structure during the compression process. The smoothing effect will cause the temporal coherence of the specular action to be destroyed during dimensionality reduction. We introduce the slice Gromov-Wasserstein distance as an alternative metric. The slice Gromov-Wasserstein distance calculates the Wasserstein distance between two one-dimensional projected distributions by projecting the high-dimensional feature distribution and the low-dimensional embedding distribution onto a randomly sampled one-dimensional subspace, and then takes the average after multiple random projections. This achieves a low-complexity approximate metric for the differences in high-dimensional geometric structures. The Gromov-Wasserstein distance of the slices is constructed as a topological rate-distortion metric. The topological rate-distortion metric is used to constrain the geometric structural differences between the original high-dimensional feature distribution and the compressed low-dimensional embedding distribution during the compression process, rather than the statistical distribution differences. By using a topology rate distortion metric, the unimodal information bottleneck layer is guided to retain the original features of the high-dimensional geometric manifold structure and action temporal causal logic during dimensionality reduction and compression, and outputs a topology-aware unimodal bottleneck embedding.

[0011] In some embodiments, a topology-aware unimodal bottleneck embedding is input to a multimodal information bottleneck layer. The multimodal information bottleneck layer collaboratively performs a contrastive learning loss and variational constraints to perform cross-modal semantic refinement on the semantically aligned visual and audio bottleneck embeddings, generating a minimally redundant multimodal bottleneck embedding, including: The topology-aware visual bottleneck embedding and the topology-aware audio bottleneck embedding are received as unimodal input embeddings. The visual bottleneck embedding and the audio bottleneck embedding are asynchronous in time, which leads to a misalignment in their semantic expression. The visual bottleneck embedding and audio bottleneck embedding are respectively spliced ​​and fused with the two branches to generate multimodal representations. Cross-modal comparison and association calculation is performed. Contrastive learning loss is introduced to narrow the semantic distance of audiovisual bottleneck embeddings under the same video context, eliminate the semantic expression misalignment caused by multimodal asynchrony, and generate semantically aligned visual bottleneck embedding and audio bottleneck embedding. The semantically aligned visual bottleneck embedding and audio bottleneck embedding are dynamically fused to construct a multimodal fusion attention feature map, which contains semantically calibrated cross-modal interaction information. The multimodal fusion attention feature map is input into the multimodal information bottleneck layer. During forward propagation computation, the multimodal variational information bottleneck loss is used to perform deep semantic purification and secondary dimensionality reduction on the fused cross-modal representation. Non-critical redundant semantic flows coexisting between the two modalities are eliminated, and the core semantic information that is highly related to the specular detection task is retained, generating a multimodal bottleneck embedding with minimal redundancy and high relevance to the prediction task.

[0012] In some embodiments, the visual bottleneck embedding and audio bottleneck embedding are respectively combined with the multimodal representation generated after splicing and fusing the two branches to perform cross-modal comparative association calculation. A contrastive learning loss is introduced to narrow the semantic distance between the audiovisual bottleneck embeddings in the same video context, eliminating semantic expression misalignment caused by multimodal asynchrony, and generating semantically aligned visual bottleneck embeddings and audio bottleneck embeddings, including: The topology-aware visual bottleneck embedding and the topology-aware audio bottleneck embedding are inherently asynchronous on the time axis. This asynchronousness manifests as a time offset between the occurrence of a visual event and the corresponding audio response, resulting in the visual representation and audio representation in the same video context being located in different positions in the semantic space. Visual bottleneck embedding and audio bottleneck embedding are spliced ​​and fused to generate a multimodal fusion representation, which serves as an anchor reference for cross-modal alignment. Contrastive learning loss function is constructed, with multimodal fusion representation as positive sample anchor point, visual bottleneck embedding and audio bottleneck embedding under the same video context as positive sample pair, and visual bottleneck embedding and audio bottleneck embedding under different video context as negative sample pair. By minimizing the contrastive learning loss function, the semantic distance between the visual bottleneck embedding and the multimodal fusion representation is reduced, and the semantic distance between the audio bottleneck embedding and the multimodal fusion representation is also reduced, so that the visual bottleneck embedding and the audio bottleneck embedding move closer to the common fusion representation center in the semantic space. Driven by the contrastive learning loss function, the temporal asynchronous offset between the visual bottleneck embedding and the audio bottleneck embedding is eliminated, and the two occupy consistent positions in the semantic space, generating semantically aligned visual bottleneck embeddings and audio bottleneck embeddings.

[0013] In some embodiments, a multimodal bottleneck is embedded into the input of a specular scorer, and the specular scorer outputs a specular detection score, including: The minimum redundancy multimodal bottleneck embedding receives the output of the multimodal information bottleneck layer. The minimum redundancy multimodal bottleneck embedding has eliminated non-critical semantic flows that coexist across modalities and retained the core semantic information that is highly relevant to the specular detection task. The minimum redundancy multimodal bottleneck is embedded into the input of the highlight scorer, which includes a linear mapping layer and a nonlinear activation unit. The highlight score prediction value corresponding to the current video segment is calculated through forward mapping. Based on the predicted value of the highlight score and the real label of the highlight annotation, the task loss value, which measures the localization performance of the model at the highlight moment, is calculated using the binary cross-entropy loss function. The task loss value is backpropagated to the multimodal information bottleneck layer and the unimodal information bottleneck layer, driving the multimodal information bottleneck layer and the unimodal information bottleneck layer to further optimize the selection and retention of core semantic information in subsequent iterations, forming a collaborative optimization closed loop of feature refinement and score prediction, and outputting the final highlight detection score result.

[0014] In a second aspect, this application also provides a video specular detection system for multimodal information bottlenecks, applicable to the method described in the first aspect, the system comprising: The data acquisition and feature extraction module is used to receive the raw video stream and audio stream, and extract the raw visual features and raw audio features using the pre-trained visual encoder and pre-trained audio encoder, respectively. The environment pseudo-label generation module is connected to the data acquisition and feature extraction module. It is used to receive the original audio features, extract the root mean square energy of the audio segment, and generate discrete environment pseudo-labels using the binarized step function. The causal attention perception module is connected to the data acquisition and feature extraction module and the environment pseudo-label generation module. It is used to receive visual raw features, audio raw features and environment pseudo-labels. Internally, it contains a parallel environment confusion network and a causal feature network. A gradient inversion layer is set in front of the input layer of the causal feature network. It uses binary cross-entropy loss, adversarial loss and orthogonal loss to drive the input features in the feature space to decouple the input features into mutually orthogonal environment branch features and causal branch features, and outputs causal features that are invariant to the environment. The unimodal topological information bottleneck compression module, connected to the causal attention perception module, is used to receive causal features that are invariant to the environment. Internally, it removes the relative entropy constraints in the standard variational information bottleneck framework and reconstructs the measure of information compression degree into a topological rate-distortion metric based on the slice Gromov-Wasserstein distance. The topological rate-distortion metric constrains the preservation of geometric manifolds during the compression process, generating a topology-aware unimodal bottleneck embedding. The multimodal semantic refinement module, connected to the single-modal topology information bottleneck compression module, is used to receive topology-aware single-modal bottleneck embeddings. It contains a contrast alignment unit and a multimodal information bottleneck layer. The contrast alignment unit uses the multimodal representation generated by splicing and fusion as an anchor point and uses contrastive learning loss to narrow the semantic distance between the visual bottleneck embedding and the audio bottleneck embedding, eliminating the semantic expression misalignment caused by multimodal asynchronous processing. The multimodal information bottleneck layer uses variational constraints to perform cross-modal semantic refinement processing on the semantically aligned visual and audio bottleneck embeddings, eliminating non-critical redundant semantic flows and generating the least redundant multimodal bottleneck embedding. The highlight scorer, connected to the multimodal semantic refinement module, is used to receive the minimum redundant multimodal bottleneck embedding and output the highlight detection score through a linear mapping layer and a nonlinear activation unit. The loss coprocessor is connected to the causal attention perception module, the single-modal topological information bottleneck compression module, the multimodal semantic refinement module, and the spectro scorer, respectively. It receives the loss components output by each module and weights and fuses the binary cross-entropy task loss, environment fitting loss, adversarial loss, orthogonal loss, slice Gromov-Wasserstein topological loss, contrast alignment loss, and multimodal variational information bottleneck loss into a global co-total loss, driving the iterative update of system parameters.

[0015] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art: Unlike existing technologies, the above-mentioned technical solution provides a video specular highlight detection method with multimodal information bottlenecks. It acquires the original visual and audio features of the video to be detected. These features are then input into a causal attention perception module. This module utilizes environmental pseudo-labels extracted from the audio signal to decouple the input features in the feature space into mutually orthogonal causal branch features and environmental branch features. After removing environmental bias interference, it outputs environment-invariant causal features. These environment-invariant causal features are then input into a single-modal information bottleneck layer constructed based on the slice Gromov-Wasserstein distance. This single-modal information bottleneck layer uses a topological rate-distortion metric to constrain geometric manifold preservation during compression, generating a topology-aware single-modal bottleneck embedding. The topology-aware single-modal bottleneck embedding is then input into a multimodal information bottleneck layer. This multimodal information bottleneck layer collaboratively learns a contrastive learning loss and variational constraints, performing cross-modal semantic refinement on the semantically aligned visual and audio bottleneck embeddings to generate a minimally redundant multimodal bottleneck embedding. Finally, the multimodal bottleneck embedding is input into a specular scorer, which outputs a specular detection score. This technical solution utilizes a causal attention perception module to decouple input features into causal and environmental branches using environmental pseudo-labels. This removes spurious correlations from environmental confounding variables at the feature level, enhancing the model's generalization ability on difficult samples. Simultaneously, by constructing a single-modal information bottleneck layer based on the slice Gromov-Wasserstein distance, and replacing traditional relative entropy constraints with a topological rate distortion metric, it preserves the high-dimensional geometric manifold and temporal causal logic of specular motion during compression and dimensionality reduction, avoiding excessive feature smoothing. While reducing feature redundancy, this significantly improves the generalization robustness and localization accuracy of video specular detection. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the video highlight detection method described in the specific implementation method; Figure 2 This is a schematic diagram of the video highlight detection system described in the specific implementation. Detailed Implementation

[0018] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1 In a first aspect, this embodiment provides a video specular highlight detection method for multimodal information bottlenecks, comprising: Obtain the original visual and audio features of the video to be detected; The original visual features and original audio features are input into the causal attention perception module. The causal attention perception module uses the environment pseudo-labels extracted from the audio signal to decouple the input features into mutually orthogonal causal branch features and environment branch features in the feature space. After removing the environmental bias interference, the output is an environment-unchanged causal feature. The environmentally invariant causal features are input into a single-modal information bottleneck layer constructed based on the slice Gromov-Wasserstein distance. The single-modal information bottleneck layer generates a topology-aware single-modal bottleneck embedding by constraining the geometric manifold preservation during the compression process through a topology rate distortion metric. The topology-aware unimodal bottleneck embedding is input into the multimodal information bottleneck layer. The multimodal information bottleneck layer collaboratively learns the loss and variational constraints, and performs cross-modal semantic refinement on the semantically aligned visual and audio bottleneck embeddings to generate a multimodal bottleneck embedding with minimal redundancy. The multimodal bottleneck is embedded into the highlight scorer, which outputs the highlight detection score.

[0020] In this embodiment, the original visual features and the original audio features are extracted from the video stream to be detected by pre-trained visual encoders and audio encoders, respectively. The obtained original visual features contain spatial semantic information of the video frame sequence, and the obtained original audio features contain temporal energy distribution information of the audio segments. The two types of features together constitute the data basis for subsequent multimodal processing.

[0021] In this embodiment, the causal attention perception module receives original visual features and original audio features as input, and simultaneously receives environmental pseudo-labels extracted from the audio signal as supervision signals. By constructing a parallel split architecture, the input features are decomposed into mutually orthogonal causal branch features and environmental branch features in the feature space, thereby achieving the removal of environmental bias interference. The output environmentally invariant causal features retain the essential semantic information related to the specular detection task.

[0022] In this embodiment, the unimodal information bottleneck layer constructed based on the slice Gromov-Wasserstein distance receives environmentally invariant causal features as input. The slice Gromov-Wasserstein distance is used to measure the geometric structural difference between the original high-dimensional feature distribution and the compressed low-dimensional embedding distribution. This is used as a topological rate distortion metric to constrain the preservation of geometric manifolds during the compression process. The generated topology-aware unimodal bottleneck embedding maintains the temporal coherence of specular motion while reducing dimensionality.

[0023] In this embodiment, the multimodal information bottleneck layer receives topology-aware single-modal bottleneck embeddings as input. Internally, through the synergistic effect of contrastive learning loss and variational constraints, it performs cross-modal semantic refinement processing on visual and audio bottleneck embeddings. The generated semantically aligned visual and audio bottleneck embeddings eliminate the semantic expression misalignment caused by multimodal asynchrony, and the output minimally redundant multimodal bottleneck embeddings eliminate non-critical semantic flows that coexist across modalities.

[0024] In this embodiment, the highlight scorer receives the minimum redundancy multimodal bottleneck embedding as input, performs forward scoring calculation through the internal linear mapping layer and nonlinear activation unit, and outputs the highlight detection score result to indicate the probability that each moment in the video belongs to a highlight moment.

[0025] This embodiment utilizes an environmental pseudo-label to decouple input features into causal and environmental branches through a causal attention perception module, thereby removing environmental bias interference. A single-modal information bottleneck layer based on slice Gromov-Wasserstein distance constrains the compression process with a topological rate distortion metric, preserving the temporal geometric manifold of the specular action. Cross-modal semantic refinement is performed through a multimodal information bottleneck layer using a collaborative contrastive learning loss and variational constraints, generating a minimally redundant multimodal bottleneck embedding. This achieves synergistic optimization of environmental bias removal, temporal topology preservation, and feature redundancy reduction, ultimately improving the generalization robustness and localization accuracy of video specular detection.

[0026] In some embodiments, the original visual features and original audio features are respectively input to the causal attention perception module. The causal attention perception module uses environmental pseudo-labels extracted from the audio signal to decouple the input features in the feature space into mutually orthogonal causal branch features and environmental branch features. After removing environmental bias interference, it outputs environmentally invariant causal features, including: The root mean square energy of the original audio features is extracted. The root mean square energy is divided into high-energy environment priors and low-energy environment priors using a binarized step function. Discrete environment pseudo-labels are generated, which are used to explicitly distinguish the source of environmental confusion signals in subsequent steps. The original visual features and original audio features are respectively input into the causal attention perception module. The causal attention perception module has a parallel splitting architecture built inside. The parallel splitting architecture includes an environment confusion network and a causal feature network. The environment confusion network and the causal feature network share the same input features but are optimized independently. The original visual or audio features are input into the environment confusion network. The binary cross-entropy loss is used to fit the environment pseudo-labels, so that the environment confusion network learns to capture the environment bias components in the features that are highly correlated with the environment pseudo-labels and outputs the environment branch features. The original visual or audio features are input into the causal feature network. A gradient inversion layer is set before the input layer of the causal feature network. The adversarial loss is used to forcibly weaken the statistical dependence between the output features of the causal feature network and the environmental pseudo-labels, blocking the penetration of environmental bias components into the causal branch and outputting causal branch features. By using an orthogonal loss function to constrain environmental branch features and causal branch features to orthogonalize in geometric space, and by calculating the ratio of the inner product of the two sets of branch features to the product of their respective moduli, the two parts of features are forced to decouple in the feature space, and the output causal features with unchanged environment are obtained.

[0027] In this embodiment, the root mean square energy of the original audio features is extracted, and the root mean square energy is divided into high-energy environmental priors and low-energy environmental priors using a binarized step function. This generates discrete environmental pseudo-labels, which are used to explicitly distinguish the source of environmental confusion signals in subsequent steps. This can be understood as approximating the strength of environmental noise by the level of audio energy, and serving as a supervision signal for feature decoupling by the causal attention perception module.

[0028] In this embodiment, the causal attention perception module has a parallel split architecture built inside. The parallel split architecture includes an environment confusion network and a causal feature network. The environment confusion network and the causal feature network share the same original visual features or original audio features as input, but each performs parameter optimization independently, thereby realizing branching processing of the same input feature under different optimization objectives.

[0029] In this embodiment, the original visual features or original audio features are input into the environment confusion network, and the environment pseudo-labels are fitted using binary cross-entropy loss. This allows the environment confusion network to learn and capture the environment bias components in the features that are highly correlated with the environment pseudo-labels. The output environment branch features are rich in environmental noise information corresponding to the level of audio energy.

[0030] In this embodiment, the original visual or audio features are input into the causal feature network. A gradient inversion layer is set before the input layer of the causal feature network. The adversarial loss is used to forcibly weaken the statistical dependence between the output features of the causal feature network and the environmental pseudo-labels. The gradient inversion layer keeps the input features unchanged during forward propagation and reverses the gradient direction calculated by the adversarial loss during backward propagation, thereby blocking the penetration of environmental bias components into the causal branch. The mutual information between the output causal branch features and the environmental pseudo-labels is minimized.

[0031] In this embodiment, the orthogonal loss function is used to constrain the environmental branch features and causal branch features to orthogonalize in the geometric space. By calculating the ratio of the inner product of the two sets of branch features to the product of their respective moduli, the cosine value of the angle between the two sets of features in the geometric space is obtained. The cosine value of the angle is added as an orthogonal constraint term to the training objective, which forces the environmental branch features and causal branch features to occupy mutually independent subspaces in the feature space. The output environmentally invariant causal features are completely stripped of the environmental bias component.

[0032] This embodiment uses an environment confusion network to fit the environment pseudo-labels using binary cross-entropy loss, thereby capturing the environment bias component; it uses a causal feature network to weaken the statistical dependence between the output features and the environment pseudo-labels using gradient inversion layers and adversarial loss, thereby blocking the penetration of environment bias; it uses orthogonal loss to constrain the environment branch features and causal branch features to orthogonalize in geometric space, thereby achieving complete decoupling in the feature space; thus, it achieves a dual decoupling mechanism at the statistical and geometric levels, and finally outputs environment-invariant causal features that are completely free from environmental bias interference.

[0033] In some embodiments, raw visual or raw audio features are input into a causal feature network. A gradient inversion layer is set before the input layer of the causal feature network. Adversarial loss is used to forcibly weaken the statistical dependence between the output features of the causal feature network and the environmental pseudo-labels, blocking the penetration of environmental bias components into the causal branch, and outputting causal branch features, including: The system receives raw visual or audio features as input features to a causal feature network, which learns causal invariant representations that are independent of environmental pseudo-label statistics. A gradient reversal layer is inserted between the input layer and the feature extraction layer of the causal feature network. The gradient reversal layer keeps the input features unchanged during forward propagation and reverses the gradient direction from the adversarial loss during backward propagation. Construct an adversarial loss function, which is used to measure the statistical dependence between the output features of the causal feature network and the pseudo-labels of the environment. The lower the adversarial loss value, the weaker the statistical dependence. During backpropagation, the gradient inversion layer reverses the gradient direction obtained from the adversarial loss calculation and passes it to the feature extraction layer of the causal feature network, causing the causal feature network to update its parameters in the direction of maximizing the adversarial loss. In other words, it forces the causal feature network to learn feature representations that are independent of the pseudo-label statistics of the environment. Through the synergistic effect of gradient inversion layer and adversarial loss, the mutual information between the causal branch features output by the causal feature network and the environmental pseudo-label is minimized, the environmental bias component is blocked outside the causal branch, and the output is a causal branch feature stripped of environmental bias.

[0034] In this embodiment, the causal feature network receives original visual features or original audio features as input features. The received original visual features or original audio features simultaneously contain causal information related to the specular detection task and environmental bias information related to the level of audio energy. The causal feature network is used to learn causal invariant representations from the input features that are independent of environmental pseudo-label statistics.

[0035] In this embodiment, a gradient reversal layer is inserted between the input layer and the feature extraction layer of the causal feature network. The gradient reversal layer keeps the input features unchanged during forward propagation and reverses the gradient direction from the adversarial loss during backward propagation. It can be understood that the gradient reversal layer does not change the feature value during the forward calculation stage, but only changes the gradient sign during the backward calculation stage, thereby realizing the adversarial learning mechanism.

[0036] In this embodiment, an adversarial loss function is constructed. The adversarial loss function is used to measure the statistical dependence between the output features of the causal feature network and the environmental pseudo-labels. The lower the adversarial loss value, the weaker the statistical dependence. It can be understood that the adversarial loss function uses the environmental pseudo-labels as a supervision signal to evaluate whether the output features of the causal feature network still contain information that can be used to distinguish between high-energy and low-energy environments.

[0037] In this embodiment, during backpropagation, the gradient reversal layer reverses the gradient direction obtained from the adversarial loss calculation and passes it to the feature extraction layer of the causal feature network, causing the causal feature network to update its parameters in the direction of maximizing the adversarial loss. That is, it forces the causal feature network to learn feature representations that are unrelated to the statistics of environmental pseudo-labels, thereby realizing a training strategy for the causal feature network that is opposite to that of conventional supervised learning in terms of optimization objectives.

[0038] In this embodiment, through the synergistic effect of the gradient inversion layer and the adversarial loss, the mutual information between the causal branch features output by the causal feature network and the environmental pseudo-labels is minimized, the environmental bias component is blocked outside the causal branch, and the output causal branch features stripped of environmental bias retain only the essential semantic information related to the specular detection task.

[0039] This embodiment reverses the gradient direction of the adversarial loss during backpropagation through a gradient reversal layer, thereby causing the causal feature network to update its parameters in the direction that maximizes the adversarial loss. This forces the causal feature network to learn feature representations that are independent of the environment pseudo-label statistics, and finally outputs causal branch features that are completely free from environmental bias, thus achieving the learning of causal invariant representations.

[0040] In some embodiments, an orthogonal loss function is used to constrain environmental branch features and causal branch features to orthogonalize in geometric space. By calculating the ratio of the inner product of the two sets of branch features to the product of their respective moduli, the two parts of features are forced to decouple in the feature space, outputting environmentally invariant causal features, including: The environmental branch features and causal features output by the environmental confusion network are received. The environmental branch features are rich in environmental bias components that are highly correlated with the environmental pseudo-labels. The statistical dependence of the causal branch features on the environmental pseudo-labels has been weakened by the gradient inversion layer and adversarial loss. Construct an orthogonal loss function, which is used to calculate the cosine of the angle between the environmental branch features and the causal branch features in geometric space. The cosine of the angle is obtained by calculating the ratio of the inner product of the two sets of branch features to the product of their respective moduli. The cosine of the included angle is added as an orthogonal constraint term to the training objective. During the optimization process, the orthogonal constraint term drives the included angle between the environmental branch features and the causal branch features in the geometric space to tend to 90 degrees, so that the two sets of features become orthogonal in the feature space. Through the synergistic effect of orthogonal constraint terms and adversarial loss, adversarial loss weakens the dependency between causal branch features and environmental pseudo-labels at the statistical level, while orthogonal constraint terms force environmental branch features and causal branch features to occupy mutually independent feature subspaces at the geometric level, forming a dual decoupling mechanism. Driven by the dual decoupling mechanism, environmental branch features and causal branch features are completely separated in the feature space. Environmental bias components are completely isolated in the environmental branch, while only environmentally invariant semantic information related to the specular detection task is retained in the causal branch, outputting environmentally invariant causal features.

[0041] In this embodiment, the environmental branch features output by the environmental confusion network are rich in environmental bias components that are highly correlated with the environmental pseudo-labels. The causal branch features output by the causal feature network have had their statistical dependence on the environmental pseudo-labels weakened by the gradient reversal layer and adversarial loss. The two sets of branch features received carry different information tendencies, providing processing objects for subsequent orthogonal constraints.

[0042] In this embodiment, an orthogonal loss function is constructed. The orthogonal loss function is used to calculate the cosine value of the angle between the environmental branch features and the causal branch features in the geometric space. The cosine value of the angle is obtained by calculating the ratio of the inner product of the two sets of branch features to the product of their respective moduli. It can be understood that the closer the cosine value of the angle is to zero, the closer the angle between the two sets of features in the geometric space is to 90 degrees, that is, the higher the degree of orthogonality.

[0043] In this embodiment, the cosine value of the included angle is added as an orthogonal constraint term to the training objective. During the optimization process, the orthogonal constraint term drives the included angle between the environmental branch features and the causal branch features in the geometric space to tend to 90 degrees, so that the two sets of features become orthogonal in the feature space, thereby realizing the directional separation of the environmental branch features and the causal branch features in the geometric space.

[0044] In this embodiment, through the synergistic effect of orthogonal constraint terms and adversarial loss, adversarial loss weakens the dependency between causal branch features and environmental pseudo-labels at the statistical level, while orthogonal constraint terms force environmental branch features and causal branch features to occupy mutually independent feature subspaces at the geometric level. The two constraints work together from different dimensions to decouple features, forming a dual decoupling mechanism.

[0045] In this embodiment, driven by the dual decoupling mechanism, the environmental branch features and the causal branch features are completely separated in the feature space. The environmental bias component is completely isolated in the environmental branch, and only the environmental invariant semantic information related to the specular detection task is retained in the causal branch. The output environmental invariant causal features are completely separated from the environmental bias at both the statistical and geometric levels.

[0046] This embodiment weakens the dependency between causal branch features and environmental pseudo-labels at the statistical level by using adversarial loss, and forces environmental branch features and causal branch features to occupy mutually independent feature subspaces at the geometric level by using orthogonal constraint terms. The two constraints together form a dual decoupling mechanism from the statistical and geometric dimensions, thereby achieving complete isolation of environmental bias components. Finally, it outputs environmentally invariant causal features that are independent of environmental bias at both the statistical and geometric levels.

[0047] In some embodiments, environmentally invariant causal features are input into a single-modal information bottleneck layer constructed based on slice Gromov-Wasserstein distance. The single-modal information bottleneck layer generates a topology-aware single-modal bottleneck embedding by constraining geometric manifold preservation during the compression process through a topology rate distortion metric, including: The received environment-invariant visual causal features and environment-invariant audio causal features are used as unimodal input features. The unimodal input features contain the temporal geometric manifold structure of specular action in the original high-dimensional space. By removing the relative entropy constraint from the standard variational information bottleneck framework, the measure of information compression is reconstructed into a topological rate-distortion metric based on the slice Gromov-Wasserstein distance. The topological rate-distortion metric is used to measure the geometric structural difference between the original high-dimensional feature distribution and the compressed low-dimensional embedding distribution. The single-modal input features are projected into a low-dimensional bottleneck embedding space. During the projection process, the slice Gromov-Wasserstein distance between the feature distributions before and after projection is calculated using the topological rate distortion metric. The slice Gromov-Wasserstein distance is then added to the training objective as a geometric constraint regularization loss. By using geometric constraint regularization loss to guide the unimodal information bottleneck layer to retain the original features of the high-dimensional geometric manifold structure and action temporal causal logic during dimensionality reduction and compression, the problem of excessive feature smoothing caused by relative entropy constraints in traditional variational information bottlenecks is avoided, and a topology-aware unimodal bottleneck embedding is generated.

[0048] In this embodiment, the single-modal information bottleneck layer receives environment-invariant visual causal features and environment-invariant audio causal features as single-modal input features. The received single-modal input features contain the temporal geometric manifold structure of specular motion in the original high-dimensional space, which can be understood as the specular motion presenting a geometric topological form with a specific temporal direction and spatial distribution in the original feature space.

[0049] In this embodiment, the relative entropy constraint is removed from the standard variational information bottleneck framework, and the measure of information compression is reconstructed into a topological rate-distortion metric based on the slice Gromov-Wasserstein distance. The topological rate-distortion metric measures the geometric structure difference between the original high-dimensional feature distribution and the compressed low-dimensional embedding distribution. It can be understood that the slice Gromov-Wasserstein distance calculates the difference between the two one-dimensional projected distributions by projecting the high-dimensional feature distribution and the low-dimensional embedding distribution onto a randomly sampled one-dimensional subspace, thereby approximating the high-dimensional geometric structure difference.

[0050] In this embodiment, the single-modal input features are projected into a low-dimensional bottleneck embedding space. During the projection process, the topological rate distortion metric is used to calculate the slice Gromov-Wasserstein distance between the feature distributions before and after projection. The slice Gromov-Wasserstein distance is added to the training objective as a geometric constraint regularization loss to ensure that the feature distributions before and after projection maintain consistency at the geometric structure level, rather than just maintaining alignment at the statistical distribution level.

[0051] In this embodiment, the geometric constraint regularization loss guides the single-modal information bottleneck layer to retain the original high-dimensional geometric manifold structure and action temporal causal logic during dimensionality reduction and compression. This can be understood as the geometric constraint regularization loss penalizing projection transformations that destroy the original geometric manifold structure during the optimization process, thereby avoiding the problem of excessive feature smoothing caused by the relative entropy constraint of the traditional variational information bottleneck. The generated topology-aware single-modal bottleneck is embedded in the low-dimensional space to maintain the original temporal topology of the specular action.

[0052] This embodiment removes the relative entropy constraint in the standard variational information bottleneck framework and reconstructs the measure of information compression into a topological rate-distortion metric based on the slice Gromov-Wasserstein distance. In this way, geometric constraint regularization loss is used to constrain the geometric structure consistency of feature distribution before and after projection during the projection process, thereby avoiding the problem of excessive feature smoothing caused by relative entropy constraint. Finally, a topologically aware single-modal bottleneck embedding is generated that retains the original high-dimensional geometric manifold structure and action temporal causal logic.

[0053] In some embodiments, the relative entropy constraint is removed from the standard variational information bottleneck framework, and the measure of information compression is reconstructed as a topological rate-distortion metric based on the slice Gromov-Wasserstein distance. This topological rate-distortion metric measures the geometric structural difference between the original high-dimensional feature distribution and the compressed low-dimensional embedding distribution, including: The relative entropy constraint term in the standard variational information bottleneck framework is identified. The relative entropy constraint term drives the compression process by measuring the probability density difference between the original feature distribution and the compressed embedding distribution, but only focuses on the alignment at the statistical distribution level, ignoring the geometric manifold structure and temporal causal logic of the data in high-dimensional space. The relative entropy constraint term is removed from the standard variational information bottleneck framework to eliminate the smoothing effect of the relative entropy constraint term on the feature geometric manifold structure during the compression process. The smoothing effect will cause the temporal coherence of the specular action to be destroyed during dimensionality reduction. We introduce the slice Gromov-Wasserstein distance as an alternative metric. The slice Gromov-Wasserstein distance calculates the Wasserstein distance between two one-dimensional projected distributions by projecting the high-dimensional feature distribution and the low-dimensional embedding distribution onto a randomly sampled one-dimensional subspace, and then takes the average after multiple random projections. This achieves a low-complexity approximate metric for the differences in high-dimensional geometric structures. The Gromov-Wasserstein distance of the slices is constructed as a topological rate-distortion metric. The topological rate-distortion metric is used to constrain the geometric structural differences between the original high-dimensional feature distribution and the compressed low-dimensional embedding distribution during the compression process, rather than the statistical distribution differences. By using a topology rate distortion metric, the unimodal information bottleneck layer is guided to retain the original features of the high-dimensional geometric manifold structure and action temporal causal logic during dimensionality reduction and compression, and outputs a topology-aware unimodal bottleneck embedding.

[0054] In this embodiment, the relative entropy constraint term in the standard variational information bottleneck framework is identified. The relative entropy constraint term drives the compression process by measuring the probability density difference between the original feature distribution and the compressed embedded distribution, but only focuses on the alignment at the statistical distribution level, ignoring the geometric manifold structure and temporal causal logic of the data in high-dimensional space. It can be understood that the relative entropy constraint term only requires that the feature distributions before and after compression be similar in probability density, but does not require that they be consistent in geometric topology.

[0055] In this embodiment, the relative entropy constraint term is removed from the standard variational information bottleneck framework to eliminate the smoothing effect of the relative entropy constraint term on the feature geometric manifold structure during compression. The smoothing effect will cause the temporal coherence of the highlight action to be destroyed during dimensionality reduction. It can be understood that the relative entropy constraint term tends to smoothly mix different types of features in the probability space during the optimization process, thereby blurring the temporal boundary and geometric contour of the highlight action in the original high-dimensional space.

[0056] In this embodiment, the slice Gromov-Wasserstein distance is introduced as an alternative metric. The slice Gromov-Wasserstein distance is calculated by projecting the high-dimensional feature distribution and the low-dimensional embedding distribution onto a randomly sampled one-dimensional subspace, respectively, and calculating the Wasserstein distance between the two one-dimensional projected distributions. The average value is then taken after multiple random projections. This achieves a low-complexity approximate measure of the differences in high-dimensional geometric structures. It can be understood as transforming the problem of measuring the differences in high-dimensional geometric structures into a statistical average of multiple one-dimensional distribution alignment problems through multiple random projections.

[0057] In this embodiment, the Gromov-Wasserstein distance of the slice is constructed as a topological rate distortion metric. The topological rate distortion metric is used to constrain the geometric structure difference between the original high-dimensional feature distribution and the compressed low-dimensional embedding distribution during the compression process, rather than the statistical distribution difference, so as to ensure that the compressed low-dimensional embedding distribution retains the geometric topological shape of the original features in the high-dimensional space.

[0058] In this embodiment, the topology rate distortion metric guides the single-modal information bottleneck layer to retain the original features of the high-dimensional geometric manifold structure and action temporal causal logic during dimensionality reduction and compression. The output topology-aware single-modal bottleneck is embedded in the low-dimensional space to restore the original temporal direction and spatial distribution relationship of the specular action.

[0059] This embodiment eliminates the smoothing effect on the feature geometric manifold structure by identifying and removing the relative entropy constraint term in the standard variational information bottleneck framework that only focuses on the alignment of statistical distributions. By introducing the slice Gromov-Wasserstein distance, which is a low-complexity approximation measure achieved by averaging multiple random projections, as an alternative metric, the geometric structure differences are constrained rather than statistical distribution differences during the compression process. The final output is a topology-aware single-modal bottleneck embedding that preserves the original high-dimensional geometric manifold structure and the causal logic of action sequence.

[0060] In some embodiments, a topology-aware unimodal bottleneck embedding is input to a multimodal information bottleneck layer. The multimodal information bottleneck layer collaboratively performs a contrastive learning loss and variational constraints to perform cross-modal semantic refinement on the semantically aligned visual and audio bottleneck embeddings, generating a minimally redundant multimodal bottleneck embedding, including: The topology-aware visual bottleneck embedding and the topology-aware audio bottleneck embedding are received as unimodal input embeddings. The visual bottleneck embedding and the audio bottleneck embedding are asynchronous in time, which leads to a misalignment in their semantic expression. The visual bottleneck embedding and audio bottleneck embedding are respectively spliced ​​and fused with the two branches to generate multimodal representations. Cross-modal comparison and association calculation is performed. Contrastive learning loss is introduced to narrow the semantic distance of audiovisual bottleneck embeddings under the same video context, eliminate the semantic expression misalignment caused by multimodal asynchrony, and generate semantically aligned visual bottleneck embedding and audio bottleneck embedding. The semantically aligned visual bottleneck embedding and audio bottleneck embedding are dynamically fused to construct a multimodal fusion attention feature map, which contains semantically calibrated cross-modal interaction information. The multimodal fusion attention feature map is input into the multimodal information bottleneck layer. During forward propagation computation, the multimodal variational information bottleneck loss is used to perform deep semantic purification and secondary dimensionality reduction on the fused cross-modal representation. Non-critical redundant semantic flows coexisting between the two modalities are eliminated, and the core semantic information that is highly related to the specular detection task is retained, generating a multimodal bottleneck embedding with minimal redundancy and high relevance to the prediction task.

[0061] In this embodiment, the multimodal information bottleneck layer receives topology-aware visual bottleneck embeddings and topology-aware audio bottleneck embeddings as unimodal input embeddings. The received visual bottleneck embeddings and audio bottleneck embeddings are asynchronous on the time axis, which can be understood as a time offset between the occurrence time of the visual event and the corresponding audio response time. This causes the visual representation and audio representation in the same video context to be in different positions in the semantic space, resulting in expression misalignment.

[0062] In this embodiment, the visual bottleneck embedding and audio bottleneck embedding are respectively compared and correlated with the multimodal representation generated after splicing and fusing the two branches. The contrastive learning loss is introduced to narrow the semantic distance between the audiovisual bottleneck embeddings in the same video context. This can be understood as using the spliced ​​and fused multimodal representation as an anchor reference, and using the visual bottleneck embedding and audio bottleneck embedding in the same video context as positive samples to move closer to it, thereby eliminating the semantic expression misalignment caused by multimodal asynchrony. The generated semantically aligned visual bottleneck embedding and audio bottleneck embedding occupy the same position in the semantic space.

[0063] In this embodiment, the semantically aligned visual bottleneck embedding and audio bottleneck embedding are dynamically fused to construct a multimodal fusion attention feature map. The constructed multimodal fusion attention feature map contains semantically calibrated cross-modal interaction information, which can be understood as dynamic fusion adaptively adjusting the contribution of different modalities through learnable weight parameters, thereby achieving an optimized combination of visual and audio information in the fusion process.

[0064] In this embodiment, the multimodal fusion attention feature map is input to the multimodal information bottleneck layer. During forward propagation computation, the multimodal variational information bottleneck loss is used to perform deep semantic purification and secondary dimensionality reduction on the fused cross-modal representation. This can be understood as the multimodal variational information bottleneck loss constraining the information capacity of the fused representation, forcing the model to eliminate non-critical redundant semantic flows between the two modalities, and retaining only the core semantic information that is highly relevant to the specular detection task. The generated multimodal bottleneck embedding with minimal redundancy and high relevance to the prediction task improves the discriminative ability of the features while reducing the feature dimensionality.

[0065] This embodiment uses contrastive learning loss to anchor the multimodal representation generated by splicing and fusion, thereby narrowing the semantic distance between visual bottleneck embedding and audio bottleneck embedding, thus eliminating the semantic expression misalignment caused by asynchronous multimodal processing. Through multimodal variational information bottleneck loss, deep semantic purification and secondary dimensionality reduction are performed on the fused cross-modal representation, thereby eliminating non-critical redundant semantic flows that coexist across modalities, and finally generating a multimodal bottleneck embedding with minimal redundancy and high relevance to the prediction task.

[0066] In some embodiments, the visual bottleneck embedding and audio bottleneck embedding are respectively combined with the multimodal representation generated after splicing and fusing the two branches to perform cross-modal comparative association calculation. A contrastive learning loss is introduced to narrow the semantic distance between the audiovisual bottleneck embeddings in the same video context, eliminating semantic expression misalignment caused by multimodal asynchrony, and generating semantically aligned visual bottleneck embeddings and audio bottleneck embeddings, including: The topology-aware visual bottleneck embedding and the topology-aware audio bottleneck embedding are inherently asynchronous on the time axis. This asynchronousness manifests as a time offset between the occurrence of a visual event and the corresponding audio response, resulting in the visual representation and audio representation in the same video context being located in different positions in the semantic space. Visual bottleneck embedding and audio bottleneck embedding are spliced ​​and fused to generate a multimodal fusion representation, which serves as an anchor reference for cross-modal alignment. Contrastive learning loss function is constructed, with multimodal fusion representation as positive sample anchor point, visual bottleneck embedding and audio bottleneck embedding under the same video context as positive sample pair, and visual bottleneck embedding and audio bottleneck embedding under different video context as negative sample pair. By minimizing the contrastive learning loss function, the semantic distance between the visual bottleneck embedding and the multimodal fusion representation is reduced, and the semantic distance between the audio bottleneck embedding and the multimodal fusion representation is also reduced, so that the visual bottleneck embedding and the audio bottleneck embedding move closer to the common fusion representation center in the semantic space. Driven by the contrastive learning loss function, the temporal asynchronous offset between the visual bottleneck embedding and the audio bottleneck embedding is eliminated, and the two occupy consistent positions in the semantic space, generating semantically aligned visual bottleneck embeddings and audio bottleneck embeddings.

[0067] In this embodiment, the multimodal information bottleneck layer receives topology-aware visual bottleneck embeddings and topology-aware audio bottleneck embeddings. The received visual bottleneck embeddings and audio bottleneck embeddings are naturally asynchronous on the time axis. This asynchronousness manifests as a time offset between the occurrence time of the visual event and the corresponding audio response time, resulting in the visual representation and audio representation in the same video context being in different positions in the semantic space. This can be understood as the visual event occurring before the audio response, or the audio response lagging behind the end of the visual event. The misalignment of the two on the time axis causes their embedding positions in the high-dimensional semantic space to deviate.

[0068] In this embodiment, the visual bottleneck embedding and the audio bottleneck embedding are spliced ​​and fused to generate a multimodal fusion representation. The generated multimodal fusion representation serves as an anchor reference for cross-modal alignment. It can be understood that the multimodal fusion representation contains both visual and audio information and is located in the middle position between the visual embedding and the audio embedding in the semantic space, providing a common reference benchmark for the alignment of the two.

[0069] In this embodiment, a contrastive learning loss function is constructed. The contrastive learning loss function uses multimodal fusion representation as positive sample anchors, visual bottleneck embeddings and audio bottleneck embeddings in the same video context as positive sample pairs, and visual bottleneck embeddings and audio bottleneck embeddings in different video contexts as negative sample pairs. It can be understood that the contrastive learning loss function brings the positive sample pairs closer to the anchors while pushing the negative sample pairs further away from the anchors, thereby forming a clustering structure centered on the anchors in the semantic space.

[0070] In this embodiment, by minimizing the contrastive learning loss function, the semantic distance between the visual bottleneck embedding and the multimodal fusion representation is reduced, and the semantic distance between the audio bottleneck embedding and the multimodal fusion representation is also reduced. This allows the visual bottleneck embedding and the audio bottleneck embedding to move closer to a common fusion representation center in the semantic space, thereby achieving synchronous calibration of the visual modality and the audio modality at the semantic level.

[0071] In this embodiment, driven by the contrastive learning loss function, the temporal asynchrony offset between the visual bottleneck embedding and the audio bottleneck embedding is eliminated, and the two occupy the same position in the semantic space. The generated semantically aligned visual bottleneck embedding and audio bottleneck embedding have completed cross-modal semantic calibration before entering the subsequent fusion process, thus avoiding the impact of asynchrony on the fusion quality.

[0072] This embodiment uses the multimodal fusion representation generated by splicing and fusion as an anchor reference. It utilizes the contrastive learning loss function to take visual bottleneck embeddings and audio bottleneck embeddings in the same video context as positive sample pairs and visual and audio embeddings in different video contexts as negative sample pairs. By minimizing the contrastive learning loss function, the visual and audio embeddings are brought closer to a common fusion representation center, thereby eliminating the semantic expression misalignment caused by temporal asynchronous offset. Finally, semantically aligned visual bottleneck embeddings and audio bottleneck embeddings occupying consistent positions in the semantic space are generated.

[0073] In some embodiments, a multimodal bottleneck is embedded into the input of a specular scorer, and the specular scorer outputs a specular detection score, including: The minimum redundancy multimodal bottleneck embedding receives the output of the multimodal information bottleneck layer. The minimum redundancy multimodal bottleneck embedding has eliminated non-critical semantic flows that coexist across modalities and retained the core semantic information that is highly relevant to the specular detection task. The minimum redundancy multimodal bottleneck is embedded into the input of the highlight scorer, which includes a linear mapping layer and a nonlinear activation unit. The highlight score prediction value corresponding to the current video segment is calculated through forward mapping. Based on the predicted value of the highlight score and the real label of the highlight annotation, the task loss value, which measures the localization performance of the model at the highlight moment, is calculated using the binary cross-entropy loss function. The task loss value is backpropagated to the multimodal information bottleneck layer and the unimodal information bottleneck layer, driving the multimodal information bottleneck layer and the unimodal information bottleneck layer to further optimize the selection and retention of core semantic information in subsequent iterations, forming a collaborative optimization closed loop of feature refinement and score prediction, and outputting the final highlight detection score result.

[0074] In this embodiment, the highlight scorer receives the minimum redundancy multimodal bottleneck embedding output by the multimodal information bottleneck layer. The received minimum redundancy multimodal bottleneck embedding has eliminated non-critical semantic flows that coexist across modalities and retained the core semantic information that is highly relevant to the highlight detection task. It can be understood that the minimum redundancy multimodal bottleneck embedding has been compressed and purified in the feature dimension, retaining only the semantic components that are most discriminative for distinguishing between highlight moments and non-highlight moments.

[0075] In this embodiment, the minimum redundancy multimodal bottleneck is embedded into the highlight scorer. The highlight scorer includes a linear mapping layer and a nonlinear activation unit. The highlight score prediction value corresponding to the current video segment is calculated through forward mapping. It can be understood that the linear mapping layer embeds the minimum redundancy multimodal bottleneck and projects it from the high-dimensional feature space to the one-dimensional scoring space. The nonlinear activation unit maps the projection result to the probability interval, thereby obtaining the highlight score prediction value between zero and one.

[0076] In this embodiment, based on the predicted highlight score and the actual highlight label, the task loss value, which measures the model's highlight moment localization performance, is calculated using the binary cross-entropy loss function. This can be understood as the binary cross-entropy loss function quantifying the highlight detection error of the model on the current video segment by comparing the difference between the predicted probability and the actual label. The lower the task loss value, the closer the prediction result is to the actual label.

[0077] In this embodiment, the task loss value is backpropagated to the multimodal information bottleneck layer and the unimodal information bottleneck layer, driving them to further optimize the selection and retention of core semantic information in subsequent iterations. This can be understood as the task loss value transmitting optimization signals to the preceding modules through the gradient backpropagation mechanism, enabling the multimodal information bottleneck layer and the unimodal information bottleneck layer to adjust their internal parameters in subsequent training to generate feature representations that are more conducive to the highlight detection task. This forms a collaborative optimization closed loop of feature refinement and score prediction, and the final highlight detection score output is used to indicate the probability that each moment in the video belongs to a highlight moment.

[0078] This embodiment backpropagates the task loss value calculated by the highlight scorer to the multimodal information bottleneck layer and the single-modal information bottleneck layer, driving the front-end module to optimize the screening and retention of core semantic information in subsequent iterations, thereby forming a collaborative optimization closed loop of feature refinement and score prediction, and finally outputting the highlight detection score result to indicate the probability that each moment in the video belongs to a highlight moment.

[0079] Please see Figure 2 In a second aspect, this embodiment also provides a video specular detection system for multimodal information bottlenecks, applicable to the method described in the first aspect, the system comprising: The data acquisition and feature extraction module is used to receive the raw video stream and audio stream, and extract the raw visual features and raw audio features using the pre-trained visual encoder and pre-trained audio encoder, respectively. The environment pseudo-label generation module is connected to the data acquisition and feature extraction module. It is used to receive the original audio features, extract the root mean square energy of the audio segment, and generate discrete environment pseudo-labels using the binarized step function. The causal attention perception module is connected to the data acquisition and feature extraction module and the environment pseudo-label generation module. It is used to receive visual raw features, audio raw features and environment pseudo-labels. Internally, it contains a parallel environment confusion network and a causal feature network. A gradient inversion layer is set in front of the input layer of the causal feature network. It uses binary cross-entropy loss, adversarial loss and orthogonal loss to drive the input features in the feature space to decouple the input features into mutually orthogonal environment branch features and causal branch features, and outputs causal features that are invariant to the environment. The unimodal topological information bottleneck compression module, connected to the causal attention perception module, is used to receive causal features that are invariant to the environment. Internally, it removes the relative entropy constraints in the standard variational information bottleneck framework and reconstructs the measure of information compression degree into a topological rate-distortion metric based on the slice Gromov-Wasserstein distance. The topological rate-distortion metric constrains the preservation of geometric manifolds during the compression process, generating a topology-aware unimodal bottleneck embedding. The multimodal semantic refinement module, connected to the single-modal topology information bottleneck compression module, is used to receive topology-aware single-modal bottleneck embeddings. It contains a contrast alignment unit and a multimodal information bottleneck layer. The contrast alignment unit uses the multimodal representation generated by splicing and fusion as an anchor point and uses contrastive learning loss to narrow the semantic distance between the visual bottleneck embedding and the audio bottleneck embedding, eliminating the semantic expression misalignment caused by multimodal asynchronous processing. The multimodal information bottleneck layer uses variational constraints to perform cross-modal semantic refinement processing on the semantically aligned visual and audio bottleneck embeddings, eliminating non-critical redundant semantic flows and generating the least redundant multimodal bottleneck embedding. The highlight scorer, connected to the multimodal semantic refinement module, is used to receive the minimum redundant multimodal bottleneck embedding and output the highlight detection score through a linear mapping layer and a nonlinear activation unit. The loss coprocessor is connected to the causal attention perception module, the single-modal topological information bottleneck compression module, the multimodal semantic refinement module, and the spectro scorer, respectively. It receives the loss components output by each module and weights and fuses the binary cross-entropy task loss, environment fitting loss, adversarial loss, orthogonal loss, slice Gromov-Wasserstein topological loss, contrast alignment loss, and multimodal variational information bottleneck loss into a global co-total loss, driving the iterative update of system parameters.

[0080] The above content can be understood by referring to the method described in the first aspect, and will not be elaborated further in this embodiment.

[0081] Unlike existing technologies, the above-mentioned technical solution provides a video specular highlight detection method with multimodal information bottlenecks. It acquires the original visual and audio features of the video to be detected. These features are then input into a causal attention perception module. This module utilizes environmental pseudo-labels extracted from the audio signal to decouple the input features in the feature space into mutually orthogonal causal branch features and environmental branch features. After removing environmental bias interference, it outputs environment-invariant causal features. These environment-invariant causal features are then input into a single-modal information bottleneck layer constructed based on the slice Gromov-Wasserstein distance. This single-modal information bottleneck layer uses a topological rate-distortion metric to constrain geometric manifold preservation during compression, generating a topology-aware single-modal bottleneck embedding. The topology-aware single-modal bottleneck embedding is then input into a multimodal information bottleneck layer. This multimodal information bottleneck layer collaboratively learns a contrastive learning loss and variational constraints, performing cross-modal semantic refinement on the semantically aligned visual and audio bottleneck embeddings to generate a minimally redundant multimodal bottleneck embedding. Finally, the multimodal bottleneck embedding is input into a specular scorer, which outputs a specular detection score. This technical solution utilizes a causal attention perception module to decouple input features into causal and environmental branches using environmental pseudo-labels. This removes spurious correlations from environmental confounding variables at the feature level, enhancing the model's generalization ability on difficult samples. Simultaneously, by constructing a single-modal information bottleneck layer based on the slice Gromov-Wasserstein distance, and replacing traditional relative entropy constraints with a topological rate distortion metric, it preserves the high-dimensional geometric manifold and temporal causal logic of specular motion during compression and dimensionality reduction, avoiding excessive feature smoothing. While reducing feature redundancy, this significantly improves the generalization robustness and localization accuracy of video specular detection.

[0082] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0083] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0084] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for detecting video specular highlights in the context of multimodal information bottlenecks, characterized in that, include: Obtain the original visual and audio features of the video to be detected; The original visual features and original audio features are respectively input into the causal attention perception module. The causal attention perception module uses the environment pseudo-labels extracted from the audio signal to decouple the input features into mutually orthogonal causal branch features and environment branch features in the feature space. After removing the environmental bias interference, the output causal features with unchanged environment are output. The environmentally invariant causal features are respectively input into a single-modal information bottleneck layer constructed based on the slice Gromov-Wasserstein distance. The single-modal information bottleneck layer generates a topology-aware single-modal bottleneck embedding by constraining the geometric manifold preservation during the compression process through topological rate distortion metric constraints. The topology-aware unimodal bottleneck embedding is input into the multimodal information bottleneck layer. The multimodal information bottleneck layer collaboratively compares the learning loss and variational constraints, and performs cross-modal semantic refinement on the semantically aligned visual and audio bottleneck embeddings to generate a multimodal bottleneck embedding with minimal redundancy. The multimodal bottleneck is embedded into the highlight scorer, which outputs the highlight detection score.

2. The video specular highlight detection method for multimodal information bottlenecks according to claim 1, characterized in that, The original visual features and original audio features are respectively input into the causal attention perception module. The causal attention perception module utilizes environmental pseudo-labels extracted from the audio signal to decouple the input features in the feature space into mutually orthogonal causal branch features and environmental branch features. After removing environmental bias interference, it outputs environmentally unchanging causal features, including: The root mean square energy of the original audio features is extracted, and the root mean square energy is divided into high-energy environment priors and low-energy environment priors using a binarized step function to generate discrete environment pseudo-labels. These environment pseudo-labels are used to explicitly distinguish the source of environmental confusion signals in subsequent steps. The original visual features and original audio features are respectively input into the causal attention perception module. The causal attention perception module has a parallel splitting architecture built inside. The parallel splitting architecture includes an environment confusion network and a causal feature network. The environment confusion network and the causal feature network share the same input features but are optimized independently. The original visual features or original audio features are input into the environment confusion network. The environment pseudo-labels are fitted using binary cross-entropy loss, so that the environment confusion network learns to capture the environment bias components in the features that are highly correlated with the environment pseudo-labels and outputs the environment branch features. The original visual or audio features are input into the causal feature network. A gradient inversion layer is set before the input layer of the causal feature network. The statistical dependence between the output features of the causal feature network and the environmental pseudo-label is forcibly weakened by adversarial loss, thereby blocking the penetration of environmental bias components into the causal branch and outputting causal branch features. The orthogonal loss function is used to constrain the environmental branch features and the causal branch features to orthogonalize in geometric space. By calculating the ratio of the inner product of the two sets of branch features to the product of their respective moduli, the two parts of features are forced to decouple in the feature space, and the causal features with unchanged environment are output.

3. The video specular highlight detection method for multimodal information bottlenecks according to claim 2, characterized in that, The original visual or audio features are input into the causal feature network. A gradient inversion layer is placed before the input layer of the causal feature network. Adversarial loss is used to forcibly weaken the statistical dependency between the output features of the causal feature network and the environmental pseudo-labels, blocking the penetration of environmental bias components into the causal branch, and outputting causal branch features, including: The original visual features or original audio features are received as input features to the causal feature network, which is used to learn causal invariant representations that are independent of environmental pseudo-label statistics. A gradient inversion layer is inserted between the input layer and the feature extraction layer of the causal feature network. The gradient inversion layer keeps the input features unchanged during forward propagation and inverts the gradient direction from the adversarial loss during backward propagation. Construct an adversarial loss function, which is used to measure the statistical dependence between the output features of the causal feature network and the environmental pseudo-labels. The lower the adversarial loss value, the weaker the statistical dependence. During backpropagation, the gradient inversion layer inverts the gradient direction calculated by the adversarial loss and passes it to the feature extraction layer of the causal feature network, so that the causal feature network updates its parameters in the direction of maximizing the adversarial loss, that is, forces the causal feature network to learn feature representations that are independent of the environmental pseudo-label statistics. Through the synergistic effect of the gradient reversal layer and the adversarial loss, the mutual information between the causal branch features output by the causal feature network and the environmental pseudo-labels is minimized, the environmental bias component is blocked outside the causal branch, and the output is a causal branch feature stripped of environmental bias.

4. The video specular highlight detection method for multimodal information bottlenecks according to claim 2, characterized in that, By using an orthogonal loss function to constrain the environmental branch features and the causal branch features to orthogonality in geometric space, and by calculating the ratio of the inner product of the two sets of branch features to the product of their respective moduli, the two parts of features are forced to decouple in the feature space, outputting environmentally invariant causal features, including: The environment branch features output by the environment confusion network and the causal branch features output by the causal feature network are received. The environment branch features are rich in environment bias components that are highly correlated with the environment pseudo-labels. The statistical dependence of the causal branch features on the environment pseudo-labels has been weakened by the gradient inversion layer and the adversarial loss. Construct an orthogonal loss function, which is used to calculate the cosine value of the angle between the environmental branch feature and the causal branch feature in geometric space. The cosine value of the angle is obtained by calculating the ratio of the inner product of the two sets of branch features to the product of their respective moduli. The cosine value of the included angle is added as an orthogonal constraint term to the training objective. During the optimization process, the orthogonal constraint term drives the included angle between the environmental branch feature and the causal branch feature in the geometric space to tend to 90 degrees, so that the two sets of features become orthogonal in the feature space. Through the synergistic effect of the orthogonal constraint term and the adversarial loss, the adversarial loss weakens the dependency between causal branch features and environmental pseudo-labels at the statistical level, while the orthogonal constraint term forces environmental branch features and causal branch features to occupy mutually independent feature subspaces at the geometric level, forming a dual decoupling mechanism. Driven by the dual decoupling mechanism, the environmental branch features and the causal branch features are completely separated in the feature space. The environmental bias component is completely isolated in the environmental branch, and only environmentally invariant semantic information related to the specular detection task is retained in the causal branch, outputting environmentally invariant causal features.

5. The video specular highlight detection method for multimodal information bottlenecks according to claim 1, characterized in that, The environmentally invariant causal features are input into a single-modal information bottleneck layer constructed based on the slice Gromov-Wasserstein distance. This single-modal information bottleneck layer generates a topology-aware single-modal bottleneck embedding by preserving the geometric manifold during the compression process through a topology rate-distortion metric constraint. The embedding includes: The received environment-invariant visual causal features and environment-invariant audio causal features are used as unimodal input features, wherein the unimodal input features contain the temporal geometric manifold structure of specular action in the original high-dimensional space; The relative entropy constraint is removed from the standard variational information bottleneck framework, and the measure of information compression is reconstructed into a topological rate-distortion metric based on the slice Gromov-Wasserstein distance. The topological rate-distortion metric is used to measure the geometric structural difference between the original high-dimensional feature distribution and the compressed low-dimensional embedding distribution. The single-modal input features are projected into a low-dimensional bottleneck embedding space. During the projection process, the slice Gromov-Wasserstein distance between the feature distributions before and after projection is calculated using the topological rate distortion metric. The slice Gromov-Wasserstein distance is then added to the training objective as a geometric constraint regularization loss. The geometric constraint regularization loss guides the unimodal information bottleneck layer to retain the original high-dimensional geometric manifold structure and action temporal causal logic during dimensionality reduction and compression, avoiding the problem of excessive feature smoothing caused by relative entropy constraints in traditional variational information bottlenecks, and generating topology-aware unimodal bottleneck embeddings.

6. The video specular highlight detection method for multimodal information bottlenecks according to claim 5, characterized in that, By removing the relative entropy constraint from the standard variational information bottleneck framework, the measure of information compression is reconstructed as a topological rate-distortion metric based on the slice Gromov-Wasserstein distance. This topological rate-distortion metric measures the geometric structural difference between the original high-dimensional feature distribution and the compressed low-dimensional embedding distribution, including: The relative entropy constraint term in the standard variational information bottleneck framework is identified. The relative entropy constraint term drives the compression process by measuring the probability density difference between the original feature distribution and the compressed embedded distribution, but only focuses on the alignment at the statistical distribution level, ignoring the geometric manifold structure and temporal causal logic of the data in high-dimensional space. The relative entropy constraint term is removed from the standard variational information bottleneck framework to eliminate the smoothing effect of the relative entropy constraint term on the feature geometric manifold structure during compression. The smoothing effect will cause the temporal coherence of the specular action to be destroyed during dimensionality reduction. The slice Gromov-Wasserstein distance is introduced as an alternative metric. The slice Gromov-Wasserstein distance is calculated by projecting the high-dimensional feature distribution and the low-dimensional embedding distribution onto a randomly sampled one-dimensional subspace, calculating the Wasserstein distance between the two one-dimensional projected distributions, and taking the average after multiple random projections, so as to achieve a low-complexity approximate metric for the differences in high-dimensional geometric structures. The Gromov-Wasserstein distance of the slice is constructed as a topological rate-distortion metric, which is used to constrain the geometric structural differences, rather than statistical distribution differences, between the original high-dimensional feature distribution and the compressed low-dimensional embedding distribution during compression. The topology rate distortion metric guides the single-modal information bottleneck layer to retain the original features of the high-dimensional geometric manifold structure and action temporal causal logic during dimensionality reduction and compression, outputting a topology-aware single-modal bottleneck embedding.

7. The video specular highlight detection method for multimodal information bottlenecks according to claim 1, characterized in that, The topology-aware unimodal bottleneck embedding is input into a multimodal information bottleneck layer. This multimodal information bottleneck layer collaboratively performs a contrastive learning loss and variational constraints, and then performs cross-modal semantic refinement on the semantically aligned visual and audio bottleneck embeddings to generate a minimally redundant multimodal bottleneck embedding, including: The topology-aware visual bottleneck embedding and the topology-aware audio bottleneck embedding are received as unimodal input embeddings. The visual bottleneck embedding and the audio bottleneck embedding are asynchronous in time, which leads to a misalignment in their semantic expression. The visual bottleneck embedding and the audio bottleneck embedding are respectively spliced ​​and fused with the multimodal representation generated by the two branches. Cross-modal comparison and association calculation is performed. Contrastive learning loss is introduced to narrow the semantic distance of the audiovisual bottleneck embeddings under the same video context, eliminate the semantic expression misalignment caused by multimodal asynchronous, and generate semantically aligned visual bottleneck embeddings and audio bottleneck embeddings. The semantically aligned visual bottleneck embedding and audio bottleneck embedding are dynamically fused to construct a multimodal fusion attention feature map, which contains semantically calibrated cross-modal interaction information. The multimodal fusion attention feature map is input into the multimodal information bottleneck layer. During forward propagation computation, the multimodal variational information bottleneck loss is used to perform deep semantic purification and secondary dimensionality reduction on the fused cross-modal representation, eliminating non-critical redundant semantic flows between the two modalities, retaining the core semantic information that is highly relevant to the specular detection task, and generating a multimodal bottleneck embedding with minimal redundancy and high relevance to the prediction task.

8. The video specular highlight detection method for multimodal information bottlenecks according to claim 7, characterized in that, The visual bottleneck embedding and the audio bottleneck embedding are respectively combined with the multimodal representation generated by splicing and fusing the two branches. Cross-modal comparative association calculation is performed, and a contrastive learning loss is introduced to narrow the semantic distance between the audiovisual bottleneck embeddings in the same video context, eliminate the semantic expression misalignment caused by multimodal asynchrony, and generate semantically aligned visual bottleneck embeddings and audio bottleneck embeddings, including: The topology-aware visual bottleneck embedding and the topology-aware audio bottleneck embedding are received. The visual bottleneck embedding and the audio bottleneck embedding have a natural asynchrony on the time axis. The asynchrony is manifested in the time offset between the occurrence time of the visual event and the corresponding audio response time, which causes the visual representation and audio representation in the same video context to be in different positions in the semantic space. The visual bottleneck embedding and the audio bottleneck embedding are spliced ​​and fused to generate a multimodal fusion representation, which serves as an anchor point reference for cross-modal alignment. Construct a contrastive learning loss function, wherein the multimodal fusion representation is used as the positive sample anchor point, the visual bottleneck embedding and audio bottleneck embedding under the same video context are used as positive sample pairs, and the visual bottleneck embedding and audio bottleneck embedding under different video contexts are used as negative sample pairs. By minimizing the contrastive learning loss function, the semantic distance between the visual bottleneck embedding and the multimodal fusion representation is reduced, and the semantic distance between the audio bottleneck embedding and the multimodal fusion representation is also reduced, so that the visual bottleneck embedding and the audio bottleneck embedding move closer to a common fusion representation center in the semantic space. Driven by the contrastive learning loss function, the temporal asynchronous offset between the visual bottleneck embedding and the audio bottleneck embedding is eliminated, and the two occupy consistent positions in the semantic space, generating semantically aligned visual bottleneck embeddings and audio bottleneck embeddings.

9. The video specular highlight detection method for multimodal information bottlenecks according to claim 1, characterized in that, The multimodal bottleneck is embedded into the highlight scorer, which outputs a highlight detection score, including: The minimum redundancy multimodal bottleneck embedding output by the multimodal information bottleneck layer is received. The minimum redundancy multimodal bottleneck embedding has eliminated non-critical semantic flows that coexist across modalities and retained the core semantic information that is highly relevant to the specular detection task. The minimum redundancy multimodal bottleneck is embedded into the highlight scorer, which includes a linear mapping layer and a nonlinear activation unit. The highlight score prediction value corresponding to the current video segment is calculated through forward mapping. Based on the predicted highlight score and the actual highlight label, the task loss value, which measures the model’s highlight moment localization performance, is calculated using the binary cross-entropy loss function. The task loss value is backpropagated to the multimodal information bottleneck layer and the single-modal information bottleneck layer, driving the multimodal information bottleneck layer and the single-modal information bottleneck layer to further optimize the selection and retention of core semantic information in subsequent iterations, forming a collaborative optimization closed loop of feature refinement and score prediction, and outputting the final specular detection score result.

10. A video specular highlight detection system for multimodal information bottlenecks, characterized in that, The system applicable to the method of any one of claims 1 to 9 comprises: The data acquisition and feature extraction module is used to receive the raw video stream and audio stream, and extract the raw visual features and raw audio features using the pre-trained visual encoder and pre-trained audio encoder, respectively. An environment pseudo-label generation module, connected to the data acquisition and feature extraction module, is used to receive the original audio features, extract the root mean square energy of the audio segment, and generate discrete environment pseudo-labels using a binarized step function. The causal attention perception module is connected to the data acquisition and feature extraction module and the environment pseudo-label generation module. It is used to receive the original visual features, original audio features and environment pseudo-labels. It contains a parallel environment confusion network and a causal feature network. A gradient inversion layer is set before the input layer of the causal feature network. It uses binary cross-entropy loss, adversarial loss and orthogonal loss to drive the input features in the feature space to decouple the input features into mutually orthogonal environment branch features and causal branch features, and outputs causal features that are invariant to the environment. A single-modal topological information bottleneck compression module, connected to the causal attention perception module, is used to receive the causal features that are invariant to the environment, internally remove the relative entropy constraints in the standard variational information bottleneck framework, reconstruct the measure of information compression degree into a topological rate-distortion metric based on the slice Gromov-Wasserstein distance, and constrain the geometric manifold preservation during the compression process through the topological rate-distortion metric to generate a topology-aware single-modal bottleneck embedding. A multimodal semantic refinement module, connected to the single-modal topology information bottleneck compression module, is used to receive the topology-aware single-modal bottleneck embedding. It includes a contrast alignment unit and a multimodal information bottleneck layer. The contrast alignment unit uses the multimodal representation generated by splicing and fusion as an anchor point and uses contrastive learning loss to narrow the semantic distance between the visual bottleneck embedding and the audio bottleneck embedding, eliminating the semantic expression misalignment caused by multimodal asynchronous processing. The multimodal information bottleneck layer uses variational constraints to perform cross-modal semantic refinement processing on the semantically aligned visual and audio bottleneck embeddings, eliminating non-critical redundant semantic flows and generating a multimodal bottleneck embedding with minimal redundancy. The highlight scorer, connected to the multimodal semantic refinement module, is used to receive the minimum redundancy multimodal bottleneck embedding and output the highlight detection score through a linear mapping layer and a nonlinear activation unit. The loss coprocessor is connected to the causal attention perception module, the single-modal topological information bottleneck compression module, the multimodal semantic refinement module, and the spectro scorer, respectively. It receives the loss components output by each module and weights and fuses the binary cross-entropy task loss, environment fitting loss, adversarial loss, orthogonal loss, slice Gromov-Wasserstein topological loss, contrast alignment loss, and multimodal variational information bottleneck loss into a global co-total loss, driving iterative updates of system parameters.

Images