A Competitive Adaptive Fusion Method for Multimodal Features

By introducing competitive weight allocation in the channel dimension and an adaptive fusion method in the time dimension, the problems of channel weight balancing and insufficient adapter adaptability in multimodal feature fusion are solved, thereby improving the information selection and dynamic modeling capabilities of multimodal feature interaction.

CN123116136APending Publication Date: 2026-11-17ZHENGZHOU UNIV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611278509.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-21
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing multimodal feature fusion methods, the balanced distribution of channel weights weakens the model's ability to highlight key semantic channels, and the adapter structure is difficult to adapt to information changes in different time segments, limiting the model's expressive ability in complex dynamic scenarios.

Method used

We employ a channel-dimensional competitive weight allocation and feature adaptive fusion method. We establish inter-channel correlations through lightweight bottleneck mapping, introduce a normalized competition mechanism to enhance the selection of key channels, and dynamically select feature processing paths in the time dimension through a hybrid expert mechanism.

Benefits of technology

It improves the efficiency of information utilization in cross-modal feature interaction and the model's expressive ability in complex dynamic scenarios, and enhances the model's response and adaptability to key semantic channels.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123116136A_ABST
    Figure CN123116136A_ABST
Patent Text Reader

Abstract

This invention discloses a competitive adaptive fusion method for multimodal features, belonging to the field of computer vision and multimodal information processing technology. Addressing the problems of independent channel weights, lack of explicit competition, and difficulty in adapting to dynamic semantic states in existing audio-visual cross-modal interactions, this invention constructs a channel-dimensional competitive weight allocation module and a feature adaptive fusion module based on a hybrid expert mechanism. The weight allocation module achieves key channel enhancement and redundant channel suppression through channel recoding, spatial dimension global aggregation, softmax competitive normalization, and centralization processing. The fusion module dynamically allocates expert weights through multiple parameter-independent adaptation experts and routing functions, achieving adaptive selection of multimodal feature processing paths. This invention improves multimodal feature selection, dynamic modeling, and cross-modal fusion capabilities while freezing backbone network parameters and controlling parameter scale, and is applicable to audio-visual semantic understanding, event recognition, and sound target localization tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multimodal information processing technology, and in particular to a multimodal feature fusion method for joint modeling of audio and video features. Specifically, this invention improves the feature representation and dynamic modeling capabilities in cross-modal audio-video interaction by performing competitive weight allocation of multimodal features based on channel dimensions and adaptive feature fusion processing. Background Technology

[0002] With the development of multimedia information acquisition and processing technologies, more and more applications require the simultaneous use of multimodal data for information analysis. For example, in video understanding tasks, not only visual information from video images can be utilized, but also corresponding audio information. Compared to single-modal information, multimodal data can describe the same event or target from different perceptual dimensions, possessing richer semantic expressive power and stronger environmental robustness. For instance, in complex scenes, when visual information is affected by occlusion or changes in lighting, audio information can still provide effective clues; and when audio signals are interfered with by noise, visual information can provide additional spatial positioning information. Therefore, by jointly utilizing multimodal information, the uncertainty of single-modal information can be compensated for to some extent, thereby improving the overall performance of intelligent perception systems.

[0003] However, due to significant differences in data generation mechanisms, representation formats, and semantic expression methods among different modalities, multimodal information in practical applications typically exhibits highly heterogeneous and dynamically changing characteristics. For example, video data originates from the optical imaging process and primarily reflects the spatial structure and motion state of a target; audio data, on the other hand, originates from the sound wave vibration process and focuses more on describing the temporal changes and frequency characteristics of the sound source. These two types of signals differ fundamentally in sampling methods, data structures, and statistical properties. Furthermore, within the same scene, different modalities may also have different semantic focuses regarding the same event. Therefore, how to effectively model and rationally integrate the features of different modalities has become a key research problem in the field of multimodal information processing.

[0004] To address this issue, existing research has explored various technical approaches. Early multimodal methods typically employed feature concatenation or linear weighting to achieve multimodal feature fusion. For example, different modal features were directly concatenated along their feature dimensions, or weighted sums of modal features using preset weights to obtain a unified multimodal representation. Due to their simplicity, these methods saw some application in early multimodal research. However, because the fusion weights are usually set manually based on experience, they are difficult to dynamically adjust according to different tasks or scenarios, thus failing to fully characterize the potentially complex relationships between different modalities.

[0005] With the development of deep learning technology, researchers are increasingly leveraging the inherent complementary properties between different modalities to more effectively model multimodal features. In audio-visual tasks, visual information typically provides clear spatial localization information, while audio information reflects the temporal variation characteristics of the sound source. Based on this complementary relationship, some studies have proposed bidirectional guidance mechanisms that guide visual features through audio features, while visual features also guide audio features in turn, thereby achieving semantic alignment during cross-modal interactions. These methods have made some progress in multimodal event recognition, audio-visual understanding, and cross-modal analysis tasks.

[0006] Building upon the bidirectional guidance mechanism, to further enhance the model's ability to model complex multimodal features, researchers introduced attention mechanisms in the channel, spatial, and temporal dimensions. By constructing attention structures across multiple dimensions, they achieved more granular feature weight allocation. For example, in the channel dimension, the model can adjust the weights of different semantic response channels; in the spatial dimension, the model can highlight key regions; and in the temporal dimension, the model can dynamically model based on the information contribution levels of different time segments. Through the joint modeling of multi-dimensional attention mechanisms, the multimodal feature interaction process is further enhanced.

[0007] Furthermore, to reduce the training cost of large-scale multimodal models, some studies have adopted the method of freezing the backbone network parameters and inserting lightweight adapter modules into the network to achieve cross-modal feature modulation. By training only the adapter module, the alignment and fusion of multimodal features in a unified semantic space can be achieved while maintaining the stability of the backbone network structure, thereby reducing the training overhead of the model in new tasks.

[0008] However, some problems still exist in the aforementioned multimodal feature fusion methods. First, in existing bidirectional guided attention structures, channel weights are typically generated by globally pooling features followed by a sigmoid function. Since the sigmoid function maps each channel response independently, there is no direct competition between different channels. Therefore, in actual training, the channel weights often exhibit a relatively similar distribution. This relatively balanced weight distribution weakens the model's ability to highlight key semantic channels, making it difficult for cross-modal guidance mechanisms to fully leverage the complementary advantages between different modalities.

[0009] Secondly, in existing lightweight adapter structures, features from different time segments are typically modulated across modally using the same set of parameters, meaning all time steps share the same adapter parameters. However, in real-world audio and video scenarios, the dominant information modality may change across different time stages. For example, visual information may be more prominent in some time segments, while audio information may be more discriminative in others. In such cases, a single adapter structure struggles to adapt to different semantic states simultaneously, thus limiting the model's ability to model complex dynamic scenes.

[0010] Therefore, it is necessary to design a multimodal feature processing method that can achieve explicit feature selection at the channel level and has dynamic adaptability in the time dimension, based on the existing multimodal fusion framework, so as to improve the information utilization efficiency in cross-modal interaction and enhance the model's expressive ability in complex dynamic scenarios. Summary of the Invention

[0011] To address the shortcomings of existing multimodal feature fusion methods in cross-modal interaction, this invention proposes a multimodal feature processing method based on competitive weight allocation of channel dimensions and adaptive feature fusion.

[0012] On the one hand, in existing bidirectional guided attention structures, channel attention is usually the first stage of cross-modal feature interaction, and the channel weights it generates directly affect the feature representation capabilities of subsequent spatial and temporal attention modules. However, in existing methods, there is a lack of direct competition between different channels, so in actual training, the weights of each channel often exhibit a relatively similar distribution. This relatively balanced weight distribution weakens the model's ability to highlight key semantic channels, making it difficult for cross-modal guidance mechanisms to fully leverage the complementary advantages between different modalities.

[0013] On the other hand, in existing multimodal models, to reduce training costs, a common approach is to freeze the backbone network and insert lightweight adapter modules to achieve cross-modal feature modulation. However, in this structure, features from different time segments are typically processed through the same set of adapter parameters, meaning all time steps share the same feature modulation scheme. But in real-world audio and video scenarios, the dominant information modality may change at different time stages. For example, visual information may be more prominent in some time segments, while audio information may be more discriminative in others. In this case, a single adapter structure struggles to adapt to different semantic states simultaneously, thus limiting the model's ability to model complex dynamic scenes.

[0014] To address the two issues mentioned above, this paper designs a multimodal feature processing method that can emphasize relevant channels and suppress irrelevant channels at the channel level, and has dynamic adaptability in the time dimension. This improves the efficiency of information utilization in cross-modal feature interaction and enhances the model's expressive ability in complex dynamic scenarios.

[0015] In the multimodal feature biased weight allocation and adaptive fusion method proposed in this invention, let the input visual features be represented as follows: Where B represents the batch size, T represents the number of time steps, C represents the number of feature channels, and H and W represent the spatial resolution. In the basic model, expanding the time dimension and batch dimension, it can be represented as: .in This indicates the number of spatial locations after unfolding. The specific technical solution is as follows:

[0016] 1. Channel Dimension Competitive Weight Allocation Module

[0017] To address the issue that channel attention weights tend to be averaged in existing bidirectional guidance structures, this invention proposes a feature modulation mechanism for achieving competitive weight allocation across channel dimensions.

[0018] First, the input features are recoded at the channel level. Specifically, a lightweight bottleneck mapping structure is used to transform the channel features, i.e., first compressing the channel dimensions, and then restoring them to the original dimensions through nonlinear mapping, which can be expressed as:

[0019] (1)

[0020] in, , Here, r represents the channel compression ratio, and σ(·) is the nonlinear activation function GELU. This bottleneck structure not only controls the number of parameters involved in the computation but also establishes more comprehensive inter-channel relationships in the low-dimensional semantic space, thereby enhancing the semantic interaction capabilities between different channels and providing more discriminative feature representations for subsequent weight calculations.

[0021] Subsequently, the recoded features are globally aggregated in the spatial dimension to obtain the global statistical response for each channel:

[0022] (2)

[0023] in, This represents the global response value of the c-th channel.

[0024] To establish competitive relationships between channels, this invention introduces a normalized competition mechanism at the channel level, using the softmax function to generate initial weights, ensuring that all channels are compared relatively within the same constraint space, satisfying... Therefore, when the weight of a certain channel increases, the weights of the other channels must decrease accordingly, thus creating an explicit competitive relationship between the channels.

[0025] Since the weights output by softmax are all positive, they can only enhance features but are difficult to suppress redundant channels. To achieve both channel enhancement and suppression simultaneously, this invention centers the generated channel weights.

[0026] (3)

[0027] Through this operation, This allows for the enhancement of critical channels and the suppression of secondary channels.

[0028] Finally, the centered weights are incremented by 1 to prevent feature sign flipping when the weights directly affect the features, and to maintain the stability of the original feature scale. The final feature modulation method is as follows:

[0029] (4)

[0030] Where α is the modulation intensity coefficient, used to control the degree of influence of channel competition weights on features.

[0031] Through the above steps, this invention achieves feature weight allocation with explicit competition in the channel dimension, thereby enhancing the model's ability to select key semantic channels.

[0032] 2. Feature Adaptive Fusion Module

[0033] To address the problem that existing adaptation modules share fixed parameters across different time segments and lack dynamic adaptability, this invention further proposes a feature adaptive fusion method based on a hybrid expert mechanism. Let a certain high-layer output feature of the backbone network be... Where t represents the time step. Construct K structurally consistent but parameter-independent adaptation expert modules. Each expert implements modulation mapping of cross-modal features:

[0034] (5)

[0035] Where k = 1, 2, …, K.

[0036] To dynamically select expert paths based on the semantic state at the current time step, the softmax function is used to generate fusion weights for each expert based on the features at the current time step.

[0037] (6)

[0038] in Let represent the scoring function corresponding to the k-th expert. This represents the weight of the k-th expert at the current time step t, satisfying... .

[0039] The final output features are represented as a multi-expert weighted fusion result:

[0040] (7)

[0041] Through this mechanism, features at different time steps can dynamically select different expert paths based on the current semantic state, thereby achieving adaptive fusion processing of multimodal features.

[0042] The beneficial effects of this invention are as follows:

[0043] 1. By introducing a normalized competition mechanism at the channel dimension, an explicit competitive relationship is achieved between channels, thereby forming a competitive weight allocation at the channel dimension. This mechanism can effectively enhance the response of key semantic channels while suppressing redundant channels, thus improving the information selection capability in the cross-modal feature interaction process.

[0044] 2. By introducing a dynamic routing mechanism based on a hybrid expert structure, adaptive feature fusion of multimodal features is achieved. This mechanism can dynamically select different feature processing paths according to different temporal semantic states, thereby improving the model's expressive ability in complex dynamic scenarios.

[0045] 3. By combining the competitive weight allocation mechanism of the channel dimension with the adaptive feature fusion mechanism, the feature selection ability and dynamic modeling ability of the model are improved at the same time while maintaining the stability of the backbone network structure and the controllability of the model parameter scale, thereby enhancing the overall effect of multimodal feature fusion. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the overall structure of the multimodal feature biased weight allocation and adaptive fusion method based on DG-SCT of the present invention.

[0047] Figure 2 This is a schematic diagram of the channel contention module in this invention. Detailed Implementation

[0048] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings.

[0049] Figure 1This is a schematic diagram of the overall structure of the multimodal feature biased weight allocation and adaptive fusion method based on DG-SCT of the present invention. The overall process is as follows: After extracting features from the audio modality and the visual modality using a frozen backbone network, biased channel weight allocation, channel attention, spatial attention, and temporal attention are performed in trainable adapters. After fusing the outputs of multiple adapters, video and audio features with complementary multimodal information are obtained and used as input for specific downstream tasks.

[0050] Figure 2 This is a schematic diagram of the channel competition module in this invention. In the diagram, Channel represents the number of channels in the input feature, and r represents the channel compression ratio. The input of this diagram consists of features extracted from the frozen backbone network, and the output is features that emphasize relevant channels and suppress irrelevant channels. The process is as follows: first, the input features are compressed and recoded using channels; then, channel competition weights are generated and applied to the input features to obtain output features with enhanced channel dimensions.

Claims

1. A competitive adaptive fusion method for multimodal features, comprising the following steps: (1) preprocessing the input video frame sequence and the corresponding audio signal to construct visual modal features and audio modal features respectively; (2) inputting the visual modal features into a visual backbone network encoder and the audio modal features into an audio backbone network encoder to obtain the corresponding multimodal feature representations, wherein the parameters of the visual backbone network encoder and the audio backbone network encoder are kept frozen; (3) inputting the multimodal features into multiple parallel DG-SCT adapters to perform cross-modal interactive processing on the features; (4) in the DG-SCT adapter, using a channel competition module to perform competitive weight allocation on the features in the channel dimension to enhance key semantic channels and suppress redundant channels; (5) sequentially performing channel attention, spatial attention and temporal attention processing on the features after channel competition weight allocation, wherein the temporal attention module includes a hybrid expert adaptation structure; (6) performing weighted fusion on the outputs of multiple DG-SCT adapters based on a routing gate mechanism to obtain the fused multimodal feature representations; (7) outputting the fused audio features and visual features.

2. The method according to claim 1, wherein the channel contention module comprises: (1) Channel recoding is performed on the input features, and a bottleneck structure is formed by linear dimensionality reduction and linear dimensionality increase; (2) Global average pooling is performed on the recoded features in the spatial dimension to obtain the global response of each channel; (3) The response of each channel is normalized by the softmax function to build the competitive relationship between channels; (4) The channel weights after normalization are centered to achieve channel enhancement and suppression.

3. The method according to claim 2, wherein the channel competition weights are scaled before being applied to the original features, and the influence intensity of the weights on the features is controlled by introducing modulation coefficients.

4. The method according to claim 1, wherein a hybrid expert mechanism is introduced into the time attention module, comprising: (1) Construct multiple expert branches with the same structure but independent parameters; (2) Generate the weights of each expert branch based on the input features using a routing function; (3) The outputs of each expert branch are weighted and fused to obtain the feature representation of the time dimension.

5. The method according to claim 4, wherein the routing function uses a softmax function to generate the weight distribution of each expert branch.

6. The method according to claim 1, wherein multiple DG-SCT adapters are configured in parallel, and each adapter includes a channel contention module, a channel attention module, a spatial attention module, and a temporal attention module.

7. The method according to claim 1, wherein the visual backbone network encoder is a Swing Transformer and the audio backbone network encoder is an HTS-AT model.

8. The method according to claim 1, wherein the multimodal features are video frame features and audio spectrum features.