Multimodal feature fusion method and related apparatus

Abstract

The application discloses a kind of based on double passage cooperation and adaptive gate's multimodal feature fusion method and related device, its method includes obtaining the feature embedding set of at least two modes and a global mode quality indication signal;At least two modal feature embedding set and indication signal are input into global signal regulation dynamic fusion passage and are treated with feature dynamic fusion, and dynamic fusion feature data is obtained;At least two modal feature embedding set is input into data-driven adaptive adjustment passage and is treated with feature regulation fusion, and regulation fusion feature data is obtained;Dynamic fusion feature data and regulation fusion feature data are input into adaptive gate integrator and are treated with feature weighted fusion, and weighted fusion feature data is obtained.In the embodiment of the application, the organic combination of "macro calibration" and "micro scheduling" is realized by double passage architecture, and the problem that single passage cannot consider global guidance and fine-grained adaptation of features is solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multimodal feature fusion method and related apparatus based on dual-path collaboration and adaptive gating. Background Technology

[0002] In multimodal intelligent systems, designing fusion strategies to fully utilize complementary information between different modalities (such as images, text, sensor data, and clinical indicators) is crucial for improving system performance. Existing fusion methods can be broadly categorized into three types: early fusion (such as feature stitching), mid-stage fusion (such as cross-modal attention), and late-stage fusion (such as decision weighting). However, most of these methods have inherent limitations: they typically employ a single, pre-defined fusion path, failing to adapt to the inherent differences in data quality among different input samples and the complex, dynamic collaborative patterns between modalities.

[0003] For example, in medical diagnosis, for some patients, MRI images may be clear and highly discriminative, while clinical scales may be noisy due to subjective factors; for others, the opposite may be true. An ideal fusion system should be able to dynamically adjust the reliance on and integration of different evidence sources for each specific case, much like an expert. Existing single-path fusion models lack this fine-grained, sample-adaptive adjustment capability, resulting in insufficient generalization ability and robustness when faced with complex and heterogeneous real-world data. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art. This invention provides a multimodal feature fusion method and related device based on dual-path collaboration and adaptive gating. Through the dual-path architecture, the feature organic combination of "macro-calibration" and "micro-scheduling" is realized, which solves the problem that a single path cannot take into account both global guidance and fine-grained adaptation of features.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a multimodal feature fusion method based on dual-pathway collaboration and adaptive gating, the method comprising: Obtain feature embedding sets for at least two modalities and an indication signal for global modal quality; The feature embedding set of at least two modalities and the dynamic fusion path of the indicator signal input global signal modulation are subjected to feature dynamic fusion processing to obtain dynamic fusion feature data; The adaptive adjustment path driven by the input data of at least two modal feature embedding sets is subjected to feature adjustment fusion processing to obtain adjustment fusion feature data; The dynamic fusion feature data and the adjusted fusion feature data are input into the adaptive gating integrator for feature weighted fusion processing to obtain weighted fusion feature data.

[0006] Optionally, the step of performing dynamic feature fusion processing on the feature embedding set of at least two modalities and the dynamic fusion path of the indicator signal input global signal modulation to obtain dynamic fused feature data includes: Features from at least two modalities are concatenated into joint features, and the basic attention weights of the dynamic fusion path are globally adjusted using the indication signal to form global adjustment weights. Based on the indication signal and the joint features, the interaction learning matrix in the dynamic fusion path is conditionally processed to form a conditional interaction learning matrix. Within the dynamic fusion pathway, the conditionalized interactive learning matrix is ​​used to perform multi-level feature refinement processing on the joint features, forming refined joint features. The joint features after fine adjustment are weighted and summed based on the global adjustment weights, and the residual connections of the joint features are introduced during the weighted summation process to form dynamic fused feature data.

[0007] Optionally, the step of conditionally processing the interaction learning matrix within the dynamic fusion path based on the indication signal and the joint features to form a conditional interaction learning matrix includes: The indication signal is encoded into a conditional vector, and the conditional vector and the joint feature are output together into the meta-network of the dynamic fusion path. The interaction learning matrix is ​​then conditionally processed to form a conditional interaction learning matrix.

[0008] Optionally, the step of using the conditionalized interaction learning matrix to perform multi-level feature refinement processing on the joint features within the dynamic fusion path to form refined joint features includes: The joint features are stacked into feature matrix data, and the feature matrix data is subjected to interactive learning processing using the conditionalized interactive learning matrix to form interactive learning feature matrix data. The interactive learning feature matrix data is locally adaptively mixed using a one-dimensional convolution function and the indicator signal to form mixed feature matrix data. The hybrid feature matrix data is subjected to global attention focusing processing based on a cross-modal multi-head attention mechanism to obtain the hybrid feature matrix data after global attention focusing; Based on the global modulation of the indication signal, the hybrid feature matrix data after global attention focusing is subjected to fine-tuning of the feature dimensions to form fine-tuned joint features.

[0009] Optionally, the step of performing feature-modal fusion processing on an adaptive adjustment path driven by feature embedding sets of at least two modalities to obtain adjusted fusion feature data includes: The features of at least two modalities are concatenated into a joint feature set. The joint matrix is ​​mapped to a predicted original adjacency matrix through the feedforward network of the adaptive adjustment path. The predicted original adjacency matrix is ​​then weighted and fused with a prior matrix reflecting prior modal relationships to form a weighted fused feature matrix. The weighted fusion feature matrix is ​​processed by a self-influence enhancement mechanism to form an enhanced feature matrix; The enhanced feature matrix is ​​subjected to interactive intensity modulation and feature aggregation processing to obtain adjusted fused feature data.

[0010] Optionally, the step of performing interactive intensity modulation and feature aggregation processing on the enhanced feature matrix to obtain adjusted fused feature data includes: Calculate the overall interaction strength scalar of the enhanced feature matrix to obtain the overall interaction strength scalar data corresponding to the enhanced feature matrix; The overall interaction intensity scalar data is used as the adjustment factor for the Softmax temperature adjustment of the independent weight prediction head; The joint features are weighted and summed based on the adjustment factor, and residual connections of the joint features are introduced during the weighted summation process to form adjusted fusion feature data.

[0011] Optionally, the formula for inputting the dynamically fused feature data and the adjusted fused feature data into the adaptive gated integrator for feature weighted fusion processing is as follows: ; in, For gated network functions; For temperature parameters; The gate value of the adaptive gate integrator is defined as follows, and its range is: ; For dynamic fusion of feature data; To adjust and fuse feature data; To set parameters; This is weighted and fused feature data.

[0012] In addition, this embodiment of the invention also provides a multimodal feature fusion device based on dual-pathway collaboration and adaptive gating, the device comprising: Acquisition module: used to acquire feature embedding sets of at least two modalities and an indication signal of global modal quality; Dynamic fusion module: used to perform dynamic feature fusion processing on the feature embedding set of at least two modes and the dynamic fusion path of the indicator signal input global signal modulation to obtain dynamic fusion feature data; The adjustment and fusion module is used to perform feature adjustment and fusion processing on an adaptive adjustment path driven by the input data of at least two modalities to obtain adjusted and fused feature data. Weighted fusion module: used to input the dynamic fusion feature data and the adjusted fusion feature data into the adaptive gating integrator for feature weighted fusion processing to obtain weighted fusion feature data.

[0013] In addition, embodiments of the present invention also provide an electronic device, including a processor and a memory, wherein the processor runs a computer program or code stored in the memory to implement the multimodal feature fusion method as described in any of the above.

[0014] In addition, embodiments of the present invention also provide a computer-readable storage medium for storing a computer program or code, which, when executed by a processor, implements the multimodal feature fusion method as described above.

[0015] In this embodiment of the invention, a dual-path architecture of "global signal regulation path" and "data-driven adjustment path" is adopted in parallel to achieve the organic combination of "macro-calibration" and "micro-scheduling", which solves the problem that a single path cannot take into account both global guidance and fine-grained adaptation. Multiple robustness enhancement mechanisms are introduced into the "data-driven adjustment path" to effectively ensure the stability, modal diversity and reliability of the learning process. Furthermore, feature weighted fusion processing is performed by an adaptive gating integrator, which can dynamically adjust the fusion logic for different cases like an expert, and significantly improve the level of intelligence. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the multimodal feature fusion method based on dual-path collaboration and adaptive gating in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structural composition of the multimodal feature fusion device based on dual-pathway collaboration and adaptive gating in an embodiment of the present invention; Figure 3This is a schematic diagram of the structural composition of the electronic device in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1, please refer to Figure 1 , Figure 1 This is a flowchart illustrating the multimodal feature fusion method based on dual-path collaboration and adaptive gating in an embodiment of the present invention.

[0020] like Figure 1 As shown, a multimodal feature fusion method based on dual-pathway collaboration and adaptive gating is proposed, the method comprising: S101: Obtain feature embedding sets for at least two modalities and an indication signal for global modal quality; In the specific implementation of this invention, it is first necessary to obtain feature embedding sets of at least two modes and a global modal quality indication signal; wherein the feature embedding sets of at least two (N≥2) modes and an optional global modal quality indication signal. The indicator signal can be used to reflect the quality or reliability of a specific modality or overall data, and its source can be external evaluation or internal learning.

[0021] S102: Perform dynamic feature fusion processing on the feature embedding set of at least two modes and the dynamic fusion path of the indicator signal input global signal modulation to obtain dynamic fusion feature data; In a specific implementation of this invention, the dynamic fusion processing of the feature embedding set of at least two modalities and the dynamic fusion path controlled by the indicator signal input global signal to obtain dynamic fusion feature data includes: concatenating features within the feature embedding set of at least two modalities into joint features, and using the indicator signal to globally adjust the basic attention weights of the dynamic fusion path to form global adjustment weights; conditionally processing the interaction learning matrix within the dynamic fusion path based on the indicator signal and the joint features to form a conditional interaction learning matrix; using the conditional interaction learning matrix within the dynamic fusion path to perform multi-level feature fine-tuning processing on the joint features to form fine-tuned joint features; and performing weighted summation processing on the fine-tuned joint features based on the global adjustment weights, and introducing residual connections of the joint features during the weighted summation processing to form dynamic fusion feature data.

[0022] Furthermore, the step of conditionalizing the interaction learning matrix within the dynamic fusion path based on the indication signal and the joint feature to form a conditional interaction learning matrix includes: encoding the indication signal into a conditional vector, outputting the conditional vector and the joint feature together into the meta-network in the dynamic fusion path, and conditionalizing the interaction learning matrix to form a conditional interaction learning matrix.

[0023] Furthermore, the step of using the conditionalized interactive learning matrix to perform multi-level feature refinement processing on the joint features within the dynamic fusion path to form refined joint features includes: stacking the joint features into feature matrix data, and using the conditionalized interactive learning matrix to perform interactive learning processing on the feature matrix data to form interactive learning feature matrix data; using a one-dimensional convolution function and the indicator signal to perform local adaptive mixing processing on the interactive learning feature matrix data to form mixed feature matrix data; performing global attention focusing processing on the mixed feature matrix data based on a cross-modal multi-head attention mechanism to obtain globally attention-focused mixed feature matrix data; and performing feature dimension refinement processing on the globally attention-focused mixed feature matrix data based on global modulation of the indicator signal to form refined joint features.

[0024] Specifically, the feature embedding sets of at least two modalities and the indicator signals are input into the dynamic fusion path of global signal modulation for dynamic feature fusion processing to obtain dynamic fused feature data; the dynamic fusion path of global signal modulation receives the feature embedding sets of at least two modalities and the indicator signals. Through signal-conditional gating, interactive matrix learning, and multi-level feature refinement, a deep interactive fusion guided by global quality signals is achieved, outputting dynamically fused feature data. .

[0025] The dynamic fusion path of global signal modulation receives feature embedding sets of at least two modes and indication signals. It emphasizes deep and refined cross-modal interaction under the guidance of global signals; the innovative use of the dynamic fusion path for global signal regulation is reflected in binding it with a clear indicator signal λ and using the indicator signal λ to achieve coordinated regulation throughout the entire link; the indicator signal λ semantically represents a measure of a certain "state" that needs to be regulated in the fusion process (e.g., the quality score, reliability estimate, or degree of uncertainty of a source data), and its numerical value (usually normalized to the [0,1] interval) is positively correlated with the level of that state; the source of the indicator signal λ is open, and it can be an external input or an internally learned parameter.

[0026] The indicator signal λ is defined as follows (open definition): The external input interface of the dynamic fusion path for global signal modulation, λ, can be used as an independent input parameter for this fusion; its specific value can be provided by any upstream processing unit or evaluation logic, for example, calculated based on the statistical properties of the feature itself (such as variance, entropy, internal correlation coefficient), or generated by a lightweight, dedicated quality evaluation subnetwork; the internally learnable parameter, in a more preferred implementation, It can be designed as a sample-dependent, adaptive parameter that can be trained end-to-end via gradient backpropagation; it will automatically learn in which data states it should be assigned... What value should be chosen to achieve the optimal fusion effect?

[0027] First, global gating initialization is based on the indicator signal. Then, the features from at least two modalities embedded in the set are concatenated into a joint feature, which is represented as follows: Basic attention weights are generated through a lightweight gating network. Subsequently, using indicator signals Perform global adjustment of this basic weight: ; in, It is a miniature network or linear function that maps the indicator signal λ to a scaling vector. Ensure dimension matching. This design employs element-wise multiplication and represents the first layer of signal-driven fusion: when the indicator signal λ is high (indicating high data quality), the original predicted values ​​of each modality weight can be preserved or enhanced; when the indicator signal λ is low (indicating questionable quality), the weights of all modalities are synchronously and moderately suppressed, laying the foundation for subsequent conservative fusion.

[0028] The interaction learning matrix within the dynamic fusion path is conditionally processed based on the indicator signal and joint features to form a conditional interaction learning matrix. This necessitates cross-modal interaction matrix learning conditionally based on the indicator signal. To model dynamic, asymmetric inter-modal information flow, the interaction learning matrix will be conditionally based on the indicator signal λ; that is, the indicator signal λ will be encoded as a conditional vector and then combined with the joint features... Input a meta-network together The details are as follows: ; in, This is the conditionalized interactive learning matrix. elements Characterization from modality Flow mode The information intensity; by adjusting the indicator signal λ, the interaction pattern can be adaptively learned; This represents the number of modes corresponding to the feature embedding set; when the indicator signal λ is low, the meta-network tends to generate a matrix closer to the identity matrix. This means suppressing cross-modal interactions and encouraging each modality to remain independent; when λ is high, it allows for the learning of more active and complex interaction patterns.

[0029] Then, multi-level feature refinement and signal modulation are performed; that is, joint features are combined. Stacked into a matrix To conduct initial interaction: .

[0030] Local adaptive blending, which applies one-dimensional convolution (Conv1D) along the modality dimension to capture local dependencies, has its update intensity directly controlled by the indicator signal λ. This is the third layer manifestation of the fusion driven by the indicator signal λ, as follows: ; in, It is a function of the indicator signal λ (e.g., a linear function). The higher λ is, the larger the step size of feature mixing and the more complete the interaction; the lower λ is, the more conservative the update. Let be the update matrix for the t-th mixing iteration.

[0031] Global attention focusing involves applying a cross-modal multi-head attention mechanism to the resulting mixed feature matrix data to capture global dependencies, thereby obtaining the globally attention-focused mixed feature matrix data. .

[0032] Next, feature recalibration gating is performed to finely adjust the feature dimensions of the attention output. This adjustment is also globally modulated by the indicator signal λ. This is the fourth layer of fusion driven by the indicator signal λ, as detailed below: ; Finally, adaptive weighted aggregation and information fidelity preservation are performed, that is, using the weights generated above. Characteristics after refining A weighted summation is performed. Simultaneously, a feature from the original splicing is introduced. Residual connection: ; in, For dynamic fusion of feature data.

[0033] This residual connection ensures that even during deep signal modulation, the most original and discriminative information of each modality is preserved, effectively preventing information loss and enhancing the stability and expressive power of the fusion result. In summary, Path 1 achieves deep fusion at multiple key levels, including initial weighting, interaction matrix learning, feature mixing intensity, and attention gating, under the coordinated and refined control of a single global indicator signal λ.

[0034] S103: Perform feature-modal fusion processing on an adaptive adjustment path driven by input data of at least two modal feature embedding sets to obtain fused adjustment feature data; In a specific implementation of this invention, the step of performing feature-modal fusion processing on an adaptive adjustment path driven by input data of feature embedding sets of at least two modalities to obtain regulated fusion feature data includes: concatenating features within the feature embedding sets of at least two modalities into joint features; mapping the joint matrix to a predicted original adjacency matrix through the feedforward network of the adaptive adjustment path; performing weighted fusion processing on the predicted original adjacency matrix and a prior matrix reflecting prior modal relationships to form a weighted fusion feature matrix; performing self-influence enhancement processing on the weighted fusion feature matrix to form an enhanced feature matrix; and performing interactive intensity modulation and feature aggregation processing on the enhanced feature matrix to obtain regulated fusion feature data.

[0035] Furthermore, the step of performing interaction intensity modulation and feature aggregation processing on the enhanced feature matrix to obtain regulated fusion feature data includes: calculating the overall interaction intensity scalar of the enhanced feature matrix to obtain the overall interaction intensity scalar data corresponding to the enhanced feature matrix; using the overall interaction intensity scalar data as the adjustment factor for the Softmax temperature adjustment of the independent weight prediction head; performing weighted summation processing on the joint features based on the adjustment factor, and introducing the residual connection of the joint features during the weighted summation processing to form regulated fusion feature data.

[0036] Specifically, the data-driven adaptive regulation pathway, as a parallel and complementary "micro-scheduler," is fundamentally based on learning a sample-specific modal relationship adjacency matrix directly from the data. It discovers personalized modal cooperation patterns that are not captured by the indicator signal λ; it complements the dynamic fusion path of global signal regulation with "macro calibration" and "micro scheduling".

[0037] First, adjacency matrix prediction and soft prior are fused, concatenating the features embedded in the feature set of at least two modalities into joint features, which are represented as follows: From the joint features In this process, the original adjacency matrix is ​​predicted through a feedforward mapping. To address the potential instability in training or the learning of relationships that violate domain knowledge that may result from a fully data-driven approach, this embodiment proposes a soft prior regularization method:

[0038] in, It is a pre-defined matrix that reflects prior modal relationships (such as the usual correlation between clinical and imaging modalities). This is a learnable mixture coefficient; this approach retains the flexibility of data-driven learning while smoothing and constraining the learning process with prior knowledge, preventing the learning of physically unreasonable strong interaction relationships, and significantly improving the robustness of training.

[0039] Furthermore, a self-influence enhancement mechanism is proposed. In cross-modal interactive learning, the features of some modalities may be over-dominated by other modalities and "annihilated." To prevent this problem and maintain the diversity of modal features, this embodiment proposes a self-influence enhancement mechanism to explicitly enhance the diagonal elements of the adjacency matrix (i.e., the self-contribution of each modality). ; in, These are learnable parameters; this mechanism does not simply increase the self-connect weights, but rather uses learnable non-negative vectors. Adaptive scaling, ensuring that each modality retains at least its own influence commensurate with its importance, is a key technique for preventing model collapse and maintaining information diversity.

[0040] Finally, interaction intensity modulation and feature aggregation are performed on the learned adjacency matrix. It reflects the overall activity of intermodal interactions; its overall interaction strength scalar is calculated. And this scalar is used as the Softmax temperature adjustment factor for an independent weighted prediction head, as follows: ; That is, when the overall intermodal interaction is strong (s is large), the weight distribution... The distribution becomes more "sharp," highlighting the dominant modality; when the interaction is weaker, the distribution becomes more "smooth," reflecting balanced fusion; this design transforms structural information (interaction intensity) into a dynamic adjustment signal for the fusion weights; finally, the adjustment features are obtained by weighted summation of the original modality embeddings and adding residual connections, as follows: ; in, To adjust and fuse feature data.

[0041] S104: Input the dynamic fusion feature data and the adjustment fusion feature data into the adaptive gating integrator for feature weighted fusion processing to obtain weighted fusion feature data.

[0042] In a specific implementation of this invention, the formula for inputting the dynamically fused feature data and the adjusted fused feature data into the adaptive gated integrator for feature weighted fusion processing is as follows: ; in, For gated network functions; For temperature parameters; The gate value of the adaptive gate integrator is defined as follows, and its range is: ; For dynamic fusion of feature data; To adjust and fuse feature data; To set parameters; This is weighted and fused feature data.

[0043] Specifically, adaptive gating integration (the key intelligent decision-making unit) is performed to obtain the outputs of two paths. (Deep interaction, global control) and After (relationship learning and robust adjustment), the final intelligent integration is achieved through a temperature scaling gating unit, which is the "decision brain" of the entire dual-pathway architecture. ; in, For gated network functions; For temperature parameters; The gate value of the adaptive gate integrator is defined as follows, and its range is: ; For dynamic fusion of feature data; To adjust and fuse feature data; To set parameters; This is weighted and fused feature data.

[0044] This design gives the model ultimate flexibility, enabling it to strike an optimal balance between "deep fusion based on feature consistency" and "robust adjustment based on data relationships".

[0045] Design details and principles: Gated networks A fusion algorithm is a small neural network (such as a multilayer perceptron) that receives the concatenation of the outputs of two pathways and learns to determine which pathway is more suitable for the current sample.

[0046] Temperature parameters : is a hyperparameter or learnable scalar greater than 0. Its key role is to adjust the "hard" or "soft" degree of gating decisions. When When the value is large, the input to the sigmoid function is compressed, and the gate value... When the value approaches 0.5, decision-making becomes "softer," encouraging the use of information from both pathways; when When the threshold is small, the decision is "hardened," with the gating value tending to be 0 or 1, explicitly choosing a path. This provides the model with the flexibility to adjust its decision-making style for different tasks or data granularities.

[0047] Adaptive Integration: Final Fusion Representation It is a convex combination of two output paths. (Gate value) Data-driven, it enables sample-level, optimized strategy selection: for samples with clear global signals requiring deep interaction, Approaching 1; for samples with complex modal relationships that require robust adjustment, It approaches 0; in most cases, it lies between the two.

[0048] The core value of this step lies in the fact that it enables the system to move away from relying on a single, fixed fusion strategy and instead dynamically and intelligently balance and integrate the two fusion philosophies of "deep interaction" and "robust adjustment" based on the unique attributes of each input sample. This is the key to achieving high-order adaptive capabilities.

[0049] In this embodiment of the invention, a dual-path architecture of "global signal regulation path" and "data-driven adjustment path" is adopted in parallel to achieve the organic combination of "macro-calibration" and "micro-scheduling", which solves the problem that a single path cannot take into account both global guidance and fine-grained adaptation. Multiple robustness enhancement mechanisms are introduced into the "data-driven adjustment path" to effectively ensure the stability, modal diversity and reliability of the learning process. Furthermore, feature weighted fusion processing is performed by an adaptive gating integrator, which can dynamically adjust the fusion logic for different cases like an expert, and significantly improve the level of intelligence.

[0050] Example 2, please refer to Figure 2 , Figure 2 This is a schematic diagram of the structure of the multimodal feature fusion device based on dual-pathway collaboration and adaptive gating in an embodiment of the present invention.

[0051] like Figure 2 As shown, a multimodal feature fusion device based on dual-pathway collaboration and adaptive gating is disclosed, the device comprising: Module 201: Used to obtain feature embedding sets of at least two modalities and an indication signal of global modal quality; In the specific implementation of this invention, it is first necessary to obtain feature embedding sets of at least two modes and a global modal quality indication signal; wherein the feature embedding sets of at least two (N≥2) modes and an optional global modal quality indication signal. The indicator signal can be used to reflect the quality or reliability of a specific modality or overall data, and its source can be external evaluation or internal learning.

[0052] Dynamic fusion module 202: used to perform feature dynamic fusion processing on the feature embedding set of at least two modes and the dynamic fusion path of the indicator signal input global signal modulation to obtain dynamic fusion feature data; In a specific implementation of this invention, the dynamic fusion processing of the feature embedding set of at least two modalities and the dynamic fusion path controlled by the indicator signal input global signal to obtain dynamic fusion feature data includes: concatenating features within the feature embedding set of at least two modalities into joint features, and using the indicator signal to globally adjust the basic attention weights of the dynamic fusion path to form global adjustment weights; conditionally processing the interaction learning matrix within the dynamic fusion path based on the indicator signal and the joint features to form a conditional interaction learning matrix; using the conditional interaction learning matrix within the dynamic fusion path to perform multi-level feature fine-tuning processing on the joint features to form fine-tuned joint features; and performing weighted summation processing on the fine-tuned joint features based on the global adjustment weights, and introducing residual connections of the joint features during the weighted summation processing to form dynamic fusion feature data.

[0053] Furthermore, the step of conditionalizing the interaction learning matrix within the dynamic fusion path based on the indication signal and the joint feature to form a conditional interaction learning matrix includes: encoding the indication signal into a conditional vector, outputting the conditional vector and the joint feature together into the meta-network in the dynamic fusion path, and conditionalizing the interaction learning matrix to form a conditional interaction learning matrix.

[0054] Furthermore, the step of using the conditionalized interactive learning matrix to perform multi-level feature refinement processing on the joint features within the dynamic fusion path to form refined joint features includes: stacking the joint features into feature matrix data, and using the conditionalized interactive learning matrix to perform interactive learning processing on the feature matrix data to form interactive learning feature matrix data; using a one-dimensional convolution function and the indicator signal to perform local adaptive mixing processing on the interactive learning feature matrix data to form mixed feature matrix data; performing global attention focusing processing on the mixed feature matrix data based on a cross-modal multi-head attention mechanism to obtain globally attention-focused mixed feature matrix data; and performing feature dimension refinement processing on the globally attention-focused mixed feature matrix data based on global modulation of the indicator signal to form refined joint features.

[0055] Specifically, the feature embedding sets of at least two modalities and the indicator signals are input into the dynamic fusion path of global signal modulation for dynamic feature fusion processing to obtain dynamic fused feature data; the dynamic fusion path of global signal modulation receives the feature embedding sets of at least two modalities and the indicator signals. Through signal-conditional gating, interactive matrix learning, and multi-level feature refinement, a deep interactive fusion guided by global quality signals is achieved, outputting dynamically fused feature data. .

[0056] The dynamic fusion path of global signal modulation receives feature embedding sets of at least two modes and indication signals. It emphasizes deep and refined cross-modal interaction under the guidance of global signals; the innovative use of the dynamic fusion path for global signal regulation is reflected in binding it with a clear indicator signal λ and using the indicator signal λ to achieve coordinated regulation throughout the entire link; the indicator signal λ semantically represents a measure of a certain "state" that needs to be regulated in the fusion process (e.g., the quality score, reliability estimate, or degree of uncertainty of a source data), and its numerical value (usually normalized to the [0,1] interval) is positively correlated with the level of that state; the source of the indicator signal λ is open, and it can be an external input or an internally learned parameter.

[0057] The indicator signal λ is defined as follows (open definition): The external input interface of the dynamic fusion path for global signal modulation, λ, can be used as an independent input parameter for this fusion; its specific value can be provided by any upstream processing unit or evaluation logic, for example, calculated based on the statistical properties of the feature itself (such as variance, entropy, internal correlation coefficient), or generated by a lightweight, dedicated quality evaluation subnetwork; the internally learnable parameter, in a more preferred implementation, It can be designed as a sample-dependent, adaptive parameter that can be trained end-to-end via gradient backpropagation; it will automatically learn in which data states it should be assigned... What value should be chosen to achieve the optimal fusion effect?

[0058] First, global gating initialization is based on the indicator signal. Then, the features from at least two modalities embedded in the set are concatenated into a joint feature, which is represented as follows: Basic attention weights are generated through a lightweight gating network. Subsequently, using indicator signals Perform global adjustment of this basic weight: ; in, It is a miniature network or linear function that maps the indicator signal λ to a scaling vector. Ensure dimension matching. This design employs element-wise multiplication and represents the first layer of signal-driven fusion: when the indicator signal λ is high (indicating high data quality), the original predicted values ​​of each modality weight can be preserved or enhanced; when the indicator signal λ is low (indicating questionable quality), the weights of all modalities are synchronously and moderately suppressed, laying the foundation for subsequent conservative fusion.

[0059] The interaction learning matrix within the dynamic fusion path is conditionally processed based on the indicator signal and joint features to form a conditional interaction learning matrix. This necessitates cross-modal interaction matrix learning conditionally based on the indicator signal. To model dynamic, asymmetric inter-modal information flow, the interaction learning matrix will be conditionally based on the indicator signal λ; that is, the indicator signal λ will be encoded as a conditional vector and then combined with the joint features... Input a meta-network together The details are as follows: ; in, This is the conditionalized interactive learning matrix. elements Characterization from modality Flow mode The information intensity; by adjusting the indicator signal λ, the interaction pattern can be adaptively learned; This represents the number of modes corresponding to the feature embedding set; when the indicator signal λ is low, the meta-network tends to generate a matrix closer to the identity matrix. This means suppressing cross-modal interactions and encouraging each modality to remain independent; when λ is high, it allows for the learning of more active and complex interaction patterns.

[0060] Then, multi-level feature refinement and signal modulation are performed; that is, joint features are combined. Stacked into a matrix To conduct initial interaction: .

[0061] Local adaptive blending, which applies one-dimensional convolution (Conv1D) along the modality dimension to capture local dependencies, has its update intensity directly controlled by the indicator signal λ. This is the third layer manifestation of the fusion driven by the indicator signal λ, as follows: ; in, It is a function of the indicator signal λ (e.g., a linear function). The higher λ is, the larger the step size of feature mixing and the more complete the interaction; the lower λ is, the more conservative the update. Let be the update matrix for the t-th mixing iteration.

[0062] Global attention focusing involves applying a cross-modal multi-head attention mechanism to the resulting mixed feature matrix data to capture global dependencies, thereby obtaining the globally attention-focused mixed feature matrix data. .

[0063] Next, feature recalibration gating is performed to finely adjust the feature dimensions of the attention output. This adjustment is also globally modulated by the indicator signal λ. This is the fourth layer of fusion driven by the indicator signal λ, as detailed below: ; Finally, adaptive weighted aggregation and information fidelity preservation are performed, that is, using the weights generated above. Characteristics after refining A weighted summation is performed. Simultaneously, a feature from the original splicing is introduced. Residual connection: ; in, For dynamic fusion of feature data.

[0064] This residual connection ensures that even during deep signal modulation, the most original and discriminative information of each modality is preserved, effectively preventing information loss and enhancing the stability and expressive power of the fusion result. In summary, Path 1 achieves deep fusion at multiple key levels, including initial weighting, interaction matrix learning, feature mixing intensity, and attention gating, under the coordinated and refined control of a single global indicator signal λ.

[0065] Adjustment and fusion module 203: used to perform feature adjustment and fusion processing on an adaptive adjustment path driven by feature embedding set of at least two modalities to obtain adjustment and fusion feature data; In a specific implementation of this invention, the step of performing feature-modal fusion processing on an adaptive adjustment path driven by input data of feature embedding sets of at least two modalities to obtain regulated fusion feature data includes: concatenating features within the feature embedding sets of at least two modalities into joint features; mapping the joint matrix to a predicted original adjacency matrix through the feedforward network of the adaptive adjustment path; performing weighted fusion processing on the predicted original adjacency matrix and a prior matrix reflecting prior modal relationships to form a weighted fusion feature matrix; performing self-influence enhancement processing on the weighted fusion feature matrix to form an enhanced feature matrix; and performing interactive intensity modulation and feature aggregation processing on the enhanced feature matrix to obtain regulated fusion feature data.

[0066] Furthermore, the step of performing interaction intensity modulation and feature aggregation processing on the enhanced feature matrix to obtain regulated fusion feature data includes: calculating the overall interaction intensity scalar of the enhanced feature matrix to obtain the overall interaction intensity scalar data corresponding to the enhanced feature matrix; using the overall interaction intensity scalar data as the adjustment factor for the Softmax temperature adjustment of the independent weight prediction head; performing weighted summation processing on the joint features based on the adjustment factor, and introducing the residual connection of the joint features during the weighted summation processing to form regulated fusion feature data.

[0067] Specifically, the data-driven adaptive regulation pathway, as a parallel and complementary "micro-scheduler," is fundamentally based on learning a sample-specific modal relationship adjacency matrix directly from the data. It discovers personalized modal cooperation patterns that are not captured by the indicator signal λ; it complements the dynamic fusion path of global signal regulation with "macro calibration" and "micro scheduling".

[0068] First, adjacency matrix prediction and soft prior are fused, concatenating the features embedded in the feature set of at least two modalities into joint features, which are represented as follows: From the joint features In this process, the original adjacency matrix is ​​predicted through a feedforward mapping. To address the potential instability in training or the learning of relationships that violate domain knowledge that may result from a fully data-driven approach, this embodiment proposes a soft prior regularization method:

[0069] in, It is a pre-defined matrix that reflects prior modal relationships (such as the usual correlation between clinical and imaging modalities). This is a learnable mixture coefficient; this approach retains the flexibility of data-driven learning while smoothing and constraining the learning process with prior knowledge, preventing the learning of physically unreasonable strong interaction relationships, and significantly improving the robustness of training.

[0070] Furthermore, a self-influence enhancement mechanism is proposed. In cross-modal interactive learning, the features of some modalities may be over-dominated by other modalities and "annihilated." To prevent this problem and maintain the diversity of modal features, this embodiment proposes a self-influence enhancement mechanism to explicitly enhance the diagonal elements of the adjacency matrix (i.e., the self-contribution of each modality). ; in, These are learnable parameters; this mechanism does not simply increase the self-connect weights, but rather uses learnable non-negative vectors. Adaptive scaling, ensuring that each modality retains at least its own influence commensurate with its importance, is a key technique for preventing model collapse and maintaining information diversity.

[0071] Finally, interaction intensity modulation and feature aggregation are performed on the learned adjacency matrix. It reflects the overall activity of intermodal interactions; its overall interaction strength scalar is calculated. And this scalar is used as the Softmax temperature adjustment factor for an independent weighted prediction head, as follows: ; That is, when the overall intermodal interaction is strong (s is large), the weight distribution... The distribution becomes more "sharp," highlighting the dominant modality; when the interaction is weaker, the distribution becomes more "smooth," reflecting balanced fusion; this design transforms structural information (interaction intensity) into a dynamic adjustment signal for the fusion weights; finally, the adjustment features are obtained by weighted summation of the original modality embeddings and adding residual connections, as follows: ; in, To adjust and fuse feature data.

[0072] Weighted fusion module 204: used to input the dynamic fusion feature data and the adjusted fusion feature data into the adaptive gating integrator for feature weighted fusion processing to obtain weighted fusion feature data.

[0073] In a specific implementation of this invention, the formula for inputting the dynamically fused feature data and the adjusted fused feature data into the adaptive gated integrator for feature weighted fusion processing is as follows: ; in, For gated network functions; For temperature parameters; The gate value of the adaptive gate integrator is defined as follows, and its range is: ; For dynamic fusion of feature data; To adjust and fuse feature data; To set parameters; This is weighted and fused feature data.

[0074] Specifically, adaptive gating integration (the key intelligent decision-making unit) is performed to obtain the outputs of two paths. (Deep interaction, global control) and After (relationship learning and robust adjustment), the final intelligent integration is achieved through a temperature scaling gating unit, which is the "decision brain" of the entire dual-pathway architecture. ; in, For gated network functions; For temperature parameters; The gate value of the adaptive gate integrator is defined as follows, and its range is: ; For dynamic fusion of feature data; To adjust and fuse feature data; To set parameters; This is weighted and fused feature data.

[0075] This design gives the model ultimate flexibility, enabling it to strike an optimal balance between "deep fusion based on feature consistency" and "robust adjustment based on data relationships".

[0076] Design details and principles: Gated networks A fusion algorithm is a small neural network (such as a multilayer perceptron) that receives the concatenation of the outputs of two pathways and learns to determine which pathway is more suitable for the current sample.

[0077] Temperature parameters : is a hyperparameter or learnable scalar greater than 0. Its key role is to adjust the "hard" or "soft" degree of gating decisions. When When the value is large, the input to the sigmoid function is compressed, and the gate value... When the value approaches 0.5, decision-making becomes "softer," encouraging the use of information from both pathways; when When the threshold is small, the decision is "hardened," with the gating value tending to be 0 or 1, explicitly choosing a path. This provides the model with the flexibility to adjust its decision-making style for different tasks or data granularities.

[0078] Adaptive Integration: Final Fusion Representation It is a convex combination of two output paths. (Gate value) Data-driven, it enables sample-level, optimized strategy selection: for samples with clear global signals requiring deep interaction, Approaching 1; for samples with complex modal relationships that require robust adjustment, It approaches 0; in most cases, it lies between the two.

[0079] The core value of this step lies in the fact that it enables the system to move away from relying on a single, fixed fusion strategy and instead dynamically and intelligently balance and integrate the two fusion philosophies of "deep interaction" and "robust adjustment" based on the unique attributes of each input sample. This is the key to achieving high-order adaptive capabilities.

[0080] In this embodiment of the invention, a dual-path architecture of "global signal regulation path" and "data-driven adjustment path" is adopted in parallel to achieve the organic combination of "macro-calibration" and "micro-scheduling", which solves the problem that a single path cannot take into account both global guidance and fine-grained adaptation. Multiple robustness enhancement mechanisms are introduced into the "data-driven adjustment path" to effectively ensure the stability, modal diversity and reliability of the learning process. Furthermore, feature weighted fusion processing is performed by an adaptive gating integrator, which can dynamically adjust the fusion logic for different cases like an expert, and significantly improve the level of intelligence.

[0081] This invention provides a computer-readable storage medium storing a computer program. When executed by a processor, this program implements the multimodal feature fusion method of any of the above embodiments. The computer-readable storage medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, the storage device includes any medium that stores or transmits information in a readable form by a device (e.g., a computer, a mobile phone), and can be a read-only memory, a disk, or an optical disk, etc.

[0082] This invention also provides a computer application running on a computer, which is used to execute the multimodal feature fusion method of any of the above embodiments.

[0083] also, Figure 3 This is a schematic diagram of the structural composition of the electronic device in an embodiment of the present invention.

[0084] This invention also provides an electronic device, such as... Figure 3 As shown. The electronic device includes a processor 302, a memory 303, an input unit 304, and a display unit 305, among other devices. Those skilled in the art will understand that... Figure 3The structural components of the illustrated electronic device do not constitute a limitation on all devices and may include more or fewer components than illustrated, or combine certain components. Memory 303 can be used to store application program 301 and various functional modules. Processor 302 runs application program 301 stored in memory 303, thereby performing various functional applications and data processing of the device. Memory can be internal memory or external memory, or both. Internal memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or random access memory. External memory may include hard disks, floppy disks, ZIP disks, USB flash drives, magnetic tapes, etc. The memory disclosed in this invention includes, but is not limited to, these types of memory. The memory disclosed in this invention is only an example and not a limitation.

[0085] Input unit 304 is used to receive signal input and user-input keywords. Input unit 304 may include a touch panel and other input devices. The touch panel can collect user touch operations on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel) and drive the corresponding connection device according to a pre-set program; other input devices may include, but are not limited to, one or more of physical keyboards, function keys (such as play control buttons, power buttons, etc.), trackballs, mice, joysticks, etc. Display unit 305 can be used to display user-input information or information provided to the user, as well as various menus of the terminal device. Display unit 305 may be in the form of a liquid crystal display, organic light-emitting diode, etc. Processor 302 is the control center of the terminal device, connecting various parts of the entire device through various interfaces and lines, and performing various functions and processing data by running or executing software programs and / or modules stored in memory 303, and calling data stored in memory.

[0086] As one embodiment, the electronic device includes: one or more processors 302, a memory 303, and one or more application programs 301, wherein the one or more application programs 301 are stored in the memory 303 and configured to be executed by the one or more processors 302, and the one or more application programs 301 are configured to execute the multimodal feature fusion method corresponding to any of the embodiments described above.

[0087] In this embodiment of the invention, a dual-path architecture of "global signal regulation path" and "data-driven adjustment path" is adopted in parallel to achieve the organic combination of "macro-calibration" and "micro-scheduling", which solves the problem that a single path cannot take into account both global guidance and fine-grained adaptation. Multiple robustness enhancement mechanisms are introduced into the "data-driven adjustment path" to effectively ensure the stability, modal diversity and reliability of the learning process. Furthermore, feature weighted fusion processing is performed by an adaptive gating integrator, which can dynamically adjust the fusion logic for different cases like an expert, and significantly improve the level of intelligence.

[0088] Furthermore, the above provides a detailed description of a multimodal feature fusion method and related apparatus based on dual-path collaboration and adaptive gating provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multimodal feature fusion method based on dual-pathway collaboration and adaptive gating, characterized in that, The method includes: Obtain feature embedding sets for at least two modalities and an indication signal for global modal quality; The feature embedding set of at least two modalities and the dynamic fusion path of the indicator signal input global signal modulation are subjected to feature dynamic fusion processing to obtain dynamic fusion feature data; The feature embedding set of at least two modalities is used to drive the adaptive adjustment path of the input data for feature adjustment fusion processing to obtain the adjustment fusion feature data; The dynamic fusion feature data and the adjusted fusion feature data are input into the adaptive gating integrator for feature weighted fusion processing to obtain weighted fusion feature data.

2. The multimodal feature fusion method according to claim 1, characterized in that, The dynamic fusion processing of the feature embedding set of at least two modalities and the dynamic fusion path of the indicator signal input global signal modulation to obtain dynamic fused feature data includes: Features from at least two modalities are concatenated into joint features, and the basic attention weights of the dynamic fusion path are globally adjusted using the indication signal to form global adjustment weights. Based on the indication signal and the joint features, the interaction learning matrix in the dynamic fusion path is conditionally processed to form a conditional interaction learning matrix. Within the dynamic fusion pathway, the conditionalized interactive learning matrix is ​​used to perform multi-level feature refinement processing on the joint features, forming refined joint features. The joint features after fine adjustment are weighted and summed based on the global adjustment weights, and the residual connections of the joint features are introduced during the weighted summation process to form dynamic fused feature data.

3. The multimodal feature fusion method according to claim 2, characterized in that, The conditionalization of the interaction learning matrix within the dynamic fusion path based on the indication signal and the joint features to form a conditionalized interaction learning matrix includes: The indication signal is encoded into a conditional vector, and the conditional vector and the joint feature are output together into the meta-network of the dynamic fusion path. The interaction learning matrix is ​​then conditionally processed to form a conditional interaction learning matrix.

4. The multimodal feature fusion method according to claim 2, characterized in that, The process of using a conditionalized interaction learning matrix to perform multi-level feature refinement on the joint features within the dynamic fusion path, forming refined joint features, includes: The joint features are stacked into feature matrix data, and the feature matrix data is subjected to interactive learning processing using the conditionalized interactive learning matrix to form interactive learning feature matrix data. The interactive learning feature matrix data is locally adaptively mixed using a one-dimensional convolution function and the indicator signal to form mixed feature matrix data. The hybrid feature matrix data is subjected to global attention focusing processing based on a cross-modal multi-head attention mechanism to obtain the hybrid feature matrix data after global attention focusing; Based on the global modulation of the indication signal, the hybrid feature matrix data after global attention focusing is subjected to fine-tuning of the feature dimensions to form fine-tuned joint features.

5. The multimodal feature fusion method according to claim 1, characterized in that, The adaptive adjustment path driven by the input data of the feature embedding set of at least two modalities is subjected to feature adjustment fusion processing to obtain adjusted fusion feature data, including: The features of at least two modalities are concatenated into a joint feature set. The joint matrix is ​​mapped to a predicted original adjacency matrix through the feedforward network of the adaptive adjustment path. The predicted original adjacency matrix is ​​then weighted and fused with a prior matrix reflecting prior modal relationships to form a weighted fused feature matrix. The weighted fusion feature matrix is ​​processed by a self-influence enhancement mechanism to form an enhanced feature matrix; The enhanced feature matrix is ​​subjected to interactive intensity modulation and feature aggregation processing to obtain adjusted fused feature data.

6. The multimodal feature fusion method according to claim 5, characterized in that, The step of performing interactive intensity modulation and feature aggregation processing on the enhanced feature matrix to obtain adjusted fused feature data includes: Calculate the overall interaction strength scalar of the enhanced feature matrix to obtain the overall interaction strength scalar data corresponding to the enhanced feature matrix; The overall interaction intensity scalar data is used as the adjustment factor for the Softmax temperature adjustment of the independent weight prediction head; The joint features are weighted and summed based on the adjustment factor, and residual connections of the joint features are introduced during the weighted summation process to form adjusted fusion feature data.

7. The multimodal feature fusion method according to claim 1, characterized in that, The formula for inputting the dynamically fused feature data and the adjusted fused feature data into the adaptive gated integrator for feature weighted fusion processing is as follows: ； in, For gated network functions; For temperature parameters; The gate value of the adaptive gate integrator is defined as follows, and its range is: ; For dynamic fusion of feature data; To adjust and fuse feature data; To set parameters; This is weighted and fused feature data.

8. A multimodal feature fusion device based on dual-pathway collaboration and adaptive gating, characterized in that, The device includes: Acquisition module: used to acquire feature embedding sets of at least two modalities and an indication signal of global modal quality; Dynamic fusion module: used to perform dynamic feature fusion processing on the feature embedding set of at least two modes and the dynamic fusion path of the indicator signal input global signal modulation to obtain dynamic fusion feature data; The adjustment and fusion module is used to perform feature adjustment and fusion processing on an adaptive adjustment path driven by the input data of at least two modalities to obtain adjusted and fused feature data. Weighted fusion module: used to input the dynamic fusion feature data and the adjusted fusion feature data into the adaptive gating integrator for feature weighted fusion processing to obtain weighted fusion feature data.

9. An electronic device comprising a processor and a memory, characterized in that, The processor runs a computer program or code stored in the memory to implement the multimodal feature fusion method as described in any one of claims 1 to 7.

10. A computer-readable storage medium for storing computer programs or code, characterized in that, When the computer program or code is executed by a processor, the multimodal feature fusion method as described in any one of claims 1 to 7 is implemented.

Images