A multi-scene cross-modal data oriented adaptive progressive fusion method and system

Abstract

The application relates to the field of artificial intelligence and multi-modal data fusion technology, and discloses a multi-scene cross-modal data adaptive progressive fusion method and system, which comprises the following steps: acquiring multi-scene cross-modal data, respectively encoding features of each modal data to obtain modal feature representation; extracting internal structure representation of each modal based on the modal feature representation, and constructing a modal relation graph according to the similarity between different modal structure representations; determining a modal grouping result and a progressive fusion sequence according to the modal relation graph; adaptively weighting and fusing the modal feature representation in the same modal group according to the progressive fusion sequence, and recursively obtaining a global fusion representation; and adjusting the influence weight of each modal according to the contribution of the global fusion representation or clustering utility. The application can improve the robustness, discriminability and explainability of multi-scene cross-modal data fusion representation, and is suitable for multi-modal representation learning and unsupervised clustering tasks.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, multimodal representation learning, and data fusion technology, and in particular to an adaptive progressive fusion method and system for multi-scenario cross-modal data. Background Technology

[0002] With the development of multi-source sensors, visual acquisition platforms, internet content platforms, and high-throughput biological data acquisition technologies, practical applications often generate multiple modalities of data simultaneously, including images, text, visual features, scene features, and gene expression features. Different modalities can describe the same object, sample, or data entity from different perspectives, making multimodal representation learning a key technology in unsupervised clustering.

[0003] Existing cross-modal fusion methods typically employ feature concatenation, fixed weighting, attention fusion, gating fusion, contrastive learning, or shared latent space learning to obtain a unified representation. These methods can achieve certain results when modal quality is stable, distribution differences are small, and the complementarity between different modalities is clear.

[0004] However, under various scenarios, the data quality, distribution, structural characteristics, and clustering contributions of different modalities often exhibit significant differences. Some modalities may generate noise due to acquisition equipment, environmental conditions, sensor drift, occlusion, missing data, or changes in data sources; some modalities may be highly redundant with other modalities, providing limited new discriminative information; and some modalities may also contain pseudo-related information that conflicts with the clustering objective. If these modalities are directly involved in a single fusion process, the fused representation is easily interfered with by low-quality information.

[0005] Existing technologies have at least the following shortcomings: First, they lack explicit modeling of the intrinsic structural relationships between modalities, making it difficult to determine which modalities are suitable for priority fusion; second, the fusion order is usually fixed, and modalities with large structural differences may be mixed prematurely; third, in unlabeled clustering scenarios, it is difficult to measure the actual contribution of a single modality to the fused representation; fourth, the suppression of low-contribution modalities, redundant modalities, or harmful modalities largely depends on soft weights, lacking contribution constraints coupled with the training objective; and fifth, the interpretability of the fusion process is insufficient, making it difficult to explain why different modalities are grouped, preferentially fused, or have their weights reduced.

[0006] Therefore, it is necessary to propose a cross-modal data fusion method that can determine the fusion order based on modal structure relationships and dynamically adjust the influence weights based on modal contributions, so as to improve the robustness, discriminativeness, and interpretability of cluster representation under multi-source heterogeneous data conditions. Summary of the Invention

[0007] The purpose of this invention is to propose an adaptive progressive fusion (TPfusion) method and system for multi-scenario cross-modal data, which solves the technical problems of existing cross-modal clustering methods under multi-source heterogeneous data conditions, such as difficulty in modeling modal structure differences, fixed fusion order, difficulty in suppressing low-contribution or harmful modalities, and insufficient robustness of clustering representation.

[0008] Specifically, the present invention provides an adaptive progressive fusion (TPfusion) method for multi-scenario cross-modal data, comprising the following steps: S1. Obtain cross-modal data from at least one application scenario, wherein the cross-modal data includes at least two modal data corresponding to the same sample object; S2. Perform feature encoding on each modal data to obtain the modal feature representation corresponding to each modality; S3. Extract structural representations based on the feature representations of each modality to characterize the intrinsic distribution structure of the modality, and construct a modality relationship graph based on the similarity between the structural representations of different modalities; S4. Determine the modal grouping results and progressive fusion order based on the modal relationship diagram, so that modalities with similar structures are fused first, and modalities with large structural differences are fused later. S5. Perform adaptive weighted fusion according to the progressive fusion order to recursively obtain the global fusion representation; S6. Adjust the influence weights according to the contribution of each modality to the global fusion representation or clustering utility, so as to reduce the impact of low-contribution modes, redundant modes or harmful modes on the fusion results; S7. Output the global fusion representation.

[0009] An adaptive progressive fusion (TPfusion) system for multi-scenario cross-modal data includes a data acquisition module, a modality encoding module, a structural representation module, a relational graph construction module, a fusion order determination module, an adaptive progressive fusion module, a contribution evaluation and weight adjustment module, and an output module.

[0010] The beneficial effects provided by this invention are as follows: It explicitly characterizes the structural similarity between modes through a modal relationship graph based on structural representation, providing an interpretable basis for the fusion order; it prioritizes the fusion of structurally similar modes through a progressive fusion mechanism, reducing information conflicts caused by premature mixing of incompatible modes; it estimates the actual contribution of each mode and suppresses the influence of low-contribution or harmful modes under unlabeled or weakly supervised conditions through contribution evaluation and contribution constraint mechanisms; and it improves the interpretability of the cross-modal fusion process by outputting intermediate information such as modal relationship graphs, modal grouping, fusion weights, and contribution values. Attached Figure Description

[0011] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is the overall framework diagram of the TPfusion invention; Figure 3 This is a system structure block diagram of the present invention; Figure 4 This is a schematic diagram of the modal structure characterization extraction and modal relationship diagram construction process of the present invention; Figure 5 This is a schematic diagram of the adaptive progressive fusion process of the present invention; Figure 6 This is a schematic diagram of the modal contribution assessment and weight adjustment process of the present invention; Figure 7 This is a diagram showing the results of the clustering comparison experiment of this invention; Figure 8 This is a graph showing the experimental results of training stability analysis of this invention; Figure 9 This is a schematic diagram of the hardware device operation according to an embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be understood that the described embodiments are only for explaining the present invention and are not intended to limit the scope of protection of the present invention.

[0013] Please refer to Figures 1 to 6 The present invention provides an adaptive progressive fusion (TPfusion) method and system for multi-scenario cross-modal data, forming a closed-loop processing flow of "structural relationship modeling - progressive fusion - contribution evaluation - weight suppression".

[0014] S1. Obtain cross-modal data from at least one application scenario.

[0015] Let the cross-modal dataset be represented by equation (1):

[0016] In equation (1), N is the number of samples, V is the number of modes, and x_i^(v) represents the data of the i-th sample in the v-th mode. Different modes can be images, text, audio, video, remote sensing images, geological attributes, sensor time series, medical images, omics data, or other data that can be digitally expressed. Multiple scenarios can correspond to different acquisition areas, equipment conditions, environmental conditions, data sources, mode combinations, or noise levels.

[0017] S2. Perform feature encoding on each modal data separately.

[0018] For each modality v, a corresponding modality encoder E_v is set to map the original modality data into a modality feature representation of a unified dimension, as shown in equation (2):

[0019] Here, E_v can be a multilayer perceptron, convolutional neural network, recurrent neural network, Transformer, graph neural network, or other feature extraction models adapted to the modality type. For missing or low-confidence modalities, compensation can be achieved using mask vectors, mean features, neighbor sample estimated features, learnable missing labels, or reconstruction constraints.

[0020] S3. Extract modal structure representations and construct modal relationship diagrams.

[0021] The feature matrix H^(v) of each modality is treated as a feature point cloud or sample relationship graph, and multi-scale structural analysis is performed on it to obtain a structural representation describing the intrinsic structure of the modality. In a preferred embodiment, the multi-scale structural analysis is a persistent cohomology analysis. A Vietoris-Rips filter is constructed for the feature point cloud of each modality, and the birth time and death time of the topological features are recorded as the distance threshold changes to form a persistence graph, expressed as Equation (3):

[0022] In equation (3), b_j represents the birth time of the topological feature, d_j represents the extinction time of the topological feature, and k represents the topological dimension. Persistent cohomology can characterize the overall structure of the modal feature space and is stable to coordinate transformations, slight perturbations, and different coding spaces.

[0023] For any two modes v and u, calculate the structural distance between their structural representations. Preferably, the bottleneck distance between persistence graphs is used, but Wasserstein distance, Euclidean distance, cosine distance, kernel distance, maximum mean difference distance, or a combination thereof can also be used. The structural distance can be expressed as equation (4):

[0024] Subsequently, a kernel function is used to convert the structural distance into intermodal affinity, expressed as equation (5):

[0025] In equation (5), τ is the bandwidth parameter, which can be determined based on the median, mean, quantile, preset value, or learnable parameter of the distance matrix. A weighted undirected modal relationship graph is constructed using modes as nodes and inter-modal affinity as edge weights, as shown below. Figure 4 As shown.

[0026] S4. Determine the modal grouping results and progressive fusion order based on the modal relationship diagram.

[0027] The modality graph is subjected to community detection, graph clustering, or hierarchical clustering to obtain modality grouping results. Modalities with high structural similarity are grouped into the same modality group and fused preferentially, while modalities with large structural differences are fused later, thus forming a hierarchical progressive fusion order from lower-level modalities to higher-level group representations. The community detection or graph clustering can be Louvain community detection, spectral clustering, hierarchical agglomerative clustering, modularity optimization, graph neural network clustering, or threshold-based graph partitioning.

[0028] In one implementation, the modality graph, modality grouping results, and progressive fusion order are periodically updated during training. The update cycle can be every 5, 10, or 20 training rounds, or it can be adaptively triggered according to changes in training loss to adapt to changes in the feature space caused by changes in modality encoder parameters.

[0029] S5, Perform adaptive progressive fusion.

[0030] For each modality group G, a learnable query vector q_G is set, and the fusion weights are calculated based on the feature representations of each modality within the group, as shown in Equation (6):

[0031] The modal feature representations within a group are weighted and aggregated according to the fusion weights to obtain the group-level fusion representation, as shown in Equation (7):

[0032] Each group-level fusion representation serves as input to the next level, continuing the adaptive weighted fusion process until a global fusion representation is obtained. The above attention fusion method can be replaced by gated fusion, multi-head attention fusion, graph attention fusion, soft-gated networks, or combinations thereof. The adaptive progressive fusion process is as follows: Figure 5 As shown.

[0033] S6. Conduct modal contribution assessment and weight adjustment.

[0034] To reduce the impact of low-quality, redundant, or harmful modes in the fusion process, this invention constructs a contribution evaluation model with modes as participants. Any combination S of modes participating in the fusion is considered a mode alliance, and only this combination is used to obtain the fusion representation and calculate the utility value U(S). The utility value can be clustering accuracy, normalized mutual information, purity, inter-cluster separation, intra-cluster compactness, Calinski-Harabasz index, Silhouette index, or other clustering quality indicators.

[0035] Preferably, the contribution of the target mode is estimated using a cooperative game contribution evaluation method based on Shapley values, expressed as equation (8):

[0036] In Equation (8), E_S represents the average of multiple modal combinations involved in the fusion. To reduce computational complexity, Monte Carlo sampling can be used to estimate the contribution, which includes sampling multiple modal combinations involved in the fusion, calculating the difference in utility value before and after adding the target modality, and averaging the difference in utility value.

[0037] The contribution constraint term is constructed based on the deviation between the contribution of each mode and the reference contribution, and is expressed as Equation (9):

[0038] In equation (9), lambda_contrib is the weight of the contribution constraint term, and phi_ref is the reference contribution. phi_ref can be the mean, median, historical moving average, or target contribution value determined based on scene configuration of all modal contributions. The contribution constraint term is fed back to the fusion module, constraining the fusion weights, attention parameters, gating parameters, or fusion control parameters corresponding to low-contribution or harmful modalities, thereby reducing their impact on the global fusion representation. This process is as follows: Figure 6 As shown.

[0039] S7. Train and output the global fusion representation.

[0040] In one embodiment, the training objective of the present invention includes clustering loss, cross-modal alignment loss, reconstruction loss, contribution constraint term, and sparse regularization term, as shown in equation (10):

[0041] Wherein, L_task is the unsupervised clustering loss; L_align is used to ensure consistency between different modal representations of the same sample and the global fusion representation; L_rec is used to constrain the modality encoder to retain the original modality information; L_contrib is used for contribution constraints; and L_sparse is used to enhance the sparsity of the fusion parameters and the modality selection capability. After training, the global fusion representation z is output, which can be used for multimodal representation learning and unsupervised clustering tasks. This embodiment verifies the technical effect with multimodal data clustering experiments.

[0042] Explanation of the diversity of experimental datasets and multi-scenario data To verify the effectiveness of this invention under multi-scenario cross-modal data conditions, clustering comparison experiments were conducted using four datasets with different data sources, modal compositions, and sample structures. Among them, the Wikipedia dataset is a cross-modal dataset of text and images, containing 693 samples, 10 categories, and 2 modalities (128-dimensional image features and 10-dimensional text features), representing the multimodal data scenario where images and text descriptions coexist in Internet content platforms; the MSRCV1 dataset is an object / scene visual dataset, containing 210 samples, 7 categories, and 6 visual feature modalities (such as color histograms, GIST, SIFT, etc.), representing the multimodal data scenario where multiple features describe visual objects or scenes; the GSE dataset is a gene expression microarray dataset, containing 8200 samples, 27 categories, and 2 modalities (gene expression features and sample annotation features), representing the multimodal fusion scenario of high-throughput omics data in bioinformatics; and the Out-Scene dataset is a natural scene dataset, containing 2688 samples, 8 categories, and 4 visual feature modalities, representing the data diversity scenario where multiple visual features coexist in natural scene data. The four datasets mentioned above show significant differences in sample size (from 210 to 8200), number of categories (from 7 to 27), number of modalities (from 2 to 6), and feature dimensions (from 10 to thousands), which can comprehensively reflect the diverse characteristics of multi-scenario cross-modal data.

[0043] Clustering comparison experiment results are as follows Figure 7 As shown. By Figure 7 It can be seen that on four datasets with different data sources, modal compositions, and sample structures—Wikipedia, MSRCV1, GSE, and Out-Scene—TPfusion achieved the best or tied-best results in terms of ACC, NMI, and PUR. Specifically, on the Wikipedia cross-modal image and text dataset, TPfusion achieved 61.33% ACC, 55.13% NMI, and 63.49% PUR; on the MSRCV1 multi-visual feature dataset, TPfusion achieved 98.57% ACC, 96.78% NMI, and 96.68% PUR; on the GSE gene expression microarray dataset, TPfusion achieved 73.67% ACC, 79.86% NMI, and 76.27% PUR; and on the Out-Scene natural scene dataset, TPfusion achieved 77.23% ACC, 62.44% NMI, and 77.23% PUR. The above results demonstrate that the structural relationship modeling, adaptive progressive fusion, and contribution evaluation mechanisms proposed in this invention can stably improve multimodal clustering performance under diverse data conditions, including cross-modal text and image data, object / scene vision, gene expression microarrays, and natural scenes.

[0044] Training stability analysis experimental results are as follows Figure 8 As shown, on the MSRCV1, GSE, and Out-Scene datasets, as the number of training rounds increases, the evaluation metrics such as ACC, NMI, PUR, and ARI generally increase and gradually stabilize, while the total loss curve generally decreases and converges, indicating that the present invention can maintain a stable training process on both small-scale and large-scale datasets. Figure 8 The dashed line in the middle corresponds to the stage update position in the training process. After the update, the evaluation index still maintains a continuous improvement or a stable state, indicating that the modality graph update, adaptive fusion parameter adjustment and contribution feedback mechanism will not destroy the training convergence.

[0045] As one application example, this invention can be used to fuse remote sensing imagery, geological attributes, and sensor monitoring data. Remote sensing imagery modalities can be encoded using convolutional neural networks or visual Transformers to obtain image representations reflecting surface texture, spatial structure, and spectral characteristics; geological attribute modalities can be encoded using multilayer perceptrons to obtain attribute representations reflecting lithology, structure, geochemical indicators, or geophysical indicators; and sensor temporal modalities can be encoded using one-dimensional convolutional networks, recurrent neural networks, or temporal Transformers to obtain temporal representations reflecting dynamic monitoring changes.

[0046] Remote sensing images may be affected by clouds, fog, shadows, or seasonal changes in different regions or under different acquisition conditions. Sensor data may drift or be missing, and geological attribute data may have scale differences. When using the method of this invention, the system first extracts the features of each modality separately, and then constructs a modal relationship diagram through structural representation. If the remote sensing image representation and the geological attribute representation are closer in sample distribution structure, they are preferentially classified into the same modal group; if a certain sensor mode has a large structural difference due to noise, the mode can be fused later or its weight can be reduced in the contribution evaluation stage.

[0047] As another implementation, when noisy modes exist in cross-modal data, the feature point cloud corresponding to these noisy modes typically lacks a stable structure, and its structural representation differs significantly from that of normal modes. Therefore, in the modality graph, the affinity between noisy modes and other modes is low. As training progresses, the contribution evaluation model periodically estimates the contribution of each mode. If adding a noisy mode to multiple mode combinations does not improve the utility value, its marginal contribution is low, and the system accordingly reduces the influence weight of this mode in the fusion process.

[0048] This invention also provides a system structure corresponding to the above method. A data acquisition module is used to acquire cross-modal data; a modality encoding module is used to obtain modal feature representations; a structural characterization module is used to extract modal structural characterizations; a relational graph construction module is used to generate a modal relational graph; a fusion order determination module is used to obtain modal grouping results and progressive fusion order; an adaptive progressive fusion module is used to obtain a global fusion representation; a contribution evaluation and weight adjustment module is used to estimate the contribution of each modality and adjust the fusion weights; and an output module is used to output the global fusion representation or input it into the clustering task module.

[0049] like Figure 9 As shown, in one hardware deployment embodiment, hardware device 401 may include processor 402 and storage medium 403. Storage medium 403 is used to store multimodal data, model parameters, and program instructions for executing the TPfusion method; after processor 402 reads and executes the program instructions, it completes processing such as modality encoding, structural representation extraction, modality relationship graph construction, adaptive progressive fusion, contribution evaluation, and weight adjustment, and outputs a global fusion representation for unsupervised clustering.

[0050] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An adaptive progressive fusion method for multi-scenario cross-modal data, characterized in that: Includes the following steps: S1. Obtain cross-modal data from at least one application scenario, wherein the cross-modal data includes at least two modal data corresponding to the same sample object; S2. Perform feature encoding on each modal data to obtain the modal feature representation corresponding to each modality; S3. Extract structural representations based on the feature representations of each modality to characterize the intrinsic distribution structure of the modality, and construct a modality relationship graph based on the similarity between the structural representations of different modalities; S4. Determine the modal grouping results and progressive fusion order based on the modal relationship diagram, so that modalities with similar structures are fused first, and modalities with large structural differences are fused later. S5. Following the progressive fusion order, adaptive weighted fusion is performed on the modal feature representations within the same modal group to obtain a group-level fusion representation. The group-level fusion representation is then used as the fusion object for the next level of fusion to perform recursive fusion, resulting in a global fusion representation. S6. Adjust the influence weight of each mode in the fusion process according to the contribution of each mode to the global fusion representation or clustering utility, so as to reduce the influence of low-contribution modes, redundant modes or harmful modes on the fusion result; S7. Output the global fusion representation.

2. The adaptive progressive fusion method for multi-scenario cross-modal data as described in claim 1, characterized in that: The cross-modal dataset mentioned in step S1 is represented by equation (1), and the modal feature encoding in step S2 is represented by equation (2): In Equation (1), N represents the number of samples, V represents the number of modes, and x_i^(v) represents the data of the i-th sample in the v-th mode; in Equation (2), E_v represents the encoder corresponding to the v-th mode, and H^(v) represents the feature matrix of the v-th mode; the cross-modal data includes at least two of the following: images, text, audio, video, remote sensing images, geological attributes, sensor time series, medical images, and omics data.

3. The adaptive progressive fusion method for multi-scenario cross-modal data as described in claim 1, characterized in that: The structural characterization described in step S3 is obtained through multi-scale structural analysis, which includes at least one of topological structural analysis, manifold structural analysis, neighborhood graph structural analysis, and distribution statistical analysis.

4. The adaptive progressive fusion method for multi-scenario cross-modal data as described in claim 3, characterized in that: The topology analysis includes persistent homology-based topology analysis, which constructs Vietoris-Rips filtering, Cech filtering, or Alpha complex filtering for each modal feature point cloud, records the birth and death times of connected components, loop structures, or high-dimensional holes, and forms a persistence graph, which is represented by Equation (3): In equation (3), b_j represents the birth time of the topological feature, d_j represents the extinction time of the topological feature, and k represents the topological dimension.

5. The adaptive progressive fusion method for multi-scenario cross-modal data as described in claim 4, characterized in that: The construction of the modal relationship diagram in step S3 includes: calculating the structural distance between the structural representations of any two modes, and generating the intermodal affinity based on the structural distance, wherein the structural distance and affinity are expressed as Equation (4) and Equation (5), respectively: In equation (4), d_B represents the bottleneck distance, and w_0 and w_1 represent the topological dimension weights; in equation (5), τ is the bandwidth parameter; a weighted undirected modal relationship graph is constructed using modalities as nodes and A_vu as edge weights.

6. The adaptive progressive fusion method for multi-scenario cross-modal data as described in claim 1, characterized in that: In step S4, the modality graph is subjected to Louvain community detection, spectral clustering, hierarchical clustering, modularity optimization or graph neural network clustering to obtain at least one modality group, and a hierarchical progressive fusion order is formed according to the modality group.

7. The adaptive progressive fusion method for multi-scenario cross-modal data as described in claim 1, characterized in that: The adaptive weighted fusion in step S5 is achieved through attention mechanisms, gating mechanisms, multi-head attention mechanisms, graph attention mechanisms, or combinations thereof, wherein the fusion weights of intra-group modalities and the group-level fusion representations are expressed as Equations (6) and (7), respectively: In equation (6), q_G represents the learnable query vector corresponding to the modality group G, h_g represents the feature representation of the modality g within the group, and d represents the feature dimension; in equation (7), h_G represents the group-level fusion representation.

8. The adaptive progressive fusion method for multi-scenario cross-modal data as described in claim 1, characterized in that: In step S6, the contribution is obtained through a contribution evaluation model based on cooperative game theory. Each modality is regarded as a participant, and the combination of modalities participating in the fusion is regarded as a modal alliance. The clustering separability or clustering quality represented by the fusion is used as the utility value. The contribution of the target modality is expressed as Equation (8): In equation (8), U represents the utility function and S represents the combination of modes participating in the fusion; the utility value includes at least one of clustering accuracy, normalized mutual information, purity, inter-cluster separation, intra-cluster compactness, Calinski-Harabasz index, and Silhouette index.

9. The adaptive progressive fusion method for multi-scenario cross-modal data as described in claim 8, characterized in that: Contribution constraint terms are constructed based on the deviation between the contribution of each modality and the reference contribution, and these contribution constraint terms are added to the training objective. The contribution constraint terms and the training objective are expressed as Equation (9) and Equation (10), respectively: In Equation (9), lambda_contrib represents the weight of the contribution constraint term, and phi_ref represents the reference contribution degree; in Equation (10), L_task represents the clustering loss, L_align represents the cross-modal alignment loss, L_rec represents the reconstruction loss, and L_sparse represents the sparse regularization term.

10. An adaptive progressive fusion system for multi-scenario cross-modal data, characterized in that: It includes a data acquisition module, a modality coding module, a structural representation module, a relationship graph construction module, a fusion order determination module, an adaptive progressive fusion module, a contribution evaluation and weight adjustment module, and an output module; Each module is used to execute the adaptive progressive fusion method for multi-scenario cross-modal data as described in any one of claims 1 to 9.

Images