A multimodal clustering method based on dynamic prototype adaptive fusion and cross-modal consistency constraints

Abstract

This invention discloses a multimodal clustering method based on dynamic prototype adaptive fusion and cross-modal consistency constraints, belonging to the field of multimodal data mining and unsupervised learning technology. This method constructs feature representations for each modality of heterogeneous data from multiple sources, such as images, text, audio, video, and sensor signals, and adaptively updates the cluster centers during training through a dynamic prototype mechanism. Furthermore, based on the clustering confidence, modal reliability, and prototype distance of samples in different modalities, it adaptively calculates the modality fusion weights to obtain sample-level clustering assignment results. Simultaneously, it introduces cross-modal consistency constraints to ensure that the cluster distribution of different modalities remains consistent within a shared semantic space. This invention can reduce the impact of noisy and missing modalities on clustering results, improving the robustness, accuracy, and interpretability of multimodal clustering.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence information processing technology, specifically involving a multimodal clustering method based on dynamic prototype adaptive fusion and cross-modal consistency constraints. Background Technology

[0002] With the rapid development of artificial intelligence, the Internet of Things, internet platforms, and smart terminal devices, data generated in real-world applications is no longer limited to a single format but simultaneously contains multiple modalities of information, including images, text, voice, video, and time-series signals. For example, in e-commerce platforms, a product typically includes product images, title text, and user reviews; in medical diagnosis, a case may simultaneously include medical images, test results, and medical record text; and in industrial monitoring, the status of equipment may be described by vibration signals, temperature signals, sound signals, and operational logs. How to effectively utilize this multi-source heterogeneous information for unsupervised clustering has become an important research problem in the fields of data mining and pattern recognition.

[0003] Clustering is a typical unsupervised learning task that aims to classify data into several categories based on the similarity between samples without human annotation. Traditional clustering methods typically calculate similarity based on single-modal features, such as K-means, spectral clustering, and hierarchical clustering. These methods are simple in structure and easy to implement, but they have significant limitations when dealing with complex multimodal data. Because different modalities have different data structures, statistical distributions, and semantic representations, directly clustering a single modality often fails to fully utilize the complementary information between multimodal data.

[0004] To improve clustering performance, several multimodal clustering methods have emerged in existing technologies. These methods typically extract features from different modalities separately, then fuse them through concatenation, weighted summation, or shared representation learning, and finally perform clustering on the fused features. However, most existing multimodal fusion methods employ fixed fusion strategies, i.e., setting fixed weights for different modalities, or simply concatenating multimodal features and processing them uniformly. This approach ignores the differences in data quality between different samples under different modalities. For example, for some samples, the image modality may be clear and highly discriminative, while the text modality may be short or contain noise; for other samples, the image modality may be blurry or severely occluded, while the text modality may provide more accurate semantic information. Fixed fusion strategies struggle to dynamically adjust modal contributions based on the individual sample characteristics, easily leading to low-quality modalities interfering with the final clustering results.

[0005] Existing multimodal clustering methods also have shortcomings in modeling cluster centers or category prototypes. Some methods use static cluster centers, which, once determined in the early stages of training, change little and are difficult to adapt to the constantly changing deep feature space during training. Especially in deep multimodal clustering scenarios, encoder parameters are continuously updated during training, and the distribution of sample features also changes accordingly. If the cluster prototype cannot reflect the true distribution of the current feature space in a timely manner, it may lead to inaccurate sample allocation, unstable training process, and clustering results converging to local suboptimal solutions.

[0006] Semantic inconsistencies also exist between different modalities. Because data from different modalities are represented differently, the same sample may be mapped to different clustering regions in different modalities. For example, in image-text data, the image modality may focus more on visual appearance similarity, while the text modality may focus more on semantic descriptive similarity; in video data, the visual modality, audio modality, and text title modality may also reflect different levels of information. Without effective cross-modal consistency constraints, clustering results from different modalities may conflict, thus reducing the accuracy and stability of the final fused clustering result.

[0007] Therefore, a novel multimodal clustering method is urgently needed that can fully utilize complementary information between different modalities under unsupervised conditions and dynamically adjust fusion weights based on sample-level modality reliability. Simultaneously, it should be able to adapt to changes in feature distribution through a dynamic prototyping mechanism and alleviate semantic inconsistencies across different modalities through cross-modal consistency constraints, thereby improving the accuracy, robustness, and stability of multimodal clustering. This invention proposes a multimodal clustering method based on dynamic prototyping adaptive fusion and cross-modal consistency constraints to address the aforementioned problems. Summary of the Invention

[0008] Purpose of the invention: To address the problems existing in the prior art, this invention provides a multimodal clustering method based on dynamic prototype adaptive fusion and cross-modal consistency constraints.

[0009] Technical Solution: To achieve the above objectives, the present invention adopts the following technical solution: a multimodal clustering method based on dynamic prototype adaptive fusion and cross-modal consistency constraints, comprising the following steps:

[0010] S1, obtain and construct containing One sample Multimodal dataset with multiple modalities , It is the first The sample at the th Data under each modality, Represents the total number of samples. Indicates the total number of modes;

[0011] S2, for each modal data Feature encoding is performed to obtain the modal features shown in the following formula.

[0012]

[0013] In the formula, For the first The sample at the th Modal representation under each mode For the first A modal feature encoder, Its parameters;

[0014] S3, using the following formula, constructs a dynamic clustering prototype set for each modality.

[0015]

[0016] In the formula, For the first A set of modal dynamic clustering prototypes. The number of cluster categories. For the first In the mode, the th A dynamic prototype of a cluster;

[0017] S4 calculates the modal-level clustering distribution based on the distance between sample features and dynamic prototypes, obtaining the sample clustering membership probability.

[0018]

[0019] in, For the sample In modality The following belongs to the The probability of each cluster, It is the first In the mode, the th A dynamic prototype of a cluster, For temperature coefficient, This represents the number of cluster categories;

[0020] S5, Generate sample-level adaptive fusion weights based on the clustering confidence and modality reliability of each modality. And calculate the fusion cluster distribution:

[0021]

[0022] in, For fused samples Belongs to the The probability of each cluster, For the sample The Each modality fusion weight, Indicates the total number of modes;

[0023] S6 utilizes a joint optimization model based on cross-modal consistency loss, prototype clustering loss, and fusion clustering loss to output the final cluster labels:

[0024]

[0025] in, For the first The final cluster label for each sample.

[0026] Furthermore, the dynamic prototype is updated using a weighted average method:

[0027]

[0028] in, For the first In the mode, the th A dynamic prototype of a cluster, express One sample, To prevent stable terms with a denominator of zero.

[0029] Furthermore, the dynamic prototype also includes a momentum update method:

[0030]

[0031] in, For momentum update After the first round In the mode, the th A dynamic prototype of a cluster, The momentum coefficient, To prevent stable terms with a denominator of zero; For the first In the mode, the th A dynamic prototype of a cluster.

[0032] Furthermore, the modal reliability is determined by the modal-level clustering entropy:

[0033]

[0034] in, For the first One sample in Clustering entropy under each modality This represents the number of clusters.

[0035] The adaptive fusion weights are calculated according to the following formula:

[0036]

[0037] in, For the first The sample corresponds to the first The fusion weights of each modality For the first One sample in Clustering entropy under each modality To achieve the temperature coefficient of fusion, Indicates the number of modes.

[0038] Furthermore, the cross-modal consistency constraint employs the divergence loss between the modal distribution and the fused distribution:

[0039]

[0040] in, To represent cross-modal consistency loss, To integrate clustered distributions, For the first Clustering distribution of each modality Let KL divergence be a metric. express One sample, express One modality.

[0041] Furthermore, the prototype clustering loss is:

[0042]

[0043] in, The prototype clustering loss is used. For the momentum update of the first In the mode, the th A dynamic prototype of a cluster, For the first The sample at the th Modal representation under each mode For the sample In modality The following belongs to the The probability of each cluster, This represents the number of cluster categories.

[0044] Furthermore, the overall optimization objective of the model is:

[0045]

[0046] in, For cluster balance constraints, For feature reconstruction constraints, These are the weighting coefficients.

[0047] Furthermore, the cluster balance constraint is as follows:

[0048]

[0049] This is used to avoid all samples being assigned to at least a few clusters; where, The number of clusters. Represents the total number of samples. For fused samples Belongs to the The probability of each cluster.

[0050] The present invention also provides a multimodal clustering system, comprising: a data input module, a modality encoding module, a dynamic prototype update module, an adaptive fusion module, a cross-modal consistency constraint module, and a clustering output module; the system is used to execute the above-described method.

[0051] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0052] (1) This invention introduces a dynamic prototype update mechanism, enabling the clustering prototype to adaptively adjust as encoder parameters and sample feature distribution change. Compared to existing methods that use fixed cluster centers or static category prototypes, this invention can more accurately characterize the constantly changing feature space structure during training, reduce erroneous clustering assignments caused by prototype lag, and improve model convergence stability. This invention utilizes view reliability weights to fuse multi-view representations and multi-view clustering distributions, enabling different samples to select more reliable information sources based on their own view quality, thereby improving the adaptability of multi-view clustering in complex data scenarios.

[0053] (2) This invention adopts a sample-level modality adaptive fusion strategy, which dynamically determines modality weights based on the clustering confidence or uncertainty of different samples under different modalities. Compared with simple feature splicing, average fusion or fixed weight fusion methods, this invention can automatically enhance the contribution of high-quality modalities for each sample and suppress the interference of noisy modalities, missing modalities or low-reliability modalities, thereby improving the clustering robustness in complex scenarios.

[0054] (3) This invention introduces cross-modal consistency constraints to ensure that the distribution of each single-modal cluster is consistent with the fused cluster distribution. Compared with existing methods that only perform fusion at the feature level, this invention not only focuses on multimodal feature representation, but also performs semantic alignment of different modalities at the clustering decision level, which can reduce the problem of the same sample being assigned to different categories in different modalities and improve the consistency of cross-modal clustering results.

[0055] (4) Through the combined effects of dynamic prototyping, modal reliability assessment, and consistency constraints, this invention enables the model to better handle common problems in multimodal data, such as modal heterogeneity, modal noise, modal quality imbalance, and semantic differences between modalities. Therefore, this invention is applicable to various multimodal clustering scenarios, such as image-text, video-audio-text, medical image-medical record text, and industrial multi-sensor signals, and has strong versatility and scalability. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart illustrating the multimodal clustering method described in this invention.

[0058] Figure 2 This is a schematic diagram of the dynamic prototype update process provided by the present invention.

[0059] Figure 3 This is a schematic diagram of the dynamic prototype adaptive fusion and cross-modal consistency constraint of the present invention. Detailed Implementation

[0060] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0061] This embodiment provides a multimodal clustering method based on dynamic prototype adaptive fusion and cross-modal consistency constraints. This method can be applied to scenarios such as image-text clustering, video clustering, speech-text clustering, medical image and clinical text clustering, and industrial multi-sensor data clustering. Let the input dataset be: ,in, Indicates the number of modes.

[0062] A multimodal clustering method based on dynamic prototype adaptive fusion and cross-modal consistency constraints includes the following steps: Indicates the first Modal There are 10 samples. Each sample is described by multiple modalities: ,in, When the data consists of both image and text modalities, ,in, For image data, For text data; when the data is in three modalities: video, audio, and text, .

[0063] Step 1): Construct feature encoders for different modalities. For the first... Each mode employs a coding network. Extracting low-dimensional semantic features:

[0064]

[0065] in, Indicates the first The sample at the th Potential representations under each modality For feature dimensions.

[0066] In practical implementations, image modalities can use convolutional neural networks, Vision Transformer, or ResNet as encoders; text modalities can use BERT, Transformer, or word vector encoders; and audio modalities can use CNNs, LSTMs, or spectral feature encoding networks. Features encoded from different modalities can be unified to the same dimension through linear mapping.

[0067]

[0068] For, the first The sample at the th The original features extracted by the encoder in each modality For the first The linear transformation matrix corresponding to each mode For the first The bias vectors corresponding to each mode are, for ease of description, referred to below as... This represents the modal features after unifying the dimensions.

[0069] Step 2): Let the preset number of clusters be... Initialize for each mode Cluster prototypes:

[0070]

[0071] in, Indicates the first In the mode, the th The prototype vectors of each cluster. We can first analyze the features of each modality. Perform K-means clustering and use the resulting cluster centers as the initial prototype:

[0072]

[0073] in, Indicates the first In the initial clustering results under the modality, belonging to the _th ... A sample set of classes.

[0074] Step 3): For the sample Calculate its first The modal feature and the first Distance between dynamic prototypes:

[0075]

[0076] Then, the sample at the Softmax function is used to obtain the value at the 1st digit. In the modality belonging to the th , The probability of each cluster:

[0077]

[0078] in, This is a temperature parameter. When... When the size is small, the cluster distribution is sharper; when When the size is large, the cluster distribution is smoother.

[0079] Step 4): To enable the clustering prototype to dynamically adjust as the feature distribution changes, the prototype is updated based on the modal-level clustering distribution. For the first... The first mode A prototype, with the update objective being to minimize the weighted intra-class distance:

[0080]

[0081] right Find the partial derivative:

[0082]

[0083] Setting the partial derivative to zero, we obtain the optimal prototype:

[0084]

[0085] in, To prevent stable terms with a denominator of zero, and to further improve training stability, a momentum update method is employed:

[0086]

[0087] in, This is the momentum coefficient.

[0088] Step 5): Different samples may have different qualities under different modes. Calculate the modal reliability based on the uncertainty of the modal clustering distribution. For samples... The Given _ modalities, the clustering entropy is defined as:

[0089]

[0090] like A smaller value indicates a more concentrated cluster distribution under this mode, making the mode identification more reliable; if A larger value indicates a more dispersed cluster distribution and higher modal uncertainty under this modality. The sample-level modality fusion weights are defined as follows:

[0091]

[0092] in, This is the fusion temperature coefficient. From the above formula, we can see that:

[0093]

[0094] Furthermore, the smaller the clustering entropy, the more likely the mode corresponds to The larger the value, the higher its weight in the final fusion clustering result.

[0095] Step 6): Based on the sample-level modality weights, perform weighted fusion of the cluster distributions of each modality to obtain the final fused cluster distribution:

[0096]

[0097] in, Indicates sample Ultimately belongs to the first The probability of each cluster. The final cluster label is:

[0098]

[0099] Step 7): To ensure consistency in clustering semantics across modalities, this embodiment introduces cross-modal consistency constraints to ensure that the clustering distribution of each modality remains consistent. Towards fusion clustering distribution Alignment. Cross-modal consistency loss is defined as:

[0100]

[0101] in:

[0102]

[0103] This constraint can reduce clustering conflicts between different modalities, enabling different modalities such as images, text, and audio to learn a consistent cluster structure.

[0104] Step 8): The total loss function includes prototype clustering loss, cross-modal consistency loss, cluster balancing loss, and reconstruction loss.

[0105]

[0106] The prototype clustering loss is:

[0107]

[0108] The cluster equilibrium loss is:

[0109]

[0110] The reconstruction loss is:

[0111]

[0112] in, ; For the first Decoder corresponding to each mode Its parameters. The values ​​can be set based on the validation set or experience, for example, 1.0, 0.1, and 0.5 respectively.

[0113] Step 9): During training, repeat the following steps:

[0114] 1. Input multimodal samples;

[0115] 2. Extract features for each modality;

[0116] 3. Calculate the distance from the sample to each dynamic prototype;

[0117] 4. Obtain the modal-level clustering distribution;

[0118] 5. Update the dynamic prototype;

[0119] 6. Calculate the sample-level modal fusion weights;

[0120] 7. Obtain the fused cluster distribution;

[0121] 8. Update the encoder parameters using backpropagation based on the joint loss function;

[0122] 9. Output cluster labels after training is complete.

[0123] Training stops when the preset number of training epochs is reached, or when the change in the loss function is less than a threshold. Final output:

[0124] ,in, For the first Cluster categories of each sample.

[0125] Example

[0126] Taking image-text bimodal data clustering as an example, the input data includes product images and product description text, and the goal is to classify products into categories without manual labeling. There are several categories. The image modal encoder uses a ResNet network, and the text modal encoder uses a Transformer network. Let the image features be:

[0127]

[0128] The text features are:

[0129]

[0130] Calculate the cluster distributions for the image modality and the text modality separately:

[0131]

[0132]

[0133] Next, calculate the uncertainties for the image modality and the text modality separately:

[0134]

[0135]

[0136] Obtain the corresponding fusion weights:

[0137]

[0138]

[0139] The final fused cluster distribution is as follows:

[0140]

[0141] The final product category is:

[0142]

[0143] When product images are clear but text descriptions are short or contain noise, the clustering entropy of the image modality is low. Increase; when the image is blurry but the text description is complete, the clustering entropy of the text modality is low. This increases the accuracy and robustness of multimodal clustering. Therefore, this invention can dynamically select more reliable modal information for different samples, thereby improving the accuracy and robustness of multimodal clustering.

Claims

1. A multimodal clustering method based on dynamic prototype adaptive fusion and cross-modal consistency constraints, characterized in that, Includes the following steps: S1, obtain and construct containing One sample Multimodal dataset with multiple modalities , It is the first The sample at the th Data under each modality, Represents the total number of samples. Indicates the total number of modes; S2, for each modal data Feature encoding is performed to obtain the modal features shown in the following formula. ， In the formula, For the first The sample at the th Modal representation under each mode For the first A modal feature encoder, Its parameters; S3, using the following formula, constructs a dynamic clustering prototype set for each modality. ， In the formula, For the first A set of modal dynamic clustering prototypes. The number of cluster categories. For the first In the mode, the th A dynamic prototype of a cluster; S4 calculates the modal-level clustering distribution based on the distance between sample features and dynamic prototypes, thus obtaining the sample clustering membership probability. ， in, For the sample In modality The following belongs to the The probability of each cluster, It is the first In the mode, the th A dynamic prototype of a cluster, For temperature coefficient, This represents the number of cluster categories; S5, Generate sample-level adaptive fusion weights based on the clustering confidence and modality reliability of each modality. And calculate the fusion cluster distribution: ， in, For fused samples Belongs to the The probability of each cluster, For the sample The Each modality fusion weight, Indicates the total number of modes; S6 utilizes a joint optimization model based on cross-modal consistency loss, prototype clustering loss, and fusion clustering loss to output the final cluster labels: ， in, For the first The final cluster label for each sample.

2. The multimodal clustering method based on dynamic prototype adaptive fusion and cross-modal consistency constraints according to claim 1, characterized in that, The dynamic prototype is updated using a weighted average method: ， in, For the first In the mode, the th A dynamic prototype of a cluster, express One sample, To prevent stable terms with a denominator of zero.

3. The multimodal clustering method based on dynamic prototype adaptive fusion and cross-modal consistency constraints according to claim 1, characterized in that, The dynamic prototype also includes a momentum update method: ， in, For momentum update After the first round In the mode, the th A dynamic prototype of a cluster, The momentum coefficient, To prevent stable terms with a denominator of zero; For the first In the mode, the th A dynamic prototype of a cluster.

4. The multimodal clustering method based on dynamic prototype adaptive fusion and cross-modal consistency constraints according to claim 1, characterized in that, The modal reliability is determined by the modal-level clustering entropy: ， in, For the first One sample in Clustering entropy under each modality The number of clusters; The adaptive fusion weights are calculated according to the following formula: ， in, For the first The sample corresponds to the first The fusion weights of each modality For the first One sample in Clustering entropy under each modality To achieve the temperature coefficient of fusion, Indicates the number of modes.

5. The multimodal clustering method based on dynamic prototype adaptive fusion and cross-modal consistency constraints according to claim 1, characterized in that, Cross-modal consistency constraints employ the divergence loss between the modal distribution and the fused distribution: ， in, To represent cross-modal consistency loss, To integrate cluster distributions, For the first Clustering distribution of each modality Let KL divergence be a metric. express One sample, express One modality.

6. The multimodal clustering method based on dynamic prototype adaptive fusion and cross-modal consistency constraints according to claim 1, characterized in that, The prototype clustering loss is: ， in, The prototype clustering loss is used. For the momentum update of the first In the mode, the th A dynamic prototype of a cluster, For the first The sample at the th Modal representation under each mode For the sample In modality The following belongs to the The probability of each cluster, This represents the number of cluster categories.

7. The multimodal clustering method based on dynamic prototype adaptive fusion and cross-modal consistency constraints according to claim 1, characterized in that, The overall optimization objective of the model is: ， in, For cluster balance constraints, For feature reconstruction constraints, These are the weighting coefficients.

8. The multimodal clustering method based on dynamic prototype adaptive fusion and cross-modal consistency constraints according to claim 7, characterized in that, The cluster balance constraint is: ， This is used to avoid all samples being assigned to at least a few clusters; where, The number of clusters. Represents the total number of samples. For fused samples Belongs to the The probability of each cluster.

9. A multimodal clustering system, characterized in that, include: The module includes a data input module, a modal coding module, a dynamic prototype update module, an adaptive fusion module, a cross-modal consistency constraint module, and a clustering output module. The system is used to perform the method according to any one of claims 1 to 8.

Images