Structural sensing pellet information bottleneck method for multi-modal data clustering
By using a structure-aware particle-sphere information bottleneck method, the particle-sphere partitioning and attribution relationships are dynamically adjusted, solving the problems of noise interference and high computational cost in multimodal clustering, and achieving efficient and robust multimodal clustering.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-03
AI Technical Summary
Existing multimodal clustering methods are susceptible to modal noise interference, have insufficient clustering robustness, and require large amounts of computation, resulting in efficiency bottlenecks and an inability to accurately segment user groups with similar interests.
The structure-aware particle-sphere information bottleneck method is adopted. By replacing sample points with particles, combining structure-aware weights and information bottleneck compression, the particle-sphere partitioning is dynamically adjusted to generate a particle-sphere set. The set of particles is then iteratively optimized to optimize their belonging relationship. An information bottleneck objective function is constructed to maximize the weighted objective function.
Significantly improves clustering performance on multimodal datasets, achieving highly robust and efficient clustering results, reducing the impact of modal noise, and lowering computational complexity.
Smart Images
Figure CN121786527A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal data processing and cluster analysis technology, and in particular to a structure-aware particle-sphere information bottleneck method for multimodal data clustering. Background Technology
[0002] In scenarios involving multimodal data processing, such as data mining and artificial intelligence, multimodal clustering requires the integration of heterogeneous modal data from different sources to achieve accurate clustering of complex objects. Taking a news recommendation platform as an example, it needs to cluster content including news headlines, images, and audio broadcasts to accurately push similar news to users. However, existing methods generally rely on fine-grained distance calculations between samples, which are susceptible to modal noise interference, such as typos in news text, blurry or distorted images, and environmental noise in audio broadcasts, leading to cluster boundary drift and insufficient cluster robustness. Furthermore, they violate the "global priority" cognitive mechanism, with computational load increasing dramatically with sample size, creating an efficiency bottleneck and failing to capture the characteristics from "coarse-grained consensus to fine-grained differences," further limiting clustering effectiveness. For example, in social network scenarios, when clustering multimodal data such as dynamic videos, avatars, and text descriptions generated by millions of users, existing methods may be computationally intensive and time-consuming, and are affected by noise such as blurred images and non-standard text descriptions, failing to accurately segment user groups with similar interests.
[0003] Therefore, to improve the efficiency and robustness of multimodal clustering, it is necessary to construct a clustering scheme that balances global noise suppression and local structure preservation, and to adapt the characteristics of multimodal data by adjusting the particle spheres. Currently, for multimodal data clustering tasks, there is still a lack of technical solutions that use particle spheres as the core and integrate structure-aware mechanisms and information bottleneck theories, making it difficult to achieve efficient clustering and stable noise resistance while reducing computational overhead. Summary of the Invention
[0004] To address the technical problems existing in the prior art, this invention proposes a structure-aware particle-sphere information bottleneck method for multimodal data clustering, achieving a highly robust and efficient multimodal clustering objective.
[0005] To achieve the above objectives, this invention provides a structure-aware particle-sphere information bottleneck method for multimodal data clustering, comprising:
[0006] A multimodal dataset containing text, images, and audio segments is obtained. Features are extracted from each modality sample in the multimodal dataset. The multimodal dataset is then divided into spheres to generate a sphere set.
[0007] For each particle, calculate the structural similarity ratio between the corresponding particle and the current cluster to which it belongs, and generate a structure-aware weight based on the structural similarity ratio;
[0008] Using the particles in the set of particles as processing units, an information bottleneck objective function is constructed, wherein the information bottleneck objective function includes an information retention term and an information compression term;
[0009] The information retention items are dynamically weighted using the structure-aware weights, and the affiliation relationship between spheres and clusters is iteratively optimized to maximize the weighted information bottleneck objective function until convergence, and the final clustering result is output.
[0010] Preferably, the multimodal dataset is divided into granular and sphere parts, including:
[0011] S1.1: Initialize the multimodal dataset into an initial sphere;
[0012] S1.2: For the initial spheres, perform a clustering algorithm for each mode to generate pseudo-labels for the corresponding modes;
[0013] S1.3: For the current particle, calculate the multimodal purity of the current particle based on the mutual information between the features extracted from each modal sample and the pseudo-label of the corresponding modality;
[0014] S1.4: If the multimodal purity is higher than the preset purity threshold, or the number of samples in the current sphere is lower than the preset sample number threshold, then the current sphere is retained; otherwise, the current sphere is split in two.
[0015] S1.5: Recursively execute S1.3 to S1.4 on the daughter particles produced after splitting until all particles meet the conditions.
[0016] Preferably, the multimodal purity of the current sphere is calculated as follows:
[0017] ;
[0018] In the formula, For the x-th sample in the i-th modality, the mutual information between features and pseudo-labels is... m is the number of samples within the sphere, and m is the number of modes. Let g be the multimodal purity of the sphere.
[0019] Preferably, calculating the structural similarity ratio between the corresponding particle and its current cluster includes:
[0020] Calculate the average first similarity between the multimodal centers of the target particle and the multimodal centers of other particles within the current cluster of the target particle;
[0021] Calculate the average second similarity between the multimodal centers of the target sphere and the multimodal centers of spheres in all other clusters;
[0022] The ratio of the average first similarity to the average second similarity is determined as the structural similarity ratio.
[0023] Preferably, the average first similarity and the average second similarity are calculated using cosine similarity.
[0024] Preferably, the structure-aware weights generated based on the structural similarity ratio are as follows:
[0025] ;
[0026] In the formula, For structural constraint hyperparameters, For structure-aware weights, The structural similarity ratio, denoted as the total number of particles, and g represents the number of particles currently involved in the calculation.
[0027] Preferably, the objective function of the information bottleneck is:
[0028] ;
[0029] In the formula, For information reservation, For information compression items, The objective function is the information bottleneck. This is a hyperparameter used to balance the relative importance of information retention and information compression terms.
[0030] Preferably, iteratively optimizing the affiliation relationship from spheres to clusters includes:
[0031] For each sphere, extract it from its current cluster to form a new independent cluster;
[0032] Calculate the merging cost when merging this sphere into each candidate cluster;
[0033] Select the candidate clusters that minimize the merging cost for merging;
[0034] Repeat the above process until the cluster affiliation of all particles no longer changes, reaching a convergent state.
[0035] Preferably, the merger cost is:
[0036] ;
[0037] In the formula, Let g be the probability weight after merging the particle g with the target cluster t, and JS be the Jensen-Shannon divergence. For merger costs, Let y be the conditional probability of feature y appearing in sphere g. For structure-aware weights, Let y be the conditional probability of feature y appearing in cluster t. For hyperparameters, Let g be the marginal probability distribution of the particle. Given a cluster t, the conditional probability that a particle g belongs to the corresponding cluster.
[0038] Compared with the prior art, the present invention has the following advantages and technical effects:
[0039] This invention, based on replacing sample points with spheres, combines structure-aware weights and information bottleneck compression at the sphere level, reducing computational complexity from the traditional sample level to the sphere level. It quantifies sphere purity through mutual information calculation using pseudo-labels and features, and recursively performs binary splitting, dynamically adjusting sphere partitioning under sample constraints. High-purity spheres accurately capture cross-modal consensus. Dynamic weights are generated based on the structural similarity ratio between spheres and spheres within the target cluster and other clusters, assigning high information retention weights to high-quality spheres and suppressing low-quality spheres, significantly reducing the impact of modal noise on cluster boundaries. Within a unified objective function, it simultaneously considers information sharing and local structural differences, achieving a balance between global consistency and modality specificity. Through the aforementioned sphere compression, structure-aware weights, and dynamic information bottleneck balancing, this invention significantly improves clustering performance on multimodal datasets compared to existing methods, achieving highly robust and efficient clustering results on multimodal heterogeneous data. Attached Figure Description
[0040] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0041] Figure 1 This is a flowchart of a structure-aware particle information bottleneck method for multimodal data clustering according to an embodiment of the present invention. Detailed Implementation
[0042] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0043] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0044] This embodiment proposes a structure-aware particle-sphere information bottleneck method for multimodal data clustering, such as... Figure 1 ,include:
[0045] Obtain a multimodal dataset containing text, images, and speech segments. Extract features from each modality sample (such as statistical feature vectors for text, visual features for images, and acoustic features for speech). Then, divide the dataset into spheres to generate a sphere set.
[0046] For each particle, calculate the structural similarity ratio between the corresponding particle and the current cluster to which it belongs, and generate a structure-aware weight based on the structural similarity ratio;
[0047] Using the particles in the set of particles as processing units, an information bottleneck objective function is constructed, wherein the information bottleneck objective function includes an information retention term and an information compression term;
[0048] The information retention items are dynamically weighted using the structure-aware weights, and the affiliation relationship between spheres and clusters is iteratively optimized to maximize the weighted information bottleneck objective function until convergence, and the final clustering result is output.
[0049] Specifically, this embodiment first recursively divides the multimodal dataset into spheres and constructs processing units that adapt to the features to reduce computational complexity. Then, using the spheres as carriers, it balances modal shared information and redundant noise based on the information bottleneck framework to improve anti-interference capability. Next, it dynamically adjusts the weights to ensure global clustering consistency and local structural integrity. Finally, it outputs the clustering results of multimodal heterogeneous data to achieve the goal of highly robust and efficient multimodal clustering.
[0050] Further, the multimodal dataset is divided into granular and sphere parts, including:
[0051] S1.1: Initialize the multimodal dataset into an initial sphere;
[0052] S1.2: For the initial spheres, perform a clustering algorithm for each mode to generate pseudo-labels for the corresponding modes;
[0053] S1.3: For the current particle, calculate the multimodal purity of the current particle based on the mutual information between the features extracted from each modal sample (modal feature vectors of each sample in multimodal data, such as text statistical features, image visual features, etc.) and the pseudo-labels of the corresponding modal.
[0054] S1.4: If the multimodal purity is higher than the preset purity threshold, or the number of samples in the current sphere is lower than the preset sample number threshold, then the current sphere is retained; otherwise, the current sphere is split in two.
[0055] S1.5: Recursively execute S1.3 to S1.4 on the daughter particles produced after splitting until all particles meet the conditions.
[0056] Specifically, the multimodal purity of the current sphere is calculated as follows:
[0057] ;
[0058] In the formula, For the x-th sample in the i-th modality, the mutual information between features and pseudo-labels is... m is the number of samples within the sphere, and m is the number of modes. Let g be the multimodal purity of the sphere.
[0059] Further, the structural similarity ratio between the corresponding particle and its current cluster is calculated, including:
[0060] Calculate the average first similarity between the multimodal centers of the target particle and the multimodal centers of other particles within the current cluster of the target particle;
[0061] Calculate the average second similarity between the multimodal centers of the target sphere and the multimodal centers of spheres in all other clusters;
[0062] The ratio of the average first similarity to the average second similarity is determined as the structural similarity ratio.
[0063] The average first similarity and the average second similarity are calculated using cosine similarity.
[0064] Furthermore, the structure-aware weights generated based on the structural similarity ratio are as follows:
[0065] ;
[0066] In the formula, For structural constraint hyperparameters, For structure-aware weights, The structural similarity ratio, denoted as the total number of particles, and g represents the number of particles currently involved in the calculation.
[0067] Specifically, the average cosine similarity between the multimodal center of a particle and the multimodal centers of other particles within its cluster is calculated; the average cosine similarity between the particle and the centers of all other clusters is also calculated; the two similarities are compared to obtain the structural similarity ratio; and the structural similarity ratio is converted into a structure-aware weight using an exponential mapping method. The purpose of this step is to preserve the differentiated information of high-quality particles having high weights and low-quality particles having low weights, thereby improving the consistency and robustness of the cluster boundaries.
[0068] Furthermore, operations that balance cross-modal information preservation and compression include:
[0069] Based on the sphere level of particles, mutual information is calculated; the target calculation logic of the information bottleneck is broken down into an information retention part and an information compression part; by adjusting the value of parameter β, the proportion of the information retention part and the information compression part is determined. The purpose of this step is to enable the same framework to amplify shared information as needed and suppress modality-specific noise, without the need to design weights separately for different modes, thus reducing the workload of parameter tuning.
[0070] Specifically, the objective function for the information bottleneck is:
[0071] ;
[0072] In the formula, For information reservation, For information compression items, The objective function is the information bottleneck. This is a hyperparameter used to balance the relative importance of information retention and information compression terms.
[0073] Furthermore, the iterative optimization of the affiliation relationship from spheres to clusters includes:
[0074] For each sphere, extract it from its current cluster to form a new independent cluster;
[0075] Calculate the merging cost when merging this sphere into each candidate cluster;
[0076] Select the candidate clusters that minimize the merging cost for merging;
[0077] Repeat the above process until the cluster affiliation of all particles no longer changes, reaching a convergent state.
[0078] Specifically, the merger cost is:
[0079] ;
[0080] In the formula, Let g be the probability weight after merging the particle g with the target cluster t, and JS be the Jensen-Shannon divergence. For merger costs, Let y be the conditional probability of feature y appearing in sphere g. For structure-aware weights, Let y be the conditional probability of feature y appearing in cluster t. For hyperparameters, Let g be the marginal probability distribution of the particle. Given a cluster t, the conditional probability that a particle g belongs to the corresponding cluster.
[0081] This embodiment significantly improves the clustering performance on multimodal datasets compared to existing methods by employing the aforementioned sphere compression, structure-aware weighting, and dynamic trade-offs over information bottlenecks, thereby achieving the goal of obtaining highly robust and efficient clustering results on multimodal heterogeneous data.
[0082] To more clearly illustrate the technical solution of the present invention, specific embodiments are provided below for description:
[0083] Step 1: Calculation of Particle Purity:
[0084] Based on a multimodal dataset, recursive splitting is performed using mutual information purity as an indicator.
[0085] In this embodiment, taking the news scenario of "urban flooding due to rainstorm" as an example, it is necessary to cluster the multimodal data released, which includes text updates, on-site photos, and audio clips, and quickly divide it into sub-topic clusters such as "rescue progress", "traffic impact", and "mutual assistance among the masses".
[0086] Input is (n is the total number of samples), This is the feature matrix corresponding to the i-th modality (including text, image, and speech features). Let m be the feature dimension of the i-th mode, and m be the number of modes.
[0087] The formula for multimodal purity is:
[0088] ;
[0089] In the formula, For the x-th sample in the i-th modality, the mutual information between the "features" and the "pseudo-labels" is... The number of samples within a granule. Let g be the multimodal purity of the sphere.
[0090] Step 2: Recursively generate a set of particles:
[0091] The dataset is initially treated as a single sphere. K-means (where K is the number of clusters) is applied to each modality of text, image, and speech to generate pseudo-labels. The multimodal purity of the initial sphere g is then calculated using the purity formula.
[0092] If the purity of the spheres is greater than the threshold (fixed at 0.6) or the number of samples is less than the minimum (fixed at 20), then no splitting is performed; otherwise, K-means binary splitting is used, and the above operation is repeated until all spheres meet the conditions, resulting in a set of spheres. ( (Total number of particles).
[0093] Step 3: Calculation of particle structure sensing weights:
[0094] Calculate the center of each sphere in three modes and integrate them to obtain the multimodal center. ,in, Let g be the center vector of the particle g in the m-th mode.
[0095] The formula for the center of the sphere g in the i-th mode is:
[0096] ;
[0097] In the formula, Let g be the center vector of the particle in the i-th mode. Let be the feature vector of the r-th sample in the i-th mode in particle g, where r is the index of the sample in particle g.
[0098] In the initial stage of iterative optimization of the objective function, clusters are initialized. For each particle g, it is extracted from the current cluster, and its average cosine similarity with particles in the target cluster is calculated. This is then compared with the average cosine similarity between the particle g and particles not belonging to the target cluster to obtain the structural similarity ratio. Structure-aware weights are generated through exponential mapping.
[0099] The formula for structure-aware weights is: ;
[0100] In the formula, For structural constraint hyperparameters, For structure-aware weights, The total number of particles;
[0101] If the similarity between a particle and particles within the target cluster is much higher than that between particles outside the target cluster, then the preservation of the particle's information is emphasized (such as "rescue operation" information); if the similarity between a particle and each cluster is ambiguous, then the weight is reduced, emphasizing information compression to achieve differentiated control.
[0102] Step 4: Construction and optimization of the information bottleneck objective function:
[0103] By embedding structure-aware weights into the information bottleneck framework, the objective function is obtained. The assignment of particles to the target cluster is iteratively optimized, and the merging cost (i.e., the increment of the objective function) for each assignment adjustment is calculated. The assignment scheme that minimizes the merging cost is selected until the objective function converges to the optimal solution.
[0104] The objective function is:
[0105] ;
[0106] In the formula, For information retention items (such as retaining multimodal core information such as "rescue" and "transportation"), For information compression (compressing redundant noise, such as irrelevant background chat text), the computational efficiency is improved by replacing samples with spheres. The hyperparameter β∈{10,50,100,500,700,900} is used to adjust the trade-off between information compression and preservation.
[0107] The iterative optimization operation adopts an "extract-merge" strategy. For each particle g, it is first extracted from its current cluster to form an independent new cluster. In order to keep the number of clusters unchanged, the loss change (i.e. merging cost) caused by assigning g to each candidate cluster t is calculated. The cluster that minimizes the merging cost is selected for merging. The above process is repeated for all spheres until the cluster assignment no longer changes, reaching a convergence state.
[0108] The formula for merger cost is:
[0109] ;
[0110] In the formula, Let g be the probability weight after merging the particle g and the target cluster t, and JS be the Jensen-Shannon divergence.
[0111] Step 5: Clustering Result Mapping and Performance Evaluation
[0112] Finally, the cluster affiliation label of each sphere is used as the final cluster label for all samples within that sphere. The effectiveness of the method is verified by calculating two metrics: clustering accuracy (ACC) and normalized mutual information (NMI), and comparing them with the true labels in the dataset. The results show that this approach outperforms the comparison methods on multiple datasets (such as SOCCER and BBC), achieving the goals of improving the robustness (resistance to noise interference such as image blurring and text typos) and efficiency (effectively reducing computation time) of multimodal clustering.
[0113] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A structure-aware particle-sphere information bottleneck method for multimodal data clustering, characterized in that, include: A multimodal dataset containing text, images, and audio segments is obtained. Features are extracted from each modality sample in the multimodal dataset. The multimodal dataset is then divided into spheres to generate a sphere set. For each particle, calculate the structural similarity ratio between the corresponding particle and the current cluster to which it belongs, and generate a structure-aware weight based on the structural similarity ratio; Using the particles in the set of particles as processing units, an information bottleneck objective function is constructed, wherein the information bottleneck objective function includes an information retention term and an information compression term; The information retention items are dynamically weighted using the structure-aware weights, and the affiliation relationship between spheres and clusters is iteratively optimized to maximize the weighted information bottleneck objective function until convergence, and the final clustering result is output.
2. The structure-aware particle-sphere information bottleneck method for multimodal data clustering according to claim 1, characterized in that, The multimodal dataset is divided into granular and sphere parts, including: S1.1: Initialize the multimodal dataset into an initial sphere; S1.2: For the initial spheres, perform a clustering algorithm for each mode to generate pseudo-labels for the corresponding modes; S1.3: For the current particle, calculate the multimodal purity of the current particle based on the mutual information between the features extracted from each modal sample and the pseudo-label of the corresponding modality; S1.4: If the multimodal purity is higher than the preset purity threshold, or the number of samples in the current sphere is lower than the preset sample number threshold, then the current sphere is retained; otherwise, the current sphere is split in two. S1.5: Recursively execute S1.3 to S1.4 on the daughter particles produced after splitting until all particles meet the conditions.
3. The structure-aware particle-sphere information bottleneck method for multimodal data clustering according to claim 2, characterized in that, The multimodal purity of the current sphere is calculated as follows: ； In the formula, For the x-th sample in the i-th modality, the mutual information between features and pseudo-labels is... m is the number of samples within the sphere, and m is the number of modes. Let g be the multimodal purity of the sphere.
4. The structure-aware particle-sphere information bottleneck method for multimodal data clustering according to claim 1, characterized in that, Calculate the structural similarity ratio between the corresponding particle and its current cluster, including: Calculate the average first similarity between the multimodal centers of the target particle and the multimodal centers of other particles within the current cluster of the target particle; Calculate the average second similarity between the multimodal centers of the target sphere and the multimodal centers of spheres in all other clusters; The ratio of the average first similarity to the average second similarity is determined as the structural similarity ratio.
5. The structure-aware particle-sphere information bottleneck method for multimodal data clustering according to claim 4, characterized in that, The average first similarity and the average second similarity are calculated using cosine similarity.
6. The structure-aware particle-sphere information bottleneck method for multimodal data clustering according to claim 1, characterized in that, Based on the structural similarity ratio, the structural awareness weights are generated as follows: ； In the formula, For structural constraint hyperparameters, For structure-aware weights, The structural similarity ratio, denoted as the total number of particles, and g represents the number of particles currently involved in the calculation.
7. The structure-aware particle-sphere information bottleneck method for multimodal data clustering according to claim 1, characterized in that, The objective function for the information bottleneck is: ； In the formula, For information reservation, For information compression items, The objective function is the information bottleneck. This is a hyperparameter used to balance the relative importance of information retention and information compression terms.
8. The structure-aware particle-sphere information bottleneck method for multimodal data clustering according to claim 1, characterized in that, Iterative optimization of the affiliation relationship from spheres to clusters includes: For each sphere, extract it from its current cluster to form a new independent cluster; Calculate the merging cost when merging this sphere into each candidate cluster; Select the candidate clusters that minimize the merging cost for merging; Repeat the above process until the cluster affiliation of all particles no longer changes, reaching a convergent state.
9. The structure-aware particle-sphere information bottleneck method for multimodal data clustering according to claim 8, characterized in that, The merger cost is: ； In the formula, Let g be the probability weight after merging the particle g with the target cluster t, and JS be the Jensen-Shannon divergence. For merger costs, Let y be the conditional probability of feature y appearing in sphere g. For structure-aware weights, Let y be the conditional probability of feature y appearing in cluster t. For hyperparameters, Let g be the marginal probability distribution of the particle. Given a cluster t, the conditional probability that a particle g belongs to the corresponding cluster.