Multiview clustering method for multimodal view noise alignment
By using encoding networks and singular value decomposition optimization methods in multimodal clustering, the view misalignment problem is solved, achieving fast and accurate multi-view clustering and improving the robustness and accuracy of multimodal data analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SICHUAN UNIV
- Filing Date
- 2024-06-25
- Publication Date
- 2026-08-04
AI Technical Summary
Existing multimodal clustering methods struggle to effectively utilize data information when faced with view misalignment issues, resulting in limited clustering accuracy. This is especially true when no category-level alignment information is available, leading to poor performance of noisy association methods.
Two encoding networks are used to encode multimodal data. The cross-view connectivity graph is constructed by matching view pairs through Euclidean distance and singular value decomposition is performed. The network is optimized using the cross-entropy loss function and the K-Means clustering algorithm is combined to obtain the clustering results.
It improves the speed and accuracy of the clustering process, effectively reduces the impact of view misalignment, and enhances the robustness of multimodal data analysis.
Smart Images
Figure CN118823404B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data analysis, and more specifically to a multi-view clustering method for aligning noise in multimodal views. Background Technology
[0002] Multimodal clustering is one of the most critical problems in the field of multimodal data analysis. It aims to learn a unified representation of information from different modalities such as text and images, grouping samples belonging to the same category together without category information, i.e., assigning each sample to one of K categories. Multimodal clustering utilizes pairwise information and similarity information between samples of the same category in the dataset to achieve accurate matching of cross-modal data. It is mainly applied to cross-modal retrieval tasks such as image-text retrieval and plays a key role in current image-text retrieval systems. In recent years, several multimodal clustering methods based on deep neural networks (DNNs) have been proposed and have achieved significant progress in various practical scenarios.
[0003] The success of current multimodal clustering methods relies on an implicit data assumption: that features from multiple views of the same sample belong to the same sample—a concept known as multi-view alignment. Multimodal clustering requires utilizing multi-view alignment information within data samples. However, in real-world scenarios, misalignment issues often arise due to sensor damage, sensor latency, or other factors. This means that errors in data acquisition and processing can lead to multiple views of the same sample in the training dataset not belonging to the same sample.
[0004] Robust multimodal clustering algorithms with incomplete information are used to handle misalignment problems. They are trained on a subset of known aligned samples, and then tested on datasets with unknown alignment relationships and missing views. This algorithm's partial view alignment learning requires training with aligned samples. The neural network is trained on known aligned portions of text, and then tested on known misaligned portions. However, obtaining aligned samples is often difficult, and reliability is hard to guarantee.
[0005] The robust cross-modal learning algorithm initializes a neural network for each modality, utilizes cross-modal contrastive learning and complementary contrastive learning loss functions, trains on noisy datasets, calculates the loss function values, and optimizes the network through backpropagation to ensure the model's robustness to noisy associations. This algorithm primarily addresses the noisy association problem in cross-modal retrieval, handling the issue of mismatches between paired text and images in retrieval tasks. In cross-modal retrieval, there is no concept of "category," and no usable category-level alignment information; only paired samples exist, requiring the search for instance-level alignment. Noisy association methods do not utilize category-level alignment information, thus limiting their effectiveness in multimodal clustering scenarios. Summary of the Invention
[0006] In view of the above-mentioned shortcomings in the prior art, the multi-view clustering method for noise alignment of multimodal views provided by the present invention takes into account both the impact of mispairing and the category-level alignment problem in the clustering problem, and can make fuller use of the information in the data to improve the clustering accuracy.
[0007] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:
[0008] A multi-view clustering method for noise alignment in multimodal views is provided, comprising the following steps:
[0009] S1. The first and second encoding networks after training are used to encode the two views of each sample to be clustered, respectively, to obtain the representation vectors corresponding to the two views of each sample to be clustered.
[0010] S2. For each representation vector corresponding to the first view, find the representation vector with the closest Euclidean distance among all representation vectors corresponding to the second view, and form a new view pair between the two corresponding views.
[0011] S3. Concatenate the representation vectors corresponding to the new view pairs into a single vector, and apply the K-Means clustering algorithm to all concatenated vectors to obtain the clustering results.
[0012] Furthermore, both the first and second encoding networks are fully connected neural networks.
[0013] Furthermore, the training methods for the first and second coding networks include the following steps:
[0014] A1. Obtain two views for each sample to be clustered to obtain a multimodal dataset; the multimodal dataset contains N samples to be clustered.
[0015] A2. Randomly sample a batch of samples from the multimodal dataset as training samples, and encode the first view corresponding to the batch of training samples through the first encoding network, and encode the second view corresponding to the batch of training samples through the second encoding network to obtain the representation vectors corresponding to the two views of the batch of training samples respectively.
[0016] A3. Based on the representation vectors corresponding to the two views of the training samples in this batch, construct cross-view connection graphs, namely the connection graph from the first view to the second view and the connection graph from the second view to the first view.
[0017] A4. Perform singular value decomposition on the connection graph from the first view to the second view to obtain the first similarity graph; perform singular value decomposition on the connection graph from the second view to the first view to obtain the second similarity graph.
[0018] A5. Based on the representation vectors corresponding to the two views of the training samples in this batch, calculate the similarity matrix between the first view and the second view, and the similarity matrix between the second view and the first view.
[0019] A6. Using the first similarity map as the supervision signal of the similarity matrix between the first view and the second view, calculate the first loss value using the cross-entropy loss function; using the second similarity map as the supervision signal of the similarity matrix between the second view and the first view, calculate the second loss value using the cross-entropy loss function.
[0020] A7. Take the sum of the first loss value and the second loss value as the final loss value, and update the first encoding network and the second encoding network through the backpropagation method until the training number is reached or the network converges, to obtain the trained first encoding network and the trained second encoding network.
[0021] Furthermore, the fully connected neural network corresponding to the first encoding network contains 5 layers, with the input dimensions of each layer being 20, 1024, 1024, 1024, and 128 respectively; the fully connected neural network corresponding to the second encoding network contains 5 layers, with the input dimensions of each layer being 59, 1024, 1024, 1024, and 128 respectively.
[0022] Furthermore, the specific method for constructing the cross-view connection diagram in step A3 includes the following steps:
[0023] A3-1. According to the formula:
[0024]
[0025]
[0026]
[0027]
[0028] Obtain the representation vector corresponding to the first view of the i-th training sample in this batch. The representation vector corresponding to the j-th training sample in the second view of this batch tags The representation vector corresponding to the i-th training sample in this batch under the second view. The representation vector corresponding to the j-th training sample in the second view of this batch tags The representation vector corresponding to the i-th training sample in this batch under the second view. The representation vector corresponding to the j-th training sample in the first view of this batch tags The representation vector corresponding to the i-th training sample in this batch under the first view. The representation vector corresponding to the j-th training sample in the first view of this batch tags This yields the label matrix of the representation vectors corresponding to the first view of the training samples in this batch, relative to the representation vectors corresponding to the second view. The label matrix of the representation vector corresponding to the second view of this batch of training samples is relative to the representation vector corresponding to the second view. The label matrix of the representation vector corresponding to the second view of this batch of training samples relative to the representation vector corresponding to the first view. The label matrix of the representation vector corresponding to the first view of this batch of training samples is relative to the representation vector corresponding to the first view. in This is the representation vector corresponding to the first view of the training samples in this batch; is the representation vector corresponding to the second view of the training samples in this batch; ∥.∥ represents the L2 norm; σ is a constant; exp represents the element-wise natural exponentiation operator with the natural constant e as the base;
[0029] A3-2. According to the formula:
[0030]
[0031]
[0032] Obtain the connection diagram from the first view to the second view. Connection diagram from the second view to the first view in This represents the transpose of a matrix.
[0033] Furthermore, the specific method for performing singular value decomposition on the two cross-view connectivity graphs in step A4 includes the following steps:
[0034] A4-1. According to the formula:
[0035]
[0036] Connection diagram from the first view to the second view Perform singular value decomposition, setting all singular values in Σ less than the threshold η to 0, to obtain a diagonal matrix. Where U is the left singular matrix; V is the right singular matrix; and ∑ is a diagonal matrix composed of singular values;
[0037] A4-2. According to the formula:
[0038]
[0039] Obtain the first similarity graph
[0040] A4-3. Obtain the first similarity map using the same method as in steps A4-1 and A4-2.
[0041] Furthermore, the expression for calculating the similarity matrix in step A5 is:
[0042]
[0043]
[0044] in This is the similarity matrix between the first view and the second view; This is the similarity matrix between the second view and the first view.
[0045] Furthermore, the expression for calculating the loss value using the cross-entropy loss function in step A6 is as follows:
[0046]
[0047]
[0048] in This is the first loss value; This is the second loss value; ρ is the cross-entropy loss function; ρ(.) represents the row-by-row softmax function.
[0049] The beneficial effects of this invention are as follows:
[0050] 1. This invention uses two views for encoding during the clustering process and obtains the clustering results through the K-Means clustering algorithm, which can quickly perform view clustering and improve the speed of data analysis.
[0051] 2. This invention utilizes the similarity between a sample in one view and all samples in another view to form the feature of that sample. The encoding network is optimized and updated through a contrastive learning method. Singular value decomposition is used in the optimization to denoise the optimization target, thereby improving the robustness of the encoding network to noise and thus improving the accuracy of view clustering.
[0052] 3. This invention can reduce the impact of misalignment by constructing a representation vector; it can also reduce the impact of misalignment by using singular value decomposition, thereby improving the accuracy of view clustering. Attached Figure Description
[0053] Figure 1 This is a flowchart illustrating the method.
[0054] Figure 2 This is a schematic diagram illustrating the misalignment issue in the embodiment. Detailed Implementation
[0055] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0056] like Figure 1 As shown, the multi-view clustering method for aligning multimodal view noise includes the following steps:
[0057] S1. The first and second encoding networks after training are used to encode the two views of each sample to be clustered, respectively, to obtain the representation vectors corresponding to the two views of each sample to be clustered.
[0058] S2. For each representation vector corresponding to the first view, find the representation vector with the closest Euclidean distance among all representation vectors corresponding to the second view, and form a new view pair between the two corresponding views.
[0059] S3. Concatenate the representation vectors corresponding to the new view pairs into a single vector, and apply the K-Means clustering algorithm to all concatenated vectors to obtain the clustering results.
[0060] Both the first and second encoding networks are fully connected neural networks. The training methods for the first and second encoding networks include the following steps:
[0061] A1. Obtain two views for each sample to be clustered to obtain a multimodal dataset; the multimodal dataset contains N samples to be clustered.
[0062] A2. Randomly sample a batch of samples from the multimodal dataset as training samples, and encode the first view corresponding to the batch of training samples through the first encoding network, and encode the second view corresponding to the batch of training samples through the second encoding network to obtain the representation vectors corresponding to the two views of the batch of training samples respectively.
[0063] A3. Based on the representation vectors corresponding to the two views of the training samples in this batch, construct cross-view connection graphs, namely the connection graph from the first view to the second view and the connection graph from the second view to the first view.
[0064] A4. Perform singular value decomposition on the connection graph from the first view to the second view to obtain the first similarity graph; perform singular value decomposition on the connection graph from the second view to the first view to obtain the second similarity graph.
[0065] A5. Based on the representation vectors corresponding to the two views of the training samples in this batch, calculate the similarity matrix between the first view and the second view, and the similarity matrix between the second view and the first view.
[0066] A6. Using the first similarity map as the supervision signal of the similarity matrix between the first view and the second view, calculate the first loss value using the cross-entropy loss function; using the second similarity map as the supervision signal of the similarity matrix between the second view and the first view, calculate the second loss value using the cross-entropy loss function.
[0067] A7. Take the sum of the first loss value and the second loss value as the final loss value, and update the first encoding network and the second encoding network through the backpropagation method until the training number is reached or the network converges, to obtain the trained first encoding network and the trained second encoding network.
[0068] The first encoding network corresponds to a fully connected neural network with 5 layers, each with input dimensions of 20, 1024, 1024, 1024, and 128 respectively. The second encoding network corresponds to a fully connected neural network with 5 layers, each with input dimensions of 59, 1024, 1024, 1024, and 128 respectively. The input dimensions of the first layer depend on the dimensions of the dataset input; in this embodiment, "20" and "59" correspond to the input dimensions of the Scene 15 dataset.
[0069] The specific method for constructing the cross-view connection diagram in step A3 includes the following steps:
[0070] A3-1. According to the formula:
[0071]
[0072]
[0073]
[0074]
[0075] Obtain the representation vector corresponding to the first view of the i-th training sample in this batch. The representation vector corresponding to the j-th training sample in the second view of this batch tags The representation vector corresponding to the i-th training sample in this batch under the second view. The representation vector corresponding to the j-th training sample in the second view of this batch tags The representation vector corresponding to the i-th training sample in this batch under the second view. The representation vector corresponding to the j-th training sample in the first view of this batch tags The representation vector corresponding to the i-th training sample in this batch under the first view. The representation vector corresponding to the j-th training sample in the first view of this batch tags This yields the label matrix of the representation vectors corresponding to the first view of the training samples in this batch, relative to the representation vectors corresponding to the second view. The label matrix of the representation vector corresponding to the second view of this batch of training samples is relative to the representation vector corresponding to the second view. The label matrix of the representation vector corresponding to the second view of this batch of training samples relative to the representation vector corresponding to the first view. The label matrix of the representation vector corresponding to the first view of this batch of training samples is relative to the representation vector corresponding to the first view. in This is the representation vector corresponding to the first view of the training samples in this batch; is the representation vector corresponding to the second view of the training samples in this batch; ∥.∥ represents the L2 norm; σ is a constant; exp represents the element-wise natural exponentiation operator with the natural constant e as the base;
[0076] A3-2. According to the formula:
[0077]
[0078]
[0079] Obtain the connection diagram from the first view to the second view. Connection diagram from the second view to the first view in This represents the transpose of a matrix.
[0080] The specific method for performing singular value decomposition on the two cross-view connectivity graphs in step A4 includes the following steps:
[0081] A4-1. According to the formula:
[0082]
[0083] Connection diagram from the first view to the second view Perform singular value decomposition, setting all singular values in ∑ that are less than the threshold η to 0, to obtain a diagonal matrix. Where U is the left singular matrix; V is the right singular matrix; and Σ is a diagonal matrix composed of singular values;
[0084] A4-2. According to the formula:
[0085]
[0086] Obtain the first similarity graph
[0087] A4-3. Obtain the first similarity map using the same method as in steps A4-1 and A4-2.
[0088] The expression for calculating the similarity matrix in step A5 is:
[0089]
[0090]
[0091] in This is the similarity matrix between the first view and the second view; This is the similarity matrix between the second view and the first view.
[0092] The expression for calculating the loss value using the cross-entropy loss function in step A6 is as follows:
[0093]
[0094]
[0095] in This is the first loss value; This is the second loss value; ρ is the cross-entropy loss function; ρ(.) represents the row-by-row softmax function.
[0096] In the specific implementation process, Figure 2 Consider a video of cake making and its accompanying narration. The images and narration text each belong to a view within the video. Correct alignment means the narration text accurately describes the image in the same frame; misalignment means the narration text's description and its corresponding image are not in the same frame; and incorrect alignment means the narration text fails to describe the image in the same frame. It can be seen that the image and text information in this video are not aligned. Retrieving corresponding image information based on misaligned text information will lead to incorrect results. Therefore, view clustering of this video can correct these errors.
[0097] In one embodiment of the present invention, Scene 15 is a dataset containing 4485 images across 15 categories. The method clusters this dataset, even if there are pairs of samples in the training data that do not actually match.
[0098] In this embodiment, the PHOG and GIST algorithms are used to extract features from the same sample, respectively, to serve as two views of that sample. Two second-encoding networks are trained using a batch of samples for 200 epochs, resulting in a trained first-encoding network and a trained second-encoding network. Based on the trained first-encoding network and the trained second-encoding network, view clustering is performed on the Scene 15 dataset to obtain the clustering results.
[0099] To demonstrate the clustering performance of our method and existing techniques on the Scene 15 dataset, we also performed clustering on the Scene 15 dataset using the DCCAE, BMVC, MvCLN, PVC, SURE, CGCN, GCFAgg, and DIVIDE clustering algorithms. The results were evaluated using accuracy (ACC), normalized mutual information (NMI), and adjusted Land coefficient (ARI), and are shown in Table 1.
[0100]
[0101]
[0102] The noise ratio in Table 1 represents the proportion of randomly shuffled views. As can be seen from Table 1, the proposed method achieves high results for datasets with various noise ratios, indicating that the method is robust to different levels of noise.
[0103] In this embodiment, another dataset used for data clustering is the NUS-WIDE dataset, which contains 9000 image-text pairs. To demonstrate the clustering performance of this method and existing techniques on the NUS-WIDE dataset, the Scene15 dataset was also clustered using the DCCAE, BMVC, MvCLN, PVC, SURE, CGCN, GCFAgg, and DIVIDE clustering algorithms. Accuracy (ACC), Normalized Mutual Information (NMI), and Adjusted Land Coefficient (ARI) were used for evaluation, and the results are shown in Table 2.
[0104] Table 2
[0105]
[0106]
[0107]
[0108] The noise ratio in Table 2 represents the proportion of views containing random shuffling. As can be seen from Table 2, the proposed method achieves high results for datasets with various noise ratios, indicating that the method is robust to different levels of noise.
[0109] In summary, this invention uses two views for encoding during the clustering process. By constructing representation vectors, the impact of misalignment can be reduced. Singular value decomposition can also reduce the impact of misalignment. By obtaining clustering results through the K-Means clustering algorithm, view clustering can be performed quickly, improving the speed of data analysis.
Claims
1. A multi-view clustering method for noise alignment in multimodal views, characterized in that, Includes the following steps: S1. The first and second encoding networks trained after training are used to encode the two views of each sample to be clustered, respectively, to obtain the representation vectors corresponding to the two views of each sample to be clustered; where the view is an image-text pair; S2. For each representation vector corresponding to the first view, find the representation vector with the closest Euclidean distance among all representation vectors corresponding to the second view, and form a new view pair between the two corresponding views. S3. Concatenate the representation vectors corresponding to the new view pairs into a single vector, and apply the K-Means clustering algorithm to all concatenated vectors to obtain the clustering results. The training methods for the first and second encoding networks include the following steps: A1. Obtain two views for each sample to be clustered to obtain a multimodal dataset; the multimodal dataset contains N samples to be clustered. A2. Randomly sample a batch of samples from the multimodal dataset as training samples, and encode the first view corresponding to the batch of training samples through the first encoding network, and encode the second view corresponding to the batch of training samples through the second encoding network to obtain the representation vectors corresponding to the two views of the batch of training samples respectively. A3. According to the formula: Obtain the connection diagram from the first view to the second view. Connection diagram from the second view to the first view ;in Represents the transpose of a matrix; This is the label matrix of the representation vectors corresponding to the first view of the training samples in this batch, relative to the representation vectors corresponding to the second view. This is the label matrix of the representation vectors corresponding to the second view of the training samples in this batch, relative to the representation vectors corresponding to the second view. This is the label matrix of the representation vectors corresponding to the second view of the training samples in this batch, relative to the representation vectors corresponding to the first view. This is the label matrix of the representation vector corresponding to the first view of the training samples in this batch, relative to the representation vector corresponding to the first view. A4. Perform singular value decomposition on the connection graph from the first view to the second view to obtain the first similarity graph; perform singular value decomposition on the connection graph from the second view to the first view to obtain the second similarity graph. A5. Based on the representation vectors corresponding to the two views of the training samples in this batch, calculate the similarity matrix between the first view and the second view, and the similarity matrix between the second view and the first view. A6. Using the first similarity map as the supervision signal of the similarity matrix between the first view and the second view, calculate the first loss value using the cross-entropy loss function; using the second similarity map as the supervision signal of the similarity matrix between the second view and the first view, calculate the second loss value using the cross-entropy loss function. A7. Take the sum of the first loss value and the second loss value as the final loss value, and update the first encoding network and the second encoding network through the backpropagation method until the training number is reached or the network converges, to obtain the trained first encoding network and the trained second encoding network.
2. The multi-view clustering method for noise alignment of multimodal views according to claim 1, characterized in that, Both the first and second encoding networks are fully connected neural networks.
3. The multi-view clustering method for noise alignment of multimodal views according to claim 2, characterized in that, The fully connected neural network corresponding to the first encoding network has 5 layers, with the input dimensions of each layer being 20, 1024, 1024, 1024, and 128 respectively; the fully connected neural network corresponding to the second encoding network has 5 layers, with the input dimensions of each layer being 59, 1024, 1024, 1024, and 128 respectively.
4. The multi-view clustering method for noise alignment of multimodal views according to claim 1, characterized in that, The label matrix is calculated as follows: According to the formula: Obtain the batch number respectively i The representation vector corresponding to each training sample in the first view Compared to this batch, the first j The representation vector corresponding to each training sample in the second view tags This batch of i The representation vector corresponding to each training sample in the second view Compared to this batch, the first j The representation vector corresponding to each training sample in the second view tags This batch of i The representation vector corresponding to each training sample in the second view Compared to this batch, the first j The representation vector corresponding to each training sample in the first view tags This batch of i The representation vector corresponding to each training sample in the first view Compared to this batch, the first j The representation vector corresponding to each training sample in the first view tags This allows us to obtain the label matrix of the representation vectors corresponding to the first view of the training samples in this batch, relative to the representation vectors corresponding to the second view. The label matrix of the representation vector corresponding to the second view of this batch of training samples relative to the representation vector corresponding to the second view. The label matrix of the representation vectors corresponding to the second view of this batch of training samples relative to the representation vectors corresponding to the first view. The label matrix of the representation vector corresponding to the first view of this batch of training samples relative to the representation vector corresponding to the first view. ;in , , , ; This is the representation vector corresponding to the first view of the training samples in this batch; This is the representation vector corresponding to the second view of the training samples in this batch; Represents the L2 norm; It is a constant; This represents the element-wise natural exponentiation operator with the natural constant e as the base.
5. The multi-view clustering method for noise alignment of multimodal views according to claim 4, characterized in that, The specific method for performing singular value decomposition on the two cross-view connectivity graphs in step A4 includes the following steps: A4-1. According to the formula: Connection diagram from the first view to the second view Perform singular value decomposition, and Medium less than the threshold The singular values are set to 0, resulting in a diagonal matrix. ;in It is a left singular matrix; It is a right singular matrix; It is a diagonal matrix composed of singular values; A4-2. According to the formula: Obtain the first similarity graph ; A4-3. Obtain the first similarity map using the same method as in steps A4-1 and A4-2. .
6. The multi-view clustering method for noise alignment of multimodal views according to claim 5, characterized in that, The expression for calculating the similarity matrix in step A5 is: in This is the similarity matrix between the first view and the second view; This is the similarity matrix between the second view and the first view.
7. The multi-view clustering method for noise alignment of multimodal views according to claim 6, characterized in that, The expression for calculating the loss value using the cross-entropy loss function in step A6 is as follows: in This is the first loss value; This is the second loss value; The cross-entropy loss function; This represents the softmax function for each row.