Non-aligned multi-view clustering method and device, electronic equipment and storage medium

Abstract

The invention relates to an unaligned multi-view clustering method and device, electronic equipment and a storage medium, and the method comprises the steps: constructing an adjacent graph for each view, and enabling the graph structure of the adjacent graph of a non-reference view to be matched with the adjacent graph of a reference view through a permutation matrix; inputting each view and the adjacent graph thereof into a first-layer graph convolutional network, and extracting a depth representation of each view; constructing a local adjacency graph of each view based on the adjacency graph of each view and the depth representation, and updating the number of neighbors of the local adjacency graph at an interval of at least one training period; and inputting the depth representation of each view and the local adjacency graph into a second-layer graph convolutional network, extracting a clustering indication matrix of each view, training and optimizing a permutation matrix based on a unified loss function, and learning adaptive weights of the views at the same time to obtain a clustering result. The cross-view consistent clustering structure is efficiently learned, and the accuracy and robustness of the misaligned multi-view clustering task are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of data processing, and particularly relates to a clustering method and device for misaligned multi-view, electronic equipment and storage medium. BACKGROUND

[0002] Clustering is an unsupervised learning method, and the purpose is to divide the samples in a data set into several subsets, so that the samples in the same cluster are as similar as possible, and the samples between different clusters are as dissimilar as possible. In the field of data processing, many data exist in the form of multi-view, for example, the same sample can have multiple views or modalities obtained from different sensors or preprocessing methods. The multi-view clustering method in the related art usually assumes that the samples across views are completely aligned, that is, the order of the samples between different views is consistent and the corresponding relationship is known. However, in actual scenarios, due to factors such as sensor differences, independent data acquisition and storage processes, this assumption cannot be met, resulting in that the corresponding relationship of the samples across views is partially known or even completely unknown, which brings great challenges to cross-view information fusion, and makes the traditional method invalid. SUMMARY

[0003] In order to solve the above technical problems, the present application provides a clustering method and device for misaligned multi-view, electronic equipment and storage medium.

[0004] This invention includes a clustering method for misaligned multiple views, comprising: acquiring sensor data including multiple views from at least one sample using multiple different methods; selecting a reference view from the multiple views and designating views other than the reference view as non-reference views, wherein the sensor data acquired using different methods constitute different views, and each view includes sensor data acquired from all samples using the same method; constructing an adjacency graph for each view, and matching the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view using a permutation matrix; inputting each view and its adjacency graph into a first-layer graph convolutional network, extracting a depth representation for each view, and constructing a first loss function and a second loss function using the depth representation; and based on the adjacency graph and depth representation of each view... The process involves constructing a local adjacency graph for each view and using this local adjacency graph to build a third loss function, while updating the number of neighbors in the local adjacency graph at least one training epoch. The depth representation of each view and the local adjacency graph are input into a second-layer graph convolutional network to extract a clustering indicator matrix for each view, and a fourth loss function is constructed using this clustering indicator matrix. A unified loss function is constructed using the first, second, third, and fourth loss functions. The permutation matrix is ​​trained and optimized based on the unified loss function to obtain the target permutation matrix, while simultaneously learning and optimizing the adaptive weights of each view to obtain the target view weights. Finally, the multi-view clustering result for the sensor data is obtained based on the target view weights and the target permutation matrix.

[0005] Optionally, before matching the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view using the permutation matrix, the method further includes: employing the Frank-Wolfe algorithm based on the formula... Solve the above permutation matrix ,in, For the first The permutation matrix of the above non-reference views, For the number of samples, each view is composed of... The data consists of sensor data acquired from each sample. Represents the trace of a matrix. This is the adjacency graph of the reference view above. For the first The adjacency graph of the aforementioned non-reference view; the above-mentioned matching of the graph structure of the adjacency graph of the aforementioned non-reference view with the adjacency graph of the aforementioned reference view by means of a permutation matrix includes: obtaining the adjacency graph of the aforementioned non-reference view after permutation by means of the aforementioned permutation matrix. .

[0006] Optionally, the first layer graph convolutional network is inputted with each view and its adjacency graph, the deep representation of each view is extracted, and the first loss function is constructed by using the deep representation, including: inputting the first layer graph convolutional network with each view and its adjacency graph, and extracting the deep representation of the non-reference view and the reference view respectively; aligning the deep representation of the non-reference view with the deep representation of the reference view through a feature permutation matrix; constructing the first loss function based on the difference between the aligned deep representation of the non-reference view and the deep representation of the reference view; the first loss function includes:

[0007] wherein, is the number of views, is the deep representation of the reference view, is the aligned deep representation of the first non-reference view, is a clustering indicator matrix.

[0008] Optionally, the second loss function is constructed by inputting the first layer graph convolutional network with each view and its adjacency graph, extracting the deep representation of each view, and using the deep representation, including: applying k-means clustering on the deep representation of the reference view to determine clustering centers; determining the clustering assignment result of each view according to the clustering centers; constructing the second loss function according to the clustering assignment result, while using the clustering centers other than the class to which the current sample belongs in the clustering assignment result as negative samples; the second loss function includes:

[0009]

[0010] wherein, is the number of samples, , is a temperature parameter, is a cosine similarity function, is an intermediate parameter, is the clustering center of the negative sample, is the total number of clustering centers.

[0011] Optionally, the third loss function includes:

[0012] wherein, is the adjacency graph of the first non-reference view, is the local adjacency graph of the th non-reference view, is a regularization parameter, denotes the trace of a matrix, is the Laplacian matrix of is the set of real numbers, is the number of samples.

[0013] Optionally, the fourth loss function includes:

[0014]

[0015] wherein, is the th row of is the th row of is the th row of is the cluster-level embedding of the reference view, is the cluster-level embedding of the aligned is a distance function for representing the matching degree or distinguishing degree of the cluster centers of two views, is a temperature parameter, is a cosine similarity function.

[0016] Optionally, the permutation matrix is trained and optimized based on the unified loss function to obtain a target permutation matrix, and each view adaptive weight is learned and optimized to obtain a target view weight, including: fixing each view adaptive weight as , and based on the unified loss function, the permutation matrix and the convolutional network parameters are trained and optimized to obtain the target permutation matrix, wherein, is the number of views; the current permutation matrix and the convolutional network parameters are fixed, and each view adaptive weight is learned and optimized by gradient descent method to obtain the target view weight.

[0017] ​Based on the same inventive concept, the application further provides a misaligned multi-view clustering device, comprising: an acquisition module, configured to acquire sensor data comprising a plurality of views from at least one sample through a plurality of different methods, select a reference view from the plurality of views, and take the views other than the reference view as non-reference views, wherein the sensor data acquired through different methods constitute different views, and each view comprises sensor data acquired from all samples through the same method; a graph structure construction module, configured to construct an adjacency graph for each view, and match the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view through a permutation matrix; a first convolutional network module, configured to input each view and the adjacency graph thereof into a first layer of graph convolutional network, extract a deep representation of each view, and construct a first loss function and a second loss function by using the deep representation; a local adjacency graph construction module, configured to construct a local adjacency graph of each view based on the adjacency graph of each view and the deep representation, and construct a third loss function by using the local adjacency graph, and update the number of neighbors of the local adjacency graph at least once every training period; a second convolutional network module, configured to input the deep representation of each view and the local adjacency graph into a second layer of graph convolutional network, extract a clustering indicator matrix of each view, and construct a fourth loss function by using the clustering indicator matrix; a loss function construction module, configured to construct a unified loss function by using the first loss function, the second loss function, the third loss function and the fourth loss function; an optimization module, configured to train and optimize the permutation matrix based on the unified loss function, obtain a target permutation matrix, and simultaneously learn and optimize an adaptive weight of each view, and obtain a target view weight; and a result output module, configured to obtain a multi-view clustering result of the sensor data according to the target view weight and the target permutation matrix.

[0018] Based on the same inventive concept, the application further provides an electronic device, comprising: a processor; a memory configured to store executable instructions; wherein the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement any one of the methods.

[0019] Based on the same inventive concept, the application further provides a computer-readable storage medium having a computer program stored thereon, wherein the storage medium stores the computer program, and when the computer program is executed by a processor, the processor implements any one of the methods.

[0020] The application has the beneficial effects that: the application provides a misaligned multi-view clustering method based on a graph permutation matrix and double-level contrastive learning, which first performs cross-view graph structure matching through a permutation matrix, then performs double-level contrastive alignment using a graph convolutional network based on the alignment, and enforces cross-view alignment at two levels of a sample level and a clustering level.Figure One The consistency is ensured; meanwhile, an adaptive local adjacency graph is constructed and view weights are learned, optimal multi-view fusion is realized, and under the condition that samples are completely misaligned, cross-view clustering structure can still be efficiently learned, and the accuracy and robustness of the misaligned multi-view clustering task are improved. Figure One BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0023] Figure 1 A flowchart of a misaligned multi-view clustering method provided by an embodiment of the present application; Figure 2 A comparison chart of experimental data of the method provided by the embodiment of the present application and the method in related art; Figure 3 A comparison chart of experimental data of the method provided by the embodiment of the present application and the method in related art; Figure 4 A structure diagram of a misaligned multi-view clustering device provided by an embodiment of the present application; Figure 5 An electronic device hardware structure diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to more clearly illustrate the above-mentioned purposes, features and advantages of the embodiments of the present application, the schemes of the embodiments of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0025] In the following description, many specific details are set forth in order to fully understand the embodiments of the present application, but the embodiments of the present application can also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some of the embodiments of the present application, not all the embodiments.

[0026] ​The misaligned multi-view clustering methods in related technologies suffer from the following technical drawbacks: most deep learning methods lack explicit sample permutation mechanisms, relying solely on contrastive learning for implicit alignment, making it difficult to handle completely misaligned multi-view data and neglecting alignment of underlying structural information; furthermore, methods in related technologies typically only perform alignment in the feature space, failing to consider both sample-level and cluster-level alignment simultaneously, leading to misalignment across views. Figure One Insufficient stamina.

[0027] In view of this, embodiments of the present invention provide a clustering method for misaligned multiple views, such as... Figure 1 As shown, it includes: S1. Sensor data comprising multiple views is acquired from at least one sample using multiple different methods. A reference view is selected from the multiple views, and views other than the reference view are designated as non-reference views. The sensor data acquired using different methods constitute different views, and each view includes sensor data acquired from all the samples using the same method.

[0028] In multi-view learning, a view refers to different representations of data. For example, for image data, one view might be pixel values, while another might be features after edge detection. Each view may contain different information. Sensor data is data generated by sensing devices through sensing, measurement, and transmission, belonging to the data type of the Internet of Things (IoT). It can be divided into real-time status data (such as speed and power) and cumulative data (such as mileage and heat consumption). Time-series data is formed through periodic collection, providing a foundation for data visualization and mining. Sensors convert physical quantities into electrical signals through detection components, and wireless transmission is achieved in conjunction with telemetry units. Real-time processing is achieved by connecting to IoT platforms using protocols such as MQTT. For example, one of the different views mentioned above might include vibration data collected using a common vibration sensor on the sample, while another view might include vibration data collected using a different vibration sensor on the same sample with higher precision or a different data output format. Yet another view might include temperature data collected using a temperature sensor on the same sample. Through multi-view learning, this information can be comprehensively utilized to improve model performance. A reference view is a baseline view selected from multiple views. In subsequent steps, other views can be compared and aligned with this baseline.

[0029] Specifically, the above S1 includes input containing Multiview dataset of samples ,common Each view consists of a set of views, each derived from... The data consists of sensor data acquired from each sample, meaning each sample includes data from... Sensor data acquired using V different methods, each sample including d features, the th view (i.e., using V different methods) Feature matrix of each view , Let be the set of real numbers, where For the first The feature dimensions of each view. The specific process of selecting a reference view from multiple views includes independently calculating the silhouette coefficient for each view and selecting the view with the largest silhouette coefficient as the reference view. Reference View It has the clearest clustering structure.

[0030] S2. Construct an adjacency graph for each view, and use a permutation matrix to match the graph structure of the adjacency graph of the non-reference view with that of the reference view.

[0031] Adjacency graphs are used to represent the relationships between data points, typically represented by adjacency matrices. The elements of an adjacency matrix represent the connection weights between nodes. Adjacency graphs can capture the structural information of the data. Permutation matrices, on the other hand, are special matrices used to adjust the row or column order of two matrices to make their structures more similar. Adjacency graphs for different views may have different structures. By aligning the adjacency graphs of non-reference views with those of the reference view using permutation matrices, structural consistency across different views can be ensured, facilitating subsequent feature extraction and fusion.

[0032] S3. Input each view and its adjacency graph into the first layer of the graph convolutional network, extract the depth representation of each view, and use the depth representation to construct the first loss function and the second loss function.

[0033] Graph convolutional networks (GCNNs) are neural networks used for graph-structured data. They learn node representations, while depth representations are high-level feature representations extracted from each view of the data. GCNNs learn these high-level feature representations, which are more expressive than the original data and helpful for subsequent clustering tasks. The loss function measures the difference between the model's predictions and the actual values. It can be viewed as the optimization objective of the GCNN. By constructing the loss function, the model's parameters can be optimized to ensure that the extracted depth representations better reflect the structure and features of the data.

[0034] S4. Construct a local adjacency graph for each view based on the adjacency graph and depth representation of each view, and use the local adjacency graph to construct a third loss function, while updating the number of neighbors in the local adjacency graph at least one training cycle interval.

[0035] The local adjacency graph is a more fine-grained adjacency graph based on the deep representation and the adjacency graph, and is used to capture the local structure. By dynamically updating the number of neighbors of the local adjacency graph, the dynamic change of the data can be adapted, and the robustness and adaptability of the model can be improved.

[0036] S5, input the deep representation and the local adjacency graph of each view into a second layer graph convolution network, extract a clustering indication matrix of each view, and construct a fourth loss function by using the clustering indication matrix.

[0037] The clustering indication matrix is a matrix representing the clustering result of the data points, and is usually used to evaluate the clustering effect. The clustering indication matrix directly reflects the clustering result of the data. By constructing a clustering-related loss function, the parameters of the model can be further optimized to ensure the quality of the clustering result. The first layer graph convolution network in S3 above is equivalent to aligning the views at the sample level, and the second layer graph convolution network in S5 is used to align the views again at the clustering level.

[0038] S6, construct a unified loss function by using the first loss function, the second loss function, the third loss function and the fourth loss function.

[0039] S7, train and optimize the permutation matrix based on the unified loss function to obtain a target permutation matrix, and simultaneously learn to optimize the adaptive weight of each view to obtain a target view weight.

[0040] The weight of the view is used to measure the importance of different views, and the optimized permutation matrix is used to adjust the graph structure of the view. In some specific embodiments, by continuously adjusting the permutation matrix, the above unified loss function can reach a minimum value or convergence, and the target permutation matrix obtained at this time can achieve the best clustering effect. The adaptive weight is used to distinguish the importance of different views in clustering, and is learned and optimized based on the adaptive learning target.

[0041] S8, obtain a multi-view clustering result of the sensor data according to the target view weight and the target permutation matrix.

[0042] In some specific embodiments, the above clustering result can be a clustering label of each sample, which is used to represent the clustering category to which each sample belongs. The embodiment of the present application provides an unaligned multi-view clustering method based on a graph permutation matrix and a double-level contrastive learning. The method first performs cross-view graph structure matching explicitly through the permutation matrix, and then performs double-level contrastive alignment using a graph convolution network based on the alignment, to enforce cross-view consistency at the sample level and the clustering level. Figure One At the same time, the adaptive local adjacency graph is constructed and the view weight is learned, the optimal multi-view fusion is realized, and the cross-view consistency can be efficiently learned even in the case of completely unaligned samples. Figure OneThe clustering structure is improved, and the accuracy and robustness of the unaligned multi-view clustering task are improved.

[0043] The clustering method for unaligned multi-views provided by the embodiment of the application can be widely applied to scenes where multi-modal data exist sample alignment problems, including but not limited to: in security monitoring, access control systems or mobile device unlocking, the same user may collect biological feature data (face image, thermal imaging, fingerprint atlas) through different sensors (such as camera, infrared sensor, fingerprint instrument), due to independent operation of the device or different data collection times, the order of the data belonging to each sample in the multi-view data is inconsistent, at this time, the clustering method provided by the embodiment of the application needs to be used to cluster and analyze the data, to improve the recognition accuracy, and is especially suitable for identity grouping in a multi-person scene.

[0044] In a hospital scene, the same patient may generate multi-view data through different examination devices, due to examination time or device differences, the initial data is in an unaligned state, at this time, the clustering method provided by the embodiment of the application needs to be used to cluster and analyze the data, to assist doctors in formulating personalized treatment plans and optimizing medical resource allocation; multi-sensor environmental monitoring and anomaly detection.

[0045] In industrial or environmental monitoring, the same area may deploy multiple sensors (such as temperature, humidity, and gas concentration sensors), independently collect data, due to sensor failure or communication delay, the multi-view data (temperature and humidity data) are unaligned, at this time, the clustering method provided by the embodiment of the application needs to be used to cluster and analyze the data, to identify environmental anomaly patterns and improve the robustness of the monitoring system.

[0046] In video monitoring or automatic driving, the same scene may collect multi-view data through multiple cameras (such as visible light, infrared, and depth cameras), due to camera position or time differences, the data samples may be unaligned, at this time, the clustering method provided by the embodiment of the application needs to be used to cluster and analyze the data, to enhance the environmental perception capability of automatic driving.

[0047] To enable the graph convolution network to perform clustering analysis, the embodiment of the application first constructs an initial adjacency graph as the network input. Specifically, S2 includes constructing an initial adjacency graph of each view by the following method and the initial adjacency graph of the reference view : S21, calculating the Euclidean distance between sample i and sample j: wherein, and are the features of sample i and sample j, respectively.

[0048] S22, Solving the graph learning optimization problem: ,in, The connection weights for samples i and j are... For the image The i-th row, This is the regularization parameter.

[0049] S23 computes the edge weights of the graph, reflecting the strength of the association between sample pairs: ,in, This represents the number of neighbors in the adjacency graph, used to control the sparsity of the adjacency graph.

[0050] S24, Through Obtain the symmetric adjacency matrix. Specifically, the adjacency matrix and the adjacency graph are not entirely the same concept, but they are related. For example, an adjacency matrix is ​​typically used in computation, and this adjacency matrix can represent an adjacency graph structure. In practice, the adjacency graph used in subsequent steps can be the symmetric adjacency graph obtained in S24.

[0051] In some embodiments, prior to the step in S2 described above of matching the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view using a permutation matrix, the method further includes: The Frank-Wolfe algorithm is used based on the formula Solving the permutation matrix ,in, For the first The permutation matrix of a non-reference view. For the number of samples, each view is composed of... The data consists of sensor data acquired from d samples, with each sample including d features. Represents the trace of a matrix. For the adjacency graph of the reference view, For the first Adjacency graph of a non-reference view.

[0052] The process in S2 above, which uses a permutation matrix to match the graph structure of the adjacency graph of the non-reference view with that of the reference view, includes: The adjacency graph after permutation of the non-reference view is obtained using the permutation matrix. .

[0053] A permutation matrix is a special square matrix that has exactly one element of 1 in each row and each column and 0s elsewhere. Such a matrix represents a rearrangement of elements. When the graph structure of the non-reference view's adjacency graph is matched with that of the reference view's adjacency graph through a permutation matrix, the principle is to rearrange the vertex order of the non-reference view's adjacency graph by using the row transformation or column transformation capability of the permutation matrix. The structure of an adjacency graph can be represented by its adjacency matrix, and a permutation matrix can change the order of the vertices by left-multiplying or right-multiplying the adjacency matrix, so that the adjacency matrix of the non-reference view is consistent in structure with the adjacency matrix of the reference view. In this way, the graph structures of the two views are matched in terms of vertex connection relationships. The above embodiments of the present application can automatically discover cross-view sample matching relationships and successfully process completely misaligned multi-view data without any prior correspondence by explicitly learning permutation matrices.

[0054] In a specific implementation, S2 includes: S25, for the non-reference view , learning a permutation matrix , so that the graph structure of its adjacency graph is matched with that of the reference view's adjacency graph , and the specific calculation formula includes:

[0055] S26, relaxing the above NP-hard integer programming problem into a real-valued optimization problem:

[0056] S27, using the Frank-Wolfe algorithm to solve the permutation matrix , which performs global exploration through Frank-Wolfe steps and local refinement through projected gradient descent.

[0057] S28, obtaining the permuted graph: (for the reference view, its graph structure remains unchanged, i.e. ).

[0058] S29, performing symmetric normalization on the permuted graph:

[0059] wherein , is a diagonal matrix, satisfying , is an identity matrix, denotes the distance between sample i and sample j in view v.

[0060] In some embodiments, S3 includes: S31. Input each view and its adjacency graph into the first layer of the graph convolutional network, and extract the depth representations of the non-reference view and the reference view respectively.

[0061] Specifically, the depth of the reference view is represented as The depth of a non-reference view is represented as , This represents the function of the first layer graph convolutional network. , To hide the representation dimension.

[0062] S32, through the characteristic permutation matrix Align the depth representation of the non-reference view with the depth representation of the reference view: ( ), This represents the depth after alignment with the non-reference view.

[0063] Specifically, before S32, it also includes: constructing a feature-level alignment optimization problem according to the following formula, and solving for the feature permutation matrix. :

[0064] in, For matrix The i-th row and j-th column, It can represent pairwise distances in the latent space.

[0065] S33. Construct a first loss function based on the difference between the depth representation of the aligned non-reference view and the depth representation of the reference view, and calculate the feature-level alignment loss.

[0066] The first loss function mentioned above include:

[0067] in, For the number of views, For the depth representation of the reference view, For the aligned first Depth representation of a non-reference view This is the clustering indicator matrix.

[0068] The above embodiments of the present application first input different views and their mutual relationship (adjacent graph) into a graph convolution network, extract deep features of each view. Then, a reference view is selected, a feature permutation matrix is solved by constructing a feature-level alignment optimization problem, and the features of non-reference views are adjusted by using this matrix to align with the features of the reference view. Finally, a loss function is constructed based on the difference between the aligned features, the feature-level alignment loss is calculated, and the effect of view feature alignment is improved by optimizing the loss function, so as to better process multi-view data.

[0069] In some embodiments, the above S3 further comprises: S34, in the depth representation of the reference view The application k-means clustering is applied to determine the cluster center, specifically, k cluster centers are obtained .

[0070] S35, determine the cluster assignment result of each view according to the cluster center. Specifically, for each sample i, determine its cluster assignment by the nearest neighbor principle .

[0071] S36, construct a second loss function according to the cluster assignment result, and use other cluster centers as negative samples. Specifically, the other cluster centers refer to the cluster centers of the categories other than the category to which the current sample belongs in the cluster assignment result.

[0072] The second loss function includes:

[0073]

[0074] wherein, is the number of samples, , is a temperature parameter, is a cosine similarity function, is an intermediate parameter, is the cluster center of the negative sample, is the total number of cluster centers, represents the sum of the cluster centers of the remaining k-1 other samples except the cluster center of the current sample, that is, the above process of using other cluster centers as negative samples.

[0075] The above embodiments of the present invention first use the k-means clustering algorithm to find cluster centers in the depth representation of the reference view. These centers represent the features of different data groups. Next, using the nearest neighbor principle, samples are assigned to corresponding cluster groups based on the similarity between each sample and the cluster center. Finally, a loss function is constructed based on these clustering assignment results. By optimizing this function, samples are made closer to their own cluster centers (positive samples) and further away from other cluster centers (negative samples), thereby improving the accuracy of clustering and the performance of the model.

[0076] Deep learning methods in related technologies (such as MVCLN and SURE) mainly rely on contrastive learning, using random samples as negative samples. This easily leads to false negative pairs (FNPs), where samples that should belong to the same class are misclassified as negative samples, introducing contradictory supervision signals and severely impacting model performance. The embodiments of this invention use other cluster centers as negative samples instead of randomly selecting all other samples, effectively mitigating the false negative pair problem in contrastive learning. This strategy improves clustering accuracy by 5-15%.

[0077] In some embodiments, based on S3, the aforementioned third loss function include:

[0078] in, For the first Adjacency graph of a non-reference view For the first A local adjacency graph of a non-reference view. For regularization parameters, Represents the trace of a matrix. This is the clustering indicator matrix. for The Laplace matrix, For the set of real numbers, The number of samples.

[0079] In some embodiments, based on S3, the aforementioned fourth loss function include:

[0080]

[0081] in, , , For cluster-level embedding of reference views, For the aligned first Cluster-level embedding of non-reference views, is a distance function used to represent the matching or distinguishing degree of the cluster centers of two views, specifically, is a variant of InfoNCE Loss, which represents the degree to which the i-th cluster center can be correctly identified as matching the i-th cluster center in another view while being distinguished from all other cluster centers given view b and view v. is a temperature parameter, is a cosine similarity function.

[0082] In implementation, on the basis of the above S3, the above S5 includes: S51, symmetrically normalizing the local adjacency graph wherein , represents the degree matrix of the graph .

[0083] S52, processing the representation and the normalized graph using a second layer graph convolutional network GCN2 to obtain a cluster indication matrix and : wherein , after being processed by a softmax function, the element represents the probability that the sample belongs to the cluster , represents the first layer graph convolutional network function.

[0084] S53, calculating the cluster-level embedding (cluster center) wherein , each row of the cluster-level embedding matrix represents the aggregated embedding of a cluster.

[0085] S54, constructing a cluster-level contrastive loss (i.e., the above fourth loss function ) to align the cluster center distributions across views.

[0086] The embodiments of the present application ensure the dual consistency of structure and semantics by constructing the above first to fourth loss functions, simultaneously performing graph structure matching and feature representation alignment, and performing contrastive learning at two levels of sample level and cluster level. Compared with the single-layer alignment method in the related art, the NMI (Normalized Mutual Information) index of the present application is improved by an average of 8-12%.

[0087] In implementation, the above unified loss function includes:

[0088] wherein , , is a weight coefficient.

[0089] In some embodiments, the S7 is implemented by two-stage optimization, specifically including: S71, in the joint learning stage, a first-stage optimization is performed: fixing the adaptive weight of each view as , and training and optimizing the permutation matrix and the convolutional network parameters based on the unified loss function to obtain a target permutation matrix, wherein is the number of views. Specifically, the first-stage optimization is used for graph structure matching, feature alignment, sample-level and cluster-level contrastive learning.

[0090] S72, in the weight refinement stage, a second-stage optimization is performed: fixing the current permutation matrix and the convolutional network parameters, and learning and optimizing the adaptive weight of each view by the gradient descent method to obtain a target view weight. Specifically, the second-stage optimization focuses on balancing the contribution of each view.

[0091] The above embodiments of the present application adopt a unified loss function and a two-stage training strategy, avoiding the multi-step optimization of traditional shallow methods and improving the calculation efficiency.

[0092] In specific implementation, the first-stage optimization is performed when iter≤0.6T (one iter represents the process of one forward propagation and one backward propagation of the model in the training process, and epoch is different, because iter is for a single data sample or a small batch of data), and the second-stage optimization is performed when iter>0.6T.

[0093] In specific implementation, when the first-stage optimization is performed, the learning rate can be set as , the parameters , and the Adam optimizer is used to update the parameters.

[0094] In specific implementation, the S72 includes: using the aligned representation to construct a new local adjacency graph , and the optimization target is the third loss function . A phased growth strategy is used to dynamically adjust the neighbor number of the local adjacency graph , specifically including: every t epochs (one epoch represents the process of completing a complete forward propagation and backward propagation of the convolutional neural network on the entire training data set), updating , until reaches the upper limit , and this strategy starts from conservative local connection and gradually expands to reveal potential relationships. The adaptive view weight is learned by the gradient descent method , by minimizing the fusion graph Laplacian, and then realizing the sum of the minimum k non-trivial eigenvalues

[0095] , wherein is the Laplacian matrix of the fusion graph , and is the kth minimum eigenvalue thereof.

[0096] The method of the embodiment of the application automatically identifies high-quality views and gives them higher weights by learning view weights, avoids the negative impact of low-quality views, and ablation experiments show that the adaptive weights improve the ARI (Adjusted Rand index) index by 3-7%.

[0097] In specific implementation, the above S8 includes: S81, a weighted fusion graph is calculated , wherein is calculated based on the optimized target permutation matrix by the method of the above embodiment. The weighted fusion graph fuses multiple adjacency matrices through weights, and then fuses the information of multiple views to form a more comprehensive graph structure.

[0098] S82, a spectral clustering algorithm is performed on the fusion graph to obtain a final clustering assignment result. The spectral clustering algorithm is a clustering method that finds natural groupings of data points by analyzing the Laplacian matrix of a graph.

[0099] S83, the clustering labels of the n samples are output.

[0100] The complete algorithm flow of the embodiment of the application is as follows: Input: multi-view data set , number of samples n , feature dimension , cluster number c , misalignment ratio , number of neighbors , stage growth interval , maximum number of iterations , view weight (initial value ), coefficient ; Output: clustering assignment result 1: initialize the number of neighbors , , select a reference view ; 2: for ​​, construct ; 3: for perform: 4: if then: 5: end loop; 6: end if 7: if then: 8: solve permutation matrix to align graph structure: 9: use get and ; 10: solve permutation matrix ; 11: align representation: ; 12: apply k-means clustering on to get cluster centers ; 13: solve ; 14: use get and ; 15: compute and ; 16: else: 17: fix network structure; 18: learn adaptive weights ; 19: end if 20: compute and ; 21: update network using Adam optimizer to minimize ; 22: if iter mod t = 0 then: 23: update ; 24: end if 25: ; 26: end for 27: perform spectral clustering on weighted graph .

[0101] The embodiments of the present application also perform comparative tests with other clustering methods in the related art, as shown in Figure 2 and Figure 3 . Figure 2For performance comparison of different data sets and methods under view partial alignment and full alignment conditions, Figure 3 For performance comparison of different data sets and methods under view full misalignment conditions), the experimental results show that on six benchmark data sets of ORL, Caltech101-20, Wiki, RGB-D, prokaryotic, MNIST-USPS and the like, the application (Ours) is superior to the latest comparative methods (OpVuC, FUMC, TUMCR, TUMC, MVC-UM, MvCLN, VITAL, SURE, PMVCR) in the related art in three evaluation indexes of ACC (Accuracy, ACC), NMI and ARI, and the advantage is more obvious especially under the full misalignment scene. Moreover, the application keeps stable high performance under three scenes of partial misalignment (η=0.5), full alignment (η=0) and full misalignment (η=1), and exhibits excellent generalization ability. The embodiment of the application supports an arbitrary number of views and sample scales, and realizes good time complexity balance through a phased growth strategy and a sparse graph structure.

[0102] The embodiment of the application also carries out time complexity analysis. The overall time complexity of the method provided by the application is , specifically, consisting of the following parts: 1) graph structure matching: each view needs to perform operations per iteration, totaling ; 2) GCN-based feature learning: two layers of GCN process the graph structure and the features, each view needs to perform per iteration, totaling ; 3) contrast alignment: calculating sample-level and clustering-level contrast loss, each iteration needs to perform , totaling .

[0103] Although the above cubic term seems significant, in practice, the Frank-Wolfe solver can quickly converge due to hot initialization and graph structure sparsity, and the actual calculation cost is significantly lower than the worst-case bound, and the phased neighbor growth mechanism dynamically adjusts the graph sparsity, further reducing the calculation overhead.

[0104] It should be noted that the method of the embodiment of the application can be executed by a single device, such as a computer or a server, etc. The method of the embodiment can also be applied to a distributed scenario, and completed by multiple devices cooperating with each other. In this distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiment of the application, and the multiple devices will interact with each other to complete the above method.

[0105] It should be noted that some embodiments of the present invention have been described above. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0106] Based on the same inventive concept, corresponding to any of the methods in the above embodiments, the present invention also provides a clustering device for misaligned multi-views, such as... Figure 4 As shown, it includes: The acquisition module 10 is used to acquire sensor data including multiple views from at least one sample using multiple different methods, select a reference view from the multiple views, and treat the views other than the reference view as non-reference views. The sensor data acquired by different methods constitute different views, and each view includes sensor data acquired from all samples using the same method. Graph structure construction module 20 is used to construct an adjacency graph for each view and to match the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view through a permutation matrix. The first convolutional network module 30 is used to input each view and its adjacency graph into the first layer graph convolutional network, extract the depth representation of each view, and use the depth representation to construct the first loss function and the second loss function; The local adjacency graph construction module 40 is used to construct a local adjacency graph for each view based on the adjacency graph and depth representation of each view, and to construct a third loss function using the local adjacency graph, while updating the number of neighbors of the local adjacency graph at least once every training period. The second convolutional network module 50 is used to input the depth representation of each view and the local adjacency graph into the second layer graph convolutional network, extract the clustering indicator matrix of each view, and construct the fourth loss function using the clustering indicator matrix. The loss function construction module 60 is used to construct a unified loss function using the first loss function, the second loss function, the third loss function, and the fourth loss function; The optimization module 70 is used to train and optimize the permutation matrix based on a unified loss function to obtain the target permutation matrix, and at the same time learn and optimize the adaptive weights of each view to obtain the target view weights. The result output module 80 is used to obtain the multi-view clustering results of the sensor data based on the target view weights and the target permutation matrix.

[0107] This invention provides an unaligned multi-view clustering device based on graph permutation matrices and two-level contrastive learning. The device first explicitly performs cross-view graph structure matching using permutation matrices, and then, based on this alignment, uses a graph convolutional network to perform two-level contrastive alignment, forcing cross-view alignment at both the sample and cluster levels.Figure One The adaptive local adjacency graph is constructed and the view weight is learned, optimal multi-view fusion is realized, cross-view clustering structure can be efficiently learned even in the case of complete misalignment of samples, and the accuracy and robustness of the misaligned multi-view clustering task are improved. Figure One

[0108] For the convenience of description, the above device is described as various modules in function and is described respectively. Of course, the functions of the modules can be implemented in one or more software and / or hardware when the present application is implemented.

[0109] The device of the above embodiment is used to implement the corresponding misaligned multi-view clustering method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here.

[0110] Figure 5 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown.

[0111] As shown in Figure 5 , the electronic device can include a processor 1101 and a memory 1102 storing computer program instructions.

[0112] Specifically, the above processor 1101 can include a central processing unit (CPU), or a specific integrated circuit (Application Specific Integrated Circuit, ASIC), or can be configured as one or more integrated circuits implementing an embodiment of the present application.

[0113] ​The memory 1102 can include mass storage for information or instructions. For example, and without limitation, the memory 1102 can include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. Storage memory 1102, where appropriate, can include removable or non-removable (or fixed) media. Storage memory 1102 can be internal or external to the integrated gateway device, as appropriate. In particular embodiments, storage memory 1102 is non-volatile solid-state memory. In particular embodiments, storage memory 1102 includes read-only memory (ROM). Where appropriate, this ROM can be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0114] The processor 1101 performs the steps of the method for clustering misaligned multi-views provided by embodiments of the application by reading and executing computer program instructions stored in the memory 1102.

[0115] In one example, the electronic device can further include a transceiver 1103 and a bus 1104. As shown, the processor 1101, the memory 1102, and the transceiver 1103 are connected by the bus 1104 and complete communication among each other. Figure 5

[0116] ​Bus 1104 includes a hardware, software, or both. By way of example and not limitation, such bus can include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side BUS (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand™ interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or another suitable bus or a combination of two or more of these. Where appropriate, bus 1104 can include one or more buses. Although the present embodiment describes and shows a particular bus, the present embodiment contemplates any suitable bus or interconnect.

[0117] The following is an embodiment of a computer-readable storage medium provided by the present embodiment, which belongs to the same inventive concept as the above-mentioned clustering method for misaligned multi-views. Details not described in the embodiment of the computer-readable storage medium can be referred to the above-mentioned embodiment of the clustering method for misaligned multi-views.

[0118] The present embodiment provides a storage medium containing computer executable instructions, which when executed by a computer processor, are used to perform a clustering method for misaligned multi-views.

[0119] Of course, the present embodiment provides a storage medium containing computer executable instructions, which computer executable instructions are not limited to the above-mentioned method operations, but can also perform related operations in the clustering method for misaligned multi-views provided by any embodiment of the present embodiment.

[0120] Through the above description of the embodiments, those skilled in the art can clearly understand that the present application can be realized by means of software and necessary universal hardware, and of course can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a part that contributes to the prior art, and the computer software product can be stored in a computer readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH, a hard disk, or an optical disc, and includes a number of instructions to make a computer cloud platform (which can be a personal computer, a server, or a network cloud platform, etc.) execute the clustering method of misaligned multi-view provided by each embodiment of the present application.

[0121] It should be noted that, in this document, the terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the above element.

[0122] The above is only a specific embodiment of the present application, which enables those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A clustering method for misaligned multi-view clustering, characterized in that, include: Sensor data comprising multiple views is acquired from at least one sample using multiple different methods. A reference view is selected from the multiple views, and views other than the reference view are designated as non-reference views. The sensor data acquired using different methods constitute different views, and each view includes sensor data acquired from all the samples using the same method. An adjacency graph is constructed for each view, and the graph structure of the adjacency graph of the non-reference view is matched with the adjacency graph of the reference view using a permutation matrix; Each view and its adjacency graph are input into the first layer of the graph convolutional network, the depth representation of each view is extracted, and the first loss function and the second loss function are constructed using the depth representation; A local adjacency graph for each view is constructed based on the adjacency graph and the depth representation, and a third loss function is constructed using the local adjacency graph, while updating the number of neighbors in the local adjacency graph at least one training epoch. The depth representation of each view and the local adjacency graph are input into the second layer graph convolutional network to extract the clustering indicator matrix of each view, and the fourth loss function is constructed using the clustering indicator matrix. A unified loss function is constructed using the first loss function, the second loss function, the third loss function, and the fourth loss function; The permutation matrix is ​​trained and optimized based on the unified loss function to obtain the target permutation matrix. At the same time, the adaptive weights of each view are learned and optimized to obtain the target view weights. The multi-view clustering result of the sensor data is obtained based on the target view weight and the target permutation matrix.

2. The method as described in claim 1, characterized in that, in Before matching the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view using a permutation matrix, the method further includes: The Frank-Wolfe algorithm is used based on the formula Solve the permutation matrix ,in, For the first The permutation matrix of the non-reference views For the number of samples, each view is composed of... The data consists of sensor data acquired from each sample. Represents the trace of a matrix. This is the adjacency graph of the reference view. For the first The adjacency graph of the non-reference views; The step of matching the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view using a permutation matrix includes: The adjacency graph of the non-reference view after permutation is obtained through the permutation matrix. .

3. The method as described in claim 1, characterized in that, Each view and its adjacency graph are input into the first layer of the graph convolutional network. The depth representation of each view is extracted, and the first loss function is constructed using the depth representation, including: Each view and its adjacency graph are input into the first layer of the graph convolutional network, and the depth representations of the non-reference view and the reference view are extracted respectively. The depth representation of the non-reference view is aligned with the depth representation of the reference view using a feature permutation matrix. The first loss function is constructed based on the difference between the depth representation of the aligned non-reference view and the depth representation of the reference view; The first loss function include: in, For the number of views, This represents the depth of the reference view. For the aligned first Depth representation of each of the non-reference views This is the clustering indicator matrix.

4. The method as described in claim 3, characterized in that, Each view and its adjacency graph are input into the first layer of the graph convolutional network. The depth representation of each view is extracted, and the second loss function is constructed using the depth representation, including: k-means clustering is applied to the depth representation of the reference view to determine multiple cluster centers; The clustering assignment result for each view is determined based on the cluster centers; The second loss function is constructed based on the clustering assignment results, and cluster centers outside the category to which the current sample belongs in the clustering assignment results are used as negative samples. The second loss function include: in, For the number of samples, for The OK for The OK, For temperature parameters, The cosine similarity function is used. For intermediate parameters, The cluster centers of the negative samples are... The total number of cluster centers.

5. The method as described in claim 3, characterized in that, The third loss function include: in, For the first The adjacency graph of the non-reference views. For the first A local adjacency graph of the non-reference views. For regularization parameters, Represents the trace of a matrix. for The Laplace matrix, For the set of real numbers, The number of samples.

6. The method as described in claim 4, characterized in that, The fourth loss function include: in, , , For cluster-level embedding of the reference view, For the aligned first Cluster-level embedding of the non-reference views This is a distance function used to represent the degree of matching or discriminative difference between the cluster centers of two views. For temperature parameters, This is the cosine similarity function.

7. The method as described in claim 1, characterized in that, The process of training and optimizing the permutation matrix based on the unified loss function to obtain the target permutation matrix, and simultaneously learning and optimizing the adaptive weights of each view to obtain the target view weights, includes: Fix the adaptive weight of each view to Simultaneously, based on the unified loss function, the permutation matrix and convolutional network parameters are trained and optimized to obtain the target permutation matrix, wherein, The number of views; By fixing the current permutation matrix and convolutional network parameters, and simultaneously learning and optimizing the adaptive weights of each view using gradient descent, the target view weights are obtained.

8. A clustering device with misaligned multiple views, characterized in that, include: An acquisition module is configured to acquire sensor data including multiple views from at least one sample using multiple different methods, select a reference view from the multiple views, and treat views other than the reference view as non-reference views, wherein sensor data acquired using different methods constitute different views, and each view includes sensor data acquired from all the samples using the same method. The graph structure construction module is used to construct an adjacency graph for each view and to match the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view through a permutation matrix. The first convolutional network module is used to input each view and its adjacency graph into the first layer graph convolutional network, extract the depth representation of each view, and use the depth representation to construct a first loss function and a second loss function; A local adjacency graph construction module is used to construct a local adjacency graph for each view based on the adjacency graph of each view and the depth representation, and to construct a third loss function using the local adjacency graph, while updating the number of neighbors of the local adjacency graph at least one training cycle. The second convolutional network module is used to input the depth representation of each view and the local adjacency graph into the second layer graph convolutional network, extract the clustering indicator matrix of each view, and construct the fourth loss function using the clustering indicator matrix; The loss function construction module is used to construct a unified loss function using the first loss function, the second loss function, the third loss function, and the fourth loss function; The optimization module is used to train and optimize the permutation matrix based on the unified loss function to obtain the target permutation matrix, and at the same time learn and optimize the adaptive weights of each view to obtain the target view weights; The result output module is used to obtain the multi-view clustering result of the sensor data based on the target view weight and the target permutation matrix.

9. An electronic device, characterized in that, include: processor; A memory for storing executable instructions; wherein the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The storage medium stores a computer program that, when executed by a processor, causes the processor to implement the method described in any one of claims 1 to 7.

Images