Clustering method and device for misaligned multi-view, electronic device and storage medium

Abstract

The application relates to a clustering method and device for misaligned multi-views, electronic equipment and a storage medium, comprising the following steps: constructing an adjacency graph for each view, and matching the graph structure of the adjacency graph of a non-reference view with the adjacency graph of a reference view through a permutation matrix; inputting each view and the adjacency graph thereof into a first-layer graph convolution network to extract a deep representation of each view; constructing a local adjacency graph of each view based on the adjacency graph and the deep representation of each view, and updating the number of neighbors of the local adjacency graph at least once every training period; inputting the deep representation and the local adjacency graph of each view into a second-layer graph convolution network to extract a clustering indication matrix of each view, training and optimizing the permutation matrix based on a unified loss function, simultaneously learning adaptive weights of the views, and obtaining a clustering result, so that a consistent clustering structure across views can be efficiently learned under the condition that samples are misaligned, and the accuracy and robustness of the misaligned multi-view clustering task are improved.

Description

Technical Field

[0001] This invention belongs to the field of data processing, specifically relating to a clustering method, apparatus, electronic device, and storage medium for misaligned multi-view clustering. Background Technology

[0002] Clustering is an unsupervised learning method that aims to divide samples in a dataset into subsets, making samples within the same cluster as similar as possible, while samples in different clusters are as dissimilar as possible. In the field of data processing, much data exists in the form of multiple views. For example, the same sample may have multiple views or modalities obtained from different sensors or preprocessing methods. Multi-view clustering methods in related technologies usually assume that cross-view samples are perfectly aligned, that is, the order of samples across different views is consistent and the correspondence is known. However, in real-world scenarios, due to factors such as sensor differences, independent data acquisition and storage processes, this assumption is often not met, resulting in cross-view sample correspondences being partially known or even completely unknown. This poses a significant challenge to cross-view information fusion, rendering traditional methods ineffective. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a clustering method, apparatus, electronic device, and storage medium for misaligned multi-view clustering.

[0004] This invention includes a clustering method for misaligned multiple views, comprising: acquiring sensor data including multiple views from at least one sample using multiple different methods; selecting a reference view from the multiple views and designating views other than the reference view as non-reference views, wherein the sensor data acquired using different methods constitute different views, and each view includes sensor data acquired from all samples using the same method; constructing an adjacency graph for each view, and matching the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view using a permutation matrix; inputting each view and its adjacency graph into a first-layer graph convolutional network, extracting a depth representation for each view, and constructing a first loss function and a second loss function using the depth representation; and based on the adjacency graph and depth representation of each view... The process involves constructing a local adjacency graph for each view and using this local adjacency graph to build a third loss function, while updating the number of neighbors in the local adjacency graph at least one training epoch. The depth representation of each view and the local adjacency graph are input into a second-layer graph convolutional network to extract a clustering indicator matrix for each view, and a fourth loss function is constructed using this clustering indicator matrix. A unified loss function is constructed using the first, second, third, and fourth loss functions. The permutation matrix is ​​trained and optimized based on the unified loss function to obtain the target permutation matrix, while simultaneously learning and optimizing the adaptive weights of each view to obtain the target view weights. Finally, the multi-view clustering result for the sensor data is obtained based on the target view weights and the target permutation matrix.

[0005] Optionally, before matching the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view using the permutation matrix, the method further includes: employing the Frank-Wolfe algorithm based on the formula... Solve the above permutation matrix ,in, For the first The permutation matrix of the above non-reference views, For the number of samples, each view is composed of... The data consists of sensor data acquired from each sample. Represents the trace of a matrix. This is the adjacency graph of the reference view above. For the first The adjacency graph of the aforementioned non-reference view; the above-mentioned matching of the graph structure of the adjacency graph of the aforementioned non-reference view with the adjacency graph of the aforementioned reference view by means of a permutation matrix includes: obtaining the adjacency graph of the aforementioned non-reference view after permutation by means of the aforementioned permutation matrix. .

[0006] Optionally, each view and its adjacency graph are input into the first layer of the graph convolutional network, the depth representation of each view is extracted, and the first loss function is constructed using the depth representation. This includes: inputting each view and its adjacency graph into the first layer of the graph convolutional network, and extracting the depth representations of the non-reference view and the reference view respectively; aligning the depth representations of the non-reference view and the reference view using a feature permutation matrix; and constructing the first loss function based on the difference between the aligned depth representations of the non-reference view and the reference view. include:

[0007]

[0008] in, For the number of views, This is a depth representation of the reference view above. For the aligned first The depth representation of the aforementioned non-reference views, This is the clustering indicator matrix.

[0009] Optionally, each view and its adjacency graph are input into the first layer of the graph convolutional network described above, the depth representation of each view is extracted, and the second loss function is constructed using the depth representation, including: applying k-means clustering to the depth representation of the reference view to determine cluster centers; determining the clustering assignment result for each view based on the cluster centers; constructing the second loss function based on the clustering assignment result, while using cluster centers outside the category to which the current sample belongs in the clustering assignment result as negative samples; the second loss function... include:

[0010]

[0011]

[0012] in, For the number of samples, , For temperature parameters, The cosine similarity function is used. For intermediate parameters, The cluster centers of the above negative samples, This represents the total number of cluster centers mentioned above.

[0013] Optionally, the third loss function mentioned above include:

[0014]

[0015] in, For the first The adjacency graph of the above non-reference views, For the first The local adjacency graph of the above non-reference views, For regularization parameters, Represents the trace of a matrix. for The Laplace matrix, For the set of real numbers, The number of samples.

[0016] Optionally, the fourth loss function mentioned above include:

[0017]

[0018]

[0019] in, The OK, The OK, The OK, For cluster-level embedding of the reference view, For the cluster-level embedding of the v-th non-reference view after alignment, This is a distance function used to represent the degree of matching or discriminative difference between the cluster centers of two views. For temperature parameters, This is the cosine similarity function.

[0020] Optionally, the above-mentioned permutation matrix is ​​trained and optimized based on the unified loss function to obtain the target permutation matrix. Simultaneously, the adaptive weights of each view are learned and optimized to obtain the target view weights. This includes fixing the adaptive weights of each view to a fixed value. Simultaneously, based on the aforementioned unified loss function, the aforementioned permutation matrix and convolutional network parameters are trained and optimized to obtain the aforementioned target permutation matrix, where, The number of views is given; the current permutation matrix and convolutional network parameters are fixed, and the adaptive weights of each view are learned and optimized through gradient descent to obtain the target view weights.

[0021] Based on the same inventive concept, this invention also provides a clustering apparatus for misaligned multiple views, comprising: an acquisition module, configured to acquire sensor data including multiple views from at least one sample using multiple different methods, select a reference view from the multiple views, and designate views other than the reference view as non-reference views, wherein sensor data acquired using different methods constitute different views, and each view includes sensor data acquired from all samples using the same method; a graph structure construction module, configured to construct an adjacency graph for each view, and match the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view using a permutation matrix; a first convolutional network module, configured to input each view and its adjacency graph into a first-layer graph convolutional network, extract a depth representation of each view, and construct a first loss function and a second loss function using the depth representation; and a local adjacency graph construction module, configured to construct a local adjacency graph based on the adjacency graph of each view. A local adjacency graph for each view is constructed using the graph and the aforementioned depth representation, and a third loss function is constructed using the aforementioned local adjacency graph. Simultaneously, the number of neighbors in the aforementioned local adjacency graph is updated at least once every training epoch. A second convolutional network module is used to input the aforementioned depth representation and the aforementioned local adjacency graph for each view into a second-layer graph convolutional network, extract the clustering indicator matrix for each view, and construct a fourth loss function using the aforementioned clustering indicator matrix. A loss function construction module is used to construct a unified loss function using the aforementioned first loss function, the aforementioned second loss function, the aforementioned third loss function, and the aforementioned fourth loss function. An optimization module is used to train and optimize the aforementioned permutation matrix based on the aforementioned unified loss function to obtain a target permutation matrix, and simultaneously learn and optimize the adaptive weights of each view to obtain target view weights. A result output module is used to obtain the multi-view clustering results for the sensor data based on the aforementioned target view weights and the aforementioned target permutation matrix.

[0022] Based on the same inventive concept, the present invention also provides an electronic device, comprising: a processor; a memory for storing executable instructions; wherein the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement any of the methods described above.

[0023] Based on the same inventive concept, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor performs any of the above-described methods.

[0024] The beneficial effects of this invention are as follows: This invention provides an unaligned multi-view clustering method based on graph permutation matrices and two-level contrastive learning. This method first explicitly performs cross-view graph structure matching using permutation matrices, and then, based on this alignment, uses a graph convolutional network to perform two-level contrastive alignment, forcing cross-view alignment at both the sample and cluster levels.Figure 1 Consistency; simultaneously constructing an adaptive local adjacency graph and learning view weights to achieve optimal multi-view fusion, enabling efficient learning of cross-view fusion even when samples are completely misaligned. Figure 1 The consistent clustering structure improves the accuracy and robustness of misaligned multi-view clustering tasks. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating a clustering method for misaligned multiple views provided in an embodiment of the present invention;

[0028] Figure 2 A comparative chart of experimental data is provided for the method of this invention and the methods in related technologies;

[0029] Figure 3 A comparative chart of experimental data is provided for another embodiment of the present invention and methods in related technologies;

[0030] Figure 4 A schematic diagram of a clustering device with misaligned multi-views provided in an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0032] To better understand the above-mentioned objectives, features, and advantages of the embodiments of the present invention, the solutions of the embodiments of the present invention will be further described below. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0033] Many specific details are set forth in the following description in order to provide a full understanding of the embodiments of the present invention, but the embodiments of the present invention may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of the embodiments of the present invention, and not all embodiments.

[0034] The misaligned multi-view clustering methods in related technologies suffer from the following technical drawbacks: most deep learning methods lack explicit sample permutation mechanisms, relying solely on contrastive learning for implicit alignment, making it difficult to handle completely misaligned multi-view data and neglecting alignment of underlying structural information; furthermore, methods in related technologies typically only perform alignment in the feature space, failing to consider both sample-level and cluster-level alignment simultaneously, leading to misalignment across views. Figure 1 Insufficient stamina.

[0035] In view of this, embodiments of the present invention provide a clustering method for misaligned multiple views, such as... Figure 1 As shown, it includes:

[0036] S1. Sensor data comprising multiple views is acquired from at least one sample using multiple different methods. A reference view is selected from the multiple views, and views other than the reference view are designated as non-reference views. The sensor data acquired using different methods constitute different views, and each view includes sensor data acquired from all the samples using the same method.

[0037] In multi-view learning, a view refers to different representations of data. For example, for image data, one view might be pixel values, while another might be features after edge detection. Each view may contain different information. Sensor data is data generated by sensing devices through sensing, measurement, and transmission, belonging to the data type of the Internet of Things (IoT). It can be divided into real-time status data (such as speed and power) and cumulative data (such as mileage and heat consumption). Time-series data is formed through periodic collection, providing a foundation for data visualization and mining. Sensors convert physical quantities into electrical signals through detection components, and wireless transmission is achieved in conjunction with telemetry units. Real-time processing is achieved by connecting to IoT platforms using protocols such as MQTT. For example, one of the different views mentioned above might include vibration data collected using a common vibration sensor on the sample, while another view might include vibration data collected using a different vibration sensor on the same sample with higher precision or a different data output format. Yet another view might include temperature data collected using a temperature sensor on the same sample. Through multi-view learning, this information can be comprehensively utilized to improve model performance. A reference view is a baseline view selected from multiple views. In subsequent steps, other views can be compared and aligned with this baseline.

[0038] Specifically, the above S1 includes input containing Multiview dataset of samples ,common Each view consists of a set of views, each view derived from... The data consists of sensor data acquired from each sample, meaning each sample includes data from... Sensor data acquired using V different methods, each sample including d features, the th view (i.e., using V different methods) Feature matrix of each view , Let be the set of real numbers, where For the first The feature dimensions of each view. The specific process of selecting a reference view from multiple views includes independently calculating the silhouette coefficient for each view and selecting the view with the largest silhouette coefficient as the reference view. Reference View It has the clearest clustering structure.

[0039] S2. Construct an adjacency graph for each view, and use a permutation matrix to match the graph structure of the adjacency graph of the non-reference view with that of the reference view.

[0040] Adjacency graphs are used to represent the relationships between data points, typically represented by adjacency matrices. The elements of an adjacency matrix represent the connection weights between nodes. Adjacency graphs can capture the structural information of the data. Permutation matrices, on the other hand, are special matrices used to adjust the row or column order of two matrices to make their structures more similar. Adjacency graphs for different views may have different structures. By aligning the adjacency graphs of non-reference views with those of the reference view using permutation matrices, structural consistency across different views can be ensured, facilitating subsequent feature extraction and fusion.

[0041] S3. Input each view and its adjacency graph into the first layer of the graph convolutional network, extract the depth representation of each view, and use the depth representation to construct the first loss function and the second loss function.

[0042] Graph convolutional networks (GCNNs) are neural networks used for graph-structured data. They learn node representations, while depth representations are high-level feature representations extracted from each view of the data. GCNNs learn these high-level feature representations, which are more expressive than the original data and helpful for subsequent clustering tasks. The loss function measures the difference between the model's predictions and the actual values. It can be viewed as the optimization objective of the GCNN. By constructing the loss function, the model's parameters can be optimized to ensure that the extracted depth representations better reflect the structure and features of the data.

[0043] S4. Construct a local adjacency graph for each view based on the adjacency graph and depth representation of each view, and use the local adjacency graph to construct a third loss function, while updating the number of neighbors in the local adjacency graph at least one training cycle interval.

[0044] Local adjacency graphs are finer-grained adjacency graphs built on depth representation and adjacency graphs. They are used to capture local structure. By dynamically updating the number of neighbors in the local adjacency graph, they can adapt to dynamic changes in data and improve the robustness and adaptability of the model.

[0045] S5. Input the depth representation and local adjacency graph of each view into the second layer graph convolutional network, extract the clustering indicator matrix of each view, and use the clustering indicator matrix to construct the fourth loss function.

[0046] The clustering indicator matrix, representing the clustering results of data points, is typically used to evaluate clustering effectiveness and directly reflects the clustering results of the data. By constructing clustering-related loss functions, the model parameters can be further optimized to ensure the quality of the clustering results. The first layer of the graph convolutional network in S3 above is equivalent to aligning the view at the sample level, while the second layer of the graph convolutional network in S5 aligns the view again at the clustering level.

[0047] S6. Construct a unified loss function using the first loss function, the second loss function, the third loss function, and the fourth loss function.

[0048] S7. Train and optimize the permutation matrix based on the unified loss function to obtain the target permutation matrix. At the same time, learn and optimize the adaptive weights of each view to obtain the target view weights.

[0049] The view weights are used to measure the importance of different views, and the optimized permutation matrix is ​​used to adjust the graph structure of the views. In some specific embodiments, by continuously adjusting the permutation matrix, the unified loss function can be minimized or converged, and the resulting target permutation matrix can achieve the best clustering effect. Adaptive weights are used to distinguish the importance of different views in clustering and are optimized based on an adaptive learning objective.

[0050] S8. Obtain the multi-view clustering results of the sensor data based on the target view weights and the target permutation matrix.

[0051] In some specific embodiments, the above clustering results can be cluster labels for each sample, used to represent the cluster category to which each sample belongs. This invention provides an unaligned multi-view clustering method based on graph permutation matrices and two-level contrastive learning. This method first explicitly performs cross-view graph structure matching using a permutation matrix, and then, based on this alignment, uses a graph convolutional network to perform two-level contrastive alignment, forcing cross-view alignment at both the sample and cluster levels. Figure 1 Consistency; simultaneously constructing an adaptive local adjacency graph and learning view weights to achieve optimal multi-view fusion, enabling efficient learning of cross-view fusion even when samples are completely misaligned. Figure 1The consistent clustering structure improves the accuracy and robustness of misaligned multi-view clustering tasks.

[0052] The clustering method for misaligned multi-view data provided in this invention can be widely applied to scenarios where multimodal data has sample misalignment problems, including but not limited to: in security monitoring, access control systems, or mobile device unlocking, the same user may collect biometric data (facial images, thermal imaging, fingerprint spectra) through different sensors (such as cameras, infrared sensors, fingerprint scanners). Due to the independent operation of the devices or different data collection times, the order of data belonging to each sample in the multi-view data is inconsistent. At this time, it is necessary to use the clustering method provided in this invention to perform cluster analysis on the data to improve the recognition accuracy, which is especially suitable for identity grouping in multi-person scenarios.

[0053] In hospital settings, the same patient may generate multiple view data through different examination devices. Due to differences in examination time or equipment, the initial data may be misaligned. In this case, the clustering method provided in this embodiment of the invention is needed to perform cluster analysis on the data, assisting doctors in formulating personalized treatment plans and optimizing the allocation of medical resources; multi-sensor environmental monitoring and anomaly detection.

[0054] In industrial or environmental monitoring, multiple sensors (such as temperature, humidity, and gas concentration sensors) may be deployed in the same area to collect data independently. Due to sensor failure or communication delay, the multi-view data (temperature and humidity data) may not be aligned. In this case, the clustering method provided in this embodiment of the invention is needed to perform cluster analysis on the data, identify abnormal environmental patterns, and improve the robustness of the monitoring system.

[0055] In video surveillance or autonomous driving, the same scene may be captured by multiple cameras (such as visible light, infrared, and depth cameras) to collect multi-view data. Due to differences in camera position or time, the data samples may not be aligned. In this case, it is necessary to use the clustering method provided in the embodiments of the present invention to perform cluster analysis on the data to enhance the environmental perception capability of autonomous driving.

[0056] To enable graph convolutional networks to perform clustering analysis, this embodiment of the invention first constructs an initial adjacency graph as network input. Specifically, S2 above includes processing each view... The initial adjacency graph of the non-reference view is constructed in the following way. and the initial adjacency graph of the reference view. :

[0057] S21. Calculate the Euclidean distance between sample i and sample j: ,in, and Let i and j be the features of sample i and sample j, respectively.

[0058] S22, Solving the graph learning optimization problem: ,in, The connection weights for samples i and j are... For the image The i-th row, This is the regularization parameter.

[0059] S23 computes the edge weights of the graph, reflecting the strength of the association between sample pairs: ,in, This represents the number of neighbors in the adjacency graph, used to control the sparsity of the adjacency graph.

[0060] S24, Through Obtain the symmetric adjacency matrix. Specifically, the adjacency matrix and the adjacency graph are not entirely the same concept, but they are related. For example, an adjacency matrix is ​​typically used in computation, and this adjacency matrix can represent an adjacency graph structure. In practice, the adjacency graph used in subsequent steps can be the symmetric adjacency graph obtained in S24.

[0061] In some embodiments, prior to the step in S2 described above of matching the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view using a permutation matrix, the method further includes:

[0062] The Frank-Wolfe algorithm is used based on the formula Solving the permutation matrix ,in, For the first The permutation matrix of a non-reference view. For the number of samples, each view is composed of... The data consists of sensor data acquired from d samples, with each sample including d features. Represents the trace of a matrix. For the adjacency graph of the reference view, For the first Adjacency graph of a non-reference view.

[0063] The process in S2 above, which uses a permutation matrix to match the graph structure of the adjacency graph of the non-reference view with that of the reference view, includes:

[0064] The adjacency graph after permutation of the non-reference view is obtained using the permutation matrix. .

[0065] A permutation matrix is ​​a special type of square matrix where each row and each column has exactly one element of 1, and the rest are 0. This matrix represents a rearrangement of elements. When a permutation matrix is ​​used to match the graph structure of the adjacency graph of a non-reference view with that of the reference view, the principle is to utilize the row or column transformation capabilities of the permutation matrix to rearrange the vertex order of the non-reference view's adjacency graph. The structure of an adjacency graph can be represented by its adjacency matrix, and the permutation matrix can change the vertex order by left-multiplying or right-multiplying the adjacency matrix, thus making the adjacency matrix of the non-reference view structurally consistent with that of the reference view. In this way, the graph structures of the two views are matched in terms of vertex connectivity. The above embodiments of the present invention, through explicit learning of the permutation matrix, can automatically discover sample matching relationships across views without any prior correspondence, successfully handling completely misaligned multi-view data.

[0066] In specific implementation, S2 above includes:

[0067] S25. For non-reference views Learning permutation matrices Make its adjacency graph The graph structure and the adjacency graph of the reference view The matching process, specifically the calculation formula, includes:

[0068]

[0069] S26. Relax the above NP-hard integer programming problem into a real-valued optimization problem:

[0070]

[0071] S27. Solve the permutation matrix using the Frank-Wolfe algorithm. The algorithm performs global exploration through the Frank-Wolfe step and then refines the local area through projective gradient descent.

[0072] S28. Obtain the permuted graph: (For the reference view, its diagram structure remains unchanged, i.e.) ).

[0073] S29. Perform symmetric normalization on the permuted graph:

[0074]

[0075] in, , For the angle matrix, satisfying , It is the identity matrix. This represents the distance between sample i and sample j in view v.

[0076] In some embodiments, S3 includes:

[0077] S31. Input each view and its adjacency graph into the first layer of the graph convolutional network, and extract the depth representations of the non-reference view and the reference view respectively.

[0078] Specifically, the depth of the reference view is represented as The depth of a non-reference view is represented as , This represents the function of the first layer graph convolutional network. , To hide the representation dimension.

[0079] S32, through the characteristic permutation matrix Align the depth representation of the non-reference view with the depth representation of the reference view: ( ), This represents the depth after alignment with the non-reference view.

[0080] Specifically, before S32, it also includes: constructing a feature-level alignment optimization problem according to the following formula, and solving for the feature permutation matrix. :

[0081]

[0082] in, For matrix The i-th row and j-th column, It can represent pairwise distances in the latent space.

[0083] S33. Construct a first loss function based on the difference between the depth representation of the aligned non-reference view and the depth representation of the reference view, and calculate the feature-level alignment loss.

[0084] The first loss function mentioned above include:

[0085]

[0086] in, For the number of views, For the depth representation of the reference view, For the aligned first Depth representation of a non-reference view This is the clustering indicator matrix.

[0087] The above embodiments of the present invention first input different views and their interrelationships (adjacency graph) into a graph convolutional network to extract the depth features of each view. Then, a reference view is selected, and a feature permutation matrix is ​​solved by constructing a feature-level alignment optimization problem. This matrix is ​​used to adjust the features of non-reference views to align with the features of the reference view. Finally, a loss function is constructed based on the aligned feature differences, and the feature-level alignment loss is calculated. By optimizing this loss function, the effect of view feature alignment is improved, thereby better handling multi-view data.

[0088] In some embodiments, S3 further includes:

[0089] S34. Depth representation in the reference view The above uses k-means clustering to determine cluster centers, specifically to obtain k cluster centers. .

[0090] S35. Determine the clustering assignment for each view based on the cluster centers. Specifically, for each sample i, determine its clustering assignment using the nearest neighbor principle. .

[0091] S36. Construct a second loss function based on the clustering assignment results, while using other cluster centers as negative samples. Specifically, other cluster centers refer to the cluster centers of categories other than the category to which the current sample belongs in the clustering assignment results.

[0092] Second loss function include:

[0093]

[0094]

[0095] in, For the number of samples, , For temperature parameters, The cosine similarity function is used. For intermediate parameters, Cluster centers for negative samples The total number of cluster centers. This means summing the cluster centers of the remaining k-1 samples other than the current sample's cluster center, which is the process described above of using other cluster centers as negative samples.

[0096] The above embodiments of the present invention first use the k-means clustering algorithm to find cluster centers in the depth representation of the reference view. These centers represent the features of different data groups. Next, using the nearest neighbor principle, samples are assigned to corresponding cluster groups based on the similarity between each sample and the cluster center. Finally, a loss function is constructed based on these clustering assignment results. By optimizing this function, samples are made closer to their own cluster centers (positive samples) and further away from other cluster centers (negative samples), thereby improving the accuracy of clustering and the performance of the model.

[0097] Deep learning methods in related technologies (such as MVCLN and SURE) mainly rely on contrastive learning, using random samples as negative samples. This easily leads to false negative pairs (FNPs), where samples that should belong to the same class are misclassified as negative samples, introducing contradictory supervision signals and severely impacting model performance. The embodiments of this invention use other cluster centers as negative samples instead of randomly selecting all other samples, effectively mitigating the false negative pair problem in contrastive learning. This strategy improves clustering accuracy by 5-15%.

[0098] In some embodiments, based on S3, the aforementioned third loss function include:

[0099]

[0100] in, For the first Adjacency graph of a non-reference view For the first A local adjacency graph of a non-reference view. For regularization parameters, Represents the trace of a matrix. This is the clustering indicator matrix. for The Laplace matrix, For the set of real numbers, The number of samples.

[0101] In some embodiments, based on S3, the aforementioned fourth loss function include:

[0102]

[0103]

[0104] in, , , For cluster-level embedding of reference views, For the aligned first Cluster-level embedding of non-reference views, This is a distance function used to represent the degree of matching or discriminative difference between the cluster centers of two views. Specifically, It is a variant of InfoNCE Loss, which represents the extent to which the i-th cluster center can be correctly identified as matching the i-th cluster center in another view, given views b and v, while also being distinguishable from all other cluster centers. For temperature parameters, This is the cosine similarity function.

[0105] In specific implementation, based on the above S3, the above S5 includes:

[0106] S51. Perform symmetric normalization on the local adjacency graph. ,in , Representation diagram The degree matrix.

[0107] S52. Use the second-layer graph convolutional network GCN2 to process the representation and normalized graph to obtain the clustering indicator matrix. and : ,in After processing by the softmax function, the elements Indicates sample Belongs to clustering The probability, This represents the function of the first layer graph convolutional network.

[0108] S53. Calculate cluster-level embeddings (cluster centers) ,in, Each row of the cluster-level embedding matrix represents the aggregate embedding of a cluster.

[0109] S54. Construct cluster-level contrastive loss (i.e., the fourth loss function mentioned above). ), aligning the distribution of cluster centers across views.

[0110] This invention constructs the first to fourth loss functions described above, simultaneously performs graph structure matching and feature representation alignment, and executes contrastive learning at both the sample and cluster levels to ensure dual consistency of structure and semantics. Compared to single-layer alignment methods in related technologies, the NMI (Normalized Mutual Information) metric of this invention is improved by an average of 8-12%.

[0111] In practice, the aforementioned unified loss function includes:

[0112]

[0113] in, , , These are the weighting coefficients.

[0114] In some embodiments, the above-mentioned S7 is implemented in a two-stage optimization manner, specifically including:

[0115] S71. In the joint learning phase, perform the first phase of optimization: fix the adaptive weights of each view to be... Simultaneously, based on a unified loss function, the permutation matrix and convolutional network parameters are trained and optimized to obtain the target permutation matrix, where, This represents the number of views. Specifically, the first stage of optimization is used for graph structure matching, feature alignment, and sample-level and cluster-level comparative learning.

[0116] S72. In the weight refinement stage, the second stage of optimization is performed: the current permutation matrix and convolutional network parameters are fixed, and the adaptive weights of each view are learned and optimized through gradient descent to obtain the target view weights. Specifically, the second stage of optimization focuses on balancing the contributions of each view.

[0117] The embodiments of the present invention employ a unified loss function and a two-stage training strategy, which avoids the multi-step optimization of traditional shallow methods and improves computational efficiency.

[0118] In practice, the first stage of optimization is performed when iter ≤ 0.6T (an iter represents the process of the model performing one forward propagation and one backward propagation on the data during training. Unlike epoch, iter is for a single data sample or a small batch of data), and the second stage of optimization is performed when iter > 0.6T.

[0119] In practice, when performing the first stage of optimization described above, a learning rate can be set. ,parameter Then, update the parameters using the Adam optimizer.

[0120] In specific implementation, S72 above includes:

[0121] Use the aligned representation Construct a new local adjacency graph The optimization objective is the third loss function mentioned above. A phased growth strategy is adopted to dynamically adjust the local adjacency graph. The number of neighbors, specifically including: updating every t epochs (one epoch represents the process of the convolutional neural network completing one full forward and backward propagation on the entire training dataset), until Reaching the upper limit This strategy starts with conservative local connections and gradually expands to reveal underlying relationships. Adaptive view weights are learned using gradient descent. By minimizing the Laplacian matrix of the fusion graph, the minimum k nontrivial eigenvalues ​​are obtained. sum:

[0122]

[0123] in For fusion graph The Laplace matrix, For its first The smallest eigenvalue.

[0124] The method described in this embodiment of the invention learns view weights, automatically identifies high-quality views and assigns them higher weights, avoiding the negative impact of low-quality views. Ablation experiments show that adaptive weights improve the ARI (Adjusted Rand index) index by 3-7%.

[0125] In specific implementation, the above S8 includes:

[0126] S81, Calculate the weighted fusion graph At this time The optimized target permutation matrix is ​​calculated using the method described in the above embodiment. The weighted fusion graph fuses multiple adjacency matrices through weights, thereby fusing information from multiple views to form a more comprehensive graph structure.

[0127] S82, regarding the fusion diagram The spectral clustering algorithm is executed to obtain the final clustering results. Spectral clustering is a clustering method that finds natural groups of data points by analyzing the Laplacian matrix of a graph.

[0128] S83, Output the cluster labels for n samples.

[0129] The complete algorithm flow of this invention embodiment is as follows:

[0130] Input: Multiview dataset Sample size n Feature Dimension , the number of clusters c Misalignment Number of neighbors Stage growth interval Maximum number of iterations View weight (Initial value) ),coefficient ;

[0131] Output: Clustering assignment results;

[0132] 1: Initialize the number of neighbors , Select Reference View ;

[0133] 2: Regarding , build ;

[0134] 3: Regarding implement:

[0135] 4: If but:

[0136] 5: Terminate the loop;

[0137] 6: End if:

[0138] 7: If but:

[0139] 8: Solving for the permutation matrix Align the diagram structure:

[0140] 9: Use get and ;

[0141] 10: Solving for the permutation matrix ;

[0142] 11: Alignment representation: ;

[0143] 12: In The cluster centers were obtained by applying k-means clustering. ;

[0144] 13: Solve ;

[0145] 14: Use get and ;

[0146] 15: Calculation and ;

[0147] 16: Otherwise:

[0148] 17: Fixed network structure;

[0149] 18: Learning Adaptive Weights ;

[0150] 19: End if:

[0151] 20: Calculation and ;

[0152] 21: Minimize using the Adam optimizer Update the network;

[0153] 22: If iter mod t = 0, then:

[0154] 23: Update ;

[0155] 24: End if:

[0156] 25: ;

[0157] 26: End of discussion for:

[0158] 27: Weighted graph Perform spectral clustering.

[0159] The embodiments of this invention also underwent comparative experiments with other clustering methods in related technologies, such as... Figure 2 and Figure 3 As shown ( Figure 2 To compare the performance of different datasets and methods under conditions of partial alignment and full alignment in the view, Figure 3 To compare the performance of different datasets and methods under the condition of complete view misalignment, experimental results show that on six benchmark datasets including ORL, Caltech101-20, Wiki, RGB-D, prokaryotic, and MNIST-USPS, our invention (Ours) outperforms the latest comparable methods (OpVuC, FUMC, TUMCR, TUMC, MVC-UM, MvCLN, VITAL, SURE, PMVCR) in three evaluation metrics: ACC (Accuracy), NMI, and ARI, especially in the case of complete misalignment. Furthermore, our invention maintains stable high performance in three scenarios: partial misalignment (η=0.5), complete alignment (η=0), and complete misalignment (η=1), demonstrating excellent generalization ability. The embodiments of our invention support any number of views and sample sizes, achieving a good balance between time complexity through a phased growth strategy and a sparse graph structure.

[0160] The embodiments of this invention also include a time complexity analysis. The overall time complexity of the method provided by this invention is... Specifically, it consists of the following parts: 1) Graph structure matching: Each view needs to be matched in each iteration. Operations, Total 2) Feature learning based on GCN: Two-layer GCN processes graph structure and features, with each view iterating in each iteration. ,total 3) Contrast Alignment: Calculate the sample-level and cluster-level contrastive loss in each iteration. ,total .

[0161] Although the above cubic terms While seemingly significant, in practice, the Frank-Wolfe solver converges quickly due to its hot initialization and graph sparsity, and its actual computational cost is significantly lower than the worst-case bound. The phased neighbor growth mechanism dynamically adjusts the graph sparsity, further reducing computational overhead.

[0162] It should be noted that the method of this embodiment can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of this embodiment, and the multiple devices will interact with each other to complete the above method.

[0163] It should be noted that some embodiments of the present invention have been described above. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0164] Based on the same inventive concept, corresponding to any of the methods in the above embodiments, the present invention also provides a clustering device for misaligned multi-views, such as... Figure 4 As shown, it includes:

[0165] The acquisition module 10 is used to acquire sensor data including multiple views from at least one sample using multiple different methods, select a reference view from the multiple views, and treat the views other than the reference view as non-reference views, wherein the sensor data acquired by different methods constitute different views, and each view includes sensor data acquired from all samples using the same method.

[0166] Graph structure construction module 20 is used to construct an adjacency graph for each view and to match the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view through a permutation matrix.

[0167] The first convolutional network module 30 is used to input each view and its adjacency graph into the first layer graph convolutional network, extract the depth representation of each view, and use the depth representation to construct the first loss function and the second loss function;

[0168] The local adjacency graph construction module 40 is used to construct a local adjacency graph for each view based on the adjacency graph and depth representation of each view, and to construct a third loss function using the local adjacency graph, while updating the number of neighbors of the local adjacency graph at least once every training period.

[0169] The second convolutional network module 50 is used to input the depth representation of each view and the local adjacency graph into the second layer graph convolutional network, extract the clustering indicator matrix of each view, and construct the fourth loss function using the clustering indicator matrix.

[0170] The loss function construction module 60 is used to construct a unified loss function using the first loss function, the second loss function, the third loss function, and the fourth loss function;

[0171] The optimization module 70 is used to train and optimize the permutation matrix based on a unified loss function to obtain the target permutation matrix, and at the same time learn and optimize the adaptive weights of each view to obtain the target view weights.

[0172] The result output module 80 is used to obtain the multi-view clustering results of the sensor data based on the target view weights and the target permutation matrix.

[0173] This invention provides an unaligned multi-view clustering device based on graph permutation matrices and two-level contrastive learning. The device first explicitly performs cross-view graph structure matching using permutation matrices, and then, based on this alignment, uses a graph convolutional network to perform two-level contrastive alignment, forcing cross-view alignment at both the sample and cluster levels. Figure 1 Consistency; simultaneously constructing an adaptive local adjacency graph and learning view weights to achieve optimal multi-view fusion, enabling efficient learning of cross-view fusion even when samples are completely misaligned. Figure 1 The consistent clustering structure improves the accuracy and robustness of misaligned multi-view clustering tasks.

[0174] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing this invention, the functions of each module can be implemented in one or more software and / or hardware components.

[0175] The apparatus of the above embodiments is used to implement the clustering method of the corresponding misaligned multi-view in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0176] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown.

[0177] like Figure 5 As shown, the electronic device may include a processor 1101 and a memory 1102 storing computer program instructions.

[0178] Specifically, the processor 1101 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention.

[0179] Memory 1102 may include a mass storage device for information or instructions. For example, and not limitingly, memory 1102 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 1102 may include removable or non-removable (or fixed) media. Where appropriate, memory 1102 may be internal or external to the integrated gateway device. In a particular embodiment, memory 1102 is a non-volatile solid-state memory. In a particular embodiment, memory 1102 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0180] The processor 1101 performs the steps of the clustering method for misaligned multi-view provided in the embodiments of the present invention by reading and executing computer program instructions stored in the memory 1102.

[0181] In one example, the electronic device may also include a transceiver 1103 and a bus 1104. Wherein, as... Figure 5 As shown, the processor 1101, memory 1102 and transceiver 1103 are connected via bus 1104 and communicate with each other.

[0182] Bus 1104 may include hardware, software, or both. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 1104 may include one or more buses. Although specific buses are described and illustrated in the embodiments of the present invention, the present invention is contemplated by any suitable bus or interconnect.

[0183] The following are embodiments of a computer-readable storage medium provided in this invention. This computer-readable storage medium and the clustering method for misaligned multiple views in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the computer-readable storage medium, please refer to the embodiments of the clustering method for misaligned multiple views described above.

[0184] This embodiment provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a clustering method for misaligned multi-view clustering.

[0185] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the above-described method operations, but can also perform related operations in the clustering method for misaligned multi-views provided in any embodiment of the present invention.

[0186] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer cloud platform (which can be a personal computer, server, or network cloud platform, etc.) to execute the clustering method for misaligned multi-views provided in the various embodiments of the present invention.

[0187] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the aforementioned element.

[0188] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments described above, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A clustering method for misaligned multi-view clustering, characterized in that, Applications in security monitoring systems include: Sensor data comprising multiple views is acquired from at least one sample using multiple different methods. A reference view is selected from the multiple views, and views other than the reference view are designated as non-reference views. The sensor data acquired using different methods constitute different views, and each view includes sensor data acquired from all the samples using the same method. The sensor data comprising multiple views acquired using multiple different methods includes facial images captured by a camera, thermal images captured by an infrared sensor, and fingerprint patterns captured by a fingerprint scanner. An adjacency graph is constructed for each view, and the graph structure of the adjacency graph of the non-reference view is matched with the adjacency graph of the reference view using a permutation matrix; Each view and its adjacency graph are input into the first layer of the graph convolutional network, the depth representation of each view is extracted, and the first loss function and the second loss function are constructed using the depth representation; A local adjacency graph for each view is constructed based on the adjacency graph and the depth representation, and a third loss function is constructed using the local adjacency graph, while updating the number of neighbors in the local adjacency graph at least one training epoch. The depth representation of each view and the local adjacency graph are input into the second layer graph convolutional network to extract the clustering indicator matrix of each view, and the fourth loss function is constructed using the clustering indicator matrix. A unified loss function is constructed using the first loss function, the second loss function, the third loss function, and the fourth loss function; The permutation matrix is ​​trained and optimized based on the unified loss function to obtain the target permutation matrix. At the same time, the adaptive weights of each view are learned and optimized to obtain the target view weights. The multi-view clustering result of the sensor data is obtained based on the target view weight and the target permutation matrix; wherein, the multi-view clustering result is used for identity grouping of the security monitoring system in multi-person scenarios.

2. The method as described in claim 1, characterized in that, in Before matching the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view using a permutation matrix, the method further includes: The Frank-Wolfe algorithm is used based on the formula Solve the permutation matrix ,in, For the first The permutation matrix of the non-reference views For the number of samples, each view is composed of... The data consists of sensor data acquired from each sample. Represents the trace of a matrix. This is the adjacency graph of the reference view. For the first The adjacency graph of the non-reference views; The step of matching the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view using a permutation matrix includes: The adjacency graph of the non-reference view after permutation is obtained through the permutation matrix. .

3. The method as described in claim 1, characterized in that, Each view and its adjacency graph are input into the first layer of the graph convolutional network. The depth representation of each view is extracted, and the first loss function is constructed using the depth representation, including: Each view and its adjacency graph are input into the first layer of the graph convolutional network, and the depth representations of the non-reference view and the reference view are extracted respectively. The depth representation of the non-reference view is aligned with the depth representation of the reference view using a feature permutation matrix. The first loss function is constructed based on the difference between the depth representation of the aligned non-reference view and the depth representation of the reference view; The first loss function include: in, For the number of views, This represents the depth of the reference view. For the aligned first Depth representation of each of the non-reference views This is the clustering indicator matrix.

4. The method as described in claim 3, characterized in that, Each view and its adjacency graph are input into the first layer of the graph convolutional network. The depth representation of each view is extracted, and the second loss function is constructed using the depth representation, including: k-means clustering is applied to the depth representation of the reference view to determine multiple cluster centers; The clustering assignment result for each view is determined based on the cluster centers; The second loss function is constructed based on the clustering assignment results, and cluster centers outside the category to which the current sample belongs in the clustering assignment results are used as negative samples. The second loss function include: in, For the number of samples, for The OK for The OK, For temperature parameters, The cosine similarity function is used. For intermediate parameters, The cluster centers of the negative samples are... The total number of cluster centers.

5. The method as described in claim 3, characterized in that, The third loss function include: in, For the first The adjacency graph of the non-reference views. For the first A local adjacency graph of the non-reference views. For regularization parameters, Represents the trace of a matrix. for The Laplace matrix, For the set of real numbers, The number of samples.

6. The method as described in claim 4, characterized in that, The fourth loss function include: in, , , For cluster-level embedding of the reference view, For the aligned first Cluster-level embedding of the non-reference views This is a distance function used to represent the degree of matching or discriminative difference between the cluster centers of two views. For temperature parameters, This is the cosine similarity function.

7. The method as described in claim 1, characterized in that, The process of training and optimizing the permutation matrix based on the unified loss function to obtain the target permutation matrix, and simultaneously learning and optimizing the adaptive weights of each view to obtain the target view weights, includes: Fix the adaptive weight of each view to Simultaneously, based on the unified loss function, the permutation matrix and convolutional network parameters are trained and optimized to obtain the target permutation matrix, wherein, The number of views; By fixing the current permutation matrix and convolutional network parameters, and simultaneously learning and optimizing the adaptive weights of each view using gradient descent, the target view weights are obtained.

8. A clustering device with misaligned multiple views, characterized in that, Applications in security monitoring systems include: An acquisition module is configured to acquire sensor data including multiple views from at least one sample using multiple different methods, select a reference view from the multiple views, and treat views other than the reference view as non-reference views. The sensor data acquired using different methods constitute different views, and each view includes sensor data acquired from all the samples using the same method. The sensor data including multiple views acquired using multiple different methods includes facial images captured by a camera, thermal images captured by an infrared sensor, and fingerprint patterns captured by a fingerprint scanner. The graph structure construction module is used to construct an adjacency graph for each view and to match the graph structure of the adjacency graph of the non-reference view with the adjacency graph of the reference view through a permutation matrix. The first convolutional network module is used to input each view and its adjacency graph into the first layer graph convolutional network, extract the depth representation of each view, and use the depth representation to construct a first loss function and a second loss function; A local adjacency graph construction module is used to construct a local adjacency graph for each view based on the adjacency graph of each view and the depth representation, and to construct a third loss function using the local adjacency graph, while updating the number of neighbors of the local adjacency graph at least one training cycle. The second convolutional network module is used to input the depth representation of each view and the local adjacency graph into the second layer graph convolutional network, extract the clustering indicator matrix of each view, and construct the fourth loss function using the clustering indicator matrix; The loss function construction module is used to construct a unified loss function using the first loss function, the second loss function, the third loss function, and the fourth loss function; The optimization module is used to train and optimize the permutation matrix based on the unified loss function to obtain the target permutation matrix, and at the same time learn and optimize the adaptive weights of each view to obtain the target view weights; The result output module is used to obtain the multi-view clustering result of the sensor data based on the target view weight and the target permutation matrix; wherein the multi-view clustering result is used for identity grouping of the security monitoring system in a multi-person scenario.

9. An electronic device, characterized in that, include: processor; A memory for storing executable instructions; wherein the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The storage medium stores a computer program that, when executed by a processor, causes the processor to implement the method described in any one of claims 1 to 7.

Images