Multi-kernel subspace clustering method based on variance-covariance subspace distance

Abstract

This invention relates to the field of machine learning and pattern recognition, specifically disclosing a multi-kernel subspace clustering method based on variance-covariance subspace distance, comprising: S1, acquiring a high-dimensional data matrix, calculating the variance-covariance matrix and its second-order statistical information, defining a preliminary VCSD kernel value and obtaining the VCSD kernel matrix through exponential transformation and normalization, and constructing a multi-kernel pool with r basic kernels; S2, constructing an optimization objective function based on non-negative matrix factorization, and solving for the local affinity feature map of the block diagonal structure under constraints; S3, capturing high-order feature associations through matrix transformation, converting tensor optimization into matrix operations; S4, solving the objective function using an alternating optimization strategy to reduce computational complexity and generate a target affinity map; S5, clustering the target affinity map and outputting the results. The technical solution proposed in this invention solves the problems of missing statistical characteristics, computational complexity, and kernel noise interference in existing methods, effectively improving clustering accuracy and efficiency.

Description

Technical Field

[0001] This invention relates to the field of machine learning and pattern recognition technology, specifically to a multi-kernel subspace clustering method based on variance-covariance subspace distance. Background Technology

[0002] Multi-kernel subspace clustering (MKSC) has significant application value in high-dimensional nonlinear data processing scenarios and can be widely used in practical tasks such as face recognition and image clustering. Unlike traditional single-kernel clustering methods, which have limitations in adapting to complex data structures, MKSC, by fusing information from multiple basis kernel functions, can more efficiently capture the nonlinear characteristics of data and shows significant advantages in clustering small- to medium-sized high-dimensional data.

[0003] Among existing high-dimensional data clustering techniques, multi-kernel subspace clustering is one of the research hotspots. Several research teams have proposed different improved MKSC methods. In 2024, Zhao et al. proposed the Pure Kernel Graph Fusion Tensor Quantum Space Clustering (PKGT) method, which constructs local affinity feature maps of block diagonal structures through non-negative matrix factorization, achieving excellent performance. In 2021, Zhang et al. proposed Low-Rank Robust Multi-Kernel Subspace Clustering (LRMKSC), which applies low-rank constraints using weighted truncated Schatten-p norms. In 2023, Wang et al. developed the Late-stage Fusion Multi-Kernel Subspace Clustering and Proxy Graph Refinement (LFMKC-PGR) method, which constructs a self-expression matrix for each basic kernel.

[0004] In terms of subspace distance metrics, Karami et al. proposed Variance-Covariance Subspace Distance (VCSD) in 2023, which evaluates feature importance by calculating the norm of the variance-covariance matrix of the deleted features and has outstanding performance in the field of unsupervised feature selection. Other methods based on neural networks, such as deep subspace clustering, can handle complex data associations but rely on a large amount of labeled data.

[0005] These MKSC methods all employ a hierarchical framework of "base kernel capture - consistent kernel optimization - affinity graph update." These methods divide the clustering task into three processing layers: First, at the highest layer, multiple base kernel functions (linear kernel, polynomial kernel, Gaussian kernel, etc.) are used to capture different nonlinear characteristics of the data; then, in the middle layer, a weight allocation strategy is designed, and an optimization algorithm learns the optimal consistent kernel to fuse information from different kernel matrices; finally, at the bottom layer, the affinity graph is iteratively updated based on the learned consistent kernel for subsequent spectral clustering. This hierarchical processing framework enables the MKSC method to effectively handle complex nonlinear data structures and effectively handle nonlinear data in specific scenarios.

[0006] However, although these methods improve the performance of single-core methods through multi-core fusion, they still have many limitations: the similarity measure of traditional MKSC methods mainly relies on Euclidean distance, principal angle, etc., lacking statistical consideration of data variance distribution and covariance relationship between features, resulting in limited feature representation ability and sensitivity to noise; methods such as PKGT require explicit tensor operations when capturing high-order feature associations, with a computational complexity as high as O(n^2). 3 However, it is difficult to adapt to large-scale data; most MKSC methods aim to optimize the optimal consistent kernel rather than directly construct the optimal affinity graph, and indirect optimization is prone to introducing kernel noise that continuously interferes with the clustering process. At the same time, although VCSD is effective in feature selection, it has not yet been applied to multi-kernel subspace clustering, and there is a gap in the application of statistical augmentation techniques; deep subspace clustering faces practical limitations such as high computational complexity, long training time, and dependence on labeled data.

[0007] Therefore, how to construct a multi-kernel subspace clustering framework that integrates statistical information, has low computational complexity, and can avoid kernel noise interference has become a key and difficult issue that urgently needs to be addressed in the field of high-dimensional data analysis. Summary of the Invention

[0008] To address the aforementioned problems in existing technologies, this invention provides a multi-kernel subspace clustering method based on variance-covariance subspace distance, which effectively solves the problem of capturing high-order feature associations through matrix methods and effectively improves clustering accuracy and efficiency.

[0009] To achieve the above objectives, this invention proposes a multi-kernel subspace clustering method based on variance-covariance subspace distance, comprising: S1. Obtain the high-dimensional data matrix Constructing the VCSD kernel matrix and multi-core pool: Calculating the variance-covariance matrix of the data. and its second-order statistical information Define the initial VCSD kernel value between samples and perform exponential transformation and normalization to form the VCSD kernel matrix. At the same time, integrate linear kernel, polynomial kernel and Gaussian kernel to construct a multi-kernel pool containing r basic kernel matrices. The high-dimensional data matrix The data are derived from face image recognition datasets, including Yale, ORL, AR, Umist, orlraws10P, and warpPIE10. S2, Feature matrix after nonnegative matrix factorization kernel mapping Furthermore, a local affinity feature map is constructed: using the base kernel matrix in the multi-kernel pool as input, an optimization objective function is constructed, and under constraints of nonnegativity, symmetry, and row randomness, a local affinity feature map with a block diagonal structure is obtained by solving the problem. ; S3. Capturing high-order feature correlations through matrix methods: Introducing representation matrices Similarity matrix Together with the global consistency matrix Z, we construct a comprehensive objective function that includes reconstruction error, local consistency, global consistency, and VCSD regularization terms. This transforms the original tensor optimization into matrix operations. The original tensor consists of r local affinity feature maps. Stacked together; S4. Solve the objective function using an alternating optimization strategy: update while keeping other variables constant. , And Z, reduce computational complexity and generate target affinity graph; S5. Perform spectral clustering on the target affinity map and output the final clustering results.

[0010] Preferably, in S1, the specific steps for defining the initial VCSD kernel value between samples and performing exponential transformation for normalization include: S11. Calculate any two samples and Difference vector The initial VCSD core value is defined as follows: ; In the formula, It is a variance-covariance matrix. This is second-order statistical information. ∈[0,1] represents the parameters that balance first-order and second-order statistical information. For two samples and The difference vector; S12. Perform an exponential transformation and normalization on the initial kernel value to obtain the positive definite VCSD kernel matrix: .

[0011] Preferably, in S1, the r basic kernel matrices included in the multi-core pool are: VCSD kernel matrix, linear kernel matrix, polynomial kernel matrices of different degrees, and Gaussian kernel matrices of different bandwidths.

[0012] Preferably, in S2, the optimization objective function and constraints of the local affinity feature map are as follows: ; ; In the formula, For the first There are basis kernel matrices, where Tr represents the trace operation. It is a vector of all 1s. For the first A local affinity feature map.

[0013] Preferably, the In the consistency index matrix The matrix is ​​an approximately symmetric projection matrix, and satisfies the following conditions: .

[0014] Preferably, in S3, the comprehensive objective function is as follows: ; In the formula, Let λ1, β, γ, and ρ be the fundamental matrix of VCSD, and let λ1, β, γ, and ρ be the regularization parameters. For the first There are three basic kernel matrices, λ1∈[0.1,10], β∈[0.01,1], γ∈[0.001,0.1], ρ∈[0,1], It is the Frobenius norm.

[0015] Preferably, in S4, the specific steps of the alternating optimization strategy include: S41, Fixed And Z, solved by eigenvalue decomposition The optimal solution; S42, Fixed And Z, will Update the projection onto the simplex constraint space; S43, Fixed and The updated value of Z is calculated by weighted average; S44. Repeat S41-S43 until the objective function converges, with the number of convergence iterations not exceeding 15.

[0016] Preferably, in S4, the computational complexity is reduced from O(n^2) to O(n^2). 3 ) decreased to O(rn) 2 ), where r is the number of base kernels in the multi-kernel pool and r∈[5,20], and n is the number of samples.

[0017] Preferably, in S5, the spectral clustering process is based on the eigenvalue decomposition of the target affinity graph, and the evaluation metrics for the clustering results include accuracy (ACC), normalized mutual information (NMI), and purity.

[0018] Therefore, this invention proposes a multi-kernel subspace clustering method based on variance-covariance subspace distance, which has the following advantages: (1) Enhance the discriminative power of similarity measurement: integrate variance-covariance statistical characteristics to provide a solid statistical foundation for similarity assessment. It can effectively capture the variance distribution of data and the covariance relationship between features, improve the limited feature expression ability of traditional measurement methods, and reduce the sensitivity of the model to noise.

[0019] (2) Improve computational efficiency and scenario adaptability: By using a matrix-based high-order association capture mechanism, the method replaces explicit tensor operations with efficient matrix operations, significantly reducing computational complexity, breaking through the bottleneck of large-scale data processing, and enabling the method to be more widely adapted to high-dimensional big data clustering scenarios.

[0020] (3) Enhance the robustness of the clustering process: Based on non-negative matrix factorization, a local affinity feature map with a block diagonal structure is directly constructed, avoiding the path of traditional methods that indirectly update the affinity map by optimizing the consistent kernel. This reduces the continuous interference of kernel noise on the clustering process from the root and improves the reliability of the clustering results.

[0021] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0022] Figure 1 This is a system architecture diagram of the multi-kernel subspace clustering method based on variance-covariance subspace distance of the present invention; Figure 2 This is a schematic diagram of the matrix-based high-order association capture mechanism of the multi-kernel subspace clustering method based on variance-covariance subspace distance of the present invention. Detailed Implementation

[0023] To make the technical solutions, advantages, and objectives of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below. The described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the protection scope of this application.

[0024] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0025] like Figures 1-2 As shown, the multi-kernel subspace clustering method based on variance-covariance subspace distance provided by this invention includes: S1. Obtain the high-dimensional data matrix Constructing the VCSD kernel matrix and multi-core pool: Calculating the variance-covariance matrix of the data. and its second-order statistical information The initial VCSD kernel value between samples is defined and subjected to exponential transformation and normalization to form a VCSD kernel matrix. At the same time, linear kernel, polynomial kernel and Gaussian kernel are fused to construct a multi-kernel pool containing r basic kernel matrices. The r basic kernel matrices contained in the multi-kernel pool are: VCSD kernel matrix, linear kernel matrix, polynomial kernel matrix of different degrees and Gaussian kernel matrix of different bandwidths. High-dimensional data matrix The data are derived from face image recognition datasets, including Yale, ORL, AR, Umist, orlraws10P, and warpPIE10. In S1, the specific steps for defining the initial VCSD kernel value between samples and performing exponential transformation for normalization include: S11. Calculate any two samples and Difference vector The initial VCSD core value is defined as follows: ; In the formula, It is a variance-covariance matrix. This is second-order statistical information. ∈[0,1] represents the parameters that balance first-order and second-order statistical information. For two samples and The difference vector; S12. Perform an exponential transformation and normalization on the initial kernel value to obtain the positive definite VCSD kernel matrix: ; S2, Feature matrix after nonnegative matrix factorization kernel mapping Furthermore, a local affinity feature map is constructed: using the base kernel matrix in the multi-kernel pool as input, an optimization objective function is constructed, and under constraints of nonnegativity, symmetry, and row randomness, a local affinity feature map with a block diagonal structure is obtained by solving the problem. The specific objective function and constraints for optimizing the local affinity feature map are as follows: ; ; In the formula, For the first There are basis kernel matrices, where Tr represents the trace operation. It is a vector of all 1s. For the first A local affinity feature map, in which... In the consistency index matrix The matrix is ​​an approximately symmetric projection matrix, and satisfies the following conditions: ; S3. Capturing high-order feature correlations through matrix methods: Introducing representation matrices Similarity matrix Together with the global consistency matrix Z, we construct a comprehensive objective function that includes reconstruction error, local consistency, global consistency, and VCSD regularization terms, transforming the original tensor optimization into matrix operations. The comprehensive objective function is as follows: ; In the formula, Let λ1, β, γ, and ρ be the fundamental matrix of VCSD, and let λ1, β, γ, and ρ be the regularization parameters. For the first There are three basic kernel matrices, λ1∈[0.1,10], β∈[0.01,1], γ∈[0.001,0.1], ρ∈[0,1], It is the Frobenius norm; S4. Solve the objective function using an alternating optimization strategy: update while keeping other variables constant. , And Z, reducing computational complexity to generate the target affinity graph; computational complexity reduced from O(n 3 ) decreased to O(rn) 2 ), where r is the number of basic kernels in the multi-kernel pool, and r∈[5,20], and n is the number of samples; In S4, the specific steps of the alternating optimization strategy include: S41, Fixed And Z, solved by eigenvalue decomposition The optimal solution; S42, Fixed And Z, will Update the projection onto the simplex constraint space; S43, Fixed and The updated value of Z is calculated by weighted average; S44. Repeat S41-S43 until the objective function converges, with the number of convergence iterations not exceeding 15. S5. Perform spectral clustering on the target affinity map and output the final clustering results.

[0026] In S5, spectral clustering is based on eigenvalue decomposition of the target affinity graph. The evaluation metrics for the clustering results include accuracy (ACC), normalized mutual information (NMI), and purity.

[0027] Example 1 like Figures 1-2 As shown, the multi-kernel subspace clustering method based on variance-covariance subspace distance provided by this invention is tested in practice on the face recognition datasets ORL and Umist to verify its clustering performance and computational efficiency.

[0028] The specific steps are as follows: S1. Data Preparation and VCSD Kernel Matrix Construction: Load the ORL and Umist face recognition datasets. ORL contains 40 categories (10 images per person), and Umist contains 20 categories (1-20 images per person). Normalize all images to 32×32 pixels and flatten them into 1024-dimensional feature vectors to form a high-dimensional data matrix. (In ORL, n=400; in Umist, n=564). Calculate the variance-covariance matrix of the data and its second-order statistical information Set balance parameters =0.5, according to the formula Calculate the initial VCSD kernel value between samples, and then use the formula Perform exponential transformation and normalization to generate the VCSD kernel matrix; Simultaneously, linear kernels, quadratic polynomial kernels, cubic polynomial kernels, Gaussian kernels with bandwidth of 0.1, and Gaussian kernels with bandwidth of 0.5 are integrated to construct a multi-kernel pool r=6 containing 6 basic kernel matrices.

[0029] S2, Local Affinity Feature Map Learning: Using the six basic kernel matrices from the multi-kernel pool as input, based on the theory of nonnegative matrix factorization, according to the formula... Construct an optimization objective function, under nonnegativity ≥0, symmetry = Randomness Solving under the 1=1 constraint yields six local affinity feature maps with block diagonal structures. ; in, As a consistency index matrix The approximate symmetric projection matrix satisfies This directly avoids nuclear noise interference.

[0030] S3, Matrix-based High-Order Association Capture: Introducing the representation matrix Similarity matrix And the global consistency matrix Z, according to the formula: ; Construct a comprehensive objective function, including a reconstruction error term and a local consistency term. =1.0, Global Consistency Item =0.2, regularization term γ=0.01 and VCSD regularization term =0.5, This transforms the original tensor optimization problem into matrix operations.

[0031] S4. Alternating optimization and target affinity graph generation: The objective function is solved using an alternating optimization strategy, with a fixed... And Z, obtained through eigenvalue decomposition Optimal solution; fixed And Z, will Project onto the simplex constraint space for updating; fix and Z is updated using a weighted average; after 12 iterations, the objective function converges, reducing the computational complexity from O(n^2) of the traditional tensor method. 3 ) decreases to O(6n 2 ), generating the final target affinity map.

[0032] S5. Spectral Clustering and Performance Evaluation: The target affinity map is decomposed into eigenvalues, and the top k feature vectors are taken. K-means clustering is performed on k, which is the number of categories. The clustering results are output, and the performance is evaluated by accuracy (ACC), normalized mutual information (NMI), and purity (Purity).

[0033] The results show that perfect clustering is achieved when all three metrics reach 1.0 on the Umist dataset; on the ORL dataset, ACC=0.96, NMI=0.92, and Purity=0.95, representing an 8.3% improvement in accuracy and a 42% improvement in computational efficiency compared to the PKGT method. Furthermore, parameter sensitivity testing indicates that when… When β∈[0.5,2.0] and β∈[0.1,0.3], the model performance fluctuation is less than 3%, demonstrating excellent robustness.

[0034] Therefore, this invention provides a multi-kernel subspace clustering method based on variance-covariance subspace distance. By fusing variance-covariance subspace distance, it provides solid statistical support for similarity measurement, effectively captures the variance distribution of data and the covariance relationship between features, and improves the discriminative ability of clustering. By leveraging a matrix-based high-order association capture mechanism, it replaces explicit tensor operations with efficient matrix operations, significantly reducing computational complexity and adapting to large-scale high-dimensional data processing scenarios. Based on non-negative matrix factorization, it directly constructs local affinity feature maps with block diagonal structures, avoiding the kernel noise interference caused by traditional methods that indirectly update affinity maps by optimizing consistent kernels, enhancing model robustness, and maintaining stable performance over a wide parameter range. It has good adaptability to different types of high-dimensional data and can be widely applied to practical tasks such as face recognition and image clustering.

[0035] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multi-kernel subspace clustering method based on variance-covariance subspace distance, characterized in that, include: S1. Obtain the high-dimensional data matrix Constructing the VCSD kernel matrix and multi-core pool: Calculating the variance-covariance matrix of the data. and its second-order statistical information Define the initial VCSD kernel value between samples and perform exponential transformation and normalization to form the VCSD kernel matrix. At the same time, integrate linear kernel, polynomial kernel and Gaussian kernel to construct a multi-kernel pool containing r basic kernel matrices. The high-dimensional data matrix The data are derived from face image recognition datasets, including Yale, ORL, AR, Umist, orlraws10P, and warpPIE10. S2, Feature matrix after nonnegative matrix factorization kernel mapping Furthermore, a local affinity feature map is constructed: using the base kernel matrix in the multi-kernel pool as input, an optimization objective function is constructed, and under constraints of nonnegativity, symmetry, and row randomness, a local affinity feature map with a block diagonal structure is obtained by solving the problem. ; S3. Capturing high-order feature correlations through matrix methods: Introducing representation matrices Similarity matrix Together with the global consistency matrix Z, we construct a comprehensive objective function that includes reconstruction error, local consistency, global consistency, and VCSD regularization terms. This transforms the original tensor optimization into matrix operations. The original tensor consists of r local affinity feature maps. Stacked together; S4. Solve the objective function using an alternating optimization strategy: update while keeping other variables fixed. , By combining Z, the computational complexity is reduced, and the target affinity graph is generated; S5. Perform spectral clustering on the target affinity map and output the final clustering results.

2. The multi-kernel subspace clustering method based on variance-covariance subspace distance according to claim 1, characterized in that, In S1, the specific steps for defining the initial VCSD kernel value between samples and performing exponential transformation for normalization include: S11. Calculate any two samples and Difference vector The initial VCSD core value is defined as follows: ； In the formula, It is a variance-covariance matrix. This is second-order statistical information. ∈[0,1] represents the parameters that balance first-order and second-order statistical information. For two samples and The difference vector; S12. Perform an exponential transformation and normalization on the initial kernel value to obtain the positive definite VCSD kernel matrix: 。 3. The multi-kernel subspace clustering method based on variance-covariance subspace distance according to claim 1, characterized in that, In S1, the r basic kernel matrices contained in the multi-core pool are: VCSD kernel matrix, linear kernel matrix, polynomial kernel matrices of different degrees, and Gaussian kernel matrices of different bandwidths.

4. The multi-kernel subspace clustering method based on variance-covariance subspace distance according to claim 1, characterized in that, In S2, the optimization objective function and constraints of the local affinity feature map are as follows: ； ； In the formula, For the first There are basis kernel matrices, where Tr represents the trace operation. It is a vector of all 1s. For the first A local affinity feature map.

5. The multi-kernel subspace clustering method based on variance-covariance subspace distance according to claim 4, characterized in that, The In the consistency index matrix The matrix is ​​an approximately symmetric projection matrix, and satisfies the following conditions: .

6. The multi-kernel subspace clustering method based on variance-covariance subspace distance according to claim 4, characterized in that, In S3, the comprehensive objective function is as follows: ； In the formula, Let λ1, β, γ, and ρ be the fundamental matrix of VCSD, and let λ1, β, γ, and ρ be the regularization parameters. For the first There are three basic kernel matrices, λ1∈[0.1,10], β∈[0.01,1], γ∈[0.001,0.1], ρ∈[0,1], It is the Frobenius norm.

7. The multi-kernel subspace clustering method based on variance-covariance subspace distance according to claim 1, characterized in that, In S4, the specific steps of the alternating optimization strategy include: S41, Fixed And Z, solved by eigenvalue decomposition The optimal solution; S42, Fixed And Z, will Update the projection onto the simplex constraint space; S43, Fixed and The updated value of Z is calculated by weighted average; S44. Repeat S41-S43 until the objective function converges, with the number of convergence iterations not exceeding 15.

8. The multi-kernel subspace clustering method based on variance-covariance subspace distance according to claim 1, characterized in that, In S4, the computational complexity is reduced from O(n 3 ) decreased to O(rn) 2 ), where r is the number of base kernels in the multi-kernel pool and r∈[5,20], and n is the number of samples.

9. The multi-kernel subspace clustering method based on variance-covariance subspace distance according to claim 1, characterized in that, In S5, spectral clustering is based on eigenvalue decomposition of the target affinity graph. The evaluation metrics for the clustering results include accuracy (ACC), normalized mutual information (NMI), and purity.

Images