A sparse representation driven large scale data integrated clustering method, system, terminal and storage medium

Abstract

The present application relates to the technical field of computer, disclose a kind of sparse representation driven large-scale data integration clustering method, system, terminal and storage medium. The method comprises: in distributed computing environment, multiple basic clusterers are run in parallel, generate multiple basic clustering results, wherein each basic clusterer carries out sparse representation clustering according to the representative point set selected from the original data set;Multiple basic clustering results are fused to generate the final clustering result.The present application improves the computing efficiency and scalability through distributed parallel processing, supports efficient processing of massive data;Utilize the wisdom of crowds effect to enhance the accuracy and robustness of clustering results;And through localized processing and non-iterative fusion optimization resource utilization, reduce energy consumption.The present application realizes collaborative promotion in efficiency, accuracy and resource optimization, and is suitable for large data application scenarios such as anomaly detection, image segmentation.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a sparse representation-driven method, system, terminal, and storage medium for large-scale data integration and clustering. Background Technology

[0002] With the advent of the big data era, how to efficiently and accurately perform unsupervised learning to partition (i.e., cluster) massive amounts of data has become a core challenge in the field of data mining. Ensemble clustering, by combining the results of multiple basic clustering methods, can effectively improve the robustness and accuracy of clustering results and is currently a research hotspot.

[0003] However, existing ensemble clustering techniques have significant limitations when dealing with large-scale data:

[0004] Single-machine computing bottleneck: The core logic of many advanced clustering algorithms is designed for single-machine environments. When the data scale exceeds the memory and computing power of a single machine, approximate strategies such as random sampling are commonly used, which can easily lead to sampling bias, fail to fully capture the global structure of the data, and affect the accuracy of the results.

[0005] Poor adaptability to distributed environments: The core operations of traditional ensemble methods (such as evidence accumulation clustering algorithms) (such as constructing a global co-occurrence matrix) rely on a global data view. In a distributed environment, frequent global data synchronization and communication are required, resulting in low parallelization efficiency, huge communication overhead, and difficulty in leveraging the advantages of distributed computing.

[0006] Limited quality of basic clusterers: Existing methods mostly rely on simple clusterers such as K-means to generate basic partitions, which have limited ability to capture complex data structures, thus restricting the improvement of ensemble results.

[0007] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0008] The main objective of this invention is to provide a sparse representation-driven method, system, terminal, and computer-readable storage medium for large-scale data integration and clustering, aiming to solve the problems of low accuracy and low efficiency in existing integration and clustering technologies when dealing with large-scale data.

[0009] To achieve the above objectives, this invention provides a sparse representation-driven large-scale data ensemble clustering method, which includes the following steps:

[0010] In a distributed computing environment, multiple basic clusterers are run in parallel to generate multiple basic clustering results, wherein each basic clusterer performs sparse representation clustering based on a set of representative points selected from the original dataset;

[0011] The multiple basic clustering results are fused to generate the final clustering result.

[0012] Furthermore, the process by which the basic clusterer performs sparse representation clustering includes:

[0013] The original dataset is randomly divided into multiple data subsets, and a clustering algorithm is applied to the multiple data subsets to generate a set of representative points.

[0014] Calculate the similarity between sample points in the original dataset and representative points in the representative point set, and construct a sparse similarity matrix that retains only the connections between each sample point and its K nearest neighbor representative points.

[0015] The sparse similarity matrix is ​​regarded as a bipartite graph, and the bipartite graph is divided by the transition cutting method to obtain the basic clustering result.

[0016] Furthermore, the step of randomly partitioning the original dataset to obtain multiple data subsets, and applying a clustering algorithm to these subsets to generate a representative point set, specifically includes:

[0017] The original dataset was divided into multiple data subsets using a random sample partitioning method.

[0018] Randomly select one data subset from the plurality of data subsets as the target data subset;

[0019] The K-means clustering algorithm is performed on the target data subset to obtain multiple cluster centers, and the multiple cluster centers are used as the representative point set.

[0020] Furthermore, the method for finding the K nearest neighbor representative points for each sample point is as follows:

[0021] An approximate nearest neighbor search technique based on inverted file index is used to accelerate the search for the K nearest neighbor representative points of each sample point.

[0022] Furthermore, the application of the transfer cutting method to partition the bipartite graph to obtain the basic clustering results includes:

[0023] Based on the bipartite graph, construct a representative point subgraph;

[0024] Eigenvectors are solved on the representative point subgraph, and the eigenvectors are used to define the coordinates of the representative points in a low-dimensional spectral embedding space.

[0025] Clustering of representative points on the coordinates yields the clustering results of the representative points.

[0026] The clustering results are passed to the entire bipartite graph to obtain the basic clustering results.

[0027] Furthermore, the process of fusing multiple basic clustering results to generate a final clustering result includes:

[0028] The quality of each of the basic clustering results is evaluated, and a corresponding weight is assigned to each of the basic clustering results based on the quality evaluation results, resulting in multiple weighted basic clustering results;

[0029] Based on all the weighted basic clustering results, a weighted object-cluster bipartite graph is constructed, and a sparse matrix is ​​used to sparsely represent the weighted object-cluster bipartite graph.

[0030] The sparse matrix is ​​spectrally segmented to obtain the final clustering result.

[0031] Furthermore, the quality of each of the basic clustering results is evaluated, and a corresponding weight is assigned to each of the basic clustering results based on the quality evaluation results, resulting in multiple weighted basic clustering results, including:

[0032] The quality of each of the basic clustering results is evaluated by calculating the normalized mutual information between any two of the basic clustering results;

[0033] The sum of the normalized mutual information between all pairs of the basic clustering results constitutes the quality vector;

[0034] The quality vector is subjected to Softmax normalization to obtain the weight vector.

[0035] Furthermore, to achieve the above objectives, the present invention also provides a sparse representation-driven large-scale data integration and clustering system, wherein the sparse representation-driven large-scale data integration and clustering system comprises:

[0036] The basic clustering module is used to run multiple basic clusterers in parallel in a distributed computing environment and generate multiple basic clustering results. Each of the basic clusterers performs sparse representation clustering based on a set of representative points selected from the original dataset.

[0037] An integrated clustering module is used to fuse multiple basic clustering results to generate a final clustering result.

[0038] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a sparse representation-driven large-scale data integration and clustering program stored in the memory and executable on the processor, wherein when the sparse representation-driven large-scale data integration and clustering program is executed by the processor, it implements the steps of the sparse representation-driven large-scale data integration and clustering method as described above.

[0039] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a sparse representation-driven large-scale data integration and clustering program, which, when executed by a processor, implements the steps of the sparse representation-driven large-scale data integration and clustering method as described above.

[0040] The beneficial effects of this invention are as follows: By running multiple basic clusterers in a distributed parallel manner, this invention effectively overcomes the computational bottleneck of traditional single-machine environments. The distributed architecture allows tasks to be executed simultaneously on multiple nodes, avoiding the limitations of single-machine memory and computing power, thereby supporting efficient processing of massive amounts of data. This invention utilizes the "collective intelligence" effect, effectively improving the quality of the final clustering by integrating multiple basic clustering results. Each basic clusterer generates differentiated results based on sparse representation clustering, thereby reducing the impact of random bias and noise. Distributed parallel processing avoids frequent global data exchange and iterative calculations in traditional fusion methods, reducing redundant operations and energy consumption. This invention significantly reduces computational resource requirements while ensuring clustering quality through localized processing (such as sparse representation based on representative points) and a non-iterative fusion mechanism. In summary, this invention achieves synergistic improvements in efficiency and accuracy by generating basic clustering results in a distributed parallel manner and intelligently fusing them. Attached Figure Description

[0041] Figure 1 This is a flowchart of a preferred embodiment of the sparse representation-driven large-scale data integration and clustering method of the present invention;

[0042] Figure 2 This is a flowchart of the sparse representation-driven large-scale data integration and clustering method of the present invention.

[0043] Figure 3 This is a flowchart of sparse representation clustering in the sparse representation-driven large-scale data integration clustering method of the present invention;

[0044] Figure 4 This is a simulation diagram of four simulation datasets with 1% data volume in the sparse representation-driven large-scale data integration and clustering method of this invention.

[0045] Figure 5 This is a structural diagram of a preferred embodiment of the sparse representation-driven large-scale data integration and clustering system of the present invention;

[0046] Figure 6 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0047] This application provides a sparse representation-driven method, system, terminal, and storage medium for large-scale data integration and clustering. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0048] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0049] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0050] The sparse representation-driven ensemble clustering (SREC) method for large-scale data ensemble clustering described in the preferred embodiment of this invention, such as... Figure 1 and Figure 2 As shown, the sparse representation-driven large-scale data ensemble clustering method includes the following steps:

[0051] S10. In a distributed computing environment, multiple basic clusterers are run in parallel to generate multiple basic clustering results, wherein each basic clusterer performs sparse representation clustering based on a set of representative points selected from the original dataset.

[0052] It should be noted that, Figure 2The relevant parameters include: the final number of clusters (representing the number of clusters into which the data is to be divided, which is the final output of the algorithm), the number of representative points (representing the number of representative points used to generate the sparse graph in the basic clusterer), the number of nearest neighbors, the number of basic clusters (representing the total number of basic partitions that need to be generated in parallel in ensemble learning), the number of partitions (representing the number of subsets into which the original dataset is divided using the random sample partitioning method in the representative point selection stage), the lower limit of the number of clusters (representing the lower limit of the number of clusters randomly generated by each basic cluster, used to inject diversity), and the upper limit of the number of clusters (representing the upper limit of the number of clusters randomly generated by each basic cluster, used to inject diversity).

[0053] Furthermore, such as Figure 3 As shown, the basic clusterer performs sparse representation clustering (SRC) by including the following steps:

[0054] A10. Randomly partition the original dataset to obtain multiple data subsets, and apply a clustering algorithm to the multiple data subsets to generate a representative point set, specifically including:

[0055] A11. Use the Random Sample Partition (RSP) method to divide the original dataset into multiple data subsets.

[0056] It should be noted that the RSP method ensures that each subset retains the statistical distribution characteristics of the original data by dividing it randomly and uniformly, thereby avoiding sampling bias.

[0057] A12. Randomly select one data subset from the plurality of data subsets as the target data subset.

[0058] A13. Perform K-means clustering algorithm on the target data subset to obtain multiple cluster centers, and use the multiple cluster centers as the representative point set.

[0059] It should be noted that the above scheme combines the efficiency of random sampling with the representativeness of K-means, significantly improving the quality and coverage of representative points.

[0060] A20. Calculate the similarity between the sample points in the original dataset and the representative points in the representative point set, and construct a sparse similarity matrix that retains only the connections between each sample point and its K nearest neighbor representative points.

[0061] In this embodiment, an approximate nearest neighbor search technique based on an inverted file index is used to accelerate the search for the K nearest neighbor representative points of each sample point.

[0062] The sparse similarity matrix is ​​calculated as follows:

[0063] if ,but ,otherwise ;

[0064] in, for of One of the neighbors, express recent A set of samples, Represents a preset positive integer, i.e., the number of "nearest neighbors"; Represents the first in the dataset j Each sample object (or data point) Represents the sparse similarity matrix B in terms of the th j line, number i The element (value) at the intersection of the columns; j and i These represent the row and column indices of the matrix, respectively. Indicates An exponential function with base 0. This represents the Euclidean distance between two vectors. This represents the ripple effect range of a single sample. In this way, the sparse similarity matrix retains only the influence range of each data point and its corresponding sample. The non-zero correlation between the nearest neighbor representative points significantly reduces storage requirements and subsequent computational overhead.

[0065] A30. Treat the sparse similarity matrix as a bipartite graph and apply a transition-cutting method to partition the bipartite graph to obtain the basic clustering results. Specifically, this includes:

[0066] A31. Treat the sparse similarity matrix as a bipartite graph, and construct a representative subgraph based on the bipartite graph.

[0067] In this embodiment, based on the bipartite graph, a significantly smaller representative point set is constructed by sampling or filtering one of the vertex sets. Subsequently, according to the topology of the original bipartite graph, the correlation strength between the representative points is calculated, thereby forming a weighted representative point subgraph. This step aims to reduce computational complexity, transforming the partitioning problem of a large-scale bipartite graph into a more manageable small-scale graph partitioning problem.

[0068] A32. Solve for the eigenvectors on the representative point subgraph, whereby the eigenvectors are used to define the coordinates of the representative points in a low-dimensional spectral embedding space.

[0069] In this embodiment, based on the representative point subgraph obtained in the previous step, its normalized Laplacian matrix is ​​calculated, and the eigenvectors corresponding to the first k smallest eigenvalues ​​(k being the preset number of clusters) of this matrix are solved. These eigenvectors collectively define the coordinates of the representative points in a low-dimensional spectral embedding space. The core purpose is to preserve the structural similarity between representative points and provide an ideal low-dimensional representation for subsequent clustering.

[0070] A33. Cluster the representative points on the coordinates to obtain the clustering results of the representative points.

[0071] In this embodiment, using the low-dimensional coordinates obtained in step B20, a clustering algorithm (such as k-means) is employed to partition all representative points, thereby obtaining the clustering results of the representative points. This step is performed in the dimensionality-reduced space, which can efficiently and accurately discover the community structure in the representative point subgraph.

[0072] A34. Pass the clustering partitioning result to the entire bipartite graph to obtain the basic clustering result.

[0073] In this embodiment, the cluster labels of representative points are used as a benchmark. For all non-representative vertices in the bipartite graph, they are assigned to the corresponding clusters based on their connection relationships with the representative points of each cluster. Common assignment strategies include assigning them to the cluster containing the representative point with the heaviest connecting edge weight, or using a majority voting principle. This step completes the mapping from the representative point subgraph to the original bipartite graph, ultimately obtaining the basic clustering result covering all vertices.

[0074] S20. Merge the multiple basic clustering results to generate the final clustering result. Specifically, this includes:

[0075] S21. Evaluate the quality of each of the basic clustering results, and assign corresponding weights to each of the basic clustering results based on the quality evaluation results, to obtain multiple weighted basic clustering results. Specifically, this includes:

[0076] S211. The quality of each basic clustering result is evaluated by calculating the normalized mutual information between any two basic clustering results.

[0077] In this embodiment, to quantify the uniqueness and consensus of each basic clustering result, the normalized mutual information between any two basic clustering results is calculated. This metric measures the degree of consistency between two clustering results; the higher the value, the more information they share, and the more reliable the result.

[0078] The formula for calculating the normalized mutual information is:

[0079] ;

[0080] in, This represents the normalized mutual information between the clustering results and the true labels. This represents the set of true labels, that is, the true class label of each sample in the dataset; This represents the set of labels for the clustering results, i.e., the clustering partitions predicted by the algorithm. Mutual information is used to quantify the shared information between two label distributions. and These represent the information entropy of the real labels and the cluster labels, respectively, and are used to measure the uncertainty of label distribution.

[0081] The formula for calculating the mutual information is:

[0082] ;

[0083] in, and These represent the number of categories (i.e., clusters) for the real labels and the cluster labels, respectively. This indicates that it belongs to the real category at the same time. i and clusters j The sample proportion Indicates belonging to the real category i The sample proportion Indicates belonging to a cluster j The sample proportion.

[0084] S212. The sum of the normalized mutual information between all pairs of the basic clustering results constitutes a quality vector.

[0085] In this embodiment, the normalized mutual information values ​​of each basic clustering result are summed with those of all other results (including itself) to obtain a comprehensive quality score. This score reflects the degree of agreement between the target basic clustering result and the overall consensus. Combining the comprehensive quality scores of all basic clustering results constitutes a quality vector. This step aims to summarize a global quality metric for each clustering result.

[0086] S213. Perform Softmax normalization on the quality vector to obtain the weight vector.

[0087] In this embodiment, the quality vector obtained above is subjected to Softmax normalization. This operation converts the quality score into a probability distribution in which the sum of all elements is 1, thereby transforming the absolute quality score into a relative weight vector. The magnitude of the weight directly represents the importance of the corresponding basic clustering result in the final ensemble.

[0088] S22. Based on all the weighted basic clustering results, construct a weighted object-cluster bipartite graph, and use a sparse matrix to sparsely represent the weighted object-cluster bipartite graph.

[0089] It should be noted that in the sparse matrix, if a sample point belongs to a base cluster, the value of the corresponding position of the sample point in the sparse matrix is ​​the weight of the basic clustering result corresponding to the base cluster; otherwise, it is zero.

[0090] Step S22 includes:

[0091] Step S221: Based on all the weighted basic clustering results, initialize a sparse matrix of all zeros, using all sample points as row indices and the union of all base clusters obtained in the previous steps as column indices. The number of rows in this matrix equals the total number of samples, and the number of columns equals the total number of unique base clusters generated by all basic clustering results, thereby establishing the object-cluster association structure framework.

[0092] Step S222: Traverse each basic clustering result and its corresponding weight. For each base cluster in the result, fill its weight value into the corresponding position in the matrix. Specifically, if a sample point belongs to the current base cluster, fill the weight of the basic clustering result corresponding to that base cluster into the intersection of the row where the sample point is located and the column where the current base cluster is located in the matrix.

[0093] Step S223: After traversing and filling the weights of all base clusters, the resulting sparse matrix is ​​the mathematical representation of the weighted object-cluster bipartite graph. In the sparse matrix, the values ​​of non-zero elements not only represent the membership relationship between the sample and the base cluster, but also reflect the relative importance of this membership relationship through the magnitude of the weights.

[0094] The formula for calculating the sparse matrix is:

[0095] ;

[0096] in, Let be the matrix elements that need to be calculated and filled, representing the first element in the weighted bipartite graph. j The sample object and the first l Connection weights between base clusters; For the first i The weights of the basic clustering results, This is a mathematical symbol representing "belongs to"; Indicates the first Each base cluster.

[0097] S23. Perform spectral segmentation on the sparse matrix to obtain the final clustering result.

[0098] It should be emphasized that the spectral segmentation adopts the transfer cutting method.

[0099] It should be noted that this step aims to transform the complex clustering structure implied by the association matrix (i.e., the sparse matrix) of the weighted object-cluster bipartite graph into a more manageable multi-way partitioning problem using spectral graph theory. Specifically, it includes the following sub-steps:

[0100] S231. Construct the normalized Laplacian matrix: Based on the weighted sparse matrix, calculate its corresponding normalized Laplacian matrix. The construction of this matrix typically involves normalizing the degree matrix of the weighted sparse matrix. The purpose is to transform the structural information of the graph into the spectral properties of the matrix, laying the foundation for subsequent dimensionality reduction analysis.

[0101] S232. Perform eigenvalue decomposition and select eigenvectors: Perform eigenvalue decomposition on the normalized Laplacian matrix and select the eigenvectors corresponding to its k smallest non-zero eigenvalues ​​(k is the preset final number of clusters). These eigenvectors together constitute a low-dimensional spectral embedding space, in which the geometric distance of data points (i.e., original objects) can more clearly reflect their similarity in the graph structure.

[0102] S233. Clustering in the spectral embedding space: Represent each data point (corresponding to each row of the sparse matrix) using the k eigenvectors obtained in step S232, thereby mapping all data points to the aforementioned low-dimensional spectral embedding space. Subsequently, cluster these data points in this low-dimensional space (e.g., using the k-means clustering algorithm) to obtain the cluster label for each data point.

[0103] S234. Output the final clustering result: The cluster labels obtained in the low-dimensional spectral embedding space will be output as the final clustering result of the original data objects.

[0104] Experimental verification and effect analysis:

[0105] To verify the effectiveness of this invention, comprehensive experiments were conducted in a specific implementation. The experiments used eight datasets, namely:

[0106] Four real-world datasets: the handwritten digit dataset (United States Postal Service, USPS), the letter recognition dataset (Letters), the mixed National Institute of Standards and Technology database (Mnist), and the forest cover type dataset (Covertype).

[0107] like Figure 4As shown, there are four simulation datasets: a moon-shaped simulation dataset (Moons), a circle-shaped simulation dataset (Circles), a stripe-shaped simulation dataset (Streaks), and a flower-shaped simulation dataset (Flower).

[0108] The data volume ranged from 72.91 to 5 million, and the dimensions ranged from 2 to 784. The experimental environment was a 5-node distributed computing framework (Hadoop) cluster with Intel Xeon Platinum 8168 processors and Python 3.6.5 as the programming language.

[0109] The algorithms compared in the experiment include: Sparse Representation Clustering (SRC), K-means clustering, Nyström method, Ultra-scalable Spectral Clustering (U-SPEC), Fast Self-supervised Clustering (FSSC), Lightweight Web-scale Spectral Clustering (LiteWSC), Evidence Accumulation Clustering (EAC), Spectral Ensemble Clustering (SEC), Ensemble Clustering by Propagating Cluster-wise Similarities with Meta-Cluster Based Consensus Function (ECPCS-MC), and Ultra-scalable Ensemble Clustering. Ensemble Clustering (U-SENC), Lightweight Web-scale Spectral Ensemble Clustering (LiteWSEC), and Sparse Representation-driven Ensemble Clustering (SREC).

[0110] Evaluation metrics include Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and Cluster Accuracy (ACC). Each algorithm is run 10 times and the average is taken to eliminate randomness.

[0111] Experimental results:

[0112] Performance of the basic clusterer SRC: Compared with algorithms such as K-means, Nyström, U-SPEC, FSSC, and LiteWSC, SRC achieves an NMI of 71.19% on real datasets (such as MNIST), outperforming U-SPEC's 69.70%, with a smaller standard deviation and higher stability. On simulation datasets, SRC achieves an NMI of nearly 100% in the Circles task, significantly outperforming traditional algorithms.

[0113] Performance of the ensemble algorithm SREC: SREC significantly outperforms SRC and advanced ensemble algorithms (such as EAC, SEC, ECPCS-MC, U-SENC, and LiteWSEC) on most datasets. For example, on MNIST, SREC's NMI is 76.32%, more than 5 percentage points higher than SRC; on the Moons dataset, SREC's NMI reaches 99.17%, while U-SENC's is only 74.66%. See Tables 1 to 5 for specific data.

[0114] Efficiency and Scalability: SREC runs in less than one-tenth the time of SEC on the Flower dataset and can process 5 million samples, demonstrating its distributed advantages. See Tables 1 to 5 for specific data.

[0115] Table 1: Dataset Information

[0116]

[0117] Table 2: NMI performance evaluation of various clustering algorithms on standard datasets (%)

[0118]

[0119] Table 3: ARI performance evaluation of various clustering algorithms on standard datasets (%)

[0120]

[0121] Table 4: ACC performance evaluation of various clustering algorithms on standard datasets (%)

[0122]

[0123] Table 5: Average running time (seconds) of various clustering algorithms on standard datasets

[0124]

[0125] Performance Analysis: This invention overcomes the single-machine bottleneck and sampling bias of traditional methods through sparse representation and distributed integration, achieving industry-leading levels in accuracy, efficiency, and scalability. Experimental data fully demonstrate its practicality in applications such as anomaly detection and image segmentation.

[0126] In summary, the beneficial effects of the present invention are at least as follows:

[0127] 1. By running multiple basic clusterers in a distributed, parallel manner, this invention effectively overcomes the computational bottleneck of traditional single-machine environments. The distributed architecture allows tasks to be executed simultaneously on multiple nodes, avoiding the limitations of single-machine memory and computing power, thus supporting efficient processing of massive datasets (such as 5 million samples). Experimental data shows that the running time of this method on the Flower dataset is only one-tenth that of traditional methods (such as the SEC algorithm), demonstrating its high efficiency. Simultaneously, the parallel design reduces the need for global data synchronization, lowers communication overhead, and enables the algorithm to better adapt to distributed computing environments, improving overall scalability.

[0128] 2. This invention leverages the "collective intelligence" effect, effectively improving the quality of the final clustering by integrating multiple basic clustering results. Each basic clusterer generates differentiated results based on sparse representation clustering, and the fusion process uses a weighted mechanism to highlight high-quality basic partitions, thereby reducing the impact of random bias and noise. For example, on the MNIST dataset, the normalized mutual information of the final clustering results reaches 76.32%, an improvement of over 5 percentage points compared to a single basic clusterer. This fusion strategy significantly improves clustering accuracy and stability, especially performing exceptionally well in complex data structures (such as non-convex distributions).

[0129] 3. Distributed parallel processing avoids the frequent global data exchange and iterative computations found in traditional ensemble methods, reducing redundant operations and energy consumption. Through localized processing (such as sparse representation based on representative points) and a non-iterative fusion mechanism, this invention significantly reduces computational resource requirements while maintaining clustering quality. Experiments show that this method can run stably on large-scale datasets such as Covertype and does not require the construction of a high-energy-consuming global co-occurrence matrix, aligning with the trend of green computing.

[0130] 4. This invention does not rely on specific data distributions or domain knowledge and is applicable to various scenarios (such as anomaly detection and image segmentation). Through a distributed integration framework, the method can flexibly adapt to challenges of different scales and data types, including high-dimensional image data (such as MNIST) and complex simulation data (such as Moons). In experimental verification, this invention outperforms traditional algorithms in NMI, ARI, and ACC metrics on multiple datasets, demonstrating its broad applicability.

[0131] In summary, this invention achieves synergistic improvements in efficiency, accuracy, resource optimization, and versatility by generating basic clustering results in a distributed and parallel manner and intelligently fusing them.

[0132] Furthermore, such as Figure 5 As shown, based on the above-mentioned sparse representation-driven large-scale data ensemble clustering method, this invention also provides a sparse representation-driven large-scale data ensemble clustering system, which includes:

[0133] The basic clustering module 51 is used to run multiple basic clusterers in parallel in a distributed computing environment and generate multiple basic clustering results, wherein each of the basic clusterers performs sparse representation clustering based on a set of representative points selected from the original dataset.

[0134] The integrated clustering module 52 is used to fuse multiple basic clustering results to generate a final clustering result, including:

[0135] The quality of each of the basic clustering results is evaluated, and a corresponding weight is assigned to each of the basic clustering results based on the quality evaluation results, resulting in multiple weighted basic clustering results;

[0136] Based on all the weighted basic clustering results, a weighted object-cluster bipartite graph is constructed, and a sparse matrix is ​​used to sparsely represent the weighted object-cluster bipartite graph.

[0137] The sparse matrix is ​​spectrally segmented to obtain the final clustering result.

[0138] Furthermore, such as Figure 6 As shown, based on the above-mentioned sparse representation-driven large-scale data integration and clustering method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0139] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a sparse representation-driven large-scale data integration and clustering program 40, which can be executed by the processor 10 to implement the sparse representation-driven large-scale data integration and clustering method of this application.

[0140] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the sparse representation-driven large-scale data integration and clustering method.

[0141] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0142] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a sparse representation-driven large-scale data integration and clustering program, which, when executed by a processor, implements the steps of the sparse representation-driven large-scale data integration and clustering method as described above.

[0143] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0144] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0145] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A sparse representation-driven method for large-scale data integration and clustering, characterized in that, The sparse representation-driven large-scale data integration and clustering method includes: In a distributed computing environment, multiple basic clusterers are run in parallel to generate multiple basic clustering results, wherein each basic clusterer performs sparse representation clustering based on a set of representative points selected from the original dataset; The multiple basic clustering results are fused to generate the final clustering result; The process by which the basic clusterer performs sparse representation clustering includes: The original dataset is randomly divided into multiple data subsets, and a clustering algorithm is applied to the multiple data subsets to generate a set of representative points. Calculate the similarity between sample points in the original dataset and representative points in the representative point set, and construct a sparse similarity matrix that retains only the connections between each sample point and its K nearest neighbor representative points. The sparse similarity matrix is ​​regarded as a bipartite graph, and the bipartite graph is partitioned by the transition cutting method to obtain the basic clustering result; The step of fusing multiple basic clustering results to generate a final clustering result includes: The quality of each of the basic clustering results is evaluated, and a corresponding weight is assigned to each of the basic clustering results based on the quality evaluation results, resulting in multiple weighted basic clustering results; Based on all the weighted basic clustering results, a weighted object-cluster bipartite graph is constructed, and a sparse matrix is ​​used to sparsely represent the weighted object-cluster bipartite graph. The sparse matrix is ​​spectrally segmented to obtain the final clustering result. The formula for calculating the sparse matrix is: ； in, Let be the matrix elements that need to be calculated and filled, representing the first element in the weighted bipartite graph. The sample object and the first Connection weights between base clusters; For the first The weights of the basic clustering results, Represents the first in the dataset One sample object, This is a mathematical symbol representing "belongs to"; Indicates the first Each base cluster.

2. The sparse representation-driven large-scale data integration and clustering method according to claim 1, characterized in that, The step of randomly partitioning the original dataset to obtain multiple data subsets, and then applying a clustering algorithm to these subsets to generate a representative point set, specifically includes: The original dataset was divided into multiple data subsets using a random sample partitioning method. Randomly select one data subset from the plurality of data subsets as the target data subset; The K-means clustering algorithm is performed on the target data subset to obtain multiple cluster centers, and the multiple cluster centers are used as the representative point set.

3. The sparse representation-driven large-scale data integration and clustering method according to claim 1, characterized in that, The method for finding the K nearest neighbor representative points for each sample point is as follows: an approximate nearest neighbor search technique based on inverted file index is used to find the K nearest neighbor representative points for each sample point.

4. The sparse representation-driven large-scale data integration and clustering method according to claim 1, characterized in that, The application of the transfer cutting method to partition the bipartite graph yields the basic clustering results, including: Based on the bipartite graph, construct a representative point subgraph; Eigenvectors are solved on the representative point subgraph, and the eigenvectors are used to define the coordinates of the representative points in a low-dimensional spectral embedding space. Clustering of representative points on the coordinates yields the clustering results of the representative points. The clustering results are passed to the entire bipartite graph to obtain the basic clustering results.

5. The sparse representation-driven large-scale data integration and clustering method according to claim 1, characterized in that, The quality of each of the basic clustering results is evaluated, and a corresponding weight is assigned to each basic clustering result based on the quality evaluation results to obtain multiple weighted basic clustering results, including: The quality of each of the basic clustering results is evaluated by calculating the normalized mutual information between any two of the basic clustering results; The sum of the normalized mutual information between all pairs of the basic clustering results constitutes the quality vector; The quality vector is subjected to Softmax normalization to obtain the weight vector.

6. A sparse representation-driven large-scale data integration and clustering system, characterized in that, The sparse representation-driven large-scale data ensemble clustering system is used to perform the steps of the sparse representation-driven large-scale data ensemble clustering method as described in any one of claims 1-5, wherein the sparse representation-driven large-scale data ensemble clustering system comprises: The basic clustering module is used to run multiple basic clusterers in parallel in a distributed computing environment and generate multiple basic clustering results. Each of the basic clusterers performs sparse representation clustering based on a set of representative points selected from the original dataset. An integrated clustering module is used to fuse multiple basic clustering results to generate a final clustering result.

7. A terminal, characterized in that, The terminal includes: a memory, a processor, and a sparse representation-driven large-scale data integration and clustering program stored in the memory and executable on the processor. When the sparse representation-driven large-scale data integration and clustering program is executed by the processor, it implements the steps of the sparse representation-driven large-scale data integration and clustering method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a sparse representation-driven large-scale data integration and clustering program, which, when executed by a processor, implements the steps of the sparse representation-driven large-scale data integration and clustering method as described in any one of claims 1-5.

Images