High-dimensional data processing method and system based on multi-criterion classification and clustering
By introducing multi-criteria classification and clustering methods in high-dimensional data processing, and using stable matching and global optimization algorithms, the problem of multi-criteria processing in the existing technology is solved, and the stability and accuracy of clustering results are achieved, which is suitable for multi-dimensional complex data scenarios.
Patent Information
- Application Number
- CN202411589104.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-08
- Publication Date
- 2025-03-04
AI Technical Summary
The prior art is difficult to process multiple criteria simultaneously in high-dimensional data, resulting in unstable clustering results and difficult to cope with the complexity of multimodal data.
A high-dimensional data processing method based on multi-criteria classification and clustering is adopted. By obtaining high-dimensional data sets, a preference list is generated, a stable matching algorithm is used to optimize the preliminary clustering results, and the class cluster boundaries are adjusted through a global optimization algorithm to output the cluster results.
Ensure that the classification results of data points are consistent under different criteria, improve the stability and accuracy of clustering results, and are suitable for multi-dimensional complex data scenarios, reducing the economic losses caused by the use of computing resources and incorrect classification.
Smart Images
Figure CN119557669A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a high-dimensional data processing method and system based on multi-criteria classification and clustering. Background Art
[0002] With the advent of the big data era, the demand for high-dimensional data processing is increasing in the fields of bioinformatics, financial analysis, medical data, complex supply chain management, etc. Existing data processing methods usually rely on a single standard, such as similarity, distance, etc. for classification and clustering, but when faced with multi-criteria requirements in multidimensional data, a single standard is difficult to effectively reflect the complex relationship between data.
[0003] In these applications, data not only contains traditional geometric distance or similarity, but also involves other key factors. For example, gene expression data analysis in bioinformatics needs to consider the relationship between different biological characteristics; in financial analysis, the risk, return, and market volatility of the investment portfolio may all become clustering criteria. Therefore, the demand for multi-criteria optimization has gradually become prominent, and the limitations of existing technologies have made it difficult to adapt to this complex data environment.
[0004] Among traditional clustering algorithms, methods such as k-means and DBSCAN are often used for classification and clustering of high-dimensional data. Although these algorithms perform well under a single criterion, due to the lack of support for multiple criteria, the clustering results are often not stable enough and have difficulty coping with the complexity of multimodal data.
[0005] In order to overcome these defects, this application proposes a high-dimensional data processing method and system based on multi-criteria classification and clustering to meet the challenges of multi-criteria classification and clustering and improve the global consistency and stability of classification results. Summary of the invention
[0006] The purpose of this application is to provide a high-dimensional data processing method and system based on multi-criteria classification and clustering, aiming to solve the problem that the prior art cannot process multiple criteria simultaneously and cannot ensure the global stability of the classification results.
[0007] To achieve the above objectives, this application provides the following technical solutions:
[0008] The present application provides a high-dimensional data processing method based on multi-criteria classification and clustering, comprising:
[0009] Obtaining a high-dimensional data set, performing feature extraction on the high-dimensional data set, and obtaining a corresponding preference list;
[0010] Perform preliminary clustering on the high-dimensional data set after feature extraction to obtain preliminary clustering results;
[0011] Based on the preference list, a stable matching algorithm is used to optimize the matching of the preliminary clustering results to obtain a matching result;
[0012] The matching results are optimized by a global optimization algorithm, and a clustering result is output.
[0013] Furthermore, in the step of obtaining a high-dimensional data set, performing feature extraction on the high-dimensional data set, and obtaining a corresponding preference list, the following steps are specifically included:
[0014] The high-dimensional data set consists of N data points D1, D2, ..., D N Composition, each data point D i Including M different eigenvectors Xi=(X i1 ,X i2 ,,...,X iM );
[0015] The eigenvectors are based on the corresponding criterion C m , generate the corresponding preference list;
[0016] Each data point D i For Criteria C m Other data points D j Generate preference order, denoted as:
[0017] Furthermore, in the step of optimizing the matching of the preliminary clustering results by using a stable matching algorithm based on the preference list to obtain the matching results, the following steps are specifically included:
[0018] Based on Criteria C m , the data points are matched using the Gale-Shapley stable matching algorithm;
[0019] Data point D i Matching the preference list with the clusters and selecting data points according to the preferences of the clusters under the criteria;
[0020] When the cluster C k The data points are generated based on the criterion C m Preference list The preference list includes the preference order of the clusters for the data points under the criteria; the matching process is:
[0021] Each data point D i According to their preference list Cluster C k Issue a matching request;
[0022] Cluster C k According to their preference list Select the received matching request and save it in criterion C m Select the preferred data points and reject other requested data points;
[0023] The rejected request data point continues to send matching requests to the next preferred cluster until all data points and clusters reach a stable matching state.
[0024] Furthermore, after obtaining the matching result, it also includes:
[0025] Matching results of combined criteria Construct a global preference stability function:
[0026]
[0027] where w m Criterion C m The weight of is the data point D i In Guideline C m Next, with cluster C(D i ) distance or difference.
[0028] Furthermore, in the step of optimizing the matching results by a global optimization algorithm and outputting the clustering results, the following steps are specifically included:
[0029] The cluster boundaries are adjusted by the global optimization algorithm, and the formula is:
[0030]
[0031] The objective function optimizes the cluster boundaries by weighted analysis of the differences between data points and clusters under different criteria and outputs the clustering results.
[0032] This application proposes a high-dimensional data processing system based on multi-criteria classification and clustering, including:
[0033] Acquisition and processing module: acquires a high-dimensional data set, performs feature extraction on the high-dimensional data set, and obtains a corresponding preference list;
[0034] Preliminary clustering module: performs preliminary clustering on the high-dimensional data set after feature extraction to obtain preliminary clustering results;
[0035] Matching module: Based on the preference list, a stable matching algorithm is used to optimize the matching of the preliminary clustering results to obtain a matching result;
[0036] Optimization and output module: optimizes the matching results through a global optimization algorithm and outputs clustering results.
[0037] The present application provides a device, which includes a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a high-dimensional data processing method based on multi-criteria classification and clustering; the processor is used to execute the program instructions stored in the memory to implement a high-dimensional data processing method based on multi-criteria classification and clustering.
[0038] The present application provides a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute a high-dimensional data processing method based on multi-criteria classification and clustering.
[0039] The present application provides a high-dimensional data processing method and system based on multi-criteria classification and clustering, which has the following beneficial effects:
[0040] (1) By introducing a multimodal stable matching algorithm, the classification results of data points under different criteria are ensured to be consistent, thus avoiding the problem of unstable classification results caused by a single standard in traditional clustering algorithms; especially when processing multidimensional complex data, the classification results are more robust and less susceptible to interference from noise and external influences;
[0041] (2) This application can process multiple criteria at the same time, such as similarity, distance, weight, etc., and is no longer limited to the distance or similarity calculation of a single dimension; by comprehensively considering multiple criteria, the accuracy and adaptability of clustering results are improved, and it is particularly suitable for fields such as bioinformatics and financial data analysis that require multi-dimensional data processing;
[0042] (3) By constructing a global optimization function, the clustering results are further optimized to ensure that the matching between data points and clusters reaches the global optimum; this not only improves the clustering accuracy, but also improves the computational efficiency of the algorithm, especially in high-dimensional data scenarios, reducing computational overhead;
[0043] (4) Unlike traditional clustering algorithms based on a single standard, the present application can still maintain high robustness in the face of data changes, and the classification and clustering results are highly consistent, avoiding the problem of over-sensitivity to initial conditions in traditional methods; in addition, the present application can adapt to different application scenarios, especially showing significant advantages when facing complex multimodal data;
[0044] (5) By improving the accuracy and efficiency of clustering, unnecessary recalculation and correction costs are reduced, and stable clustering results can be obtained in a shorter time. For data-intensive application scenarios, such as finance and supply chain management, the use of computing resources can be effectively reduced, and the economic losses caused by misclassification can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1A flowchart of a high-dimensional data processing method based on multi-criteria classification and clustering according to Example 1 of the present application;
[0046] Figure 2 A flowchart of a high-dimensional data processing method based on multi-criteria classification and clustering according to Example 1 of the present application;
[0047] Figure 3 This is a structural diagram of a high-dimensional data processing system based on multi-criteria classification and clustering according to Embodiment 2 of the present application;
[0048] Figure 4 This is a schematic diagram of the device structure of Example 3 of the present application;
[0049] Figure 5 This is a schematic diagram of the storage medium structure of Example 4 of the present application. DETAILED DESCRIPTION
[0050] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0051] The following is an analysis of the solutions in the prior art in combination with related technologies.
[0052] Among the existing clustering technologies, classic k-means, hierarchical clustering, DBSCAN and other algorithms have their limitations when processing high-dimensional data, especially in a multi-criteria environment.
[0053] k-means clustering algorithm: k-means is a commonly used distance-based clustering method. Its core idea is to achieve classification by minimizing the distance between data points and cluster centers. However, k-means clustering is based on only a single criterion, usually the Euclidean distance, and is difficult to handle complex data containing multiple criteria. At the same time, k-means is sensitive to the initial center point, easily falls into local optimality, and does not handle noise and outliers well [1].
[0054] DBSCAN clustering algorithm: DBSCAN performs well in discovering clusters of arbitrary shapes, can identify high-density areas and distinguish noisy data. However, DBSCAN has limited application under multi-criteria conditions, has strong parameter dependence, and is difficult to adapt to the clustering needs of high-dimensional and multi-criteria data [2].
[0055] Challenges in multi-criteria data processing: Currently, some studies have explored how to process complex data through weighted or multi-criteria optimization methods. For example, some improved clustering algorithms consider multiple criteria in the clustering process by setting weights for different criteria. However, these methods usually rely on manually set weight parameters, lack global optimality and adaptability, and are difficult to ensure the stability of the results [3].
[0056] In order to overcome the shortcomings of these traditional methods, researchers have begun to explore how to optimize the clustering process through multimodal stable matching in recent years. The stable matching algorithm was first applied to the marriage matching problem (Gale-Shapley algorithm), and in recent years it has been extended to multiple fields such as resource allocation and task scheduling. By combining stable matching with clustering, globally stable clustering results can be achieved under multiple criteria, avoiding local optimal solutions and ensuring the consistency of clustering results in multiple dimensions [4].
[0057] [1]Hartigan, JA, & Wong, MA (1979). Algorithm AS136: AK-means clustering algorithm. Journal of the Royal Statistical Society: Series C (AppliedStatistics), 28 (1), 100-108.
[0058] [2]Ester, M., Kriegel, HP, Sander, J., & Xu,
[0059] [3]Kaufman,L.,&Rousseeuw,PJ(2009).Finding Groups in Data: AnIntroduction to Cluster Analysis.John Wiley&Sons.
[0060] [4]Gale, D., & Shapley, LS (1962). College admissions and the stability of marriage. The American Mathematical Monthly, 69 (1), 9-15.
[0061] Among the existing classification and clustering technologies, traditional algorithms such as k-means and DBSCAN mainly rely on a single standard to classify and cluster data. However, in the face of multimodal and high-dimensional data, these methods have many shortcomings, especially under multi-criteria conditions, it is difficult to ensure the global stability and consistency of clustering results. Specific shortcomings include:
[0062] Limitations of a single criterion: Traditional clustering algorithms mostly divide data sets based on distance or similarity and cannot handle complex data with multiple dimensions and multiple standards. For example, in bioinformatics or financial analysis, data often has multiple dimensions, and clustering under a single criterion is prone to ignore other key factors, resulting in biased classification results.
[0063] Lack of multimodal support: In the face of multimodal data, existing clustering methods lack an effective mechanism to integrate information from different data dimensions or modalities. Traditional methods cannot fully utilize the correlation between multiple criteria, resulting in unstable clustering results and difficulty in dealing with possible conflicts between criteria.
[0064] Unstable clustering results: Clustering methods in existing technologies, such as k-means, are sensitive to initial conditions and easily fall into local optimality, resulting in inconsistent clustering results under different criteria. The complexity of high-dimensional data further exacerbates this problem, and the stability and robustness of clustering results are difficult to guarantee.
[0065] Disadvantages of multi-criteria optimization: Although some existing technologies attempt to process multi-criteria data by weighting or setting parameters, these methods often rely on manually set weights and are difficult to achieve global optimization. In addition, as the dimension and complexity of the data increase, the clustering efficiency and accuracy of existing technologies under multi-criteria decrease significantly.
[0066] In view of the shortcomings of the existing technology, this application solves the multi-criteria classification and clustering problems in high-dimensional data by combining multimodal stable matching technology, overcomes the limitations of the existing technology in processing multidimensional data, and significantly improves the stability and accuracy of the results.
[0067] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0068] Example 1
[0069] See also Figure 1, is a flow chart of a high-dimensional data processing method based on multi-criteria classification and clustering in Example 1 of the present application; the steps include:
[0070] S1: Obtain a high-dimensional data set, perform feature extraction on the high-dimensional data set, and obtain a corresponding preference list.
[0071] In this embodiment, first, feature extraction is performed on the acquired high-dimensional data set. The high-dimensional data set consists of N data points D1, D2, ..., D N Composition, each data point D i Including M different eigenvectors Xi=(X i1 ,X i2 ,,...,X iM ); these features come from different modalities or have different criteria, such as distance, similarity, weight, etc.
[0072] For each eigenvector, according to the criterion C to which it belongs m , generate the corresponding preference list. Each data point D i For Criteria C m Other data points D j Generate preference order, denoted as: Preference List is based on the data points in criterion C m The purpose is to provide preference information for the subsequent stable matching process.
[0073] S2: Perform preliminary clustering on the high-dimensional data set after feature extraction to obtain preliminary clustering results.
[0074] In this embodiment, a traditional clustering algorithm is used to perform preliminary classification of data. For example, a k-means clustering algorithm is selected to firstly group the data points according to a preset criterion to generate K initial clusters C1, C2, ..., C k . Each cluster contains a set of data points: C k ={D i ∣D i Belong to cluster k, k = 1, 2, ..., K}; the results of the preliminary clustering will be used as the initial conditions for subsequent matching optimization.
[0075] S3: Based on the preference list, a stable matching algorithm is used to perform matching optimization on the preliminary clustering result to obtain a matching result.
[0076] In this embodiment, in order to further optimize the preliminary clustering results and ensure that the classification results of data points under all criteria are stable and consistent, a multimodal matching optimization based on the Gale-Shapley stable matching algorithm is introduced. For each criterion, the data points are matched by the Gale-Shapley algorithm, each data point sends a matching request with the cluster according to the preference list, and the data point is selected according to the preference of the cluster under the criterion.
[0077] Assume that cluster C k The data points are generated based on the criterion C m Preference list The preference list includes the preference order of the clusters for the data points under the criteria. The matching process is:
[0078] Each data point D i According to their preference list Cluster C k Issue a matching request;
[0079] Cluster C k According to their preference list Select the received matching request and save it in criterion C m Select the preferred data points and reject other requested data points;
[0080] The rejected request data point continues to send a matching request to the next preferred cluster, and the process is repeated until all data points and clusters reach a stable matching state.
[0081] Furthermore, the above matching process is performed under each criterion to generate a stable matching result. This application constructs a global preference stability function by combining the matching results of all criteria:
[0082]
[0083] where w m Criterion C m The weight of is the data point D i In Guideline C m Next, with cluster C(D i ). By minimizing this function, the matching results under all criteria are guaranteed to achieve optimal consistency in the global scope.
[0084] S4: Optimizing the matching results through a global optimization algorithm and outputting clustering results.
[0085] In this embodiment, after the multimodal stable matching process is completed, a global optimization algorithm is used to further improve the global optimality of the matching result. The goal of global optimization is to adjust the cluster boundaries so that the final clustering result is more accurate. Specifically:
[0086]
[0087] The objective function considers the differences between data points and their clusters under different criteria by weighting, further optimizes the cluster boundaries, and finally outputs the clustering results.
[0088] In summary, Example 1 of the present application first performs multi-criteria feature extraction on a high-dimensional data set to generate a preference list for each data point under different criteria; then, a traditional clustering algorithm is used for preliminary classification to provide a basis for subsequent optimization; a multimodal stable matching algorithm is introduced to optimize the matching results of each data point under multiple criteria to ensure the global consistency and stability of the clustering results; finally, the cluster boundaries are further adjusted through a global optimization algorithm to improve the accuracy and efficiency of the clustering results. This application processes multi-criteria and multi-modal data, significantly improving the stability and accuracy of clustering results, and is suitable for complex data scenarios such as bioinformatics and financial analysis, showing strong technical advantages and application prospects.
[0089] Example 2
[0090] See also Figure 3 , is a structural diagram of a high-dimensional data processing system based on multi-criteria classification and clustering in Example 2 of the present application; the specific contents include:
[0091] Acquisition and processing module: acquires a high-dimensional data set, performs feature extraction on the high-dimensional data set, and obtains a corresponding preference list;
[0092] Preliminary clustering module: performs preliminary clustering on the high-dimensional data set after feature extraction to obtain preliminary clustering results;
[0093] Matching module: Based on the preference list, a stable matching algorithm is used to optimize the matching of the preliminary clustering results to obtain a matching result;
[0094] Optimization and output module: optimizes the matching results through a global optimization algorithm and outputs clustering results.
[0095] Example 3
[0096] See also Figure 4 , is a schematic diagram of the device structure of Embodiment 3 of the present application. The device 50 includes a processor 51 and a memory 52 coupled to the processor 51 .
[0097] The memory 52 stores program instructions for implementing the above-mentioned high-dimensional data processing method based on multi-criteria classification and clustering.
[0098] The processor 51 is used to execute program instructions stored in the memory 52 to implement high-dimensional data processing based on multi-criteria classification and clustering.
[0099] The processor 51 may also be referred to as a CPU (Central Processing Unit).
[0100] The processor 51 may be an integrated circuit chip with signal processing capabilities. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0101] Example 4
[0102] See also Figure 5 , which is a schematic diagram of the structure of the storage medium of Example 4 of the present application. The storage medium of the embodiment of the present application stores a program file 61 that can implement all the above methods, wherein the program file 61 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a computer, a server, a mobile phone, a tablet, and other devices.
[0103] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.
[0104] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.
[0105] Although the embodiments of the present application have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the appended claims and their equivalents.
[0106] Of course, the present invention may have many other implementations. Based on this implementation, other implementations obtained by ordinary technicians in this field without any creative work are all within the scope of protection of the present invention.
Claims
1. A high-dimensional data processing method based on multi-criteria classification and clustering, characterized in that: include: Obtaining a high-dimensional data set, performing feature extraction on the high-dimensional data set, and obtaining a corresponding preference list; Perform preliminary clustering on the high-dimensional data set after feature extraction to obtain preliminary clustering results; Based on the preference list, a stable matching algorithm is used to optimize the matching of the preliminary clustering results to obtain a matching result; The matching results are optimized by a global optimization algorithm, and a clustering result is output.
2. A high-dimensional data processing method based on multi-criteria classification and clustering according to claim 1, characterized in that: The step of obtaining a high-dimensional data set, performing feature extraction on the high-dimensional data set, and obtaining a corresponding preference list specifically includes the following steps: The high-dimensional data set consists of N data points D1, D2, ..., D N Composition, each data point D i Including M different eigenvectors Xi=(X i1 ,X i2 ,,...,X iM ); The eigenvectors are based on the corresponding criterion C m , generate the corresponding preference list; Each data point D i For Criteria C m Other data points D j Generate preference order, denoted as:
3. The high-dimensional data processing method based on multi-criteria classification and clustering according to claim 1, characterized in that: In the step of optimizing the matching of the preliminary clustering results by using a stable matching algorithm based on the preference list to obtain the matching results, the following steps are specifically included: Based on Criteria C m , the data points are matched using the Gale-Shapley stable matching algorithm; Data point D i Matching the preference list with the clusters and selecting data points according to the preferences of the clusters under the criteria; When the cluster C k The data points are generated based on the criterion C m Preference list The preference list includes the preference order of the clusters for the data points under the criteria; the matching process is: Each data point D i According to their preference list Cluster C k Issue a matching request; Cluster C k According to their preference list Select the received matching request and save it in criterion C m Select the preferred data points and reject other requested data points; The rejected request data point continues to send matching requests to the next preferred cluster until all data points and clusters reach a stable matching state.
4. The high-dimensional data processing method based on multi-criteria classification and clustering according to claim 1, characterized in that: After getting the matching results, it also includes: Matching results of combined criteria Construct a global preference stability function: where w m Criterion C m The weight of is the data point D i In Guideline C m Next, with cluster C(D i ) distance or difference.
5. The high-dimensional data processing method based on multi-criteria classification and clustering according to claim 1, characterized in that: The step of optimizing the matching results by a global optimization algorithm and outputting the clustering results specifically includes the following steps: The cluster boundaries are adjusted by the global optimization algorithm, and the formula is: The objective function optimizes the cluster boundaries by weighted analysis of the differences between data points and clusters under different criteria and outputs the clustering results.
6. A system for high-dimensional data processing method based on multi-criteria classification and clustering according to claim 1, characterized in that: include: Acquisition and processing module: acquires a high-dimensional data set, performs feature extraction on the high-dimensional data set, and obtains a corresponding preference list; Preliminary clustering module: performs preliminary clustering on the high-dimensional data set after feature extraction to obtain preliminary clustering results; Matching module: Based on the preference list, a stable matching algorithm is used to optimize the matching of the preliminary clustering results to obtain a matching result; Optimization and output module: optimizes the matching results through a global optimization algorithm and outputs clustering results.
7. A device, characterized in that: The device includes a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a high-dimensional data processing method based on multi-criteria classification and clustering as described in any one of claims 1-5; and the processor is used to execute the program instructions stored in the memory to implement a high-dimensional data processing method based on multi-criteria classification and clustering.
8. A storage medium, characterized in that: The invention stores program instructions executable by a processor, wherein the program instructions are used to execute a high-dimensional data processing method based on multi-criteria classification and clustering as described in any one of claims 1 to 5.