Automatic matching and classification method based on multifactor dimensionality reduction

Abstract

The present invention provides an automatic matching and classification method based on multifactor dimensionality reduction, used for carrying out dimensionality reduction on sets of data classified by labels and reflecting the clustering effect of each dimension, and characterized by using clustering analysis to compare label classes, and automatically matching and evaluating the clustering effect. By means of the method, features after dimensionality reduction are applied to a label classification task, and the classification effect is measured by means of evaluation indexes. The performance of a classification model is observed by means of feature dimensionality reduction, and whether the dimensionality reduction has an obvious effect on the classification result is determined by means of automatic matching.

Description

A method of automatic matching and classification based on multi-factor dimensionality reduction Technical Field

[0001] The present invention relates to the field of data analysis, and in particular to a method for automatic matching and classification based on multi-factor dimensionality reduction. Background Art

[0002] Multi-factor dimensionality reduction is a technique that reduces the dimensionality of data by combining multiple related factors or features to retain the most important information. This can help simplify data analysis and model building, and can also help determine the contribution of each factor to a particular dimension.

[0003] Currently, for labeled classification datasets, after performing multi-factor dimensionality reduction, it's unclear whether a particular dimension significantly affects the classification results. This requires visualization tools. This consumes significant manpower and resources, and manual analysis inevitably leads to erroneous judgments.

[0004] Summary of the Invention

[0005] The present invention aims to overcome the above-mentioned defects. Through automatic matching, data analysts only need to provide a purity ratio standard, and an algorithm can automatically output whether a certain dimension after dimensionality reduction has achieved the clustering effect.

[0006] The present invention provides an automatic matching classification method based on multi-factor dimensionality reduction, which is used to reduce the dimensionality of a labeled classification data set and reflect the clustering effect of each dimension. The method is characterized by:

[0007] Use cluster analysis to compare label categories and automatically match and evaluate the clustering effect.

[0008] Furthermore, the present invention provides a method for automatic matching and classification based on multi-factor dimensionality reduction, characterized by comprising the following steps:

[0009] S1. Obtain the dataset;

[0010] S2. Perform multi-factor dimensionality reduction analysis based on the characteristics of the data set to obtain multi-dimensional data;

[0011] It is preferred to perform dimensionality reduction analysis through an EDA system, and the specific algorithm used is PCA.

[0012] S3. Perform cluster analysis on the dimensional data of S2 to obtain clustered data;

[0013] The preferred clustering method used is the dbsacn algorithm.

[0014] S4. Integrate the clustering data and label data in S3;

[0015] Specifically, the primary key in the clustered data is associated with the primary key in the label data and integrated using the dataframe.

[0016] S5. Group the integrated data of S4 into clusters and calculate the classification proportion of each label in each cluster;

[0017] S6. Measure the dimensionality reduction effect based on the preset purity ratio standard.

[0018] Furthermore, the present invention provides a method for automatic matching and classification based on multi-factor dimensionality reduction, characterized by comprising the following steps:

[0019] The dimensionality reduction analysis of S2 includes the following steps:

[0020] Furthermore, the present invention provides a method for automatic matching and classification based on multi-factor dimensionality reduction, characterized by comprising the following steps:

[0021] The cluster analysis of S3 includes the following steps:

[0022] Furthermore, the present invention provides a method for automatic matching and classification based on multi-factor dimensionality reduction, characterized by comprising the following steps:

[0023] The data integration of S4 includes the following steps:

[0024] Furthermore, the present invention provides a method for automatic matching and classification based on multi-factor dimensionality reduction, characterized by comprising the following steps:

[0025] The method for measuring the dimensionality reduction effect in step S6 is:

[0026] S6-1. Sort the classification proportion of each label in each cluster;

[0027] S6-2. Determine whether there is a label category with a significantly higher proportion than other categories. If so, it indicates that the dimension after dimensionality reduction has an impact on the classification results.

[0028] The judgment of significant high can be set according to different data sets. Generally speaking, if the proportion (purity) corresponding to each label in different label categories reaches more than 80%, it can be considered that the dimension after dimensionality reduction has an impact on the classification result.

[0029] For example: two or more label categories, a, b, c

[0030] The overall performance is that the higher the proportion of a certain category within the cluster, the better, and it has high cohesion characteristics.

[0031] Function and effect of the present invention:

[0032] The present invention develops an algorithm that uses a clustering algorithm to compare label categories and automatically match and evaluate the clustering effect after performing multi-factor dimensionality reduction, which can reflect the dimensionality reduction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG1 is a flow chart of the method for automatic matching and classification based on multi-factor dimensionality reduction involved in this embodiment.

[0034] Figures 2 and 3 show the comparison process of traditional solutions. For labeled classification datasets, after multi-factor dimensionality reduction, visualization tools are used to observe whether a certain dimension has a significant impact on the classification results. DETAILED DESCRIPTION

[0035] The present invention is susceptible to various modifications and embodiments, and thus specific embodiments are illustrated and described in the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments, but rather should be understood to encompass all modifications, equivalents, and even substitutes that fall within the spirit and technical scope of the present invention.

[0036] As shown in Figure 1, the main principle of the automatic matching classification algorithm based on multi-factor dimensionality reduction of the present invention is to cluster the dimensional data after dimensionality reduction, automatically determine the proportion of label classification in each cluster, and thus determine whether this dimension produces an obvious classification effect.

[0037] The specific analysis method is:

[0038] Step 1: Prepare the features of the data set and perform a multi-factor dimensionality reduction algorithm to obtain multi-dimensional data. The results are shown in the following table:

[0039] Step 2: Use the reduced dimension data through a clustering algorithm (k-means / dbsacn) to obtain clustered data. The results are shown in the following table:

[0040] Step 3: Integrate the clustered data with the label data. The result example is shown in the following table:

[0041] Step 4: Group by cluster and calculate the label classification ratio in each cluster. The result example is shown in the following table:

[0042] Step 5: Measure whether the partitioning effect is obvious based on the given purity ratio standard

[0043] If the proportion of one label category in each cluster is very high and the proportion of other label categories is very low, it means that the dimension after dimensionality reduction has a significant impact on the classification results.

[0044] Comparative experiment:

[0045] Traditionally, visualization tools in EDA can be used to show the proportion of each category after grouping, which requires manual judgment. The following is a screenshot of the visualization tool analysis in the self-developed EDA system:

[0046] The significant clustering is shown in Figure 2.

[0047] The situation where the clustering is not obvious is shown in Figure 3.

[0048] In other words, traditional solutions require humans to select different dimensions to see the effects.

[0049] This patent automatically determines whether a certain dimension is good or bad for clustering by calculating indicators in the program, which is obviously more advantageous than traditional solutions.

[0050] The effects and effects of this embodiment are as follows:

[0051] Using the above method, the reduced features are applied to the label classification task, and the classification effect is measured using evaluation metrics. The performance of the classification model is observed by reducing the feature dimension, and automatic matching is used to determine whether the dimensionality reduction has a significant impact on the classification results.

[0052] While the above description focuses on the embodiments, this is merely illustrative and does not limit the present invention. Persons skilled in the art will readily appreciate that various modifications and applications not illustrated above are possible without departing from the essential characteristics of the embodiments. For example, the various components specifically illustrated in the embodiments may be implemented with modifications. Furthermore, any differences associated with such modifications and applications should be construed as being within the scope of the present invention as defined in the appended claims.

Claims

1. A method for automatic matching classification based on multi-factor dimensionality reduction, which is used to reduce the dimensionality of a labeled classification data set and reflect the clustering effect of each dimension, characterized by: Use cluster analysis to compare label categories and automatically match and evaluate the clustering effect.

2. The method for automatic matching and classification based on multi-factor dimensionality reduction according to claim 1, characterized in that: The following steps are included: S1. Get the dataset; S2. Perform multi-factor dimensionality reduction analysis based on the characteristics of the data set to obtain data of multiple dimensions; S3. The dimension data of S2 is subjected to cluster analysis to obtain clustered data; S4. Integrate the clustering data and label data of S3; S5. Group the integrated data of S4 into groups, and calculate the classification proportion of each label in each group; S6. Measure the dimensionality reduction effect based on the preset purity ratio standard.

3. The method for automatic matching and classification based on multi-factor dimensionality reduction according to claim 1, characterized in that: The dimensionality reduction analysis of S2 comprises the following steps:

4. The method for automatic matching and classification based on multi-factor dimensionality reduction according to claim 1, characterized in that: The cluster analysis of S3 includes the following steps:

5. The method for automatic matching and classification based on multi-factor dimensionality reduction according to claim 1, characterized in that: The data integration of S4 includes the following steps:

6. The method for automatic matching and classification based on multi-factor dimensionality reduction according to claim 1, characterized in that: The method for measuring the dimensionality reduction effect in step S6 is: S6-1. Sort the classification proportion of each label in each group; S6-2. Determine whether there is a label category with a proportion significantly higher than other categories. If so, it indicates that the dimension after dimensionality reduction has an impact on the classification result.

Images