A three-branch text clustering method based on improved game theory rough set

Abstract

The present invention provides a text three-branch clustering method based on improved game theory rough sets, comprising: collecting text data, preprocessing the text data, establishing a text data set, establishing a corresponding rough set according to the text data set, clustering the rough set to obtain clustering information, clustering the text data set according to the clustering information to obtain a plurality of data clusters, clustering the text data set based on the improved game theory rough set text three-branch clustering, processing the missing of values ​​in the data set, significantly reducing the calculation time of the text three-branch clustering, improving the versatility of the model, and effectively and efficiently improving the accuracy of the text three-branch clustering.

Description

Technical Field

[0001] The invention relates to the technical field of data mining, and in particular to a text three-branch clustering method based on improved game theory rough sets. Background Art

[0002] Clustering technology is widely used in machine learning, data analysis, artificial intelligence and other fields. It can be used to solve many practical problems, such as market segmentation, recommendation systems, image analysis, text classification, etc.

[0003] The core idea of ​​clustering technology is to find natural groups in the data set, which are composed of data points with similar properties. Clustering algorithms usually represent data points as n-dimensional vectors and use distance metrics to calculate the similarity between data points. Once the similarity is measured, the algorithm can divide the data points into different clusters.

[0004] In modern data science, clustering algorithm has become a very important technology because it can process large-scale data sets and discover structures in them. There are many clustering algorithms to choose from, including hierarchical clustering, k-means clustering, DBSCAN clustering, etc. These algorithms have their own advantages and disadvantages, and can be selected according to the needs of different problems.

[0005] Due to incomplete, missing or damaged data, the general clustering effect is not very good. Three-branch clustering is a good solution to deal with the uncertainty in clustering caused by missing values. The core idea of ​​this method is to place an object in an uncertain class when it is not certain whether an object belongs to a cluster, and postpone the decision on the object. However, there is a key issue in this method, that is, how to determine the threshold of three-branch clustering, which is generally a fixed value.

[0006] Therefore, the present invention provides a text three-branch clustering method based on improved game theory rough sets. Summary of the invention

[0007] The invention discloses a text three-branch clustering method based on improved game theory rough sets. The text data set is clustered based on the text three-branch clustering method based on improved game theory rough sets, and the missing of values ​​in the data set is processed, which significantly reduces the calculation time of the text three-branch clustering, improves the versatility of the model, and effectively and efficiently improves the accuracy of the text three-branch clustering.

[0008] The present invention provides a text three-branch clustering method based on improved game theory rough sets, comprising:

[0009] Step 1: Collect text data;

[0010] Step 2: Preprocess the text data and establish a text dataset;

[0011] Step 3: Establish the corresponding rough set based on the text data set;

[0012] Step 4: Cluster the rough set to obtain clustering information, and perform text clustering on the text data set based on the clustering information to obtain several data clusters.

[0013] In one practicable manner,

[0014] The step 2 comprises:

[0015] Step 21: Eliminate the first non-important words whose occurrence frequency in the text data is less than a preset frequency to obtain target text data;

[0016] Step 22: Record a preset amount of target text data as one category and establish a standard data set;

[0017] Step 23: Eliminate the second non-important words belonging to the preset part of speech in the standard data set to establish a text data set.

[0018] In one practicable manner,

[0019] The step 3 comprises:

[0020] Step 31: Obtain the first set of features corresponding to each text data set respectively;

[0021] Step 32: Divide the text data set into a complete text data set and a defective text data set according to the first set of features;

[0022] Step 33: Convert the complete text data set and the defect text data set into a complete rough set and a defect rough set;

[0023] Step 34: The complete rough set and the defective rough set are uniformly recorded as rough sets.

[0024] In one practicable manner,

[0025] The step 4 comprises:

[0026] Step 41: Obtain the second set features corresponding to each rough set;

[0027] Step 42: record the rough set whose second set features are defect features as defect rough set, and record the rough set whose second set features are complete features as complete rough set;

[0028] Step 43: Obtain defect rough sets and complete rough sets, and establish clustering information;

[0029] Step 44: Obtain a corresponding clustering algorithm according to the clustering information, and perform text clustering according to the clustering algorithm to obtain a number of data clusters.

[0030] In one practicable manner,

[0031] The step 44 comprises:

[0032] Step 441: using consistent manifold approximation and projection to reduce the dimension of each defect rough set, obtain a corresponding compensation rough set, and generate a compensation text data set according to the compensation rough set;

[0033] Step 442: extracting a target clustering algorithm from a preset clustering algorithm library according to the clustering information;

[0034] Step 443: clustering the compensated text data set and the complete text data set using a target clustering algorithm to obtain a number of data clusters.

[0035] In one practicable manner,

[0036] Also includes:

[0037] Get defect rough set;

[0038] Using the preset sample data segments to traverse the defect rough set, the defect data segments on the defect rough set are obtained;

[0039] The data segments except the defective data segments in the defect rough set are recorded as non-defective data segments;

[0040] Obtaining a data ratio between defective data segments and non-defective data segments, and establishing an initial threshold value group according to the data ratio;

[0041] The target clustering algorithm is modified by using the initial threshold group, and the compensated text dataset and the complete text dataset are clustered using the modified target clustering algorithm to obtain several data clusters.

[0042] In one practicable manner,

[0043] Also includes:

[0044] The initial threshold group includes a first initial threshold and a second initial threshold, and the value range of the first initial threshold and the second initial threshold is (0, 1);

[0045] The initial threshold group is combined with the target clustering algorithm, the first initial threshold and the second initial threshold are adjusted within the range of (0, 1), and a clustering result trend graph is established during the adjustment process;

[0046] The target trend point corresponding to the target clustering result in the clustering result trend graph;

[0047] A first value and a second value corresponding to the target trend point are obtained, and the target clustering algorithm is corrected using the first value and the second value.

[0048] In one practicable manner,

[0049] Also includes:

[0050] Each data cluster is named separately and transmitted to the designated terminal for display.

[0051] The beneficial effects that can be achieved by the present invention are as follows: by collecting text data, then preprocessing it to establish a text data set, and then establishing a corresponding rough set, clustering the rough set is achieved to obtain a plurality of data clusters. In this way, the missing values ​​in the data set can be processed before clustering, the calculation time of the three-branch text clustering is significantly reduced, the versatility of the model is improved, and the accuracy of the three-branch text clustering is effectively and efficiently improved.

[0052] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0053] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0055] Figure 1 The figure is a schematic diagram of the workflow of a text three-branch clustering method based on an improved game theory rough set in an embodiment of the present invention;

[0056] Figure 2 Schematic diagram of the workflow of step 3 of a text three-branch clustering method based on improved game theory rough sets in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0058] Example 1

[0059] This embodiment provides a text three-branch clustering method based on improved game theory rough set, such as Figure 1 As shown, including:

[0060] Step 1: Collect text data;

[0061] Step 2: Preprocess the text data and establish a text dataset;

[0062] Step 3: Establish the corresponding rough set based on the text data set;

[0063] Step 4: Cluster the rough set to obtain clustering information, and perform text clustering on the text data set based on the clustering information to obtain several data clusters.

[0064] The working principle and beneficial effects of the above technical solution are as follows: by collecting text data, then preprocessing it to establish a text data set, and then establishing a corresponding rough set, the text data is clustered by clustering the rough set, and a number of data clusters are obtained. In this way, the missing values ​​in the data set can be processed before clustering, which significantly reduces the calculation time of the text three-branch clustering, improves the versatility of the model, and effectively and efficiently improves the accuracy of the text three-branch clustering.

[0065] Example 2

[0066] On the basis of Example 1, the text three-branch clustering method based on improved game theory rough set, the step 2 comprises:

[0067] Step 21: Eliminate the first non-important words whose occurrence frequency in the text data is less than a preset frequency to obtain target text data;

[0068] Step 22: Record a preset amount of target text data as one category and establish a standard data set;

[0069] Step 23: Eliminate the second non-important words belonging to the preset part of speech in the standard data set to establish a text data set.

[0070] In this example, the preset frequency may be 3 times;

[0071] In this example, the first non-important word refers to a word that appears less than a preset frequency;

[0072] In this example, the preset part of speech may be a conjunctive adverb;

[0073] In this example, the second non-important word represents a linking adverb;

[0074] In this example, the preset number may be 50.

[0075] The working principle and beneficial effects of the above technical solution are as follows: In order to speed up clustering and avoid interference of unnecessary factors on clustering results, unimportant words in text data are removed before clustering analysis, and a standard data set is established, thereby reducing the workload of clustering work and improving the efficiency of clustering.

[0076] Example 3

[0077] On the basis of Example 1, the text three-branch clustering method based on improved game theory rough set is as follows: Figure 2 As shown, the step 3 includes:

[0078] Step 31: Obtain the first set of features corresponding to each text data set respectively;

[0079] Step 32: Divide the text data set into a complete text data set and a defective text data set according to the first set of features;

[0080] Step 33: Convert the complete text data set and the defect text data set into a complete rough set and a defect rough set;

[0081] Step 34: The complete rough set and the defective rough set are uniformly recorded as rough sets.

[0082] In this example, the first feature set represents the features of the text data set, which is used to distinguish different text data sets. The "first" is to distinguish it from the feature sets of the subsequent rough sets, and does not have the function of comparison and sorting.

[0083] In this example, the complete text data set means that all data positions in the text data set contain text data and do not contain missing data;

[0084] In this example, defective text data means that the text data set contains blank data bits, and data is missing at the blank data bits;

[0085] In this example, the complete rough set refers to the rough set obtained by classifying all complete text data sets into one category;

[0086] In this example, the defect rough set represents a rough set obtained by classifying all defect text data sets into one category.

[0087] The working principle and beneficial effects of the above technical solution are as follows: by obtaining the set features of the text data set, using the set features to classify the text data set and converting the classified data set into a rough set, the text data is roughly classified by establishing a rough set, thereby facilitating subsequent clustering analysis.

[0088] Example 4

[0089] On the basis of Example 1, the text three-branch clustering method based on improved game theory rough set, the step 4 comprises:

[0090] Step 41: Obtain the second set features corresponding to each rough set;

[0091] Step 42: record the rough set whose second set features are defect features as defect rough set, and record the rough set whose second set features are complete features as complete rough set;

[0092] Step 43: Obtain defect rough sets and complete rough sets, and establish clustering information;

[0093] Step 44: Obtain a corresponding clustering algorithm according to the clustering information, and perform text clustering according to the clustering algorithm to obtain a number of data clusters.

[0094] In this example, the second feature set represents the features of the rough set, where "second" is used to distinguish it from the features of the text data set mentioned above and does not have the function of sorting or comparison.

[0095] The working principle and beneficial effects of the above technical solution are as follows: by utilizing the set characteristics of rough sets, rough set classification is performed, clustering information is established, and then the clustering information is used to match the clustering algorithm, and the clustering algorithm is used to cluster the clustering algorithm to obtain several data clusters, thus completing the clustering work.

[0096] Example 5

[0097] On the basis of Example 4, the text three-branch clustering method based on improved game theory rough set, step 44, includes:

[0098] Step 441: using consistent manifold approximation and projection to reduce the dimension of each defect rough set, obtain a corresponding compensation rough set, and generate a compensation text data set according to the compensation rough set;

[0099] Step 442: extracting a target clustering algorithm from a preset clustering algorithm library according to the clustering information;

[0100] Step 443: clustering the compensated text data set and the complete text data set using a target clustering algorithm to obtain a number of data clusters.

[0101] The working principle and beneficial effects of the above technical solution are as follows: by using consistent manifold approximation and projection to reduce the dimension of the defective rough set, the corresponding compensating rough set can be generated according to the dimension reduction result, so as to establish a compensated text data set, and clustering is performed in combination with the target clustering algorithm proposed in the preset clustering algorithm library to obtain data clusters. By reducing the dimension of the defective rough set, it is compensated and then clustered, which handles the missing values ​​in the data set and significantly reduces the calculation time of the three-branch text clustering.

[0102] Example 6

[0103] On the basis of Example 5, the text three-branch clustering method based on improved game theory rough set further includes:

[0104] Get defect rough set;

[0105] Using the preset sample data segments to traverse the defect rough set, the defect data segments on the defect rough set are obtained;

[0106] The data segments except the defective data segments in the defect rough set are recorded as non-defective data segments;

[0107] Obtaining a data ratio between defective data segments and non-defective data segments, and establishing an initial threshold value group according to the data ratio;

[0108] The target clustering algorithm is modified by using the initial threshold group, and the compensated text dataset and the complete text dataset are clustered using the modified target clustering algorithm to obtain several data clusters.

[0109] In this example, the preset data segment may be a blank data segment with a length of 1.

[0110] The working principle and beneficial effects of the above technical solution are as follows: In order to improve the accuracy of clustering when clustering is performed using a target clustering algorithm, the defective data segment is first divided into multiple data segments using a preset sample data segment, and then the number of defective data segments and non-defective data segments are compared, an initial threshold group is established, and this initial threshold group is used to correct the target clustering algorithm, and finally cluster analysis is performed. This not only improves the accuracy of cluster analysis, but also broadens the scope of application of cluster analysis, and cluster analysis can be performed on different types of text data.

[0111] Example 7

[0112] On the basis of Example 6, the text three-branch clustering method based on improved game theory rough set further includes:

[0113] The initial threshold group includes a first initial threshold and a second initial threshold, and the value range of the first initial threshold and the second initial threshold is (0, 1);

[0114] The initial threshold group is combined with the target clustering algorithm, the first initial threshold and the second initial threshold are adjusted within the range of (0, 1), and a clustering result trend graph is established during the adjustment process;

[0115] The target trend point corresponding to the target clustering result in the clustering result trend graph;

[0116] A first value and a second value corresponding to the target trend point are obtained, and the target clustering algorithm is corrected using the first value and the second value.

[0117] In this example, the initial threshold is set as (α, β) = (1, 0). Different thresholds will affect the accuracy and versatility. The larger the values ​​of α and β are within a limited range, the better the three-branch clustering effect. There are three ways to adjust the threshold: ① reduce the threshold α, recorded as α↓, ② increase the threshold β, recorded as β↑, ③ reduce α and increase β at the same time, recorded as α↓β↑;

[0118] The accuracy of clustering objects with missing values ​​is defined as precision, and its formula is:

[0119]

[0120] Among them, Accuracy (α, β) represents accuracy, Correcthy clustered objects represents the number of correctly clustered objects, and Total clustered objects represents the total number of clustered objects;

[0121] The accuracy of the objects actually being clustered is defined as generalizability, which is given by:

[0122]

[0123] Among them, Generality(α, β) represents generality, Totalclusteredobjects represents the total clustered objects, and Totalobjects inU represents the total number of objects in U;

[0124] Take the median of α and β in (1,0) and (0,1) respectively and calculate the values ​​of Accuracy(α,β) and Generality(α,β) so that the values ​​of these two formulas are as large as possible at the same time. Let the new values ​​of α and β be a and b respectively. Next time, take the median of α and β in (1,a) and (0,b) respectively. Repeat the above steps until the set number of iterations Q is reached, or until Accuracy(α,β)≤Generality(α,β);

[0125] According to the above process, we calculate and determine the appropriate threshold (α, β), which can divide the object into Inside (C k )、Partical(C k )、Outside(C k )Three parts:

[0126] Inside(C k )={o i ∈U|e(c k ,o i )≥α},

[0127] Partical(Ck )={o i ∈U|β<e(c k ,o i )<α},

[0128] Outside(C k )={o i ∈U|e(c k ,o i )≤β}.

[0129] Using the distance formula (i represents the i-th object, a is the attribute of the object, is the value of the ath attribute of the ith object) to determine the distance from all objects in the data set to each missing value, and the normal object o can be calculated through this distance i The nearest neighbor, and then through the evaluation function:

[0130]

[0131] Among them, Number ofo i neighbors belonging to c k Indicates that the object q belongs to class c in the domain k Total neighbors of i represents the number of q domains;

[0132] Quantification object i and the relationship between clusters ckand;

[0133] Use the following formula to transform the object o i Classify into corresponding areas

[0134] Inside(C k )={o i ∈U|e(c k ,o i )≥α},

[0135] Partical(C k )={o i ∈U|β<e(c k ,o i )<α},

[0136] Outside(C k )={o i ∈U|e(c k ,o i )≤β}.

[0137] Among them, Inside represents the core domain, Partical represents the boundary domain, and Outside represents the external domain.

[0138] The working principle and beneficial effects of the above technical solution are as follows: in order to improve the clustering effect, the two initial thresholds are adjusted accordingly within the range of the initial threshold, and then a trend chart of the clustering results during the adjustment process is obtained. The specific data values ​​of the first initial threshold and the second initial threshold are selected according to the trend chart, and then clustering is performed, thereby improving the effectiveness of the clustering algorithm.

[0139] Example 8

[0140] On the basis of Example 1, the text three-branch clustering method based on improved game theory rough set further includes:

[0141] Each data cluster is named separately and transmitted to the designated terminal for display.

[0142] The working principle and beneficial effects of the above technical solution are as follows: In order to distinguish different data clusters, different data clusters are named and then transmitted to a designated terminal for display, so as to facilitate user viewing.

[0143] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A text three-branch clustering method based on improved game theory rough set, characterized in that: include: Step 1: Collect text data; Step 2: Preprocess the text data and establish a text dataset; Step 3: Establish the corresponding rough set based on the text data set; Step 4: Cluster the rough set to obtain clustering information, and perform text clustering on the text data set based on the clustering information to obtain several data clusters; The step 4 comprises: Step 41: Obtain the second set features corresponding to each rough set; Step 42: record the rough set whose second set of features is defective features as defective rough set, and record the rough set whose second set of features is complete features as complete rough set; Step 43: Obtain defect rough sets and complete rough sets, and establish clustering information; Step 44: Obtain a corresponding clustering algorithm according to the clustering information, and perform text clustering according to the clustering algorithm to obtain a number of data clusters; The step 44 comprises: Step 441: using consistent manifold approximation and projection to reduce the dimension of each defect rough set, obtain a corresponding compensation rough set, and generate a compensation text data set according to the compensation rough set; Step 442: extracting a target clustering algorithm from a preset clustering algorithm library according to the clustering information; Step 443: clustering the compensated text data set and the complete text data set using a target clustering algorithm to obtain a number of data clusters; Also includes: Get defect rough set; Using the preset sample data segments to traverse the defect rough set, the defect data segments on the defect rough set are obtained; The data segments except the defective data segments in the defect rough set are recorded as non-defective data segments; Obtaining a data ratio between defective data segments and non-defective data segments, and establishing an initial threshold value group according to the data ratio; The target clustering algorithm is modified by using the initial threshold group, and the compensated text data set and the complete text data set are clustered by using the modified target clustering algorithm to obtain several data clusters; The initial threshold group includes a first initial threshold and a second initial threshold, and the value range of the first initial threshold and the second initial threshold is (0, 1); The initial threshold group is combined with the target clustering algorithm, and the first initial threshold and the second initial threshold are adjusted in the range of (0, 1); Determining an appropriate threshold ,The threshold divides the object into three parts: core domain, boundary domain and external domain.

2. The text three-branch clustering method based on improved game theory rough set as claimed in claim 1, characterized in that: The step 2 comprises: Step 21: Eliminate the first non-important words whose occurrence frequency in the text data is less than a preset frequency to obtain target text data; Step 22: Record a preset amount of target text data as one category and establish a standard data set; Step 23: Eliminate the second non-important words belonging to the preset part of speech in the standard data set to establish a text data set.

3. The text three-branch clustering method based on improved game theory rough set as claimed in claim 1, characterized in that: The step 3 comprises: Step 31: Obtain the first set of features corresponding to each text data set respectively; Step 32: Divide the text data set into a complete text data set and a defective text data set according to the first set of features; Step 33: Convert the complete text data set and the defect text data set into a complete rough set and a defect rough set; Step 34: The complete rough set and the defective rough set are uniformly recorded as rough sets.

4. The text three-branch clustering method based on improved game theory rough set as claimed in claim 1, characterized in that: Also includes: The initial threshold group includes a first initial threshold and a second initial threshold, and the value range of the first initial threshold and the second initial threshold is (0, 1); The initial threshold group is combined with the target clustering algorithm, the first initial threshold and the second initial threshold are adjusted within the range of (0, 1), and a clustering result trend graph is established during the adjustment process; The target trend point corresponding to the target clustering result in the clustering result trend graph; A first value and a second value corresponding to the target trend point are obtained, and the target clustering algorithm is corrected using the first value and the second value.

5. The text three-branch clustering method based on improved game theory rough set as claimed in claim 1, characterized in that: Also includes: Each data cluster is named separately and transmitted to the designated terminal for display.

Images