Text clustering method and device, nonvolatile storage medium and electronic device

Abstract

The application discloses a text clustering method and device, a nonvolatile storage medium and an electronic device. The method comprises the following steps: clustering a text to be clustered according to a first algorithm to obtain a plurality of clustering clusters, and determining a keyword of the clustering cluster according to a second algorithm; determining a connection matrix according to the clustering cluster and the keyword, wherein the connection matrix is a symmetric matrix; performing normalization processing on the connection matrix to obtain a first target connection matrix, decomposing the first target connection matrix according to an average value of the first target connection matrix to obtain a second target connection matrix; and merging the second target connection matrix to generate a target clustering result. The application solves the technical problem that, in the existing clustering algorithm, a clustering model divides different expression methods of the same category into different categories due to a large number of clustering categories, thereby leading to inaccurate clustering.

Description

Technical Field

[0001] This application relates to the field of text clustering technology, and more specifically, to a text clustering method and apparatus, a non-volatile storage medium, and an electronic device. Background Technology

[0002] A simple and practical clustering algorithm for text clustering is the k-means algorithm. Although this algorithm cannot guarantee that it will obtain the optimal clustering result, it has a very good practical effect.

[0003] However, in actual experiments, the following problems were found in the results of K-means clustering:

[0004] 1) If the number of clusters is set to be large, the model will categorize different expressions of frequently occurring problems into different categories. For example, regarding the complaint of "not receiving the discount," some users might directly complain that "I didn't get a discount of ** yuan when I paid," while others might say more specifically, "When I participated in a certain package discount activity, I couldn't enjoy the discount after using YiPay." These are actually the same problem, but due to the high degree of difference in customer expressions, the model might incorrectly classify them as different categories.

[0005] 2) Setting the number of clusters to a smaller number can improve the problem in 1) to some extent, but it introduces another new problem: since sentence encoding based on word vector mean cannot fully extract the semantic meaning of the sentence, it inevitably introduces noise or bias. Therefore, if the number of clusters is small, the model may incorrectly cluster samples from different categories into one category.

[0006] There is currently no effective solution to the above problems. Summary of the Invention

[0007] This application provides a text clustering method and apparatus, a non-volatile storage medium, and an electronic device to at least solve the technical problem in existing clustering algorithms where the clustering model classifies different representations of the same category into different categories due to the large number of clusters, resulting in inaccurate clustering.

[0008] According to one aspect of the embodiments of this application, a text clustering method is provided, comprising: clustering the text to be clustered according to a first algorithm to obtain multiple clusters, and determining the keywords of the clusters according to a second algorithm; determining a connection matrix based on the clusters and keywords, wherein the connection matrix is ​​a symmetric matrix; normalizing the connection matrix to obtain a first target connection matrix; decomposing the first target connection matrix according to the average value of the first target connection matrix to obtain a second target connection matrix; and merging the second target connection matrices to generate a target clustering result.

[0009] Optionally, the connection matrix is ​​determined based on the clusters and keywords, including: determining a first connection matrix based on the initial connection matrix, the frequency of the first target word, the criticality of the target words in the first cluster, and the length of the target intersection, wherein the criticality of the target words in the first cluster is determined according to a first algorithm; determining a second connection matrix based on the initial connection matrix, the frequency of the second target word, the criticality of the target words in the second cluster, and the length of the target intersection, wherein the criticality of the target words in the first cluster is determined according to a second algorithm; and determining the connection matrix based on the first connection matrix, the second connection matrix, and the length of the target intersection.

[0010] Optionally, before determining the first connection matrix, the method further includes: generating an initial connection matrix, wherein the initial connection matrix is ​​a K·K connection matrix with a value of 0, where K is the number of clusters; obtaining the keywords of the first cluster and the keywords of the second cluster in the multiple clusters, and determining the target intersection of the keywords of the first cluster and the keywords of the second cluster; obtaining the first target word frequency of the target word based on the occurrence frequency of the target word in the text to be clustered in the first cluster and the occurrence frequency of all words in the text to be clustered, wherein the first target word frequency is the word frequency of the target word in the direction from the first cluster to the second cluster; obtaining the second target word frequency of the target word based on the occurrence frequency of the target word in the text to be clustered in the second cluster and the occurrence frequency of all words in the text to be clustered, wherein the second target word frequency is the word frequency of the target word in the direction from the second cluster to the first cluster.

[0011] Optionally, K is the number of clusters within the target range randomly generated according to the first algorithm.

[0012] Optionally, normalizing the connection matrix to obtain a first target connection matrix includes: transforming the connection matrix into a first connection matrix, wherein the first connection matrix is ​​an asymmetric matrix; performing a first processing on the first connection matrix to obtain a second connection matrix, wherein the first processing includes at least one of the following: matrix broadcasting and matrix shuffling, wherein the first processing is used to remove the first target number from each row of the first connection matrix; performing a second processing on the second connection matrix to obtain a third connection matrix, wherein the second processing includes at least: calculating the exponent of the second target number of each element in the second connection matrix; performing a third processing on the third connection matrix to obtain a fourth connection matrix, wherein the third processing includes at least: summing the elements in each row of the third connection matrix; and performing a fourth processing on the fourth connection matrix to obtain the first target connection matrix, wherein the fourth processing includes at least: determining the dimension of the fourth connection matrix based on the dimension of the connection matrix.

[0013] Optionally, before merging the second target connection matrix, the method further includes: removing abnormal clusters from the second target connection matrix, wherein the abnormal clusters are subsets of a set consisting of multiple clusters.

[0014] Optionally, the first target connection matrix is ​​decomposed according to the average value of the first target connection matrix to obtain the second target connection matrix, including: traversing all elements in the first target connection matrix, when the value of an element is greater than the average value, the element is assigned a first number, and when the value of an element is not greater than the average value, the element is assigned a second number; and determining the second target connection matrix based on the first number and the second number.

[0015] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the storage medium including a stored program, wherein the program controls the device where the storage medium is located to execute the above text clustering method when it runs.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes the above-described text clustering method during runtime.

[0017] In this embodiment, the text to be clustered is clustered according to a first algorithm to obtain multiple clusters, and keywords of the clusters are determined according to a second algorithm. A connection matrix is ​​determined based on the clusters and keywords, wherein the connection matrix is ​​a symmetric matrix. The connection matrix is ​​normalized to obtain a first target connection matrix. The first target connection matrix is ​​decomposed according to the average value of the first target connection matrix to obtain a second target connection matrix. The second target connection matrices are merged to generate the target clustering result. By constructing the connection matrix and normalizing the connection matrix, the purpose of optimizing the K-Means clustering method is achieved, thereby realizing a more accurate clustering effect for text. This solves the technical problem in existing clustering algorithms where the clustering model classifies different representations of the same category into different categories due to a large number of clusters, resulting in inaccurate clustering. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a flowchart of a text clustering method according to an embodiment of this application;

[0020] Figure 2 This is a diagram showing the effect of text clustering based on relevant technologies;

[0021] Figure 3 This is a diagram showing the effect of another text clustering method based on related technologies;

[0022] Figure 4 This is a diagram illustrating the effect of a text clustering method implemented according to this application.

[0023] Figure 5 This is a diagram illustrating the effect of another text clustering method implemented according to this application;

[0024] Figure 6 This is a diagram showing the effect of another text clustering method based on related technologies;

[0025] Figure 7 This is a diagram illustrating the effect of another text clustering method implemented according to this application;

[0026] Figure 8 It is a contour coefficient comparison chart based on relevant technologies;

[0027] Figure 9 This is a contour coefficient comparison diagram implemented according to this application;

[0028] Figure 10 It is a contour coefficient comparison chart in a low-dimensional data space based on relevant technologies;

[0029] Figure 11 This is a contour coefficient comparison diagram in a low-dimensional data space according to an embodiment of this application;

[0030] Figure 12 This is a structural diagram of a text clustering device according to an embodiment of this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0034] Clustering refers to the process of dividing a given set of objects into different subsets, with the goal of making the elements within each subset as similar as possible, and the elements between different subsets as dissimilar as possible. These subsets are also called clusters, and generally have no intersection.

[0035] Text clustering, also known as document clustering, refers to the clustering analysis of documents and is widely used in text mining and information retrieval. Initially, text clustering was only used for text archiving, but later, many new uses were discovered, such as improving search results and generating synonyms.

[0036] Silhouette coefficient: A method for evaluating the quality of clustering. The silhouette coefficient combines two factors: cohesion and separation. It can be used to evaluate the impact of different algorithms or different ways of running algorithms on the clustering results, based on the same original data.

[0037] Clustering is commonly used for data preprocessing or archiving similar data. Its process, except for not requiring labeled data, is similar to most known tasks: extracting features and then feeding them to a machine learning algorithm. Through clustering, websites can provide users with generalized recommendations. For example, science fiction films might be automatically grouped into the same cluster. After a user watches one of these films, the website will recommend several other films most similar to it. This recommendation is not personalized for each user because clustering does not consider individual user preferences; it only extracts features from the films themselves. Generalized recommendations are particularly helpful for new users because newly registered users have little viewing history, making it difficult to predict their preferences. In this case, recommending similar films through clustering is often a smooth "cold start" strategy. With a small amount of manual sampling, clustering can also automatically filter samples containing certain common characteristics. For example, apps with artificially inflated ratings often have a high ratio of positive reviews to downloads, but low daily active users and retention rates. The specific values ​​of these metrics are difficult to determine manually, but there should be a fixed range. After grouping newly released apps into several clusters through clustering, a few samples are randomly selected from each cluster for manual review. The clusters of apps identified by manual review manipulation likely contain many more similar apps, thus narrowing the scope of the checks and reducing labor costs.

[0038] Clustering can also play a role in text preprocessing. For example, before annotating the corpus, it is usually necessary to select a certain number of representative documents as samples from the raw corpus. Suppose N documents need to be annotated, these raw documents can be clustered into N clusters, and one document can be randomly selected from each cluster. Taking advantage of the property that the elements within each cluster are similar, clustering can even be used for text deduplication.

[0039] In related technologies, the k-means algorithm is a simple and practical clustering algorithm for text clustering. While it cannot guarantee the optimal clustering result, its practical performance is excellent. The basic idea of ​​the K-means algorithm is to cluster objects based on k points in space, grouping them according to their nearest neighbors. Iteratively updating the values ​​of each cluster center until the best clustering result is obtained. The steps of the K-means algorithm are as follows:

[0040] 1. Select k initial samples as initial cluster centers a = a1, a2, ... a k ;

[0041] 2. For each sample i in the dataset, calculate its distance to the k cluster centers and assign it to the cluster c corresponding to the cluster center with the smallest distance. (i) ∶=argminμ||c (i) -μ j μ||2 ;

[0042] 3. For each category a j Recalculate its cluster centers That is, the centroid of all samples belonging to this class;

[0043] 4. Repeat steps 2 and 3 until a certain termination condition is met (number of iterations, minimum error change, etc.).

[0044] Disadvantages of the K-means algorithm:

[0045] 1. The K value needs to be set manually, and different K values ​​will yield different results;

[0046] 2. It is sensitive to the initial cluster centers; different selection methods will yield different results.

[0047] 3. Sensitive to outliers;

[0048] 4. Samples can only be classified into one class, making it unsuitable for multi-class classification tasks;

[0049] 5. Not suitable for highly discrete classifications, classifications with imbalanced sample classes, or classifications with non-convex shapes.

[0050] Because the K-Means clustering method requires specifying the number of clusters in advance, and the number of clusters has a significant impact on the clustering effect, it is necessary to try multiple values ​​based on business logic and continuously optimize the number of clusters before clustering. Therefore, the results of K-means clustering have the following problems:

[0051] 1. When the number of K samples is set relatively small, the model will mix up samples of different types, making it difficult to correct. For example... Figure 2 As shown.

[0052] 2. When the number of K is large, the model splits samples of the same class into several classes. For example... Figure 3 As shown, at this point, categories 1 and 2, and categories 3 and 4 need to be merged back into two new categories, making the correction more difficult than problem 1. Figure 2 The situation is low.

[0053] To address the issue of clustering models classifying different representations of the same category into different categories due to a large number of clusters, this application provides relevant solutions in its embodiments, such as... Figure 4 As shown, this application embodiment addresses the case of a large number of K clusters by mining the characteristics of each cluster and merging highly correlated clusters through a certain mechanism to obtain the ideal clustering result.

[0054] According to an embodiment of this application, a method embodiment for text clustering is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0055] Figure 1 This is a flowchart of a text clustering method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0056] Step S102: Cluster the text to be clustered according to the first algorithm to obtain multiple clusters, and determine the keywords of the clusters according to the second algorithm.

[0057] According to an optional embodiment of this application, the K-means algorithm is used to cluster the text to be clustered. The K-means algorithm randomly assigns a K value within a specific range. In the K-means algorithm, the K value represents the number of categories, i.e., the number of clusters. The text-rank algorithm is used to obtain the keywords of the clusters generated by the K-means algorithm. These keywords can be domain keywords. For example, in the field of big data, domain keywords can include: data resource utilization, data governance, data assetization, data circulation, data security, etc.

[0058] Step S104: Determine the connection matrix based on the clusters and keywords, wherein the connection matrix is ​​a symmetric matrix.

[0059] According to another optional embodiment of this application, the specific calculation steps for determining the connection matrix are as follows:

[0060] First, initialize a K·K connection matrix C with a value of 0, where K represents the initial number of clusters in the K-means algorithm.

[0061]

[0062] Then, the initial connection matrix is ​​filled by calculating the degree of connection between categories according to the following formula.

[0063] I = W S ∩W T (2)

[0064]

[0065]

[0066]

[0067]

[0068] (2) In the formula, I represents the intersection of keywords in the S-th cluster and the T-th cluster, W S and W T These represent the keywords in their respective clusters;

[0069] (3) In the formula The frequency of keywords in category S pointing in the direction T. The number of times a keyword in cluster S appears in the text to be clustered is represented by N, and the total number of times all words appear in the text is represented by N. Equation (3) is a normalization process, the purpose of which is to eliminate the difference in document length.

[0070] (4) This indicates that the i-th word in the direction T points to S affects the connectivity between the two clusters, where L represents the importance of the i-th word in cluster S, which can be determined using the K-means algorithm. I It is the length of the intersection of the keywords of the two clusters, and similarly, equation (5) This indicates the impact of the j-th word in the direction S points to T on the connectivity between the two clusters.

[0071] Equation (6) represents the summation of word-based connectivity in both directions (S to T and T to S), followed by summation in various clusters to obtain the final connectivity between the two clusters S and T. TS This is the connection matrix determined in step S104 based on the clusters and keywords. The advantage of calculating in equation (6) is that it takes into account the interaction between the word units and the cluster units in the intersection of keywords of the two clusters. The interaction at the word unit mainly involves the influence in the corresponding direction and is subject to normalized constraints. It also takes into account the mutual influence between clusters.

[0072] Step S106: Normalize the connection matrix to obtain the first target connection matrix. Decompose the first target connection matrix according to the average value of the first target connection matrix to obtain the second target connection matrix.

[0073] In some optional embodiments of this application, a symmetric softmax function is used to normalize the connection matrix to a 0-1 space to facilitate subsequent calculations. The softmax calculation formula is as follows:

[0074] C TS (K,K)→C TS (1,K·K) (7)

[0075]

[0076]

[0077]

[0078] m_softmax=(m _exp / m _sum [1,K·K]→[K,K] (11)

[0079] Equation (7) serves to first transform the K*K symmetric matrix C TS (K,K) is converted to C of 1*K·K. TS (1, K·K) facilitates subsequent calculations because if we calculate the original matrix, we cannot guarantee the symmetry of the matrix while ensuring the effect of softmax. At the same time, if we guarantee the symmetry of the matrix, we cannot guarantee softmax. Therefore, we need to transform the shape of the matrix first.

[0080] (8) is used to prevent the infimum from occurring during the calculation process. Each row of the transformed matrix is ​​subtracted from the largest number in its own row. Matrix broadcasting and matrix recombination are used here.

[0081] Equation (9) calculates each value in the matrix to the power of e, Equation (10) performs a summation operation on each row in the matrix, and Equation (11) divides each original data in the matrix by the sum of each row and restores the shape of the matrix to its original shape.

[0082] In some optional embodiments of this application, the normalized connection matrix is ​​decomposed according to a threshold. In this embodiment, the threshold used is the average value of the matrix, and the decomposition is performed according to the following formula and matrix:

[0083]

[0084] The conversion result of the above formula is as follows:

[0085]

[0086] Step S108: Merge the second target connection matrix to generate the target clustering result.

[0087] As an optional embodiment of this application, according to step S106, the second target connection matrix L ST As shown below:

[0088]

[0089] As another optional embodiment of this application, the pseudocode for the merging algorithm is as follows:

[0090] Input: A normalized symmetric matrix L ST =[K,K],L ST This represents the symmetric matrix after normalization matrix decomposition. The matrix has a size of K*K, where K represents the number of cluster centers set by the preceding K-means clustering algorithm.

[0091] Output: An array of cluster centers after initial merging, where each element represents a set.

[0092] 01:for i←1to Input.shape do

[0093] 02:{row_i.set()and row_i.add(i)}

[0094] 03:for j←1to Input.shape do

[0095] 05: if Input[i][j]=1do

[0096] 06:{row_i.add(j)}

[0097] 07:end if

[0098] 08: if len(Output) = 0 do

[0099] 09:{Output.append(row_i))

[0100] 10:else

[0101] 11:for k←1to Output

[0102] 12:if row_i.issubset(k)do

[0103] 13:{break};

[0104] 14:elif k.issubset(row_i)do

[0105] 15:{k=value}

[0106] 16:else

[0107] 17: if index = len(Output) - 1 do

[0108] 18:{Output.append(row_i)}

[0109] 19:end if

[0110] 20:end if

[0111] 21:end for

[0112] 22:end for

[0113] 23:end for

[0114] 24: return Output

[0115] Through the second target connection matrix L ST As can be seen, the first, fifth, and sixth clusters can be merged into a new cluster; the second cluster is a separate category; and the third and fourth clusters can be merged into another new cluster. The final clustering result is shown in the figure below. Figure 5 As shown.

[0116] Based on the above steps, by constructing a connection matrix and normalizing it, the K-Means clustering method is optimized, thereby achieving a more accurate clustering effect for text.

[0117] According to an optional embodiment of this application, the connection matrix is ​​determined based on clusters and keywords by the following method: a first connection matrix is ​​determined based on an initial connection matrix, a first target word frequency, the keyness of the target words in the first cluster, and the length of the target intersection, wherein the keyness of the target words in the first cluster is determined according to a first algorithm; a second connection matrix is ​​determined based on an initial connection matrix, a second target word frequency, the keyness of the target words in the second cluster, and the length of the target intersection, wherein the keyness of the target words in the first cluster is determined according to a second algorithm; and the connection matrix is ​​determined based on the first connection matrix, the second connection matrix, and the length of the target intersection.

[0118] According to another optional embodiment of this application, the initial connection matrix is ​​a K·K connection matrix with a value of 0, where K represents the initial number of clusters in the K-means algorithm. The first target word frequency is the ratio of the occurrence frequency of the target word in the text to be clustered in the first cluster to the occurrence frequency of all words in the text to be clustered. The criticality of the target word in the first cluster can be determined according to the K-means algorithm. The target intersection is the intersection of the target words in the S-th cluster and the T-th cluster.

[0119] In some optional embodiments of this application, before determining the first connection matrix, the following method can be used: generating an initial connection matrix, wherein the initial connection matrix is ​​a K·K connection matrix with a value of 0, where K is the number of clusters; obtaining the keywords of the first cluster and the keywords of the second cluster in the multiple clusters, and determining the target intersection of the keywords of the first cluster and the keywords of the second cluster; obtaining the first target word frequency of the target word based on the occurrence frequency of the target word in the text to be clustered in the first cluster and the occurrence frequency of all words in the text to be clustered, wherein the first target word frequency is the word frequency of the target word in the direction from the first cluster to the second cluster; obtaining the second target word frequency of the target word based on the occurrence frequency of the target word in the text to be clustered in the second cluster and the occurrence frequency of all words in the text to be clustered, wherein the second target word frequency is the word frequency of the target word in the direction from the second cluster to the first cluster.

[0120] In some optional embodiments of this application, K is the number of clusters within the target range randomly generated according to the first algorithm.

[0121] According to an optional embodiment of this application, normalizing a connection matrix to obtain a first target connection matrix includes the following steps: transforming the connection matrix into a first connection matrix, wherein the first connection matrix is ​​an asymmetric matrix; performing a first processing on the first connection matrix to obtain a second connection matrix, wherein the first processing includes at least one of the following: matrix broadcasting and matrix shuffling, the first processing being used to remove a first target number from each row of the first connection matrix; performing a second processing on the second connection matrix to obtain a third connection matrix, wherein the second processing includes at least: calculating the exponent of a second target number for each element in the second connection matrix; performing a third processing on the third connection matrix to obtain a fourth connection matrix, wherein the third processing includes at least: summing the elements in each row of the third connection matrix; and performing a fourth processing on the fourth connection matrix to obtain the first target connection matrix, wherein the fourth processing includes at least: determining the dimension of the fourth connection matrix based on the dimension of the connection matrix.

[0122] According to another optional embodiment of this application, a symmetric softmax is used to normalize the connection matrix to a 0-1 space to facilitate subsequent calculations. Softmax, literally, can be divided into two parts: soft and max. Max, as the name suggests, means the maximum value. The core of softmax lies in "soft," which implies flexibility, as opposed to "hard." In many scenarios, we need to find the element with the largest value among all elements in an array; essentially, this involves calculating the hardmax. The formula for hardmax is as follows:

[0123]

[0124] The biggest characteristic of hardmax is that it selects only the largest value, i.e., a black-and-white approach. However, this approach is often unreasonable in practice. For example, in text classification, an article contains various thematic information, and we would prefer to obtain the probability value (confidence score) of the article for each possible text category, which can be simply understood as the likelihood of belonging to the corresponding category. Therefore, the concept of softmax is used here. The meaning of softmax is that instead of uniquely determining a single maximum value, it assigns a probability value to each output classification result, representing the likelihood of belonging to each category. The softmax expression is as follows:

[0125]

[0126] Where z i Let C be the output value of the i-th node, and C be the number of output nodes, i.e., the number of categories. The softmax function can convert the output values ​​of multi-class classification into a probability distribution ranging from [0,1] to 1. Softmax can thus normalize the output space to the 0,1 space, mitigating computational difficulties and data invalidation caused by differences in data volume.

[0127] In some optional embodiments of this application, before merging the second target connection matrix, the method further includes: removing abnormal clusters from the second target connection matrix, wherein the abnormal clusters are subsets of a set consisting of multiple clusters.

[0128] As an optional embodiment of this application, an abnormal cluster is a cluster that can be classified individually and is located in a set composed of multiple clusters. For example, the original classification is 1; 2; 3; 4; 5; 6; 7; 8; 9; 10, a total of 10 categories (clusters). After reclassification, it becomes 123; 2; 456; 789; 10. At this time, category (cluster) 2 is an abnormal cluster. In order to ensure the accuracy of clustering, the abnormal cluster needs to be removed.

[0129] According to an optional embodiment of this application, the pseudocode for removing abnormal clusters (cluster islands) is as follows:

[0130] Input: An array of initial merged cluster centers, where each element represents a set.

[0131] Output: An array of cluster centers after removing data silos, where each element represents a set.

[0132] 01: define big_set, Output

[0133] 02:for i←1to Input

[0134] 03: if len(i)>1do

[0135] 04:big_set=set.union(i,big_set)

[0136] 05:end if

[0137] 06:end for

[0138] 07:for j←1to Input

[0139] 08: if len(j) = 1 do

[0140] 09:if j not∈big_set

[0141] 10: Output.append(j)

[0142] 12:end if

[0143] 13:else:

[0144] 14: Output.append(j)

[0145] 15:end if

[0146] 16:end for

[0147] 17: return Output

[0148] In some optional embodiments of this application, the first target connection matrix is ​​decomposed according to the average value of the first target connection matrix to obtain the second target connection matrix, including the following steps: traversing all elements in the first target connection matrix, when the value of an element is greater than the average value, the element is assigned a first number, and when the value of an element is not greater than the average value, the element is assigned a second number; and determining the second target connection matrix based on the first number and the second number.

[0149] As another optional embodiment of this application, all elements at all positions in the first target connection matrix are traversed. When the value of an element exceeds the average value of the first target connection matrix, the value of that element is 1; when the value of an element does not exceed the average value of the first target connection matrix, the value of that element is 0. The decomposition process is as follows:

[0150]

[0151] Where, matrix C STThe value of the element in the first row and second column of matrix C is 0.22, which is less than the value of the element in matrix C. ST If the average value is 0, then the value of the element in the first row and second column of the decomposed matrix is ​​0.

[0152] According to an optional embodiment of this application, in order to evaluate the effectiveness of this application, two clustering evaluation methods are introduced below. The first method directly visualizes the features of the clustered data and the themes of the cluster centers. This allows direct observation of the entire data distribution space and simultaneously observes the clustering effect of the data. This evaluation method is as follows: Figure 6 and Figure 7 As shown; the second evaluation method is to introduce the profile coefficient as an indicator to evaluate this application and related technologies, such as Figure 8 As shown in ,9,10,11.

[0153] Figure 6 This is the result of clustering using the traditional K-means algorithm, such as Figure 6 As shown, the initial cluster centers are 100. It can be seen that many categories are very close together and need to be merged. However, due to the nature of the K-means algorithm, the number of cluster centers must be fixed each time K-means is initialized. Since the clustering algorithm is an unsupervised algorithm and the number of cluster centers in the data to be processed is uncertain, the randomly initialized K value cannot accurately match the true distribution of the data. As a result, many data that should be clustered together are divided into multiple clusters.

[0154] Figure 7 This is a clustering effect diagram of this application, such as... Figure 7 As shown, this application effectively merges many redundant categories. On the premise of effectively merging the original K-means redundant clusters, it also achieves the technical effect of making the data clustering effect linearly separable without affecting the original reasonably distributed clusters.

[0155] According to another optional embodiment of this application, the core idea of ​​the silhouette coefficient is to determine the relative magnitude of the inter-class distance and the intra-class distance. If the inter-class distance is greater than the intra-class distance, the clustering result is good; otherwise, it is bad. This idea of ​​the silhouette coefficient is very similar to Fisher's linear discriminant analysis, both of which determine the relative magnitude of the inter-class distance and the intra-class distance. The difference is that the silhouette coefficient is used to measure the quality of the clustering result, while the comparison of inter-class and intra-class distances in Fisher's linear discriminant analysis is used to reduce the original dimensionality data to a one-dimensional linear space. This dimensionality reduction has a prerequisite: that the categories can be well distinguished in the reduced space.

[0156] 1. Calculate the average distance a(i) from sample i to other samples in the same cluster. The smaller a(i) is, the more likely sample i should be clustered into that cluster. a(i) is called the intra-cluster dissimilarity of sample i. The mean a(i) of all samples in cluster C is called the cluster dissimilarity of cluster C.

[0157] 2. Calculate the distance from sample i to another cluster C. j The average distance b of all samples ij , referred to as sample i and cluster C j The dissimilarity between clusters is defined as the dissimilarity between samples i:

[0158] b(i)=min{b(i)1,b(i)2,...,b(i) k}b(i)

[0159] The larger b(i) is, the less likely sample i belongs to other clusters.

[0160] 3. Define the silhouette coefficient of sample i based on the intra-cluster dissimilarity a(i) and inter-cluster dissimilarity a(i) of sample i:

[0161]

[0162] 4. Judgment:

[0163] If s(i) is close to 1, it indicates that the clustering of sample i is reasonable;

[0164] If s(i) is close to -1, it means that sample i should be classified into another cluster.

[0165] If s(i) is approximately 0, it means that sample i is on the boundary between the two clusters.

[0166] In some optional embodiments of this application, the silhouette coefficient is a standard evaluation method for clustering models. The silhouette coefficient can describe the clustering effect from the following two aspects:

[0167] 1. A silhouette coefficient of -1 indicates poor clustering results, +1 indicates compact clustering, and 0 indicates cluster overlap.

[0168] 2. The larger the silhouette coefficient, the more compact the instances within a cluster are and the greater the distance between clusters, which is the standard concept of clustering.

[0169] A comparison chart of profile coefficients in related technologies is shown below. Figure 8 As shown, based on the definition of the silhouette coefficient and the criteria for judging the clustering effect, it can be seen that... Figure 8Since the initial model set the k value to be relatively large, at 100, the proportion of silhouette coefficients less than 0 is relatively large and the distribution is relatively scattered. At this time, the clustering effect is not ideal, and the final clustering result does not conform to the true distribution space of the data. Many clusters can actually be merged into one space.

[0170] Figure 9 After keyword normalization and connection matrix decomposition and merging, the model merges many clusters of data with values ​​less than 0 together to form new clusters, resulting in a significant improvement in clustering performance. Figure 9 The 11th cluster in the model effectively merged many scattered data points that were less than zero and originally belonged to other clusters.

[0171] Figure 10 This refers to a clustering scenario where the cluster center is set to 20 in related technologies. In this case, many data points do not actually belong to this category, so the silhouette coefficients of many data points are less than 0.

[0172] Figure 11 This is the optimized and corrected result of this application (Key-means model). After Key-means optimization, the original clusters are adjusted to 8. At this point, except for one data point less than 0 in cluster 0, all other categories are correctly assigned to the correct clusters. Figure 11 It can be seen that the Key-means model effectively optimizes the shortcomings of related techniques (K-means model), can correctly correct the problem of fixed cluster centers caused by the inability of K-means to determine cluster centers, and the cluster data distribution after Key-means optimization is more consistent with the real data distribution space.

[0173] Figure 12 This is a structural diagram of a text clustering device according to an embodiment of this application, such as... Figure 12 As shown, the device includes:

[0174] The first determining module 1202 is used to cluster the text to be clustered according to the first algorithm to obtain multiple clusters, and to determine the keywords of the clusters according to the second algorithm.

[0175] The second determining module 1204 is used to determine the connection matrix based on the clusters and keywords, wherein the connection matrix is ​​a symmetric matrix;

[0176] The first generation module 1206 is used to normalize the connection matrix to obtain the first target connection matrix, and decompose the first target connection matrix according to the average value of the first target connection matrix to obtain the second target connection matrix.

[0177] The second generation module 1208 is used to merge the second target connection matrix to generate the target clustering result.

[0178] It should be noted that, Figure 12 Preferred embodiments of the shown examples can be found in [reference needed]. Figure 1 The relevant descriptions of the embodiments shown will not be repeated here.

[0179] This application also provides a non-volatile storage medium, which includes a stored program, wherein the program, when running, controls the device where the storage medium is located to execute the above text clustering method.

[0180] A program for a non-volatile storage medium to perform the following functions: clustering the text to be clustered according to a first algorithm to obtain multiple clusters, and determining the keywords of the clusters according to a second algorithm; determining a connection matrix based on the clusters and keywords, wherein the connection matrix is ​​a symmetric matrix; normalizing the connection matrix to obtain a first target connection matrix; decomposing the first target connection matrix according to the average value of the first target connection matrix to obtain a second target connection matrix; merging the second target connection matrices to generate the target clustering result.

[0181] This application also provides an electronic device, including a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the above-described text clustering method during runtime.

[0182] The processor is used to run a program that performs the following functions: clustering the text to be clustered according to a first algorithm to obtain multiple clusters, and determining the keywords of the clusters according to a second algorithm; determining a connection matrix based on the clusters and keywords, wherein the connection matrix is ​​a symmetric matrix; normalizing the connection matrix to obtain a first target connection matrix; decomposing the first target connection matrix according to the average value of the first target connection matrix to obtain a second target connection matrix; and merging the second target connection matrices to generate the target clustering result.

[0183] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0184] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0185] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0186] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0187] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0188] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0189] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A text clustering method, characterized in that, include: The text to be clustered is clustered according to the first algorithm to obtain multiple clusters, and the keywords of the clusters are determined according to the second algorithm. A connection matrix is ​​determined based on the clusters and the keywords. Specifically, a first connection matrix is ​​determined based on an initial connection matrix, the frequency of a first target word, the criticality of the target word in the first cluster, and the length of the target intersection. The criticality of the target word in the first cluster is determined according to the first algorithm. A second connection matrix is ​​determined based on the initial connection matrix, the frequency of a second target word, the criticality of the target word in the second cluster, and the length of the target intersection. The criticality of the target word in the second cluster is determined according to the first algorithm. The connection matrix is ​​a symmetric matrix, determined based on the first connection matrix, the second connection matrix, and the length of the target intersection. The connection matrix is ​​normalized to obtain a first target connection matrix. The first target connection matrix is ​​then decomposed based on its average value to obtain a second target connection matrix. Specifically, all elements in the first target connection matrix are iterated over; if the value of an element is greater than the average value, the element is assigned a first value; if the value of an element is not greater than the average value, the element is assigned a second value. The second target connection matrix is ​​then determined based on the first value and the second value. The second target connection matrix is ​​merged to generate the target clustering result.

2. The text clustering method according to claim 1, characterized in that, Before determining the first connectivity matrix, the method further includes: Generate the initial connection matrix, wherein the initial connection matrix is ​​a K·K connection matrix with a value of 0, and K is the number of clusters; Obtain the keywords of the first cluster in the plurality of clusters and the keywords of the second cluster in the plurality of clusters, and determine the target intersection of the keywords of the first cluster and the keywords of the second cluster; The first target word frequency of the target word is obtained based on the number of occurrences of the target word in the text to be clustered in the first cluster and the total number of occurrences of all words in the text to be clustered. The first target word frequency is the word frequency of the target word in the direction from the first cluster to the second cluster. The second target word frequency of the target word is obtained based on the number of occurrences of the target word in the text to be clustered in the second cluster and the total number of occurrences of all words in the text to be clustered. The second target word frequency is the word frequency of the target word in the direction from the second cluster to the first cluster.

3. The text clustering method according to claim 2, characterized in that, K is the number of clusters within the target range that are randomly generated according to the first algorithm.

4. The text clustering method according to claim 1, characterized in that, The connection matrix is ​​normalized to obtain the first target connection matrix, which includes: The connection matrix is ​​transformed into a first connection matrix, wherein the first connection matrix is ​​an asymmetric matrix; The first connection matrix is ​​subjected to a first process to obtain a second connection matrix, wherein the first process includes at least one of the following: matrix broadcasting and matrix shuffling, and the first process is used to remove the first target number from each row of the first connection matrix; The second connection matrix is ​​subjected to a second processing to obtain a third connection matrix, wherein the second processing includes at least: calculating the exponent of the second target number of each element in the second connection matrix; The third connection matrix is ​​subjected to a third processing to obtain a fourth connection matrix, wherein the third processing includes at least: summing the elements in each row of the third connection matrix; The fourth connection matrix is ​​subjected to a fourth processing to obtain the first target connection matrix, wherein the fourth processing includes at least: determining the dimension of the fourth connection matrix based on the dimension of the connection matrix.

5. The text clustering method according to claim 1, characterized in that, Before merging the second target connectivity matrix, the method further includes: Remove abnormal clusters from the second target connection matrix, wherein the abnormal clusters are subsets of the set consisting of multiple clusters.

6. A text clustering device, characterized in that, include: The first determining module is used to cluster the text to be clustered according to the first algorithm to obtain multiple clusters, and to determine the keywords of the clusters according to the second algorithm. The second determining module is used to determine a connection matrix based on the clusters and the keywords. Specifically, it determines a first connection matrix based on an initial connection matrix, a first target word frequency, the criticality of the target words in the first cluster, and the length of the target intersection, wherein the criticality of the target words in the first cluster is determined according to the first algorithm; it determines a second connection matrix based on the initial connection matrix, a second target word frequency, the criticality of the target words in the second cluster, and the length of the target intersection, wherein the criticality of the target words in the second cluster is determined according to the first algorithm; and it determines the connection matrix based on the first connection matrix, the second connection matrix, and the length of the target intersection, wherein the connection matrix is ​​a symmetric matrix. A first generation module is used to normalize the connection matrix to obtain a first target connection matrix, and decompose the first target connection matrix according to the average value of the first target connection matrix to obtain a second target connection matrix. Specifically, it iterates through all elements in the first target connection matrix, assigning a first value to an element when the value of the element is greater than the average value, and assigning a second value to an element when the value of the element is not greater than the average value. The second target connection matrix is ​​then determined based on the first value and the second value. The second generation module is used to merge the second target connection matrix to generate the target clustering result.

7. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein, when the program is executed, it controls the device where the non-volatile storage medium is located to perform the text clustering method according to any one of claims 1 to 5.

8. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, executes the text clustering method according to any one of claims 1 to 5.

Images