An internal evaluation method for hot events based on unsupervised clustering

Abstract

The present invention discloses a method for internal evaluation of hot events for unsupervised clustering, which solves the problems of low efficiency and low accuracy in the current method for discovering telecommunications hot events based on internal evaluation of clustering. The method is to obtain the feature space of telecommunications complaint data by pre-processing it using a pre-trained model, and then obtain the clustering results using the k-Means clustering algorithm. The compactness of the clustering results is judged by measuring the average differential entropy of the hot event clusters, and the separability of the clustering results is judged by measuring the differential entropy of the space formed by the cluster centers of each hot event cluster, and finally the method for internal evaluation of telecommunications complaint hot events with better clustering results is screened out. By using differential entropy to measure the compactness and separability of clustering results, the method has low time complexity and can more effectively evaluate clustering results. This can improve the efficiency and accuracy of discovering telecommunications hot events.

Description

Technical Field

[0001] The present invention belongs to the field of unsupervised learning under machine learning, and in particular relates to an internal evaluation method for hot events oriented to unsupervised clustering. Background Art

[0002] In real-world hot event discovery systems, users typically predefine common event types. However, users cannot know all event types, and existing systems are unable to detect new events of undefined types. For example, telecom operators receive numerous complaints daily, such as "signal issues," "phone bill issues," and "broadband issues." If service personnel have defined these types of complaint events in the system, the system can identify complaints of this type, enabling rapid responses and improving service quality. However, operators' services are not limited to these three types of complaint events. For example, if a complaint regarding "5G" arises and service personnel have not yet categorized this event type, the system will be unable to identify it, hindering timely responses to these complaints and further impacting service quality.

[0003] Event discovery aims to uncover previously unknown hot events. It is essentially a clustering process. Clustering refers to the process of dividing a given data set into clusters, with the goal of making data within the same cluster as similar as possible and data between different clusters as dissimilar as possible. With the development of deep learning, current research has shown that applying classic clustering algorithms to well-defined sample feature representations can achieve better clustering results. This is primarily because deep neural networks can effectively extract nonlinear features from samples. While such features are effective for clustering, clustering results are also affected by factors other than feature representation. For example, different clustering algorithms can produce different clustering results for a given dataset. Even for the same clustering algorithm, different parameter choices can affect the final clustering results. Therefore, in an effective clustering analysis, evaluating the quality of different clustering results and selecting the best one for subsequent application is essential. Currently, most clustering evaluation methods have been proposed. They can be broadly divided into two categories: external evaluation and internal evaluation. External evaluation methods assume that the true cluster information is known and use this supervisory signal to measure how well the resulting clustering results match the true clusters. However, such prior knowledge is often difficult to obtain in reality. Therefore, internal evaluation methods become the only option to evaluate clustering results.

[0004] Internal cluster evaluation is typically based on two aspects when measuring clustering results: (1) compactness, which measures the closeness of samples within the same cluster; and (2) separation, which measures the separation between clusters. Currently, most internal evaluation methods use variance and distance strategies to measure compactness and separation. However, these two strategies have some drawbacks that limit the performance of these metrics.

[0005] First, given two clusters, the same distance calculation result does not guarantee that the compactness of the two clusters is the same. In particular, suppose there are two clusters A and B, where there are 5 samples in cluster A and 3 samples in cluster B. The two clusters are clustered into two cubes, and each cube represents the volume corresponding to the vector space composed of the feature representations of the samples in the cluster. Assuming that the volume of the vector space of the two clusters is the same and the average pairwise distance of each cluster is also the same, the evaluation method based on the average pairwise distance believes that they have the same compactness. However, from the perspective of density, cluster A should be more compact than cluster B. Although some methods have also proposed density-based concepts, the way they calculate density is still based on distance.

[0006] Secondly, the calculation method of closeness based on variance usually assumes that the smaller the variance, the higher the closeness. However, in some cases, this calculation method is not accurate. Suppose there are two clusters A and B, where the covariance matrix of cluster A is The covariance matrix of cluster B is We can see that the variance of the variables in cluster A and cluster B is both 5. In fact, covariance is a more appropriate measure of cluster compactness. The covariance between the variables in cluster A is smaller than the covariance between the variables in cluster B. Further calculating the determinant of the covariance matrix shows that the determinant of cluster B is smaller than that of cluster A. Therefore, cluster B is more compact than cluster A.

[0007] Finally, most existing methods require pairwise distances when calculating closeness and separation. However, this computational approach is time-consuming and time-consuming. Based on these findings, addressing the effectiveness and efficiency issues of existing methods is essential for hotspot event discovery based on internal cluster evaluation. Summary of the Invention

[0008] The purpose of the present invention is to provide a hot event internal evaluation method for unsupervised clustering to address the problems of low efficiency and accuracy in the existing hot event discovery based on unsupervised clustering internal evaluation.

[0009] The object of the present invention is achieved through the following technical solution: a method for internal evaluation of hot events for unsupervised clustering, comprising the following steps:

[0010] (1) Preprocess a batch of complaint tickets to remove extra spaces and punctuation marks. Then use the BERT model to obtain the feature representation of each complaint ticket. The feature representations of all complaint tickets constitute the feature space.

[0011] (2) Using the k-Means clustering algorithm initialized with different parameter k values ​​to act on the feature space of this batch of complaint tickets, several non-overlapping subsets are obtained, namely clustering results; each subset is called an event cluster; the range of the k value is 2 to Where N is the number of complaint tickets in a batch;

[0012] (3) The compactness of the clustering results is determined by measuring the average differential entropy of all event clusters; the separability of the clustering results is determined by measuring the differential entropy of the space formed by the cluster centers of each event cluster; the distribution of the compactness and separability is the normalized value under different clustering results;

[0013] (4) Subtract the separation from the compactness described in step (3), and use the value obtained after the subtraction to internally evaluate the clustering result effect, and obtain the optimal clustering result, that is, obtain the hot events in the complaint ticket.

[0014] Furthermore, the step (3) of judging the compactness of the clustering result by measuring the average differential entropy of the event cluster is specifically as follows:

[0015] Let H be the feature space, where H = {h1,…,h N} T ; Assume that N complaint tickets are divided into k event clusters, namely H1,…,H k , let the feature matrix H k Obeying the multivariate Gaussian distribution, the compactness of these k event clusters is defined as the average differential entropy of the k event clusters:

[0016]

[0017] where DiffEntropy(·) is the differential entropy of the multivariate Gaussian distribution, and i is the index of the i-th event cluster.

[0018] Furthermore, the differential entropy of the space formed by the cluster centers of each event cluster in step (3) is used to determine the separability of the clustering results:

[0019] Let μ k is the cluster center of the kth event cluster, that is, Among them, C k is the kth event cluster, h i is the feature representation of the i-th complaint work order under the k-th event cluster;

[0020] Let the cluster centers of these k event clusters obey the multivariate Gaussian distribution, and the separability of the k event clusters is defined as the differential entropy of the characteristic matrix composed of the k cluster centers, that is: DiffEntropy(H μ ),

[0021] Among them, H μ ={μ1,…,μ k} T , that is, the characteristic subspace formed by each cluster center.

[0022] Furthermore, the internal evaluation in step (4) is: based on compactness and separation, the internal evaluation of k event clusters is compactness norm Minus separability norm The tightness norm and separability norm They represent the normalized values ​​under different clustering results respectively; the internal evaluation method adopts the form of compactness minus separability. If there is a series of clustering results for comparison, before subtracting compactness and separability, they will be normalized separately so that they are in the same range; when the maximum and minimum normalization is performed, the range of internal evaluation is obtained. The smaller the value of internal evaluation within the range, the better the clustering result; after obtaining the clustering result, each event cluster in the clustering result is statistically analyzed to screen out the optimal clustering result, that is, to obtain the hot events in the complaint ticket.

[0023] The beneficial effects of the present invention are as follows:

[0024] This paper provides a new internal evaluation method based on differential entropy to measure the compactness and separation of event clusters in clustering results, enabling the selection of better clustering results. This metric also has low time complexity, making it more suitable for large-scale, high-dimensional datasets. This further addresses the high time complexity and low accuracy of existing hot event discovery methods based on unsupervised clustering internal evaluation, improving the efficiency and accuracy of hot event discovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the method steps in cluster verification of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be described in detail below based on preferred embodiments, and the purpose and effects of the present invention will become more apparent. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] This paper addresses the shortcomings of current hot event discovery methods based on unsupervised clustering internal evaluation by developing a new internal evaluation method. Existing internal evaluation methods based on distance and variance cannot measure the true density of clusters. Furthermore, distance-based metrics have high time complexity when applied to large-scale datasets. This paper proposes a new internal evaluation method based on the concept of differential entropy. This method judges the quality of clustering results by measuring their purity. This metric can effectively measure cluster density and overcomes the shortcomings of existing methods.

[0028] The present invention provides a method for internal evaluation of hot events oriented to unsupervised clustering; Figure 1 The following steps are shown:

[0029] (1) Preprocess a batch of complaint tickets to remove extra spaces and punctuation marks. Then use the BERT model to obtain the feature representation of each complaint ticket. The feature representations of all complaint tickets constitute the feature space.

[0030] (2) Then, the k-Means clustering algorithm initialized with different parameter k values ​​is applied to the feature space to obtain several non-overlapping subsets, i.e., clustering results; each subset is called an event cluster; the range of the k value is generally 2 to Where N is the number of complaint tickets in a batch;

[0031] (3) The compactness of the clustering results is determined by measuring the average differential entropy of all event clusters; the separability of the clustering results is determined by measuring the differential entropy of the space formed by the cluster centers of each event cluster; the distribution of the compactness and separability is the normalized value under different clustering results;

[0032] (4) Based on the normalized values ​​of compactness and separation under different clustering results, the normalized value of separation is subtracted from the normalized value of compactness in step (3), and the value obtained after subtraction is used to internally evaluate the effect of the clustering result.

[0033] In step (1), the feature representation H of a batch of complaint tickets X is obtained through the pre-training model f(·), that is, H = f(X).

[0034] In step (2), the k-Means clustering algorithm is initialized with different parameter k values ​​and applied to the feature matrix H to obtain different clustering results.

[0035] In step (2), for different clustering results, the internal evaluation of the clustering result can be obtained by calculating the compactness and separation.

[0036] The compactness mentioned above specifically refers to: let H be the feature space, where H = {h1,…,h N} T Assume that N samples are divided into k event clusters, namely H1,…,H k The compactness of these k event clusters is defined as the average differential entropy of the k event clusters, and the expression of compactness is:

[0037]

[0038] Among them, DiffEntropy(H k ) represents the differential entropy of the k-th event cluster; i represents the index of the i-th event cluster;

[0039] Given a batch of complaint tickets, we can get each ticket x through the pre-training model f(·) i The characteristic representation of h i =f(x i ). Then for the k-th event cluster we can get Then H k is composed of the feature representations corresponding to the work orders in the kth event cluster |C k |×d matrix, where |C k | is the number of complaint tickets in the kth cluster. H k Assume that the feature matrix H k Obeying the multivariate Gaussian distribution, the differential entropy DiffEntropy (H) of the multivariate Gaussian distribution is expressed as:

[0040]

[0041] Among them, |Σ| is the determinant of the covariance matrix. Therefore, through this formula we can get the differential entropy of an event cluster.

[0042] Furthermore, the separability specifically refers to: let μ k is the cluster center of the kth event cluster, that is, h i ∈H k Among them, C k is the kth event cluster, h i is the feature representation of the i-th complaint ticket under the k-th event cluster. The separability of k event clusters can be defined as the differential entropy of the feature matrix composed of k cluster centers. The expression of separability is:

[0043] DiffEntropy(H μ )

[0044] Among them, H μ ={μ1,…,μ k} T ; C krepresents the kth event cluster; h i Represents the feature representation of the i-th complaint ticket; here we also assume that the cluster centers of these k event clusters obey the multivariate Gaussian distribution.

[0045] In step (3), the compactness and separation under different clustering results are normalized respectively; for example, the maximum and minimum normalization is performed on the compactness and separation respectively;

[0046] Furthermore, based on closeness and separation, the internal evaluation of k event clusters can be defined as:

[0047] Internal evaluation = tightness norm - Separability norm

[0048] The tightness norm and separability norm The distribution represents the normalized values ​​under different clustering results.

[0049] Furthermore, internal evaluations can be obtained for different clustering results. Smaller internal evaluation values ​​indicate better clustering results. Therefore, using this criterion, the optimal clustering results can be selected based on internal evaluations.

[0050] In addition to the BERT model, the pre-trained model may also use the SBERT model or the SimCSE model.

[0051] According to the calculation formula of differential entropy of multivariate Gaussian distribution, we can see that the differential entropy of multivariate Gaussian distribution is proportional to the determinant of the covariance matrix. The determinant of the covariance matrix is ​​usually called the generalized variance. For a data set of fixed size N, the generalized variance is proportional to the square of the volume spanned by the d deviation vectors, that is,

[0052] Generalized variance = |Σ| = (N-1) -d (volume) 2

[0053] Where volume is the volume spanned by the deviation vector. Based on the above formula, we can understand why our internal evaluation can avoid the defects of distance-based and variance-based methods. In particular, The form of explicates the true density in a given vector space.

[0054] Considering that there are many public datasets currently, the present invention selects several datasets in the text field as datasets for embodiments of the present invention.

[0055] Table 1 Statistical results of the text dataset designed by the present invention

[0056] Dataset Division Number of samples Number of categories SearchSnippets training set 1,2340 8 Biomedical training set 20,000 20 StackOverflow training set 20,000 20 WebofSeicence-11967 training set 11,967 7 Yahoo! Answers training set 1,400,000 70

[0057] The datasets of the present invention have been established and summarized, with statistical results shown in Tables 1 and 2. In Table 1, the SearchSnippets dataset consists of 12,340 search snippets extracted from web pages; StackOverflow is a subset of a competition dataset published by Kaggle, containing 20,000 question descriptions; Biomedical is a subset of PubMed data published by BioASQ, containing 20,000 paper titles; Web of Science-11967 is a document classification dataset containing 11,967 documents; and Yahoo! Answers is a collection of questions published by Yahoo!.

[0058] By extracting features from the data using a pre-trained model, initializing the k-Means clustering algorithm with different parameter k values, and applying it to the feature matrix, different clustering results can be obtained. The internal evaluation method is then used to evaluate each clustering result, identifying the optimal clustering result and identifying the hot events. Therefore, the internal evaluation method of the present invention can be applied to discovering hot events in all telecommunications fields.

[0059] Those skilled in the art may modify the technical solutions described in the aforementioned examples or replace some of the technical features with equivalents. Any modifications, equivalents, etc. made within the spirit and principles of the invention shall be included in the scope of protection of the invention.

Claims

1. A hot event internal evaluation method for unsupervised clustering, characterized by: The following steps are involved: (1) Preprocess a batch of complaint tickets to remove extra spaces and punctuation marks from the text, and use the BERT model to obtain the feature representation of each complaint ticket. The feature representations of all complaint tickets constitute the feature space; (2) Using the k-Means clustering algorithm initialized with different parameter k values ​​to act on the feature space of this batch of complaint tickets, several non-overlapping subsets are obtained, namely clustering results; each subset is called an event cluster; the range of the k value is 2 to Where N is the number of complaint tickets in a batch; (3) The compactness of the clustering results is determined by measuring the average differential entropy of all event clusters; the separability of the clustering results is determined by measuring the differential entropy of the space formed by the cluster centers of each event cluster; the distribution of the compactness and separability is the normalized value under different clustering results; (4) Subtract the separation from the compactness described in step (3), and use the value obtained after the subtraction to internally evaluate the clustering result effect, and obtain the optimal clustering result, that is, obtain the hot events in the complaint ticket.

2. The method for internal evaluation of hot events based on unsupervised clustering according to claim 1, characterized in that: The method of judging the compactness of the clustering result by measuring the average differential entropy of the event cluster in step (3) is specifically as follows: Let H be the feature space, where H = {h1,…,h N } T ; Assume that N complaint tickets are divided into k event clusters, namely H1,…,H k , let the feature matrix H k Obeying the multivariate Gaussian distribution, the compactness of these k event clusters is defined as the average differential entropy of the k event clusters: where DiffEntropy(·) is the differential entropy of the multivariate Gaussian distribution, and i is the index of the i-th event cluster.

3. The method for internal evaluation of hot events based on unsupervised clustering according to claim 1, characterized in that: The differential entropy of the space formed by the cluster centers of each event cluster in step (3) is used to determine the separability of the clustering results: Let μ k is the cluster center of the kth event cluster, that is, Among them, C k is the kth event cluster, h i is the feature representation of the i-th complaint work order under the k-th event cluster; Let the cluster centers of these k event clusters obey the multivariate Gaussian distribution, and the separability of the k event clusters is defined as the differential entropy of the characteristic matrix composed of the k cluster centers, that is: DiffEntropy(H μ ), Among them, H μ ={μ1,…,μ k } T , that is, the characteristic subspace formed by each cluster center.

4. The method for internal evaluation of hot events based on unsupervised clustering according to claim 1, characterized in that: The internal evaluation in step (4) is: Based on compactness and separation, the internal evaluation of k event clusters is compactness norm Minus separability norm The tightness norm and separability norm They represent the normalized values ​​under different clustering results respectively. The internal evaluation method adopts the form of compactness minus separation. If there is a series of clustering results for comparison, before subtracting compactness and separation, they will be normalized so that they are in the same range. After performing maximum and minimum normalization, the range of internal evaluation is obtained. The smaller the internal evaluation value within the range, the better the clustering result. After obtaining the clustering results, statistics are performed on each event cluster in the clustering results to filter out the optimal clustering results, that is, to obtain the hot events in the complaint tickets.

Images