Method and apparatus for determining number of clusters, and computer device

Abstract

Embodiments of the present application provide a method and device for determining a cluster number and a computer device, wherein the method comprises: determining a plurality of first cluster numbers, and clustering sample data based on each first cluster number to obtain a clustering result corresponding to each first cluster number; for the clustering result corresponding to each first cluster number, evaluating the clustering effect of the clustering result to obtain an evaluation result corresponding to the first cluster number; and determining a second cluster number according to the plurality of first cluster numbers and the evaluation result corresponding to each first cluster number. Through the embodiments of the present application, the cluster number can be accurately determined, and the accurate cluster number can improve the accuracy of data clustering.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus and computer equipment for determining the number of clusters. Background Technology

[0002] Clustering is the process of dividing a dataset into different clusters according to specific rules, maximizing the similarity of individuals within the same cluster and minimizing the similarity between different clusters. Clustering algorithms are used in many knowledge domains, which often require identifying "natural associations" in specific data. The definition of natural associations depends on the specific domain and application and can take many forms. Typical applications include: in biology, it can be used to derive the classification of plants and animals; in geographic information, it can be used to identify similar regions in observation databases.

[0003] Generally, clustering algorithms are implemented by first determining the number of clusters, and then clustering the data according to the number of clusters. The number of clusters affects the accuracy of clustering, so how to accurately determine the number of clusters is a hot research topic. Summary of the Invention

[0004] This application provides a clustering processing method, apparatus, computer equipment, and storage medium, which can accurately determine the number of clusters. An accurate number of clusters can improve the accuracy of data clustering.

[0005] On the one hand, embodiments of this application provide a method for determining the number of clusters, the method comprising:

[0006] Determine multiple numbers of first clusters, and cluster the sample data based on each number of first clusters to obtain the clustering results corresponding to each number of first clusters;

[0007] For each clustering result corresponding to the first cluster number, the clustering effect is evaluated to obtain the evaluation result corresponding to the first cluster number.

[0008] The number of second clusters is determined based on the number of first clusters and the evaluation results corresponding to each number of first clusters.

[0009] On one hand, embodiments of this application provide an apparatus for determining the number of clusters, which includes units for implementing the method for determining the number of clusters provided in embodiments of this application. In one embodiment, the apparatus for determining the number of clusters includes:

[0010] A processing unit is used to determine the number of multiple first clusters;

[0011] Clustering unit, used to cluster sample data based on each first cluster number, to obtain the clustering result corresponding to each first cluster number;

[0012] An evaluation unit is used to evaluate the clustering effect of the clustering results corresponding to each first number of clusters, and obtain the evaluation result corresponding to the first number of clusters.

[0013] The processing unit is further configured to determine the number of second clusters based on the plurality of first cluster numbers and the evaluation results corresponding to each first cluster number.

[0014] On one hand, embodiments of this application provide a computer device, including: a processor and a memory, the memory storing executable program code, and the processor being used to call the executable program code to implement the method for determining the number of clusters provided in embodiments of this application.

[0015] Accordingly, embodiments of this application also provide a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to implement the method for determining the number of clusters provided in embodiments of this application.

[0016] Accordingly, this application also provides a computer program product, which includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, implements the method for determining the number of clusters provided in this application.

[0017] This application embodiment determines multiple first cluster numbers, clusters the sample data based on each first cluster number, and evaluates the clustering effect of the first clustering results corresponding to each first cluster number. Based on the multiple first cluster numbers and the evaluation results corresponding to each first cluster number indicating the clustering effect, the number of clusters with excellent clustering effect can be accurately determined. An accurate number of clusters can improve the accuracy of data clustering. Furthermore, this application embodiment can automatically evaluate the clustering effect of sample data under different cluster numbers, thus improving the efficiency and accuracy of clustering effect evaluation. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the architecture of a clustering processing system provided in an embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating a method for determining the number of clusters provided in an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of a clustering result provided in an embodiment of this application;

[0022] Figure 4 This is a flowchart illustrating a method for determining the inter-cluster clustering effect value provided in an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of a clustering method provided in an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of a device for determining the number of clusters provided in an embodiment of this application;

[0025] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0027] This application provides a method for determining the number of clusters to accurately identify the number of clusters with excellent clustering results and to automatically evaluate the clustering effectiveness. An accurate number of clusters can improve the accuracy of data clustering. The method for determining the number of clusters provided in this application can be implemented based on one or more technologies, including AI (Artificial Intelligence) and cloud technology. The method for determining the number of clusters provided in this application is applicable to various fields. Figure 1 The clustering processing system shown includes a clustering processing device 10 and a data terminal 11. A communication connection can be established between the clustering processing device 10 and the data terminal 11.

[0028] In feasible implementations, the clustering processing device 10 can be a physical server or a cloud server, or a terminal device such as a smartphone, tablet, laptop, or desktop computer. The data terminal 11 is a device or apparatus that can provide data, such as a database, terminal device, or server, etc. There can be one or more data terminals.

[0029] The clustering method provided in this application includes, but is not limited to, the following steps:

[0030] The clustering processing device 10 acquires data to be classified from the data terminal 11, forming a dataset. Sample data is then obtained from this dataset; this sample data can be all the data in the dataset or a portion of the data sampled from the dataset. The clustering processing device 10 determines multiple first cluster numbers, which indicate the final number of clusters in the clustering process. The clustering processing device 10 performs clustering on the sample data under each first cluster number, obtaining the clustering results for each first cluster number. The clustering algorithm used can be one that requires a pre-set final number of clusters. The clustering processing device 10 evaluates the clustering results for each first cluster number, obtaining an evaluation result corresponding to each first cluster number to indicate the clustering effect. In feasible implementations, the evaluation can be based on intra-cluster dispersion and inter-cluster distance. Based on the multiple first cluster numbers and the evaluation results corresponding to each first cluster number, the clustering processing device 10 determines a second cluster number.

[0031] This application embodiment can accurately determine the number of clusters with excellent clustering effect based on multiple first cluster numbers and the evaluation results corresponding to each first cluster number used to indicate the clustering effect. The accurate number of clusters can improve the accuracy of data clustering. In addition, this application embodiment can automatically evaluate the clustering effect of sample data under different cluster numbers, which can improve the efficiency and accuracy of clustering effect evaluation.

[0032] The above provides a brief overview of the method for determining the number of clusters provided in the embodiments of this application. The specific implementation of this method for determining the number of clusters will be described in detail below.

[0033] Please see Figure 2 , Figure 2 This is a flowchart illustrating a method for determining the number of clusters provided in an embodiment of this application. The method for determining the number of clusters described in this application can be implemented by the clustering processing device in the aforementioned clustering processing system. This method for determining the number of clusters includes, but is not limited to, the following steps:

[0034] S201. Determine the number of multiple first clusters.

[0035] In this embodiment, the cluster number (such as the first cluster number) is used to indicate the final number of clusters after clustering. For example, if the cluster number is 5, then after clustering the data with a cluster number of 5, the clustering result will be that the data is divided into 5 clusters. The cluster number can be an integer greater than 1.

[0036] In one embodiment, the method for determining multiple first cluster numbers may be: determining a baseline cluster number and determining multiple values ​​based on a preset value; for each value, performing a preset operation with the baseline cluster number to obtain a first cluster number.

[0037] In possible implementations, the number of first clusters to be determined is P, where P is an integer greater than 1 (P can be, for example, 5 or 7, etc.). The preset value is X, where X is a positive integer (X can be, for example, 1, 2, or 3, etc.). The multiple values ​​determined based on the preset value can be P integer values ​​from X to X+P-1. The preset operation is multiplication, and the baseline number of clusters is M, where M is a positive integer (M can be, for example, 3, 4, or 5, etc.). Therefore, the P first cluster numbers can be X×M, (X+1)×M, ..., (X+P-2)×M, and (X+P-1)×M.

[0038] For example, if P is 5 and X is 1, then the number of the P first clusters are M, 2M, 3M, 4M, and 5M, respectively. If M is 3, then the number of the P first clusters are 3, 6, 9, 12, and 15, respectively.

[0039] In feasible embodiments, the number of multiple first clusters may not conform to the pattern described above, and the values ​​of the multiple first clusters may be integers that are at least partially discontinuous and greater than 1, set based on experience. For example, the values ​​of the multiple first clusters may be 2, 5, 9, 10, and 15.

[0040] S202. Cluster the sample data based on the number of each first cluster to obtain the clustering result corresponding to the number of each first cluster.

[0041] The sample data is clustered based on each first cluster number to obtain the clustering result corresponding to each first cluster number. In other words, the sample data is clustered separately under each first cluster number to obtain the clustering result of the sample data under each first cluster number.

[0042] In this embodiment, the number of sample data is multiple. The multiple sample data are clustered based on each first cluster number to obtain a clustering result corresponding to each first cluster number. That is, for each of the determined multiple first cluster numbers, the multiple sample data are clustered under that first cluster number to obtain a clustering result for the multiple sample data under that first cluster number. For each clustering result corresponding to a first cluster number, the clustering result includes multiple clusters, and each cluster includes one or more sample vectors used to represent the sample data. The number of multiple clusters is the same as the first cluster number.

[0043] For example, see Figure 3 , Figure 3 This is a schematic diagram illustrating a clustering result provided in an embodiment of this application. Assuming the number of the first cluster is 3 and the number of sample data is 15, the clustering result of the 15 sample data with the first cluster number of 3 can be as follows: Figure 3 As shown. Figure 3 The clustering results shown include three clusters: 31, 32, and 33. Cluster 31 contains five sample vectors (A, B, C, E, and F); cluster 32 contains four sample vectors (D, G, K, and I); and cluster 33 contains six sample vectors (H, S, J, L, V, and Z). These 15 sample vectors from clusters 31, 32, and 33 represent 15 data samples, with each sample vector representing one data sample.

[0044] In feasible implementations, clustering multiple sample data based on a first clustering number can be achieved by: determining the sample vector of each sample data in the multiple sample data, clustering the sample vectors of the multiple sample data under the first clustering number, and obtaining the clustering result. In feasible implementations, a language model can be used to extract features from the sample data to obtain sample vectors. The language model can be a BERT (Bidirectional Encoder Representation from Transformers) model or a Word2Vec model, etc. BERT is a pre-trained language model that uses a bidirectional encoder representation from a Transformer, a neural network architecture based on a self-attention mechanism. Word2Vec is a model that learns semantic knowledge in an unsupervised manner from a large amount of text data and can be used to generate word vectors.

[0045] In feasible embodiments, sample data can be obtained by sampling from a dataset. The clustering processing device acquires the dataset, which includes multiple unclassified data points. It then samples (which can be random sampling) these unclassified data points to obtain sample data. For example, assuming the dataset includes N unclassified data points, n unclassified data points are randomly sampled from these N unclassified data points as sample data, where N is an integer greater than 1, and n can take the following values: or, symbol Indicates rounding down, sign This indicates rounding up. In other possible embodiments, the sample data could also be all the data to be classified in the dataset.

[0046] In feasible embodiments, the clustering algorithm used for data clustering in this application embodiment may be a clustering algorithm that requires pre-setting the final number of clusters, such as K-means clustering algorithm or K-means++ clustering algorithm, etc.

[0047] K-means clustering is an iterative clustering algorithm. It requires a pre-input of the final number of clusters, K, and randomly selects K objects as the initial centroids of these K clusters. Then, it merges nearby points to form new centroids. The iterative principle is that intra-cluster points are sufficiently close, and inter-cluster points are sufficiently far apart, until the condition for K clusters is met. As the principle of K-means clustering is clear, the first step before clustering is initializing the K cluster centers. This is why the convergence of K-means clustering heavily depends on the initialization of the cluster centers. If the cluster centers are not initialized properly, the clustering results will contain serious errors and will likely not converge to the global minimum.

[0048] The K-means++ clustering algorithm is an improvement on the K-means clustering algorithm. The K-means++ algorithm improves the method of initializing cluster centers, which includes the following steps: First, randomly select a sample from the dataset as the first initial cluster center. Second, calculate the shortest distance between each sample and the existing cluster centers, then calculate the probability of each sample being selected as the next cluster center, and finally select the sample corresponding to the highest probability value as the next cluster center. Samples farther from existing cluster centers are more likely to be selected as the next cluster center. Third, repeat step two until K cluster centers are selected. Fourth, calculate and compare the distances of the remaining samples to the K cluster centers, and assign them to different clusters corresponding to these centers, thus completing the clustering of the dataset.

[0049] S203. For each clustering result corresponding to the first cluster number, evaluate the clustering effect and obtain the evaluation result corresponding to the first cluster number.

[0050] For each first cluster number, the clustering results are evaluated to obtain the evaluation result corresponding to that first cluster number. In other words, the clustering results of the sample data under each first cluster number are evaluated separately, yielding the evaluation result for each first cluster number. The evaluation result can be used to indicate the clustering effect of the sample data under the first cluster number.

[0051] In this embodiment of the application, as described above, for each first number of clusters, the clustering result of the sample data under the first number of clusters includes multiple clusters, each cluster including one or more sample vectors used to represent the sample data, and the number of multiple clusters is the same as the first number of clusters. For each clustering result corresponding to the first number of clusters, the clustering effect is evaluated to obtain the evaluation result corresponding to the first number of clusters. This can be achieved by: determining the inter-cluster clustering effect value between every two clusters in the multiple clusters included in the clustering result; determining the average value of the inter-cluster clustering effect value based on the inter-cluster clustering effect value between every two clusters; and determining the evaluation result corresponding to the first number of clusters based on the average value of the inter-cluster clustering effect value.

[0052] In feasible embodiments, the method for determining the inter-cluster clustering effect value between every two clusters in the multiple clusters included in the clustering results may include, for example: Figure 4 The following steps are shown:

[0053] S401, determine the cluster dispersion of the first cluster. The first cluster is any one of the multiple clusters included in the clustering result.

[0054] In a feasible embodiment, determining the cluster dispersion of the first cluster may include the following steps:

[0055] S11. Determine the center sample vector and the first sample vector from the sample vectors included in the first cluster.

[0056] The central sample vector can be the cluster center of the first cluster, and the distance between the first sample vector and the central sample vector satisfies the distance condition.

[0057] In feasible implementations, satisfying the distance condition can mean that the sample vector in the cluster is the one farthest from the center sample vector; that is, the first sample vector can be the sample vector in the first cluster that is farthest from the center sample vector. For example, please refer to [link to relevant documentation]. Figure 5 , Figure 5 This is a schematic diagram of a clustering method provided in an embodiment of this application, as shown below. Figure 5 As shown, sample vector A is the center sample vector of the cluster, and sample vector F in the cluster is the farthest from sample vector A, that is, sample vector F is the first sample vector.

[0058] In other feasible implementations, satisfying the distance condition may also mean satisfying a distance threshold. For example, the first sample vector can be any sample vector in the first cluster whose distance to the center sample vector is greater than the distance threshold.

[0059] Optionally, the first sample vector can also be the sample vector in the first cluster that is second farthest from the center sample vector. Alternatively, the first sample vector can be the average vector determined from multiple (e.g., 2 or 3) sample vectors in the first cluster that are farthest from the center sample vector, and so on. It should be noted that the embodiments of this application do not limit the method of determining the first sample vector from the first cluster.

[0060] S12. Determine the path distance from the center sample vector through the second sample vector to the first sample vector.

[0061] The second sample vector can be any sample vector in the first cluster other than the center sample vector and the first sample vector. There can be one or more second sample vectors. Optionally, the second sample vector can refer to all sample vectors in the first cluster other than the center sample vector and the first sample vector.

[0062] In a feasible implementation, the path distance of all paths from the center sample vector through the second sample vector to the first sample vector can be determined by finding the minimum path distance among all path distances and then determining the minimum path distance as the path distance from the center sample vector through the second sample vector to the first sample vector.

[0063] In other feasible embodiments, the average path distance can be calculated based on the path distances of all paths, and this average path distance can be determined as the path distance from the center sample vector through the second sample vector to the first sample vector. Alternatively, the smallest number of path distances (e.g., 2 or 3) can be determined from the path distances of all paths, and the average path distance can be calculated based on these smallest number of path distances. This average path distance can then be determined as the path distance from the center sample vector through the second sample vector to the first sample vector. It should be noted that the embodiments of this application do not limit the method for determining the path distance from the center sample vector through the second sample vector to the first sample vector.

[0064] For example, suppose Figure 5 The cluster shown is the first cluster. Figure 5 The sample vector A in the data is the center sample vector of the first cluster. Figure 5 The sample vector F is the furthest from the sample vector A, and is therefore the first sample vector. Figure 5 The sample vectors B, C, and E in the dataset are the second sample vectors. Figure 5 The first cluster shown has multiple paths from the center sample vector A through the second sample vectors (including sample vectors B, C, and E) to the first sample vector F, which are as follows:

[0065] Path 1: A→B→C→E→F, the path distance of path 1 d1(A→F)=d(A,B)+d(B,C)+d(C,E)+d(E,F);

[0066] Path 2: A→B→E→C→F, the path distance of path 2 d2(A→F)=d(A,B)+d(B,E)+d(E,C)+d(C,F);

[0067] Path 3: A→C→B→E→F, the path distance of path 3 d3(A→F)=d(A,C)+d(C,B)+d(B,E)+d(E,F);

[0068] Path 4: A→C→E→B→F, the path distance of path 4 d4(A→F)=d(A,C)+d(C,E)+d(E,B)+d(B,F);

[0069] Path 5: A→E→B→C→F, the path distance of path 5 d5(A→F)=d(A,E)+d(E,B)+d(B,C)+d(C,F);

[0070] Path 6: A→E→C→B→F, the path distance of path 6 d6(A→F)=d(A,E)+d(E,C)+d(C,B)+d(B,F);

[0071] Where d represents distance, d(A→F) represents the path distance from the center sample vector A through the second sample vector to the first sample vector F, and d(B,C) represents the distance between sample vector B and sample vector C, which can be Euclidean distance, cosine distance, etc. The rest are similarly expressed and will not be repeated here.

[0072] Assuming that the path distance d2(A→F) of path 2 is the minimum among the path distances of the above 6 paths, then the path distance d2(A→F) of path 2 can be determined as the path distance from the center sample vector A through the second sample vector to the first sample vector F. Alternatively, the average path distance of path distances d1(A→F), d2(A→F), d3(A→F), d4(A→F), d5(A→F), and d2(A→F) can be calculated, and this average path distance can be determined as the path distance from the center sample vector A through the second sample vector to the first sample vector F.

[0073] S13. Determine the cluster dispersion of the first cluster based on the distance and path distance between the central sample vector and the first sample vector.

[0074] The distance between the center sample vector and the first sample vector can be either the cosine distance or the Euclidean distance. Figure 5 The length d(A,F) of the dashed line AF shown is the Euclidean distance between the center sample vector A and the first sample vector F. It should be noted that the distance involved in this embodiment can be cosine distance or Euclidean distance, or other types of distance; this embodiment does not limit this. This path distance is the path distance determined in S12 from the center sample vector through the second sample vector to the first sample vector.

[0075] In a feasible implementation, the cluster dispersion of the first cluster can be determined using Formula 1, as shown below:

[0076]

[0077] Where η represents the cluster dispersion, R represents the center sample vector in the cluster, and r represents the first sample vector in the cluster. d(R, r) represents the distance between the center sample vector R and the first sample vector r, and D(R→r) represents the path distance from the center sample vector R through the second sample vector in the cluster to the first sample vector r. The ratio of D(R→r) to d(R, r) measures the dispersion of samples within a cluster. The smaller the ratio, the smaller the dispersion of samples within the cluster, that is, the smaller the distance between samples within the cluster, and the better the clustering effect.

[0078] S402, determine the cluster dispersion of the second cluster. The second cluster is any cluster other than the first cluster among the multiple clusters included in the clustering result.

[0079] The method for determining the cluster dispersion of the second cluster is similar to the method for determining the cluster dispersion of the first cluster in S401. For details on the implementation, please refer to the relevant description above.

[0080] S403, determine the inter-cluster distance between the first cluster and the second cluster.

[0081] In feasible embodiments, the cluster average vector of the first cluster may be determined based on the sample vectors included in the first cluster, which may be the average of all sample vectors in the first cluster; the cluster average vector of the second cluster may be determined based on the sample vectors included in the second cluster, which may be the average of all sample vectors in the second cluster; the distance between the cluster average vector of the first cluster and the cluster average vector of the second cluster (such as cosine distance, Euclidean distance, etc.) may be determined; and the inter-cluster distance between the first cluster and the second cluster may be determined based on the distance between the cluster average vector of the first cluster and the cluster average vector of the second cluster.

[0082] Optionally, the distance between the average cluster vector of the first cluster and the average cluster vector of the second cluster can be directly determined as the inter-cluster distance between the first cluster and the second cluster. Alternatively, the distance between the average cluster vector of the first cluster and the average cluster vector of the second cluster can be multiplied by a preset weight and used as the inter-cluster distance between the first cluster and the second cluster.

[0083] S404. Based on the cluster dispersion of the first cluster, the cluster dispersion of the second cluster, and the inter-cluster distance, determine the inter-cluster clustering effect value between the first cluster and the second cluster.

[0084] In a feasible implementation, Formula 2 can be used to determine the inter-cluster clustering effect value between the first cluster and the second cluster, as shown in Formula 2 below:

[0085]

[0086] Where ω represents the inter-cluster clustering effect value between clusters. This represents the cluster dispersion of cluster i (such as the first cluster). This represents the cluster dispersion of cluster j (e.g., the second cluster). d(i,j) represents the inter-cluster distance between cluster i and cluster j. A larger distance indicates a greater difference between the samples in cluster i and cluster j, and a better inter-cluster clustering effect. Therefore, for Generally speaking, the smaller the numerator and the larger the denominator, that is, the smaller the value of ω, the better the conditions for excellent clustering effect are met: the closer the samples in the same cluster are, and the farther the distance between clusters is.

[0087] Steps S401-S404 above elaborate on the method for determining the inter-cluster clustering effect value between any two clusters in the multiple clusters included in the clustering result. Assuming the first number of clusters is K, where K is an integer greater than 1, the clustering result of the sample data under the first number of clusters includes K clusters. For these K clusters, the number of pairwise cluster combinations is... (C represents a combination) For each pairwise clustering combination, the inter-cluster clustering effect value of that combination can be determined using the methods shown in steps S401-S404 above. That is, a total of [number] clustering effect values ​​need to be determined. Clustering effect values ​​between clusters, and then based on the determined The clustering effect value among clusters determines the evaluation result corresponding to the first cluster number, used to indicate the clustering effect. Among these,

[0088] In feasible implementations, the determined The average value of the clustering effect among the clusters is taken as the evaluation result corresponding to the first number of clusters. Based on the preceding information, Formula 3 can be used to evaluate the clustering effect of the sample data at the first number of clusters, thus determining the evaluation result corresponding to the first number of clusters to indicate the clustering effect. Formula 3 is shown below:

[0089]

[0090] Among them, S K S represents the evaluation result. K The smaller the value, the better the clustering result of the data under the corresponding number of clusters. That is, the closer the sample vectors in the same cluster are, and the farther the distance between clusters is.

[0091] It should be noted that the above method of evaluating the clustering effect of sample data at any first cluster number to determine the evaluation result corresponding to the first cluster number has been described in detail. By using a similar method to evaluate the clustering effect of sample data at each first cluster number, the evaluation result corresponding to each first cluster number can be obtained to indicate the clustering effect.

[0092] As described above, the method for determining the number of clusters provided in this application includes an automated way of evaluating clustering effect. It mainly evaluates the clustering effect based on intra-cluster dispersion and inter-cluster distance. This allows for the automatic evaluation of clustering effect for data under different numbers of clusters, thereby improving the efficiency and accuracy of clustering effect evaluation.

[0093] S204. Determine the number of second clusters based on the number of multiple first clusters and the evaluation results corresponding to each number of first clusters.

[0094] It can be that the evaluation results that meet the set conditions are determined from the evaluation results corresponding to each first cluster number, and the first cluster number corresponding to the evaluation results that meet the set conditions among multiple first cluster numbers is determined as the second cluster number.

[0095] In a feasible implementation, the set condition can refer to the minimum value. That is, it can be that the evaluation result with the minimum value is determined from the evaluation results corresponding to each first cluster number, and the first cluster number corresponding to the evaluation result with the minimum value (that is, the first cluster number with the best clustering effect) is determined as the second cluster number.

[0096] In some embodiments, after the multiple first cluster numbers are arranged in order (e.g., in ascending order), the numerical relationship between two adjacent cluster numbers satisfies a first numerical relationship threshold. Satisfying the first numerical relationship threshold means that the absolute value of the numerical difference between two adjacent cluster numbers is less than or equal to the first numerical relationship threshold. The first numerical relationship threshold can be determined based on the absolute value of the numerical difference between any two adjacent cluster numbers. For example, if the ordered arrangement of the multiple first cluster numbers is 4, 8, 10, 15, and 20, and the absolute values ​​of the numerical differences between any two adjacent cluster numbers are 4, 2, 5, and 5 respectively, then the maximum value of 5 or the average value of 4 can be determined as the first numerical relationship threshold. Optionally, the first numerical relationship threshold can also be determined directly based on the baseline cluster number used to determine the multiple first cluster numbers. For example, if the baseline cluster number is M, and the multiple first cluster numbers are M, 2M, 3M, 4M, and 5M respectively, then the first numerical relationship threshold can be the value M of the baseline cluster number.

[0097] The system checks whether a first numerical relationship threshold meets the filtering criteria. The filtering criteria can be less than or equal to a set threshold. For example, if the set threshold is 1, and the first numerical relationship threshold is 1, then the first numerical relationship threshold meets the filtering criteria; if the first numerical relationship threshold is 4, then the first numerical relationship threshold does not meet the filtering criteria.

[0098] If the first numerical relationship threshold meets the screening criteria, it indicates that the granularity of the multiple first cluster numbers is relatively fine. Fine granularity means that, after the multiple first cluster numbers are arranged in order, the absolute value of the difference between any two adjacent cluster numbers is small. For example, if the values ​​of the multiple first cluster numbers are consecutive, namely 5, 6, 7, 8, and 9, then the determined second cluster number is very likely to be the cluster number with the optimal clustering effect, and can be directly used as the final cluster number.

[0099] If the first numerical relationship threshold does not meet the screening criteria, it indicates that the granularity of the multiple first cluster numbers is relatively coarse. Coarse granularity can refer to a large absolute value of the difference between all or some of the values ​​of two adjacent cluster numbers after the multiple first cluster numbers are arranged in order. For example, the values ​​of the multiple first cluster numbers are not consecutive, and are 5, 10, 15, and 20 respectively. In this case, the determined second cluster numbers are very likely not the optimal cluster numbers. To ensure that the optimal cluster numbers are determined, multiple third cluster numbers can be determined based on the second numerical relationship threshold and the second cluster numbers, and the third cluster numbers can replace the first cluster numbers. Then, clustering of the sample data based on each first cluster number and subsequent steps can be performed.

[0100] In a feasible implementation, the method for determining the number of multiple third clusters based on the second numerical relationship threshold and the number of second clusters can be as follows: based on the ordered arrangement of the multiple first cluster numbers, determine the value T of the first cluster number preceding the second cluster number and the value W of the first cluster number following the second cluster number; determine the multiple third cluster numbers based on the second numerical relationship threshold and the integer values ​​between T and W. The second numerical relationship threshold can be a filtering condition; for example, the second numerical relationship threshold can be less than or equal to a set threshold, i.e., the second numerical relationship threshold can be less than the first numerical relationship threshold.

[0101] For example, if the number of the second cluster is 10, the number of the first cluster preceding the second cluster is T=7, and the number of the first cluster following the second cluster is W=13, then if the second numerical relationship threshold is 1, four third cluster numbers can be determined, and these four third cluster numbers can be 8, 9, 10, 11, and 12 respectively. If the second numerical relationship threshold is 2, three third cluster numbers can be determined, and these three third cluster numbers can be 8, 10, and 12 respectively.

[0102] The third cluster number is used to replace the first cluster number, and the sample data is clustered based on each of the first cluster numbers, and subsequent steps are performed. That is, the following steps are performed based on the third cluster number:

[0103] The sample data is clustered based on the number of each third cluster, resulting in clustering results corresponding to each number of third clusters. The specific implementation of this step is similar to the implementation of clustering the sample data based on the number of each first cluster in step S202 above, and will not be repeated here.

[0104] For each clustering result corresponding to the third cluster number, the clustering effect is evaluated to obtain the evaluation result corresponding to the third cluster number. The specific implementation of this step is similar to the implementation of S202 above, which evaluates the clustering effect for each clustering result corresponding to the first cluster number to obtain the evaluation result corresponding to the first cluster number, and will not be repeated here.

[0105] The new number of second clusters is determined based on the number of third clusters and the evaluation results corresponding to each number of third clusters. This can be achieved by selecting evaluation results that meet set conditions from the evaluation results corresponding to each number of third clusters, and then determining the number of third clusters corresponding to these evaluation results as the new number of second clusters. In feasible implementations, the set conditions can refer to a minimum value; that is, the evaluation result with the smallest value can be selected from the evaluation results corresponding to each number of third clusters, and the number of third clusters corresponding to this smallest value (i.e., the number of third clusters with the optimal clustering effect) is determined as the new number of second clusters.

[0106] Since the numerical relationship between any two adjacent cluster numbers satisfies the second numerical relationship threshold after the multiple third cluster numbers are arranged in order, and the second numerical relationship threshold satisfies the screening condition, it indicates that the granularity of the multiple third cluster numbers is relatively fine. At this time, the new second cluster number determined is very likely to be the cluster number with the best clustering effect, and the new second cluster number can be used as the final cluster number.

[0107] For multiple first cluster numbers, the sample data performs best under the second cluster number. For multiple third cluster numbers determined based on the second cluster number, the data performs best under the new second cluster number. That is to say, the new second cluster number determined above is the most accurate (i.e. the best) cluster number, and the clustering results of the data under the new second cluster number have higher accuracy.

[0108] As described above, the method for determining the number of clusters provided in this application includes a two-stage clustering approach to determine the most accurate number of clusters. If the first stage uses a coarse-grained clustering method, the optimal number of clusters is initially determined based on the evaluation results indicating the clustering effect corresponding to the coarse-grained clustering method. In the second stage, a fine-grained clustering method is used, and the most accurate number of clusters is determined based on the evaluation results indicating the clustering effect corresponding to the fine-grained clustering method. Based on the most accurate number of clusters, the accuracy of data clustering can be improved, thereby resulting in better quality clustering results.

[0109] In feasible embodiments, if the sample data is randomly sampled from a dataset containing multiple unclassified data, then for the dataset containing multiple unclassified data, the most accurate number of clusters is the second number of clusters or a new second number of clusters determined using the above method. Therefore, the multiple unclassified data included in the dataset can be clustered at this number of clusters to obtain the clustering results for the multiple unclassified data at this number of clusters, and the clustering results for the multiple unclassified data included in the dataset at this number of clusters are optimal. If the sample data consists of all the unclassified data in the dataset, then the clustering results obtained by clustering the sample data at this number of clusters can be directly determined as the optimal clustering results for the dataset.

[0110] The implementation of the method for determining the number of clusters provided in the embodiments of this application has been described in detail above. To better understand the method for determining the number of clusters provided in the embodiments of this application, one implementation of the method for determining the number of clusters provided in the embodiments of this application is introduced below by way of example:

[0111] Step 1: Assume the dataset contains N data points to be classified. Randomly sample n data points from the dataset as sample data. The value of n can be n = N / 10. If N / 10 is not an integer, it can be rounded up or down.

[0112] Step 2: Initially set the number of clusters K to the following values: K = m, K = 2*m, K = 3*m, K = 4*m, K = 5*m. The value of m can be 3, 4, or 5, and can be set empirically. The initial number of clusters has a relatively coarse granularity. Use the K-means clustering algorithm or K-means++ clustering algorithm to cluster the n sample data under different K values, obtaining the clustering results for the n sample data at cluster numbers m, 2*m, 3*m, 4*m, and 5*m respectively. The clustering results include multiple clusters, and each cluster includes one or more sample vectors representing the sample data.

[0113] Step 3: Evaluate the clustering effect of each clustering result obtained in Step 2. The evaluation result S is when the number of clusters is K. K The calculation methods are shown in Formulas 4 and 5 below:

[0114]

[0115] Among them, R i R represents the center sample vector of the i-th cluster in the clustering results corresponding to the number of clusters K; j r represents the center sample vector of the j-th cluster in the clustering result; i R represents the distance of the sample vector from the center in the i-th cluster. i The farthest sample vector; r j R represents the distance of the sample vector from the center in the j-th cluster. j The farthest sample point; d(R) i r i R represents the center sample vector in the i-th cluster. i With sample vector r i The distance; d(R) j r j R represents the center sample vector in the j-th cluster. j With sample vector r j Distance; D(R) i →r i R represents the center sample vector in the i-th cluster. i After passing through all intermediate sample vectors within the cluster to the sample vector r i The distance, where the intermediate sample vector refers to the distance in the i-th cluster excluding the center sample vector R. i and sample vector r i Sample vectors other than those in the D(R) j →r j R represents the center sample vector in the j-th cluster. j After passing through all intermediate sample vectors within the cluster to the sample vector r j The distance, where the intermediate sample vector refers to the distance in the j-th cluster excluding the center sample vector R. j and sample vector r j The sample vectors other than the sample vectors; min() means taking the minimum value.

[0116] d(i,j) represents the distance between the i-th cluster and the j-th cluster in the clustering results. In a feasible implementation, the cluster mean vector V of the i-th cluster is... i It equals the average of all sample vectors in the i-th cluster, and the cluster mean vector V of the j-th cluster is... j The value of d(i,j) is equal to the average of all sample vectors in the j-th cluster, which is V.i With V j The distance between them. Clustering is performed with K clusters, and the pairwise combinations of cluster i and cluster j are: species, therefore S K The value represents the average of the sums of all combinations.

[0117] Formula Explanation: This represents the center sample vector R in the i-th cluster. i The vector from all intermediate sample points within the cluster to the sample vector r that is furthest from it. i The minimum distance between the center sample vector R and the center sample vector R. i and sample vector r i The ratio of the distances between the two measures the dispersion of samples within a cluster (or the data corresponding to the samples within a cluster). The smaller the ratio, the smaller the dispersion of samples within a cluster, that is, the smaller the distance between samples, and the better the clustering effect. and The representation is similar and will not be repeated here. d(i,j) measures the distance between cluster i and cluster j. The larger the distance, the greater the difference between the samples contained in cluster i and cluster j, and the better the inter-cluster clustering effect.

[0118] Therefore, for Generally speaking, the smaller the numerator, the larger the denominator, i.e., S K The smaller the value, the better the clustering effect is met: the samples in the same cluster are closer together, and the distance between clusters is farther.

[0119] Step 4: Evaluate the clustering effect of different K values ​​based on Step 3, assuming S K =S 3*m The evaluation result is minimized when K = 3*m, meaning that among K = m, K = 2*m, K = 3*m, K = 4*m, and K = 5*m, the clustering effect is optimal when the number of clusters K = 3*m. To ensure the optimal number of clusters is determined, the number of clusters K can be set again to: K = 2*m+1, K = 2*m+2, ..., K = 4*m-1, with finer granularity. The K-means clustering algorithm or K-means++ clustering algorithm is then used to cluster the n sample data under the different newly set K values. The value of m is the same as the value of m in the initial clustering.

[0120] Step 5: Evaluate the clustering effect of each clustering result obtained in Step 4. The processing method is similar to that described in Step 3, and will not be repeated here. In K = 2*m+1, K = 2*m+2, ..., K = 4*m-1, assume S... K =S 3*m+2The optimal clustering result is achieved when the number of clusters, K = 3*m + 2, is minimized. Therefore, it can be considered that setting the number of clusters to K = 3*m + 2 yields the best clustering result for the aforementioned n sample data. Since these n sample data are randomly sampled from a dataset containing N data points to be classified, the optimal number of clusters for a dataset containing N data points to be classified is also K = 3*m + 2.

[0121] Step 6: Cluster the dataset containing N data points to be classified with the number of clusters K = 3*m+2 to obtain the best clustering result for the dataset.

[0122] For example, in chatbot business scenarios, it is necessary to establish and enrich a customer dialogue knowledge base (i.e., to mine customer dialogue, identify the categories of frequently asked questions, and identify the dialogue options within each category) to improve the service quality of the chatbot and enhance the user experience. During the establishment or enrichment phase of the customer dialogue knowledge base, manual extraction and classification of dialogue options from large amounts of historical dialogue data is typically required, which is very labor-intensive. Using the cluster number determination method provided in this application embodiment, only the historical dialogue data needs to be provided to the cluster number determination device to obtain the classified customer dialogue options.

[0123] In a feasible implementation, a batch of historical dialogue data is retrieved to form a first dataset, and 1 / 10 of the data is randomly sampled from this dataset to form a second dataset. The initial values ​​for the number of clusters K are set as follows: K = m, K = 2*m, K = 3*m, K = 4*m, K = 5*m, where m can be 3, 4, or 5. The second dataset is then clustered using either the K-means clustering algorithm or the K-means++ clustering algorithm at the different initial K values ​​(i.e., different numbers of clusters). The clustering results are then evaluated to obtain the evaluation results S for each different K value (i.e., different numbers of clusters). K Take S KThe minimum value of K is found, at which point the clustering effect is best. The purpose of this step is to initially determine the optimal number of clusters. Then, based on the optimal K value determined in the previous step, a second clustering is performed on the second dataset with different values ​​of K at a finer granularity. (The first clustering is to initially determine the optimal K value, so the granularity of the K value is relatively coarse. For example, if m=5, the K values ​​for the first clustering are 5, 10, 15, 20, and 25. If the optimal K value determined by the first clustering is 15, then the K values ​​for the second clustering can be 11, 12, 13, 14, 15, 16, 17, 18, and 19, with a finer granularity than the first clustering. This is used for the second clustering to determine the most accurate K value.) The clustering effect of the second clustering is evaluated, and the evaluation results S under different K values ​​(i.e., different numbers of clusters) are obtained. K Take S K The minimum value of K is the most accurate K value. Finally, clustering the first dataset with the number of clusters K yields the best clustering result for the first dataset, thus completing the classification of historical dialogue data.

[0124] Please see Figure 6 , Figure 6 This is a schematic diagram of a device for determining the number of clusters provided in an embodiment of this application. The device for determining the number of clusters described in this embodiment corresponds to the clustering processing device described above. This device includes units for implementing the method for determining the number of clusters provided in the embodiment of this application. In one embodiment, the device for determining the number of clusters may include:

[0125] Processing unit 601 is used to determine the number of multiple first clusters;

[0126] Clustering unit 602 is used to cluster the sample data based on each first cluster number to obtain the clustering result corresponding to each first cluster number;

[0127] Evaluation unit 603 is used to evaluate the clustering effect of the clustering results corresponding to each first cluster number, and obtain the evaluation result corresponding to the first cluster number.

[0128] The processing unit 601 is further configured to determine the number of second clusters based on the plurality of first cluster numbers and the evaluation results corresponding to each first cluster number.

[0129] In one embodiment, the plurality of first cluster numbers are arranged in order, and the numerical relationship between two adjacent cluster numbers satisfies a first numerical relationship threshold.

[0130] The processing unit 601 is further configured to: if the first numerical relationship threshold does not meet the screening conditions, determine multiple third cluster numbers based on the second numerical relationship threshold and the second cluster number, replace the first cluster number with the third cluster number, and perform clustering of the sample data based on each first cluster number and subsequent steps.

[0131] In one embodiment, the processing unit 601 is specifically used to: determine a baseline number of clusters, and determine multiple values ​​based on preset values; for each value, perform a preset operation with the baseline number of clusters to obtain a first number of clusters.

[0132] In one embodiment, the plurality of first cluster numbers are arranged in order, and the processing unit 601 is specifically used to: determine the value T of the first cluster number that is one position before the second cluster number and the value W of the first cluster number that is one position after the second cluster number from the plurality of first cluster numbers; and determine a plurality of third cluster numbers based on the second numerical relationship threshold and the integer values ​​between T and W.

[0133] In one embodiment, for each first cluster number, the clustering result includes multiple clusters, each cluster including one or more sample vectors for representing the sample data, and the number of multiple clusters is the same as the first cluster number;

[0134] The evaluation unit 603 is specifically used to: determine the inter-cluster clustering effect value between every two clusters in the plurality of clusters; determine the average value of the inter-cluster clustering effect value based on the inter-cluster clustering effect value between every two clusters; and determine the evaluation result corresponding to the first number of clusters based on the average value of the inter-cluster clustering effect value.

[0135] In one embodiment, the plurality of clusters includes a first cluster and a second cluster. The evaluation unit 603 is specifically used to: determine the cluster dispersion of the first cluster and the cluster dispersion of the second cluster; determine the inter-cluster distance between the first cluster and the second cluster; and determine the inter-cluster clustering effect value between the first cluster and the second cluster based on the cluster dispersion of the first cluster, the cluster dispersion of the second cluster, and the inter-cluster distance.

[0136] In one embodiment, the evaluation unit 603 is specifically configured to: determine a center sample vector and a first sample vector from the sample vectors included in the first cluster; the distance between the first sample vector and the center sample vector satisfies a distance condition; determine the path distance from the center sample vector through a second sample vector to the first sample vector; the second sample vector is a sample vector in the first cluster other than the center sample vector and the first sample vector; and determine the clustering dispersion of the first cluster based on the distance between the center sample vector and the first sample vector and the path distance.

[0137] In one embodiment, the evaluation unit 603 is specifically configured to: determine the cluster average vector of the first cluster based on the sample vectors included in the first cluster; determine the cluster average vector of the second cluster based on the sample vectors included in the second cluster; and determine the inter-cluster distance between the first cluster and the second cluster based on the distance between the cluster average vector of the first cluster and the cluster average vector of the second cluster.

[0138] It is understood that the functions of each functional unit of the cluster number determination device provided in the embodiments of this application can be specifically implemented according to the methods in the above method embodiments, and will not be repeated here.

[0139] In feasible embodiments, the cluster number determination device provided in this application embodiment can be implemented in software. The cluster number determination device can be stored in memory and can be software in the form of programs and plug-ins, and includes a series of units, including a processing unit, a clustering unit and an evaluation unit; wherein, the processing unit, the clustering unit and the evaluation unit are used to implement the cluster number determination method provided in this application embodiment.

[0140] In other feasible embodiments, the cluster number determination device provided in this application embodiment can also be implemented in a combination of hardware and software. As an example, the cluster number determination device provided in this application embodiment can be a processor in the form of a hardware decoding processor, which is programmed to execute the cluster number determination method provided in this application embodiment.

[0141] This application embodiment determines multiple first cluster numbers, clusters the sample data based on each first cluster number, and evaluates the clustering effect of the first clustering results corresponding to each first cluster number. Based on the multiple first cluster numbers and the evaluation results corresponding to each first cluster number indicating the clustering effect, the number of clusters with excellent clustering effect can be accurately determined. An accurate number of clusters can improve the accuracy of data clustering. Furthermore, this application embodiment can automatically evaluate the clustering effect of sample data under different cluster numbers, thus improving the efficiency and accuracy of clustering effect evaluation.

[0142] Please see Figure 7 , Figure 7 This is a schematic diagram of a computer device provided in an embodiment of this application. The computer device described in this embodiment includes a processor 701 and a memory 702. Optionally, the computer device may further include a communication interface 703. The processor 701, memory 702, and communication interface 703 can be connected via a bus or other means; this embodiment uses a bus connection as an example.

[0143] The processor 701 (or CPU, Central Processing Unit) is the computing and control core of the computer device. It can parse various instructions and process various data within the computer device. For example, the CPU can parse power-on / off commands sent by the user and control the computer device to perform power-on / off operations; it can also transmit various interactive data between internal structures of the computer device. The memory 702 is the storage device in the computer device, used to store programs and data. It is understood that the memory 702 here can include the computer device's built-in memory, or it can include extended memory supported by the computer device. The memory 702 provides storage space for the computer device's operating system, which may include, but is not limited to, Android, iOS, Windows Phone, etc. This application does not limit this. The communication interface 703 may optionally include standard wired interfaces or wireless interfaces (such as Wi-Fi, mobile communication interfaces, etc.), controlled by the processor 701 for sending and receiving data. The communication interface 703 can also refer to various interfaces used to implement internal computer communication.

[0144] In this embodiment of the application, the processor 701 performs the following operations by running the executable program code in the memory 702:

[0145] A plurality of first cluster numbers are determined, and the sample data is clustered based on each first cluster number to obtain a clustering result corresponding to each first cluster number; for each clustering result corresponding to the first cluster number, the clustering effect is evaluated to obtain an evaluation result corresponding to the first cluster number; based on the plurality of first cluster numbers and the evaluation result corresponding to each first cluster number, a second cluster number is determined.

[0146] In one embodiment, the plurality of first cluster numbers are arranged in order, and the numerical relationship between two adjacent cluster numbers satisfies a first numerical relationship threshold; the processor 701 is further configured to: if the first numerical relationship threshold does not meet the screening conditions, determine a plurality of third cluster numbers according to a second numerical relationship threshold and a second cluster number, replace the first cluster number with the third cluster number, and perform clustering of the sample data based on each first cluster number and subsequent steps.

[0147] In one embodiment, when the processor 701 determines a plurality of first cluster numbers, it is specifically used to: determine a baseline cluster number and determine a plurality of values ​​based on a preset value; for each value, perform a preset operation on the value and the baseline cluster number to obtain a first cluster number.

[0148] In one embodiment, the plurality of first cluster numbers are arranged in order. When the processor 701 determines the plurality of third cluster numbers based on the second numerical relationship threshold and the second cluster number, it is specifically used to: determine the value T of the first cluster number that is one position before the second cluster number and the value W of the first cluster number that is one position after the second cluster number from the plurality of first cluster numbers; and determine the plurality of third cluster numbers based on the second numerical relationship threshold and the integer values ​​between T and W.

[0149] In one embodiment, for each clustering result corresponding to a first number of clusters, the clustering result includes multiple clusters, each cluster including one or more sample vectors for representing the sample data, and the number of multiple clusters is the same as the first number of clusters; when the processor 701 evaluates the clustering effect of each clustering result corresponding to a first number of clusters and obtains the evaluation result corresponding to the first number of clusters, it is specifically used to: determine the inter-cluster clustering effect value between every two clusters in the multiple clusters; determine the average value of the inter-cluster clustering effect value based on the inter-cluster clustering effect value between every two clusters; and determine the evaluation result corresponding to the first number of clusters based on the average value of the inter-cluster clustering effect value.

[0150] In one embodiment, the plurality of clusters includes a first cluster and a second cluster. When the processor 701 determines the inter-cluster clustering effect value between every two clusters in the plurality of clusters, it is specifically configured to: determine the cluster dispersion of the first cluster and the cluster dispersion of the second cluster; determine the inter-cluster distance between the first cluster and the second cluster; and determine the inter-cluster clustering effect value between the first cluster and the second cluster based on the cluster dispersion of the first cluster, the cluster dispersion of the second cluster, and the inter-cluster distance.

[0151] In one embodiment, when the processor 701 determines the cluster dispersion of the first cluster, it is specifically configured to: determine a center sample vector and a first sample vector from the sample vectors included in the first cluster; the distance between the first sample vector and the center sample vector satisfies a distance condition; determine the path distance from the center sample vector through a second sample vector to the first sample vector; the second sample vector is a sample vector in the first cluster other than the center sample vector and the first sample vector; and determine the cluster dispersion of the first cluster based on the distance between the center sample vector and the first sample vector and the path distance.

[0152] In one embodiment, when the processor 701 determines the inter-cluster distance between the first cluster and the second cluster, it is specifically configured to: determine the cluster average vector of the first cluster based on the sample vectors included in the first cluster; determine the cluster average vector of the second cluster based on the sample vectors included in the second cluster; and determine the inter-cluster distance between the first cluster and the second cluster based on the distance between the cluster average vectors of the first cluster and the second cluster.

[0153] In specific implementations, the processor 701, memory 702, and communication interface 703 described in the embodiments of this application can execute the implementation of the clustering processing device described in the clustering number determination method provided in the embodiments of this application, or the implementation of the clustering number determination device provided in the embodiments of this application, which will not be repeated here.

[0154] This application embodiment determines multiple first cluster numbers, clusters the sample data based on each first cluster number, and evaluates the clustering effect of the first clustering results corresponding to each first cluster number. Based on the multiple first cluster numbers and the evaluation results corresponding to each first cluster number indicating the clustering effect, the number of clusters with excellent clustering effect can be accurately determined. An accurate number of clusters can improve the accuracy of data clustering. Furthermore, this application embodiment can automatically evaluate the clustering effect of sample data under different cluster numbers, thus improving the efficiency and accuracy of clustering effect evaluation.

[0155] This application also provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to implement the method for determining the number of clusters provided in this application. The specific implementation method can be referred to the foregoing description and will not be repeated here.

[0156] This application also provides a computer program product, which includes a computer program or computer instructions. The computer program or computer instructions may be stored in a computer-readable storage medium. The processor of the computer device reads the computer program or computer instructions from the computer-readable storage medium and executes the computer program or computer instructions, so that the computer device implements the method for determining the number of clusters provided in this application. The specific implementation method can be referred to the foregoing description, and will not be repeated here.

[0157] It should be noted that the data collection and processing described in this application should be strictly in accordance with the requirements of relevant laws and regulations, obtaining the informed consent or separate consent of the personal information subject, and conducting subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0158] For the sake of simplicity, the foregoing method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0159] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0160] The above-disclosed embodiments are only some of the embodiments of this application, and should not be construed as limiting the scope of this application. Therefore, any equivalent changes made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A method for determining the number of clusters, characterized in that, The method includes: Determine multiple numbers of first clusters, and cluster the sample data based on each number of first clusters to obtain the clustering results corresponding to each number of first clusters; For each clustering result corresponding to the first cluster number, the clustering effect is evaluated to obtain the evaluation result corresponding to the first cluster number. The number of second clusters is determined based on the number of first clusters and the evaluation results corresponding to each number of first clusters.

2. The method of claim 1, wherein, The plurality of first cluster numbers are arranged in order, and the numerical relationship between the numbers of two adjacent clusters satisfies a first numerical relationship threshold; the method further includes: If the first numerical relationship threshold does not meet the screening criteria, then multiple third cluster numbers are determined based on the second numerical relationship threshold and the second cluster number, and the third cluster number is used to replace the first cluster number. Then, clustering of the sample data based on each first cluster number and subsequent steps are performed.

3. The method of claim 1, wherein, Determining the number of multiple first clusters includes: Determine the baseline number of clusters, and determine multiple values ​​based on preset values; For each value, a preset operation is performed between the value and the baseline cluster number to obtain a first cluster number.

4. The method of claim 2, wherein, The number of the plurality of first clusters is arranged in order, and the determination of the number of the plurality of third clusters based on the second numerical relationship threshold and the number of the second clusters includes: Determine the value T of the first cluster number that is one position before the second cluster number from the plurality of first cluster numbers, and the value W of the first cluster number that is one position after the second cluster number; The number of multiple third clusters is determined based on the second numerical relationship threshold and the integer values ​​between T and W.

5. The method according to any one of claims 1 to 4, characterized in that, For each first cluster number, the clustering result includes multiple clusters, each cluster includes one or more sample vectors used to represent the sample data, and the number of multiple clusters is the same as the first cluster number. The process of evaluating the clustering results for each first cluster number, and obtaining the evaluation results corresponding to the first cluster number, includes: Determine the inter-cluster clustering effect value between every two clusters in the plurality of clusters; Based on the inter-cluster clustering effect value between each pair of clusters, determine the average value of the inter-cluster clustering effect value; The evaluation result corresponding to the number of the first clusters is determined based on the average value of the clustering effect values ​​among the clusters.

6. The method of claim 5, wherein, The plurality of clusters includes a first cluster and a second cluster. Determining the inter-cluster clustering effect value between every two clusters within the plurality of clusters includes: Determine the cluster dispersion of the first cluster and the cluster dispersion of the second cluster; Determine the inter-cluster distance between the first cluster and the second cluster; Based on the cluster dispersion of the first cluster, the cluster dispersion of the second cluster, and the inter-cluster distance, the inter-cluster clustering effect value between the first cluster and the second cluster is determined.

7. The method of claim 6, wherein, Determining the cluster dispersion of the first cluster includes: A center sample vector and a first sample vector are determined from the sample vectors included in the first cluster; the distance between the first sample vector and the center sample vector satisfies a distance condition; Determine the path distance from the center sample vector through the second sample vector to the first sample vector; the second sample vector is the sample vector in the first cluster other than the center sample vector and the first sample vector; The clustering dispersion of the first cluster is determined based on the distance between the central sample vector and the first sample vector and the path distance.

8. The method of claim 6, wherein, Determining the inter-cluster distance between the first cluster and the second cluster includes: The cluster average vector of the first cluster is determined based on the sample vectors included in the first cluster. The cluster average vector of the second cluster is determined based on the sample vectors included in the second cluster; The inter-cluster distance between the first cluster and the second cluster is determined based on the distance between the cluster average vector of the first cluster and the cluster average vector of the second cluster.

9. A cluster number determination apparatus characterized by comprising: a cluster number determination unit operable to determine a cluster number based on a distance between a cluster center and a data point. The device includes: A processing unit is used to determine the number of multiple first clusters; Clustering unit, used to cluster sample data based on each first cluster number, to obtain the clustering result corresponding to each first cluster number; An evaluation unit is used to evaluate the clustering effect of the clustering results corresponding to each first cluster number, and obtain the evaluation result corresponding to the first cluster number. The processing unit is further configured to determine the number of second clusters based on the plurality of first cluster numbers and the evaluation results corresponding to each first cluster number.

10. A computer device, comprising: include: A processor and a memory, the memory storing executable program code, the processor being used to call the executable program code to implement the method for determining the number of clusters as described in any one of claims 1-8.

Images