A Prediction Method for the Number of Knowledge Service Entity Clusters Based on Deep Learning

Abstract

The present invention relates to a method for predicting the number of entity clusters in knowledge services based on deep learning, belonging to the technical field of data mining applications and information processing. This method is to solve the technical problem of how to effectively predict the number of entity clusters under a large amount of literature and data in network knowledge service applications, and improve the accuracy and efficiency of information data mining and pushing. By combining an autoencoder with the Monte Carlo method, it can accurately and automatically estimate the number of clusters in a high-dimensional data set, effectively overcome the technical defect that the current number of clusters needs to be estimated manually, significantly improve the information recommendation ability of the system in aspects such as hot topic recommendation, search, and question answering, and recommend more relevant information to users. The present invention has broad application prospects in the fields of text mining and fine-grained knowledge services, etc.

Description

Technical Field

[0001] The present invention relates to a method for predicting the number of clusters of knowledge service entities, in particular to a method for predicting the number of clusters of knowledge service entities based on deep learning, and belongs to the technical field of big data mining applications and information processing. Background Art

[0002] In the Internet information era, the literature analysis technology in the field of knowledge services plays an important role in promoting scientific and technological innovation, and helps technicians explore deeper information data relationships. Among them, entity analysis in literature analysis is of even greater significance and role. With the rapid development of big data and computer information processing technologies, there is an increasing need for accurate and reliable methods to cluster and analyze massive literature and entities in the literature, so as to help the system provide more accurate and efficient services such as hot topic recommendations, searches, and question answering for users, and thus recommend more relevant information to users. Especially when facing knowledge service entity clustering with unknown quantity and content, the selection of the number of clusters and the processing means are of crucial importance.

[0003] In the existing entity clustering processing methods related to knowledge services, whether it is partitioning clustering represented by K-means, density clustering represented by DBSCAN, hierarchical clustering represented by Chameleon clustering, graph clustering represented by spectral clustering, or model clustering represented by Gaussian mixture model (GMM), the number of clusters needs to be used as an initial input, and then the entities are clustered. This results in the fact that although traditional methods have their respective advantages in clustering scenarios, the number of clusters, as a prior condition, often has a profound impact on the clustering results. Even if subsequent clustering steps are combined with deep learning for analysis, once there is a large error in the artificially estimated number of clusters, it will also have a very bad impact on the clustering results. In addition, when facing large-scale, high-dimensional, and complex data as mentioned above, directly applying traditional methods often fails. There is a need to be able to input a more accurate number of clusters to provide good conditions for the clustering work.

[0004] In recent years, thanks to the booming development of deep learning, deep neural networks have made it possible to extract large-scale and deep features due to their hierarchical structure and non-linear mapping ability. Therefore, how to use deep learning to accurately predict the number of clusters, improve the accuracy and efficiency of knowledge service entity clustering, and improve the information recommendation performance of the system in aspects such as hot topic recommendations, searches, and question answering has become a technical problem to be solved urgently. Summary of the Invention

[0005] The object of the present invention is to address the deficiencies existing in the prior art and to solve the technical problem of how to effectively predict the number of entity clusters under a vast amount of literature and data in network knowledge service applications, and improve the accuracy and efficiency of information data mining and pushing. Creatively, a method for predicting the number of knowledge service entity clusters based on deep learning is proposed.

[0006] The innovation of this method lies in: creatively combining the autoencoder and the Monte Carlo method, which can accurately and automatically estimate the number of clusters in a high-dimensional dataset, effectively overcoming the technical defect that the current number of clusters needs to be estimated manually, and significantly improving the information recommendation ability of the system in aspects such as hot topic recommendation, search, and question answering, and recommending more relevant information to users.

[0007] The technical solution adopted by the present invention is as follows:

[0008] A method for predicting the number of knowledge service entity clusters based on deep learning, comprising the following steps:

[0009] Step 1: Extract dataset features. It includes the following steps:

[0010] Step 1.1: Extract dataset features through an autoencoder.

[0011] First, import the normalized dataset. Through the autoencoder, complete the encoding process from the input layer to the hidden layer to obtain the dimensionality-reduced features, as shown in Equation (1):

[0012] h = gθ1(x) = α(W1x + b1) (1)

[0013] where h represents the latent variable, g represents the activation function, θ1 represents the weight affecting the node in the activation function, x represents the input; α, W1, and b1 are all calculation parameters of the activation function in the autoencoder network and are obtained through gradient descent training.

[0014] After the above processing, the high-dimensional input x is encoded into the low-dimensional latent variable h.

[0015] Preferably, the dimension of the latent feature space can be set to 16.

[0016] Step 1.2: Use the T-SNE algorithm to perform visual dimensionality reduction on the features.

[0017] T-SNE (t-distributed stochastic neighbor embedding) is a machine learning algorithm for dimensionality reduction. This method defines a non-linear mapping from the feature space P to the two-dimensional feature space Z. It calculates the difference between the mismatches of P and Z in terms of pairwise distances and the corresponding probability distributions of P and Z by minimizing the principle of asymmetry. The resulting feature space Z is more suitable for estimating the number of clusters.

[0018] Thus, the two-dimensional feature representation of the dataset is obtained.

[0019] Step 2: Perform Gaussian mixture clustering (GMM) on the dataset features and calculate various parameters.

[0020] Specifically, it includes the following steps:

[0021] Step 2.1: Create a set of GMM models.

[0022] Preset a range of values for the number of clusters K, where n is a positive integer and n ≥ 2, and create a set of models based on the range of K values.

[0023] Establish GMM clustering models for the visualized dimensionality reduction results under different K values to form a set of models U.

[0024] Step 2.2: Calculate the relevant parameters for each K value.

[0025] Let n be the number of observations, RSS be the residual sum of squares, k be the number of model parameters, and L be the likelihood function. Calculate the AIC value (Akaike Information Criterion, abbreviated as AIC) and BIC value (Bayesian Information Criterion, abbreviated as BIC) for each model through Equation 2 and Equation 3 as a direct evaluation method for the model. Among them, the AIC value reflects the appropriate number of clusters, and the BIC value reflects the simplicity and effectiveness of the model. Specifically as follows:

[0026] AIC = 2k + nln(RSS / n) (2)

[0027] BIC = kln(n) - 2ln(L) (3)

[0028] where ln() represents calculating the natural logarithm.

[0029] Calculate the silhouette coefficient and the CH value as an indirect evaluation method for the model. Among them, the within-cluster dissimilarity is a(i), the between-cluster dissimilarity is b(i), and the mean of s(i) for all samples is called the silhouette coefficient of the clustering result. The calculation formula is shown in Equation 4; the CH value, that is, the Calinski Harabasz index, is calculated as shown in Equation 5:

[0030]

[0031]

[0032] Among them, n represents the number of clusters, k represents the current class, trB(k) represents the trace of the between-class scatter matrix, and trW(k) represents the trace of the within-class scatter matrix.

[0033] Step 2.3: Judge the size of the corresponding K through the parameter values.

[0034] Take the positions of the maximum values and peaks of the silhouette coefficient and the CH value, and the positions of the minimum values of AIC and BIC to obtain the values of K corresponding to each parameter.

[0035] Step 3: Use the Monte Carlo method to obtain the predicted result of the number of clusters.

[0036] Step 3.1: Eliminate the interference effect.

[0037] Considering the influence of noise points and data dimensions, the above four parameters - the silhouette coefficient, the CH value, AIC, and BIC - are not very stable in some cases. Therefore, combining the K values under the silhouette coefficient, the CH value, AIC, and BIC parameters calculated in a certain previous calculation, first take the mean value of K once as the predicted result of this time. Then, use the Monte Carlo method to design the number of iterations. After iterating N times, find the mean value of K.

[0038] For example, set the number of iterations to 100 times, and the result K after each iteration can be obtained, making the number of clusters k in the result closer and closer to the true value. That is: the fluctuation becomes less and less obvious, and the result value K becomes more and more stable. In this way, combine the above parameter results together, which not only satisfies the situation where the samples within the cluster are more "gathered", but also satisfies the situation where the clusters are more "separated". By observing the above four parameters, the final number of clusters K can be obtained.

[0039] Step 3.2: The K value of the last iteration is the predicted value of the overall number of clusters, which is used as the final predicted result of the number of clusters.

[0040] Cluster the information data according to the number of clusters K to complete the accurate recommendation of the information data.

[0041] Beneficial effects

[0042] The method of the present invention has the following advantages compared with the prior art:

[0043] 1. The present invention is a work for determining the number of clusters in a high-dimensional dataset. Compared with existing methods, it can process more types of datasets. It can not only be applied to image datasets, but also to text datasets and audio datasets.

[0044] 2. The present invention outputs the number of clusters K of the high-dimensional dataset with higher accuracy. This method considers more parameters for evaluating the value of the number of clusters K. Therefore, it has better robustness on any type of dataset. At the same time, it can also enable other clustering methods to improve the existing clustering effect based on the results of this method.

[0045] 3. The present invention provides a better embedded feature space for the visualization of various types of datasets. It enables various types of datasets to be better visualized in a two-dimensional space, facilitating clustering analysis in more downstream tasks.

[0046] The present invention has broad application prospects in the fields of text mining and fine-grained knowledge services. Description of the drawings

[0047] Figure 1 is the overall flowchart of the present invention.

[0048] Figure 2 is the flowchart of feature extraction of the present invention.

[0049] Figure 3 is the flowchart of calculating parameters and the Monte Carlo method of the present invention. Detailed implementation manners

[0050] The method of the present invention will be further described in detail below with reference to the drawings and embodiments.

[0051] Embodiment

[0052] Taking the handwritten image dataset MNIST as an example, the experiment is carried out using Python throughout. As Figure 1 shown, it includes the following steps:

[0053] Step 1: Read the dataset and extract the features of the dataset.

[0054] Step 1.1: Call the mnist dataset in the tensorflow package to obtain the dataset variables.

[0055] Step 1.2: Classify it into a test set and a training set according to the labels it comes with.

[0056] Step 1.3: Uniformly normalize the dataset to make it conform to the format for input into the neural network model. Since the pixels of the image are 28 * 28, each data item has a dimension of 784 bits.

[0057] In this embodiment, there are a total of 70,000 data items. In this embodiment, the processing result of the 784 - dimensional data of a certain image is:

[0058] [0.,0.,0.,0.,0.......,0.32941177,0.7254902,0.62352943,0.5921569,0.23529412,0.14117648,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.87058824,0.99607843,0.99607843,0.99607843,0.99607843,0.94509804,0.7764706,0.7764706,0.7764706,0.7764706,0.7764706,0.7764706,0.7764706,0.6666667,0.20392157,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.2627451,0.44705883,0.28235295,0.44705883,0.6392157,0.8901961,0.99607843,0.88235295,0.99607843,0.99607843,0.99607843,0.98039216,0.8980392,0.99607843,0.99607843,0.54901963,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.06666667,0.25882354,0.05490196,0.2627451,0.2627451,0.2627451,0.23137255,0.08235294,0.9254902,0.99607843,0.41568628,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.3254902,0.99215686,0.81960785,0.07058824,0.,0.,0.......0.,0.,0.,0.]

[0059] Step 1.4: Define an autoencoder neural network, including an input layer, a hidden layer, and an output layer. Among them, the dimensions of the input layer and the output layer are defined as 784, and the dimension of the hidden layer is 16.

[0060] Step 1.5: Write an autoencoder model.

[0061] Call the keras method in the tensorflow package to generate layers with corresponding dimensions. Among them, the activation function of the hidden layer is relu, and the activation function of the output layer is sigmoid.

[0062] Step 1.6: Set the calculation method of the loss function loss to MSE. Set the number of training iterations per time to 100.

[0063] Step 1.7: Substitute the processing result in Step 1.3 into the defined autoencoder for fitting training to obtain the result of 16-dimensional feature extraction of the hidden layer output. As Figure 2 shown.

[0064] In this embodiment, the 16-dimensional data result after feature extraction of the above data is as follows: [4.41753, 14.25602, 4.26799, 15.39900, 4.58232, 5.21208, 2.20098, 6.41697, 10.73474, 3.91638, 9.31799, 7.08892, 7.15096, 11.15065, 5.47077, 4.20849]

[0065] Step 1.8: Use T-SNE for dimensionality reduction.

[0066] Call the T-SNE method in sklearn to define the T-SNE dimensionality reduction process.

[0067] Step 1.9: Perform T-SNE dimensionality reduction processing on the 16-dimensional data to visualize it in a two-dimensional plane, which is also convenient for subsequent Gaussian mixture clustering (GMM) operation of the data.

[0068] In this embodiment, the T-SNE dimensionality reduction result t of the above data is as follows: [-15.20039, 2.30614]

[0069] Step 2: Establish a Gaussian mixture clustering model and calculate various parameters.

[0070] Step 2.1: Define the value range of the predicted value K of the number of clusters.

[0071] In this embodiment, the value range of K is defined as [1, 20]. The GMM method in the sklearn package is called to establish GMM clustering models with different K values for the dimensionality reduction result t respectively, and a set U of 20 models is formed in total.

[0072] Step 2.2: For all the models in the set U, calculate their respective AIC, BIC, Silhouette coefficient, and CH value.

[0073] In this embodiment, when the K value = 8, the values of the above four parameters are: 197426.35, 197803.16, 0.4513, 12645.93

[0074] Step 2.3: Record the K value when the Silhouette coefficient and CH value are the largest and at the peak, and record the K value at the turning point for AIC and BIC.

[0075] In this embodiment, the K values of a certain calculation result are taken as: 9, 9, 10, 10 respectively.

[0076] Step 3: Use the Monte Carlo method to obtain the clustering number prediction result. As Figure 3 shown.

[0077] Step 3.1: Use the Monte Carlo method for the above calculated parameter values to calculate the average value K of each parameter result.

[0078] This is to eliminate the influence of noise points, etc. on the data, and gradually make the predicted K value approach the true value. <�

[0079] Specifically, first calculate the mean value of K for one experiment. For example, the value of K in this experiment in Step 2.3 is 9.5. Then, set the number of iterations to 100, and start taking the results from the 10th iteration, that is, calculate the average value of the first 10 experiment results. In this embodiment, the experiment results of 100 iterations are:

[0080] [9.469388,9.466495,9.476562,9.476842,9.476064,9.47043,9.476413,9.470769,9.502778,9.508427,9.505682,9.5,9.505814,9.511765,9.53869,9.518072,9.550854,9.578395,9.5625,9.562025,9.590385,9.574026,9.573684,9.58,9.593243,9.576027,9.572222,9.589437,9.596429,9.592754,9.585294,9.585075,9.584848,9.569231,9.553125,9.564286,9.559677,9.571311,9.5875,9.574576,9.582759,9.591228,9.626786,9.636364,9.618519,9.633019,9.619231,9.614706,9.64,9.661224,9.626042,9.6,9.616304,9.616667,9.708864,9.7,9.702381,9.70561,9.70875,9.70641,9.700263,9.706757,9.727778,9.735714,9.722059,9.784848,9.760938,9.789677,9.8,9.817241,9.853571,9.861852,9.886538,9.889,9.882083,9.943478,9.968182,9.959524,9.96,9.910526,9.931667,9.920588,9.965625,9.95,9.95,9.982692,9.983333,9.990091,9.9975,9.999722]

[0081] Step 3.2: Take the final K value as the predicted value of the overall number of clusters K, and use it as the prediction result.

[0082] In this embodiment, take 9.9. Round up, K = 10.

[0083] The above steps are the whole process of one experiment. A total of more than 10 experiments are completed, and the accuracy rate is calculated, that is, the number of times when the prediction result of 10 is correctly obtained in the experiment. Among the total of 10 experiments, the prediction accuracy rate is 9 / 10, and the average predicted value is 9.9 (±0.4).

[0084] To illustrate the clustering number prediction effect of the present invention, this experiment was carried out under the same conditions, using the same data set, and comparing two types of methods respectively. The first type of method is the clustering prediction method based on a convolutional neural network, and the second method is the clustering number prediction method of the present invention.

[0085] For this image data set, the prediction accuracy of the prior art neural network-based prediction method is 9 / 10, but for the text data set 20NewsGroup and the audio data set Urbansound8K, the prediction accuracies are 2 / 10 and 1 / 10 respectively. The prediction accuracies of the method of the present invention for these two data sets are 6 / 10 and 8 / 10 respectively.

[0086] The experiment shows the effectiveness of the method for predicting the clustering number of knowledge service entities based on deep learning proposed by the present invention.

Claims

1. A method for predicting the number of clusters of knowledge service entities based on deep learning, characterized in that, It includes the following steps: Step 1: Extract the features of the dataset; it includes the following steps: Step 1.1: Extract the features of the dataset through an autoencoder; First, import the normalized dataset; through the autoencoder method, complete the encoding process from the input layer to the hidden layer to obtain the dimensionality-reduced features, as shown in Equation (1): h = gθ1(x) = α(W1x + b1) (1) where h represents the latent variable, g represents the activation function, θ1 represents the weight affecting the node in the activation function, x represents the input; α, W1, and b1 are all calculation parameters of the activation function in the autoencoder network and are obtained through gradient descent training; After the above processing, the high-dimensional input x is encoded into the low-dimensional latent variable h; Step 1.2: Use the T-SNE algorithm to perform visual dimensionality reduction on the features to obtain the two-dimensional feature representation of the dataset; Step 2: Perform Gaussian mixture clustering GMM on the dataset features and calculate various parameters; Step 2.1: Create a set of GMM models; Set a range of values for the number of clusters K, where n is a positive integer and n ≥ 2, and create a model set based on the range of K values; Establish GMM clustering models under different K values for the visual dimensionality reduction results to form a set U of models; Step 2.2: Calculate the relevant parameters for each K value; Let n be the number of observations, RSS be the residual sum of squares, k be the number of model parameters, and L be the likelihood function; calculate the AIC value and BIC value for each model through Equations (2) and (3) as a direct evaluation means of the model. Among them, the AIC value reflects the appropriate number of clusters, and the BIC value reflects the simplicity and effectiveness of the model; specifically as follows: AIC = 2k + nln(RSS / n) (2) BIC = kln(n) - 2ln(L) (3) where ln() represents calculating the natural logarithm; Calculate the silhouette coefficient and CH value as an indirect evaluation means of the model; among them, the within-cluster dissimilarity is a(i), the between-cluster dissimilarity is b(i), and the mean of s(i) of all samples is called the silhouette coefficient of the clustering result, and the calculation formula is as shown in Equation (4); the CH value, that is, the Calinski Harabasz index, is calculated as shown in Equation (5): where n represents the number of clusters, k represents the current class, trB(k) represents the trace of the between-class scatter matrix, and trW(k) represents the trace of the within-class scatter matrix; Step 2.3: Determine the size of the corresponding K through the parameter values; Take the positions of the maximum values and peaks of the silhouette coefficient and CH value, and the minimum values of AIC and BIC to obtain the K value ranges corresponding to each parameter; Step 3: Use the Monte Carlo method to obtain the clustering number prediction result; Step 3.1: Eliminate the interference effect; Combined with the K value under the silhouette coefficient, CH value, AIC, and BIC parameters calculated in a certain calculation in Step 2, first take the mean of K once as the prediction result of this time; then, use the Monte Carlo method to design the number of iterations. After iterating N times, calculate the mean of the K values; The K value of the last iteration is the predicted value of the overall clustering number, which is used as the final clustering number prediction result; Cluster the information data according to the number of clusters K to complete the precise recommendation of the information data.

2. The method for predicting the number of clusters of knowledge service entities based on deep learning according to claim 1, characterized in that The dimension of the latent feature space is 16.

Images