Short text clustering method based on singular value decomposition and domain pre-training

Abstract

The application discloses a short text clustering method based on singular value decomposition and field pre-training, and steps are as follows: short texts are acquired as a data set, and the data set is preprocessed; a CBOW model is used to extract word vector features, including frequent words and corresponding documents; the frequent words and the documents are formed into a matrix form, the matrix is decomposed by using an SVD method, then a GCN representation is constructed, corresponding graph structure vectors are formed, a RoBERTa model is used to perform basic training on the documents, new semantic subtasks are added for training, NLP semantic vectors are generated, two vectors are combined, a D-D-Kmeans algorithm is used for clustering, and clustering results are obtained. The application can improve the clustering convergence speed and accuracy, can resist noise interference well, and has strong robustness.

Description

Technical Field

[0001] This invention belongs to the field of computer graph data processing technology, specifically relating to a short text clustering method based on singular value decomposition and domain pre-training. Background Technology

[0002] With the rapid development of network information technology, text information on the internet is expanding rapidly. How to accurately and quickly classify and cluster this text information is a crucial research topic. For example, identifying spam text messages and classifying information typically requires scanning tens of thousands of web pages, a difficult and time-consuming process. This problem can be solved through document clustering. Clustering is the process of grouping similar items into one class and different items into different categories; it is an important data analysis tool. Data clustering methods divide data into different unknown classes based on their inherent properties or characteristics, ensuring that data within the same class has the highest possible similarity.

[0003] Short text is a common content format, including text messages, user comments, and Weibo topics. Cluster analysis of short text has significant application value, such as opinion mining from user comments, topic detection on social media, and public opinion early warning. However, due to the characteristics of short text data, such as high feature dimensionality, difficulty in feature extraction, and high levels of noise, traditional clustering algorithms are not very accurate and are slow. This is mainly due to four reasons: First, for unstructured or semi-structured short text data, the high dimensionality of text vectors leads to inaccurate clustering results; second, random selection of initial cluster centers can result in high algorithm time consumption; third, the sparse features of short text data make the clustering process susceptible to noise, resulting in poor algorithm robustness; and fourth, the semantic gap problem—the semantic information is insufficient, and the semantics expressed within the text cannot be fully extracted during pre-training. Summary of the Invention

[0004] The purpose of this invention is to provide a short text clustering method based on singular value decomposition and domain pre-training, which solves the problems of inaccurate clustering results, random selection of initial cluster centers and poor algorithm robustness in the prior art.

[0005] The technical solution adopted in this invention is:

[0006] A short text clustering method based on singular value decomposition and domain pre-training includes the following steps:

[0007] Step 1: Obtain short texts as a dataset and preprocess the dataset;

[0008] Step 2: For the preprocessed dataset, use the CBOW model to extract word vector features, including frequent words and their corresponding documents;

[0009] Step 3: Construct matrix Q using word vector features via TF-IDF algorithm, perform singular value decomposition on matrix Q, and then convolve it using GCN to form the corresponding graph structure vector;

[0010] Step 4: Build the RoBERTa model, pre-train the feature vectors extracted in Step 2, extract the Token-Document Relation Prediction subtask, add the IRRelevance subtask for retraining, and output the NLP semantic vector.

[0011] Step 5: Concatenate the graph structure vector with the NLP semantic vector, and use the DD-Kmeans algorithm to perform clustering to obtain the clustering results.

[0012] The invention is further characterized by:

[0013] Step 1 specifically involves using the Natural Language Toolkit (NLTK) to perform tokenization, stop word removal, part-of-speech tagging, and stemming on the short text dataset.

[0014] Step 2 specifically includes:

[0015] Step 2.1: Map the preprocessed dataset to a two-dimensional vector space;

[0016] Step 2.2: Build the CBOW model. Input a one-hot vector of size V into the CBOW model. In this vector, only the index of the corresponding word is 1, and all other positions are 0. We can denote the input vector as x.

[0017] The input layer passes through a matrix W of size V*N. V*N After multiplication, we obtain the hidden layer vector h of size N. The result of the multiplication is actually derived from the matrix W. V*N Extract the vector from the k-th row (where k is the index of 1 in a one-hot vector), which is the word vector corresponding to the word.

[0018]

[0019] Step 2.3: The hidden layer is then multiplied by an N*V matrix W′ to obtain a V-dimensional output layer vector u; where the j-th element u in the output layer vector... j It is the j-th column vector v′ in matrix W′. wj Product with the hidden layer vector h:

[0020] u=hW′

[0021]

[0022] Step 2.4: The output vector u is processed by softmax to obtain the predicted probability of each word in this table. The word with the highest output probability is the result of this prediction, i.e., the frequent word. The frequent word and its corresponding document constitute the word vector feature.

[0023] Step 3 specifically includes:

[0024] Step 3.1: Sort the frequent words in the word vector features in descending order, build an FP-Tree, and mine the FP-Tree to obtain the word frequency set; build an IDF model based on the word frequency, and call the model to obtain the inverse text frequency index (IDF value) of the frequent words;

[0025] Step 3.2: Obtain the term frequency matrix (TF matrix) for each frequent word. The term frequency matrix (TF matrix) consists of two columns: the first column is the label, and the second column is the IDF value of the frequent word with column names added.

[0026] Step 3.3: Map the matrix set obtained in Step 3.2 to the vector space and convert it into a TF-IDF term frequency matrix Q of m×n.

[0027] Step 3.4: Perform singular value decomposition on matrix Q to obtain its singular value matrix Σ;

[0028] Step 3.5: Build the GCN model and input the singular value matrix into the GCN model to output the graph structure vector.

[0029] Step 3.4 includes:

[0030] Step 3.4.1, represent matrix Q as:

[0031] Q=U∑V T

[0032] Where U is an m×m matrix, Σ is an m×n singular value matrix, with all elements except those on the main diagonal being 0. Each element on the main diagonal is called a singular value, and V is an n×n matrix. Both U and V are unitary matrices, meaning they satisfy:

[0033] U T U = I, V T V = I

[0034] Step 3.4.2: Perform matrix multiplication between the transpose of Q and Q, which will result in an n×n square matrix Q. T Q, against the Q formation T The eigenvalues ​​and eigenvectors obtained by performing eigendecomposition on Q satisfy the following equation:

[0035] (QT Q)v i =λ i v i

[0036] Obtain matrix Q T The n eigenvalues ​​of Q and their corresponding n eigenvectors v, Q T All eigenvectors of Q span an n×n matrix V. We call each eigenvector in V a right singular vector of Q, and λ is the eigenvalue of the matrix.

[0037] Step 3.4.3: Perform matrix multiplication on Q and its transpose, which will result in an m×m square matrix QQ. T QQ of the square T Eigenvalue decomposition yields eigenvalues ​​and eigenvectors that satisfy the following equation:

[0038] (QQ T )u i =λ i u i

[0039] Get the matrix QQ T Given m eigenvalues ​​and their corresponding m eigenvectors u, QQ T All the eigenvectors of Q span an m×m matrix U, and we call each eigenvector in U a left singular vector of Q.

[0040] Step 3.4.4: Since Σ is 0 except for the singular values ​​on the diagonal, we only need to find σ for each singular value. Because:

[0041]

[0042] It can be known that σ i =Av i / u i Find each singular value σ, and then find the singular value matrix Σ.

[0043] Step 3.5 includes:

[0044] The GCN model performs convolution operations on graph data, fusing attribute information of nodes and structural information between nodes. By stacking multiple layers of networks, it extracts information from the multi-order neighborhoods of nodes to construct GCN representations, forming the corresponding graph structure vectors.

[0045]

[0046] in It involves adding an identity matrix of the same size to the adjacency matrix of an undirected graph and performing a self-connection operation. The purpose is to preserve the information of the nodes in the graph as the information propagates through each layer of the graph convolutional neural network. yes The degree matrix, H (l) This is the activation unit matrix of the l-th layer in a graph convolutional neural network. Before training, the activation unit matrix is ​​the feature matrix X, W. (l) These are the weight parameters of the l-th layer in a graph convolutional neural network.

[0047] Step 4 includes:

[0048] Step 4.1: Build the RoBERTa model by importing the training set from the word vector features output in Step 2 into the RoBERTa model and initializing the RoBERTa pre-trained weights.

[0049] Step 4.2: Set the output embedding vector dimension and the number of training dataset iterations;

[0050] Step 4.3: The model transforms each character in the entity training set into a one-dimensional vector to obtain a sentence's segment vector and position vector. The segment vector and position vector are then used as input to the deep learning model, and finally, a vector that integrates the semantic information of the entire text is output.

[0051] Step 4.4: Add the extracted Capitalization Prediction subtask to the RoBERTa model, and divide the above output vector into five layers. The first three layers are trained using the RoBERTa model.

[0052] Step 4.5: Add the extracted Token-Document RelationPrediction subtask to the RoBERTa model, and train the remaining two times.

[0053] Step 4.6: Add the IR Relevance subtask to the RoBERTa model, and then train the vectors from Step 4.3 as a whole, outputting the training results, which are the NLP semantic vectors.

[0054] Step 5 includes:

[0055] Step 5.1: Concatenate the graph structure vector generated in Step 3 and the NLP semantic vector generated in Step 4 as input, and calculate the σ radius and weights of the n input data points:

[0056] Calculate the Euclidean distance between two vectors:

[0057]

[0058] dis is the formula for calculating the distance between two vectors, X gi Let X be the coordinate representation of the GCN vector of the i-th vector. biLet represent the coordinates of the i-th NLP vector. The distance between the coordinates of two vectors is calculated using GCN coordinates and NLP coordinates, respectively. TF-IDF is used to count the frequency of word occurrences. If x∈X, the density dens(x) of point x is calculated using the following formula:

[0059]

[0060] From the data above, the radius σ can be calculated:

[0061]

[0062] Where, p k_neares(k) Indicates the relationship with point p i The k nearest points; dis() represents the Euclidean distance between the points; dens() represents the vector density; delta() represents the error between the distance and the density; w1 and w2 are constant coefficients; q is the object p in the σ-neighborhood.

[0063] weight ω p The calculation formula is as follows:

[0064]

[0065] Where range represents the size of the dataset's vector space, and its calculation method is the same as that for Euclidean distance, with the following formula:

[0066]

[0067] Step 5.2: Select the data point with the largest weight as the first cluster center;

[0068] Step 5.3: Calculate the centroid index for each data point and select the data point with the largest centroid index for pre-classification;

[0069] Step 5.4: Calculate the average BWP index value of the n data points after pre-classification; the calculation formula is as follows:

[0070]

[0071] Step 5.5: If the average value of the BWP index increases, this data point becomes the cluster center, and the data in its σ neighborhood will not participate in the selection of the next center point. Proceed to step 5.6. If the average value of the BWP index decreases, proceed to step 5.7.

[0072] Step 5.6: If there are data points that can be clustered, proceed to step 5.3; otherwise, proceed to step 5.8.

[0073] Step 5.7: Use the generated center points as the initial cluster centers until the positions of the cluster centers no longer change, and then output the clustering results.

[0074] The beneficial effects of this invention are:

[0075] Compared to the classical clustering algorithm that randomly selects initial cluster centers, the algorithm in this paper uses a vectorized representation of the word frequency matrix of the original dataset, performs matrix decomposition using the singular value decomposition algorithm, performs convolution using the GCN model, and trains it using the RoBERTa model to generate graph structures and NLP semantic vectors. Finally, it uses the DD-Kmeans algorithm to determine the initial cluster centers, which can improve the clustering convergence speed and accuracy, better resist noise interference, and has strong robustness. Attached Figure Description

[0076] Figure 1 This is the overall block diagram of the short text clustering method based on singular value decomposition and domain pre-trained model of the present invention;

[0077] Figure 2 This is an architecture diagram of the CBOW model in this invention;

[0078] Figure 3 This is a schematic diagram of the convolutional neural network in this invention;

[0079] Figure 4 This is a schematic diagram of the Transformer encoding unit in this invention. Detailed Implementation

[0080] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0081] This invention provides a short text clustering method based on singular value decomposition and domain pre-training, such as... Figures 1-4 This includes the following steps:

[0082] Step 1: Obtain the short texts as a dataset and preprocess the dataset. The specific preprocessing steps are as follows:

[0083] The short text dataset was processed using the Natural Language Toolkit (NLTK) to perform operations such as tokenization, stop word removal, part-of-speech tagging, and stemming.

[0084] Step 2, Feature Extraction: The text information is vectorized using the CBOW model within the word2vec word vector model in natural language processing, and word vector features are extracted. The word vector features include frequent words and their corresponding documents.

[0085] The CBOW model consists of an input layer, a hidden layer, and an output layer. The input layer takes word variables as input, the hidden layer sums the vectors from the input layer, and the output layer removes the previous hidden layer and replaces it with a Huffman tree.

[0086] In the word2vec word vector model, two deep model architectures are proposed to learn distributed representations of words. These architectures attempt to minimize computational complexity, as much of this complexity arises from the non-linear hidden layers in the model. However, the proposed word2vec architectures may be able to be trained on more efficient data.

[0087] The Word2vec model's architecture is similar to a feedforward neural network language model, which consists of an embedding layer, hidden layers, a projection layer, and an output layer. The non-linear hidden layers are removed, and the projection layer is shared by all words (not just the projection matrix); therefore, all words are projected to the same position (their vectors are averaged). This architecture is called a bag-of-words model because the order of already appearing words does not affect the projection; simultaneously, upcoming words are also utilized. In the following sections, the best performance is achieved by constructing a log-linear classifier with four future words and four historical words at the input, where the training criterion is correctly classifying the current (intermediate) word. The training complexity is:

[0088] Q = N × D + D × log2(V)

[0089] This model architecture is further called the Continuous Bag-of-Words Model (CBOW) because, unlike the standard bag-of-words model, it uses a continuously distributed representation of context. The model architecture is as follows: Figure 2 As shown. In this model, the weight matrix between the input layer and the projection layer is shared for all word positions in the same way as in neural network language models.

[0090] Using the word2vec word vector model, the textual description information of nodes can be represented as vectors, while preserving as much of the information contained in the text as possible. Then, the generated word vectors are used as feature vectors of the nodes and input into a graph representation learning model, enabling better embedding representation of the nodes.

[0091] Step 2.1: Map the preprocessed dataset to a two-dimensional vector space;

[0092] Step 2.2: Build the CBOW model. Input a one-hot vector of size V into the CBOW model. In this vector, only the index of the corresponding word is 1, and all other positions are 0. We can denote the input vector as x.

[0093] The input layer passes through a matrix W of size V*N. V*N After multiplication, we obtain the hidden layer vector h of size N. The result of the multiplication is actually derived from the matrix W. V*N Extract the vector from the k-th row (in a one-hot vector, the index of 1 is k), which is the word vector corresponding to the word. That is:

[0094]

[0095] Step 2.3: The hidden layer is then multiplied by an N*V matrix W′ to obtain a V-dimensional output layer vector u. The j-th element u in the output layer vector... j It is the j-th column vector v′ in matrix W′. wj Product with the hidden layer vector h:

[0096] u=hW′

[0097]

[0098] Step 2.4: The output vector u is processed using softmax to obtain the predicted probability of each word in this table, i.e., the acquired word vector features, where the input w... I Output w j The probability is:

[0099]

[0100] The word with the highest output probability is the frequent word, and the frequent words and their corresponding documents constitute the word vector features.

[0101] Step 3: Combine the frequent words in the word vector features with the documents to form a matrix. Use FP-Tree to mine the number of frequent words and TF-IDF algorithm to construct matrix Q. Use singular value decomposition to decompose matrix Q. Use GCN (Graph Convolutional Neural Network) to convolve the decomposed matrix to form the corresponding graph structure vector.

[0102] Graph Convolutional Neural Networks (GCNs) are the most classic model in graph neural networks. A GCN is a first-order local approximation of spectral graph convolution, mimicking the convolution operation on graph data. It integrates the attribute information of nodes and the structural information between nodes, resulting in a multi-layered GCN. In a GCN, each layer's convolution operation processes only one hop of neighborhood information around a node. By stacking multiple layers, information from multiple orders of neighborhoods of nodes can be extracted, such as... Figure 3 As shown.

[0103] Before being input into the network, the adjacency matrix of graph data needs to be normalized. The purpose is to maintain the original distribution of H when information propagates in each layer of the graph convolutional neural network, and to prevent nodes with excessively high or low degrees from having a significant impact on the feature distribution.

[0104] Singular Value Decomposition (SVD) is a widely used algorithm in machine learning. It can be used not only for feature decomposition in dimensionality reduction algorithms, but also for recommendation systems and natural language processing. It is the cornerstone of many machine learning algorithms.

[0105] Step 3.1: Sort the frequent words in the word vector features in descending order, build an FP-Tree, and mine the FP-Tree to obtain the word frequency set; build an IDF model based on the word frequency, and call the model to obtain the inverse text frequency index (IDF value) of the frequent words;

[0106] Step 3.2: Obtain the term frequency matrix (TF matrix) for each frequent word. The term frequency matrix (TF) consists of two columns: the first column is the label, and the second column is the IDF value of the frequent word with column names added.

[0107] Step 3.3: Map the matrix set obtained in Step 3.2 to the vector space and convert it into a TF-IDF term frequency matrix Q of m×n.

[0108] Step 3.4: Perform singular value decomposition on matrix Q to obtain its singular value matrix Σ;

[0109] SVD decomposes the original dataset matrix Data into three matrices: U, Σ, and V. T If the original matrix is ​​m rows and n columns, then U, Σ, and V T These are respectively m rows and m columns, m rows and n columns, and n rows and n columns. Of particular note is the matrix Σ, which has only diagonal elements; all other elements are 0. A convention is that the diagonal elements of Σ are arranged in descending order. These diagonal elements are called singular values.

[0110] The geometric meaning of SVD decomposition is that any matrix A can be transformed into a diagonal matrix Σ under a series of rotations and translations, where the geometric meaning of unitary matrices U and V is the superposition of a series of rotations and translations. Furthermore, the geometric meaning of singular values ​​is that matrix A maps a unit sphere (with radius 1) in vector space to an ellipsoid, and the length of each axis of the ellipsoid is the magnitude of each singular value.

[0111] The specific steps in step 3.4 are as follows:

[0112] Step 3.4.1, matrix Q can be represented as:

[0113] Q=U∑V T

[0114] Where U is an m×m matrix, Σ is an m×n singular value matrix, with all elements except those on the main diagonal being 0. Each element on the main diagonal is called a singular value, and V is an n×n matrix. Both U and V are unitary matrices, meaning they satisfy:

[0115] U T U = I, V T V = I

[0116] Step 3.4.2: Perform matrix multiplication between the transpose of Q and Q, which will result in an n×n square matrix Q. T Q, against the Q formation T The eigenvalues ​​and eigenvectors obtained by performing eigendecomposition on Q satisfy the following equation:

[0117] (Q T Q)v i =λ i v i

[0118] Obtain matrix Q T The n eigenvalues ​​of Q and their corresponding n eigenvectors v, Q T All eigenvectors of Q span an n×n matrix V. We call each eigenvector in V a right singular vector of Q, and λ is the eigenvalue of the matrix.

[0119] Step 3.4.3: Perform matrix multiplication on Q and its transpose, which will result in an m×m square matrix QQ. T QQ of the square T Eigenvalue decomposition yields eigenvalues ​​and eigenvectors that satisfy the following equation:

[0120] (QQ T )u i =λ i u i

[0121] Get the matrix QQ T Given m eigenvalues ​​and their corresponding m eigenvectors u, QQ T All the eigenvectors of Q span an m×m matrix U, and we call each eigenvector in U a left singular vector of Q.

[0122] Step 3.4.4: Since Σ is 0 except for the singular values ​​on the diagonal, we only need to find σ for each singular value. Because:

[0123]

[0124] It can be known that σ i =Avi / u i Find each singular value σ, and then find the singular value matrix Σ.

[0125] Step 3.5: Build the GCN model by inputting the singular value matrix, an N×F0 matrix representing the graph structure, into the GCN (Graph Convolutional Neural Network) model. The singular value matrix Σ has dimensions N×F0, where N is the number of nodes and F0 is the number of input features per node; the output is a graph structure vector.

[0126] Specifically, the GCN model performs convolution operations on graph data, fusing attribute information of nodes and structural information between nodes. By stacking multiple layers of networks, it extracts information from the multi-order neighborhoods of nodes to construct GCN representations, forming the corresponding graph structure vectors.

[0127]

[0128] in It involves adding an identity matrix of the same size to the adjacency matrix of an undirected graph and performing a self-connection operation. The purpose is to preserve the information of the nodes in the graph as the information propagates through each layer of the graph convolutional neural network. yes The degree matrix, H (l) This is the activation unit matrix of the l-th layer in a graph convolutional neural network. Before training, the activation unit matrix is ​​the feature matrix X, W. (l) These are the weight parameters of the l-th layer in a graph convolutional neural network.

[0129] Step 4: Construct the RoBERTa model, employing a RoBERTa + semantic subtask pre-training method. The document is trained twice: first on the RoBERTa model, then a second training iteration using the semantic subtask, and finally, a final training iteration using both RoBERTa and the semantic subtask. This generates NLP semantic vectors.

[0130] In step 4, the RoBERTa model is an encoder structure based on a bidirectional Transformer model. Its structure is mainly a stack of Transformers. As long as the corresponding token is fed into RoBERTa, each Transformer layer outputs a corresponding number of hidden vectors, which are passed down layer by layer until the final output.

[0131] The Transformer encoder unit primarily utilizes a multi-layer self-attention mechanism, where the Self-Attention mechanism learns a weight for each word in the input vector. RoBERTa is a bidirectional connection in the Transformer encoder, capable of capturing the left and right contexts from all layers; after adding a fully connected layer, a pre-trained RoBERTa model is generated. After training, the RoBERTa model without the fully connected layer can be used for various NLP tasks, as shown in the image. Figure 4 As shown.

[0132] The RoBERTa model can learn the contextual relationships between words in a corpus. When processing a word, it can simultaneously consider the information of the preceding and following words to obtain the semantic features of the word as a whole.

[0133] Step 4.1: Build the RoBERTa model by importing the training set from the word vector features output in Step 2 into the RoBERTa model and initializing the RoBERTa pre-trained weights.

[0134] Step 4.2: Set the output embedding vector dimension and the number of training dataset iterations;

[0135] Step 4.3: The model transforms each character in the entity training set into a one-dimensional vector to obtain a sentence's segment vector and position vector. The segment vector and position vector are then used as input to the deep learning model, and finally, a vector that integrates the semantic information of the entire text is output.

[0136] Step 4.4: Add the extracted CapitalizationPrediction subtask to the RoBERTa model, and divide the above output vector into five layers. The first three layers are trained using the RoBERTa model.

[0137] Step 4.5: Add the extracted Token-Document RelationPrediction subtask to the RoBERTa model, and train the remaining two times.

[0138] Step 4.6: Add the IR Relevance subtask to the RoBERTa model, and then train the vectors from Step 4.3 as a whole, outputting the training results, which are the NLP semantic vectors.

[0139] Step 5: Concatenate the graph structure vector with the NLP semantic vector, and use the DD-Kmeans algorithm to perform clustering to obtain the clustering results.

[0140] The DD-Kmeans algorithm, based on distance and density, is used to cluster two vectors. This algorithm employs a weighted approach to balance density and distance, solving the problem globally to ensure that the selected cluster centers better reflect the data distribution and reduce the number of iterations. The weights for each point are calculated based on the dataset size and the distances between data points. The DD-Kmeans algorithm selects cluster centers based on the minimax principle, avoiding the local optima problem caused by randomness in initial cluster center selection. This approach better handles outliers and low-density regions, and automatically determines the number of cluster centers using weights and the BWP (Browser-Wide Potential) metric.

[0141] In the DD-Kmeans algorithm, all distance calculations use Euclidean distance, and the radius of each data point p is adaptively calculated using a greedy strategy. The DD-Kmeans algorithm selects cluster centers sequentially as follows: Based on the minimax principle, the data point with the largest BWP value is selected as the experimental cluster center for pre-classification, i.e., all data points are assigned to the class represented by the nearest cluster center. The change in the average BWP value of all data points before and after pre-classification is compared. If the average BWP value increases, this point becomes the cluster center, and data points within the -neighborhood of the new cluster center do not participate in the selection of subsequent cluster centers. Simultaneously, the generation of a new cluster center may change the nearest cluster center of a data point; therefore, the cluster center values ​​of all data points are updated after each cluster center is generated. If the average BWP value decreases or no data point can be selected, the selection of cluster centers stops. This method automatically determines k cluster centers.

[0142] The selection of centroids in the DD-Kmeans algorithm is similar to the actual clustering effect: there is a certain distance between cluster centers, and the density of points around the cluster centers is relatively high. Observing the weight calculation formula, we can see that the greater the density around a data point, the greater its weight. Applying the minimax principle to the centroid index, if a data point has a large weight and is far from the cluster center, it is more likely to be selected as a centroid. At the beginning of clustering, there are no centroids, and the centroid index lacks the θ parameter and cannot be calculated. The more data points a data object has within a given spatial range, the more beneficial it is to converge the objective function if that data object is chosen as the cluster center. Therefore, selecting the point with the largest weight as the first centroid helps improve the cluster compactness.

[0143] Distance is defined as the distance from a sample point to the centroid. The smaller the sum of the distances from all sample points to the centroid, the more similar the samples in the cluster are considered, and the smaller the intra-cluster differences are. All distance calculations use Euclidean distance, and the formula for the distance between two vectors is:

[0144]

[0145] Where: X gi Let X be the coordinate representation of the GCN vector of the i-th vector. bi Let w1, w2, n1, and n2 be the coordinate representation of the i-th NLP vector, where w1, w2, n1, and n2 are constant coefficients. The distance between the coordinates of two vectors is calculated using both GCN coordinates and NLP coordinates.

[0146] Density is defined as the frequency of a word appearing in a region, and TF-IDF is used to count the number of times a word appears.

[0147] If x∈X, the formula for calculating the density at point x is:

[0148]

[0149] Where n i,j The denominator is the number of times the word appears in the data, while the total number of times all words appear in the data is represented by the denominator.

[0150] The radius of the data points is continuously and adaptively calculated using a greedy strategy. The calculation formula is as follows:

[0151]

[0152] Where, p k_neares(k) Indicates the relationship with point p i The k nearest points; dis() represents the Euclidean distance between the points, dens() represents the vector density, and delta() represents the error between the distance and the density. The value of k is generally 4 in two-dimensional spatial clustering; in other cases, it can be a value equal to a fraction of the dataset. (n is the total number of data samples; (This indicates rounding down). Based on the distance between object p and its σ-neighborhood object q, the weight of p is calculated, and the weight is processed to obtain the centroid index C for each data point. p The calculation formula is:

[0153] C p =ω p *θ p

[0154] Where, ω p θ represents the weight of data point p, reflecting the neighborhood density of point p; p Let p be the distance between data point p and its nearest center point i. The calculation formulas are as follows:

[0155]

[0156]

[0157] Where: k represents the number of existing centroids; m represents the number of data objects in the σ-neighborhood of data point p; range represents the size of the dataset vector space, calculated in the same way as the Euclidean distance calculation, using the following formula:

[0158]

[0159] Where: x represents the dimension of the dataset; max and min represent the maximum and minimum values ​​of the corresponding features in the dataset; 2 This represents the square of the Euclidean distance; the range value is essentially the modulus of the entire dimensional range of the dataset. Each data point within the σ-neighborhood of point p will have a value of ω. p The contribution value ranges from 0 to 1, with a larger contribution value the closer to point p. Furthermore, ω... p The larger the value of θ, the more data there is around point p, and the more concentrated the data is; p The larger the value, the farther point p is from the already generated cluster centers. ω p With θ p The center point index C obtained by multiplication p The larger the value, the denser the clusters, and the greater the dissimilarity between the two clusters. The time consumption of the DD-Kmeans algorithm is mainly determined by the number of iterations. Selecting cluster centers using centroid indices can effectively reduce the number of iterations and improve the algorithm's time performance.

[0160] The Cluster Effectiveness metric (BWP) is used as the evaluation metric, and the decision to continue selecting centroids is based on changes in the average BWP value. The formula for calculating the average BWP value is:

[0161]

[0162] Where: n is the size of the dataset; b(j,i) and ω(j,i) are defined as follows: Given a dataset S with n data objects, assuming the n data objects are divided into k classes, the inter-class distance b(j,i) of object i in class j is defined as the minimum distance from this sample to the average distance of samples in each of the other classes, and the intra-class distance ω(j,i) of object i in class j is defined as the average distance from this data object to other data objects in class j. The formulas are as follows:

[0163]

[0164]

[0165] Where: c and j represent class labels; n c The number of elements in class c; This represents the p-th data object of class c; Let b(j,i) represent the i-th object in the j-th class. The larger b(j,i) is, the better the inter-class separation; the smaller ω(j,i) is, the better the intra-class compactness. Therefore, the larger the BWP index value, the better the clustering effect.

[0166] The specific steps of step 5 are as follows:

[0167] Step 5.1: Concatenate the graph structure vector generated in Step 3 and the NLP semantic vector generated in Step 4 as input, and calculate the radius σ and weight ω of the n input data points in the following manner. p ;

[0168] Calculate the Euclidean distance between two vectors:

[0169]

[0170] dis is the formula for calculating the distance between two vectors, where: X gi Let X be the coordinate representation of the GCN vector of the i-th vector. bi Let represent the coordinates of the i-th NLP vector. The distance between two vector coordinates is calculated using GCN coordinates and NLP coordinates respectively, with w1, w2, n1, and n2 being constant coefficients. TF-IDF is used to count the frequency of word occurrences. If x∈X, the density dens(x) of point x is calculated using the following formula:

[0171]

[0172] From the data above, the radius σ can be calculated:

[0173]

[0174] weight ω p The calculation formula is as follows:

[0175]

[0176] Where range represents the size of the dataset's vector space, and its calculation method is the same as that for Euclidean distance, with the following formula:

[0177]

[0178] Step 5.2: Select the data point with the largest weight as the first cluster center;

[0179] Step 5.3: Calculate the centroid index for each data point and select the data point with the largest centroid index for pre-classification;

[0180] Step 5.4: Calculate the average BWP index value of the n data points after pre-classification. The calculation formula is as follows:

[0181]

[0182] Step 5.5: If the average value of the BWP index increases, this data point becomes the cluster center, and the data in its σ neighborhood will not participate in the selection of the next center point. Proceed to step 5.6. If the average value of the BWP index decreases, proceed to step 5.7.

[0183] Step 5.6: If there are data points that can be clustered, proceed to step 5.3; otherwise, proceed to step 5.8.

[0184] Step 5.7: Use the generated center points as the initial cluster centers until the positions of the cluster centers no longer change, and then output the clustering results.

Claims

1. A short text clustering method based on singular value decomposition and domain pre-training, characterized by the following steps: include: Step 1: Obtain short texts as a dataset and preprocess the dataset; Step 2 involves extracting word vector features from the preprocessed dataset using the CBOW model, including frequent words and their corresponding documents. Specifically, Step 2 includes: Step 2.1: Map the preprocessed dataset to a two-dimensional vector space; Step 2.2: Build the CBOW model. Input a one-hot vector of size V into the CBOW model. In this vector, only the index of the corresponding word is 1, and all other positions are 0. We can denote the input vector as x. The input layer passes through a Matrix of size After multiplication, we obtain the hidden layer vector h of size N. The result of multiplication is actually derived from the matrix. Extract the vector from the k-th row, which is the word vector corresponding to the word. Step 2.3, the hidden layer then passes through a... Matrix of size After multiplication, we obtain the output layer vector of size V. u ; where the first in the output layer vector j element It is a matrix The first in column vector Product with the hidden layer vector h: Step 2.4: The output vector u is processed by softmax to obtain the predicted probability of each word in this table. The word with the highest output probability is the result of this prediction, i.e., the frequent word. The frequent word and its corresponding document constitute the word vector feature. Step 3: Construct matrix Q using word vector features via the TF-IDF algorithm. After performing singular value decomposition on matrix Q, convolve it using GCN to form the corresponding graph structure vector. Specifically, step 3 involves: Step 3.1: Sort the frequent words in the word vector features in descending order, build an FP-Tree, and mine the FP-Tree to obtain the word frequency set; build an IDF model based on the word frequency, and obtain the IDF value of frequent words through the IDF model; Step 3.2: Obtain the word frequency matrix for each frequent word. The word frequency matrix consists of two columns: the first column is the label, and the second column is the IDF value of the frequent word with column names added. Step 3.3: Map the matrix set obtained in Step 3.2 to the vector space and convert it into a TF-IDF term frequency matrix Q of m×n. Step 3.4 involves performing singular value decomposition on matrix Q to obtain its singular value matrix Σ; step 3.4 includes: Step 3.4.1, represent matrix Q as: Where U is an m×m matrix, Σ is an m×n singular value matrix, all elements except those on the main diagonal are 0, and each element on the main diagonal is called a singular value, and V is an n×n matrix; both U and V are unitary matrices, that is, they satisfy: Step 3.4.2: Perform matrix multiplication between the transpose of Q and Q, which will result in an n×n square matrix. against the formation Eigenvalue decomposition yields eigenvalues ​​and eigenvectors that satisfy the following equation: Obtain the matrix Given n eigenvalues ​​and their corresponding n eigenvectors v, we will... All the eigenvectors of Q span an n×n matrix V. We call each eigenvector in V a right singular vector of Q, and λ is the eigenvalue of the matrix. Step 3.4.3: Perform matrix multiplication on Q and its transpose, which will result in an m×m square matrix. against the formation Eigenvalue decomposition yields eigenvalues ​​and eigenvectors that satisfy the following equation: Obtain the matrix Given m eigenvalues ​​and their corresponding m eigenvectors u, we will... All the eigenvectors of Q span an m×m matrix U, and we call each eigenvector in U a left singular vector of Q. Step 3.4.4: Since Σ is 0 except for the singular values ​​on the diagonal, we only need to find σ for each singular value. Because: but Find each singular value σ, and then find the singular value matrix Σ. Step 3.5: Establish the GCN model, inputting the singular value matrix into the GCN model to output the graph structure vector; Step 3.5 includes: The GCN model performs convolution operations on graph data, fusing attribute information of nodes and structural information between nodes. By stacking multiple layers of networks, it extracts information from the multi-order neighborhoods of nodes to construct GCN representations, forming the corresponding graph structure vectors. in It involves adding an identity matrix of the same size to the adjacency matrix of an undirected graph and performing a self-connection operation. The purpose is to preserve the information of the nodes in the graph as the information propagates through each layer of the graph convolutional neural network. yes The degree matrix, H (l) It is the first in a graph convolutional neural network l The activation unit matrix of a layer; the activation unit matrix of the network before training is the feature matrix. X , W (l) These are the weight parameters of the l-th layer in a graph convolutional neural network; Step 4: Build the RoBERTa model, pre-train the feature vectors extracted in Step 2, extract the Token-Document Relation Prediction subtask, add the IR Relevance subtask for retraining, and output the NLP semantic vector. Step 5: Concatenate the graph structure vector with the NLP semantic vector, and use the DD-Kmeans algorithm to perform clustering to obtain the clustering results.

2. The short text clustering method based on singular value decomposition and domain pre-training as described in claim 1, characterized in that, Step 1 specifically involves using the Natural Language Processing (NLTK) toolkit to sequentially perform tokenization, stop word removal, part-of-speech tagging, and stemming on the short text dataset.

3. The short text clustering method based on singular value decomposition and domain pre-training as described in claim 1, characterized in that, Step 4 includes: Step 4.1: Build the RoBERTa model by importing the training set from the word vector features output in Step 2 into the RoBERTa model and initializing the RoBERTa pre-trained weights. Step 4.2: Set the output embedding vector dimension and the number of training dataset iterations; Step 4.3: The model transforms each character in the entity training set into a one-dimensional vector to obtain a sentence's segment vector and position vector. The segment vector and position vector are then used as input to the deep learning model, and finally, a vector that integrates the semantic information of the entire text is output. Step 4.4: Add the extracted Capitalization Prediction subtask to the RoBERTa model, and divide the above output vector into five layers. The first three layers are trained using the RoBERTa model. Step 4.5: Add the extracted Token-Document Relation Prediction subtask to the RoBERTa model, and train the remaining two times. Step 4.6: Add the IR Relevance subtask to the RoBERTa model, and then train the vectors from Step 4.3 as a whole, outputting the training results, which are the NLP semantic vectors.

4. The short text clustering method based on singular value decomposition and domain pre-training as described in claim 1, characterized in that, Step 5 includes: Step 5.1: Concatenate the graph structure vector generated in Step 3 and the NLP semantic vector generated in Step 4 as input, and calculate the σ radius and weights of the n input data using the following method: Calculate the Euclidean distance between two vectors: dis Here is the formula for calculating the distance between two vectors: X gi Let be the coordinate representation of the GCN vector of the i-th vector. X bi For the first i The coordinate representation of an NLP vector; the distance between the coordinates of two vectors is calculated using GCN coordinates and NLP coordinates respectively; if x∈X, the density dens(x) of point x is calculated using the following formula: From the data above, the radius σ can be calculated: in, Representation and point Recent k One point; dis ( ) represents the Euclidean distance between points; ( ) represents the density of a vector. delta() This represents the error between distance and density, where w1 and w2 are constant coefficients. q For object p Objects within the σ-neighborhood; weight The calculation formula is as follows: in, range The vector space representing the size of the dataset is calculated using the same method as the Euclidean distance calculation, and the formula is: Step 5.2: Select the data point with the largest weight as the first cluster center; Step 5.3: Calculate the centroid index for each data point and select the data point with the largest centroid index for pre-classification; Step 5.4: Calculate the average BWP index value of the n data points after pre-classification; the calculation formula is as follows: Step 5.5: If the average value of the BWP index increases, this data point becomes the cluster center, and the data in its σ neighborhood will not participate in the selection of the next center point. Proceed to step 5.

6. If the average value of the BWP index decreases, proceed to step 5.

7. Step 5.6: If there are data points that can be clustered, proceed to step 5.3; otherwise, proceed to step 5.

8. Step 5.7: Use the generated center points as the initial cluster centers until the positions of the cluster centers no longer change, and then output the clustering results.

Images