A text clustering model training method and a text clustering method based on the same

Abstract

This invention provides a training method for a text clustering model. The method includes: S1, obtaining a training set and an initial model, wherein the initial model includes a pre-trained BERT model, a pre-trained deep coding model, and a clustering module, wherein the training set includes text data of multiple categories, and the pre-trained deep coding model includes an encoder, a hidden layer, and a decoder; S2, using the training set obtained in step S1 to iteratively train the initial model obtained in step S1 until convergence, wherein the pre-trained BERT model, the encoder of the pre-trained deep coding model, and the clustering module in the converged initial model constitute the final text clustering model. This invention solves the problem of separation between embedding and clustering in traditional deep clustering methods, and improves the clustering effect of the text clustering model.

Description

Technical Field

[0001] This invention relates to the field of machine learning, specifically to the field of text clustering, and more specifically, to a training method for a text clustering model and a text clustering method based thereon. Background Technology

[0002] There are polysemous text samples, such as patent texts and microblog texts. Among them, clustering patent texts is quite difficult. Reference [1] records that patent texts contain a large amount of innovative information and advanced technology studied by scholars. Analyzing patent texts can provide information on the current status and development of related patent technology research. Patent text analysis can utilize existing text classification, clustering and other methods. Clustering is essentially a grouping. The standard for grouping is that the similarity between samples within a group is as high as possible, while the similarity between samples between groups is as low as possible. Using cluster analysis on text samples such as patent texts helps to analyze information that is not easily obtained directly from text samples. For patent texts, it can also effectively understand the latest dynamic changes of patents.

[0003] Reference [2] discloses that patent texts contain patent-specific domain vocabulary, technical terms, and synonyms and antonyms. Traditional vectorization methods cannot effectively solve the problem of polysemy in patent texts. That is, complete semantic information is difficult to extract accurately, which will also directly affect the effect of subsequent clustering processing. For example, Reference [3] records that existing deep clustering methods are prone to separation of clustering and embedding. There is a problem that clustering cannot adapt to embedding because clustering is not used to induce embedding. In terms of vectorization representation of patent texts, the technical solution in Reference [4] uses the TF-IDF (Term-Frequency-Inverse-Document-Frequency) method for vector representation. However, if there is a lot of patent content, the dimension of the vector space generated by TF-IDF will increase as the length of the patent increases, which will consume a lot of time when applied. Reference [5] proposes to use word2vec for patent vector representation. However, word2vec is a static word embedding model and cannot solve the problem of polysemy in patent texts.

[0004] To address the problem of polysemy in text samples, ELMo (embedding from language models) in reference [6] uses an unsupervised bidirectional language model for pre-training, dynamically adjusting the word embedding according to the context of the word, effectively distinguishing the different semantics of the same word in different contexts. However, ELMo uses a Long-Short Term Memory (LSTM) feature extractor, which has relatively weak feature extraction capabilities. In contrast, BERT (bidirectional encoder representations from transformers) in reference [7] uses a transformer feature extractor and utilizes a multi-head attention mechanism. For the same word, the output of the corresponding 12-layer Transformer is different in different contexts. BERT's integrated feature fusion method obtains more complete representation information of the word and more effectively solves the problem of polysemy.

[0005] In order to achieve better clustering results for text samples, existing technologies use deep clustering methods that combine classic clustering algorithms with deep learning. For example, reference [8] utilizes the temporal memory capability of LSTM and the nonlinear feature extraction capability of autoencoders to perform automatic feature extraction and nonlinear dimensionality reduction, and then uses the k-means clustering algorithm for cluster analysis. However, this sequential clustering method makes clustering adapt to features and depends heavily on the quality of feature extraction. In order to improve this situation, the literature [9] proposed DEC (deep embedding cluster) to learn feature representation and cluster assignment simultaneously using a deep neural network, which can effectively avoid the shortcomings of sequential clustering. The literature

[10] improved upon DEC and proposed the IDEC algorithm. This algorithm performs joint clustering, learns embedded features suitable for clustering, and maintains the local structure by merging an autoencoder. Specifically, IDEC (improved deep embedded clustering) is an improvement on the DEC algorithm. It addresses the problem that in the DEC training stage, discarding the decoding layer will cause the encoding layer to distort the embedding space, thereby reducing the retention of original data features in feature extraction and resulting in poor clustering effect. Therefore, IDEC improves the clustering effect by jointly training the reconstruction loss and the clustering loss to retain as many original data features as possible.

[0006] In summary, the clustering of text samples, especially polysemous texts (e.g., patent texts with specific domain vocabulary, technical terms, synonyms, and antonyms), in existing technologies is highly dependent on the selection of initial centroids. Furthermore, the traditional vectorization methods used cannot effectively address the polysemous word problem in patent texts, making it more difficult to accurately extract complete semantic information. This also makes it easy for the downstream deep clustering process to experience the problem of feature embedding and clustering process separation, affecting the clustering effect of text samples.

[0007] References:

[0008] Literature[1]Ashouri S, Mention AL, Smyrnios K

[0009] Literature [2] QiJ, LeiL, Zheng K, et al. Patent analytic citation-based vsm: Challenges and applications [J]. IEEE Access, 2020, 8: 17464-17476.

[0010] Reference [3] DENG Xiang, YULu. An overview of deep clustering algorithms[J]. Communications Technology, 2021, v.54; No.356(08): 1807-1814. (i.e., Deng Xiang, Yu Lu. An overview of deep clustering algorithms[J]. Communications Technology, 2021, v.54; No.356(08): 1807-1814.)

[0011] Reference [4] Shen Y C, Wang M Y, Yang Y C. Discovering the potential opportunities of scientific advancement and technological innovation: A case study of smart health monitoring technology[J]. Technological Forecasting and Social Change, 2020, 160: 120225.

[0012] Reference [5] Lee C, Jeon D, Ahn J M, et al. Navigating a product landscape for technology opportunity analysis: A word2vec approach using an integrated patent - product database[J]. Technovation, 2020, 96: 102140.

[0013] Reference [6] Gupta H, Patel M. Study of extractiVe text summarizer using the elmo embedding[C] / / 202o Fourth International Conference on I - SMAC (IoT in Social, Mobile, Analytics and ClOud)(I - SMAC). IEEE, 2020: 829 - 834.

[0014] Reference [7] Devlin J, Chang M W, Lee K, et al. BERT: Pre - training of Deep Bidirectional Transformers for Language Understanding[J]. 2018.

[0015] Reference [8] PANG Chanjun, YU Jianming, FENG Changyou, et al. Power load clustering modeling and characterization analysis based on LSTM autoencoder. (i.e., Pang Chuanjun, Yu Jianming, Feng Changyou, et al. Power load clustering modeling and characterization analysis based on LSTM autoencoder [J]. Automation of Electric Power Systems, 2020, 44(23): 57-63.)

[0016] Literature [9] Guo X, Zhu E, Liu X, et al.Deep embedded clustering with dataaugmentation[C] / / Asian conference on machine learning.PMLR, 2018: 550-565.

[0017] Literature

[10] Guo X, Gao L, Liu X, et al. Improved deep embedded clustering with l0cal structure preservation [C] / / Ijcai.2017: 1753-1759. Summary of the Invention

[0018] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a text clustering model training method based on this text clustering method.

[0019] This application achieves the objective of the invention through the following methods:

[0020] According to a first aspect of the present invention, the present invention provides a training method for a text clustering model, the method comprising: S1, obtaining a training set and an initial model, the initial model comprising a pre-trained BERT model, a pre-trained deep coding model, and a clustering module, wherein the training set comprises text data of multiple categories, and the pre-trained deep coding model comprises an encoder, a hidden layer, and a decoder; S2, using the training set obtained in step S1 to iteratively train the initial model obtained in step S1 until convergence, wherein the pre-trained BERT model, the encoder of the pre-trained deep coding model, and the clustering module in the converged initial model constitute the final text clustering model.

[0021] According to some embodiments of the present invention, the clustering module consists of a single hidden layer and a clustering submodule with autoencoder properties.

[0022] According to some embodiments of the present invention, in step S1, the pre-trained BERT model is used to encode text data in the training set into text vectors corresponding to the text data; the encoder in the pre-trained deep coding model is used to perform dimensionality reduction processing on the text vectors output by the pre-trained BERT model to obtain dimensionality-reduced text vectors; the clustering module is used to perform autoencoder clustering on the dimensionality-reduced text vectors output by the pre-trained deep coding model; and the hidden layer and decoder reconstruction are used to reconstruct the dimensionality-reduced text vectors output by the pre-trained deep coding model.

[0023] According to some embodiments of the present invention, in the method, the total loss is calculated and the parameters of the initial model are updated based on the total loss during each iteration of training the initial model:

[0024]

[0025]

[0026] Among them, L DAE-CM This represents the total loss of the initial model, where β represents the reconstruction loss weight parameter of the encoder. Initially, β = 0.5. For the reconstruction loss of DAE, It is the reconstruction loss of the i-th vector in the input text, where N is the number of all input vectors corresponding to the input text in one training iteration, and x i This represents the i-th input vector in a single training iteration. This represents the vector reconstructed from the i-th input vector in a single training iteration. z represents the reconstruction loss of the clustering module CM. i This represents the i-th input vector in the clustering module CM. Let represent the vector reconstructed from the i-th input vector. γ is used to sparsify the parameters of a text clustering model, avoiding the learning of non-salient features in the text. ik This represents the posterior probability, i.e., the probability that the i-th data point comes from the k-th cluster, where K is the final number of clusters generated by text clustering, and k represents the k-th cluster generated by clustering. Dirichlet priors are used to balance the cluster assignments. λ||μ represents the average attractiveness of k clusters, i.e., the average value of each pair of data points in the k clusters. T μ-I k ||1 represents the Lagrange orthogonal constraint, that is, using the Lagrange theorem to solve the orthogonal constraint problem, ensuring the sparsity of the parameters in the formula, where λ represents the Lagrange coefficient, μ T I represents the transpose of the cluster center matrix μ. kThis represents the identity matrix with eigenvalue k, where k is the kth cluster generated by the clustering process.

[0027] According to some embodiments of the present invention, the clustering submodule with autoencoder properties employs a likelihood function with the following rules:

[0028]

[0029] in, The reconstruction loss is denoted by x, where N is the total number of input vectors. i This represents the i-th input vector. Let represent the vector reconstructed from the i-th input vector. Indicates sparsification γ ik To prevent the cluster centroid from shifting, γ ik ||μ represents the posterior probability, i.e., the probability that the i-th data point comes from the k-th cluster. k || 2 These represent the parameters of the sparsity model, effectively preventing cluster centroids from deviating from the data points. It is a term formed by combining and transforming formulas, where γ il With γ ik They have the same meaning, namely, the probability that the i-th data point comes from the l-th cluster, μ. l With μ k They represent the same meaning, namely, the center vector of the l-th cluster. μ represents the center vector of the k-th cluster. k The transpose of , where K is the number of clusters ultimately generated by text clustering. This indicates that Dirichlet priors are used to efficiently balance cluster assignments, where α k Denotes the Dirichlet prior parameters of the k-th cluster class. For γ k Take the logarithm to balance the calculation results, where The average attraction of the k clusters is represented by K, where K represents the final number of clusters generated by text clustering, and k represents the kth cluster generated by the clustering.

[0030] According to some embodiments of the present invention, the clustering module further includes a Gaussian mixture model.

[0031] According to a second aspect of the present invention, the present invention provides a text clustering method, the text clustering method comprising: T1, acquiring text data to be clustered; T2, using the training method as described in the first aspect to obtain a final text clustering model to perform clustering processing on the text data to be clustered obtained in step T1 to obtain clustering results.

[0032] Compared with the prior art, the advantages of the present invention are as follows: The present invention uses the BERT model to initialize the vector of the patent to solve the problem of polysemy in the patent text. It uses the encoder in the deep coding model to directly connect with the clustering model to form a clustering module CM, which is embedded into the deep autoencoder DAE to form DAE-CM. This solves the problem of separation between embedding and clustering in traditional deep clustering methods, and improves the clustering effect of the text clustering model. Attached Figure Description

[0033] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0034] Figure 1 This is a schematic diagram of the network structure of a BERT model;

[0035] Figure 2 A schematic diagram of a network structure for a depth encoder;

[0036] Figure 3 This is a DAE-CM model provided according to an embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of a network structure based on a text clustering model according to an embodiment of the present invention;

[0038] Figure 5 To illustrate the clustering performance of the BERT+DAE-CM clustering model compared with the DAE-CM model, DKM model, and ELMo+DAE-CM model according to embodiments of the present invention, a comparative clustering effect diagram is provided. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0040] According to the background art, existing technologies for clustering text samples, especially polysemous texts, are highly dependent on the choice of initial centroids. Deep clustering also easily suffers from the problem of feature embedding separating from the clustering process, affecting the clustering effect of text samples. Therefore, this invention provides a text clustering method based on a text clustering model. It inputs polysemous text samples into a BERT model for multi-semantic vector transformation to obtain a context-specific text vector training set. Then, it inputs this context-specific text vector training set into the BERT+DAE-+CM text clustering model for clustering processing, obtaining at least one cluster. Thus, this invention can solve the problems of clustering being highly dependent on initial centroids and the separation of feature embedding from the clustering process. According to an embodiment of the present invention, a training method for a text clustering model is provided. The method includes: S1, obtaining a training set and an initial model, wherein the initial model includes a pre-trained BERT model, a pre-trained deep coding model, and a clustering module, wherein the training set includes text data of multiple categories, and the pre-trained deep coding model includes an encoder, a hidden layer, and a decoder; S2, using the training set obtained in step S1 to iteratively train the initial model obtained in step S1 until convergence, wherein the pre-trained BERT model, the encoder of the pre-trained deep coding model, and the clustering module in the converged initial model constitute the final text clustering model. It should be noted that the training set used in this invention is Reuters10k. The following will explain in detail the four aspects of model pre-training, initial model construction and training, text clustering, and model validation.

[0041] I. Model Pre-training

[0042] like Figure 1As shown, the BERT model is a bidirectional encoder based on Transformer. Transformer can capture long-range sequence features by caching previous outputs and using relative positions, leveraging a longer historical record. The BERT model uses a multi-head attention mechanism to obtain context-specific text vector representations. It expands the dimension of the input embedding, then performs multi-head partitioning. For each partitioned head, self-attention is performed with other words in the input to obtain a new vector representation. The vector representations after self-attention of each head are simply concatenated. To prevent gradient vanishing and gradient exploding problems that may occur with deep Transformer, residual connections are used. In the attention mechanism, Q (Query), K (Key), and V (Value) are essentially linear transformations of the input X, multiplied by three weight matrices. The dot product of Q and K represents the similarity between elements of Q and K, finding K similar to the query Q. V represents the features after the linear transformation of the input; multiplying by V yields the weighted features. Query and Key are used to transfer information between tokens, while Value itself is information extracted from the current token.

[0043] This embodiment uses a publicly available training set to pre-train the BERT model until convergence, enabling fine-tuning of the BERT model. For example, this embodiment uses patent text as text data and can call upon a pre-trained BERT model. A pre-trained model refers to a dynamic vector model trained on a large-scale internet corpus. Therefore, fine-tuning is performed using the patent text. For patent text sentences exceeding a set length, truncation is randomly set from the beginning or end of the sentence. Punctuation marks, particles, prepositions, and special symbols that do not contain much actual semantic meaning are removed. Then, word segmentation is performed to divide the patent sentence into single-character sequences, which serve as input for fine-tuning the BERT model. The fine-tuning process involves using the filtered and processed text to fine-tune the transform parameters of the BERT model based on the domain knowledge of the patent-specific text. The resulting initial vector possesses the semantics of the patent context. In this embodiment, when the BERT model converges, if the loss value of the validation set does not decrease after more than 1000 batches, training will automatically terminate early. A brief explanation of the text vector representation of the BERT model follows.

[0044] Given text input data {x1, x2, ..., xn}, dot products are performed with the three learnable weight matrices wq1, wk1, and wv1 to obtain q1q2...qn, k1k2...kn, and v1v2...vn, respectively. Then, the score is calculated using the dot product of qi and ki, where qi is used to find important information within x1, x2...xn. Finally, the score is normalized by dividing by... As shown in equation (1). Finally, the obtained value is multiplied by {v1v2...vn} to obtain a weighted value z1, which is the word vector of the input x1. Similarly, z2 and zn can be calculated. The above operation uses vector operations of the attention mechanism, while the BERT model uses a multi-head attention mechanism, that is, the process in the attention mechanism is performed h times, as shown in equation (2), and then the output z is calculated. i Combining these, we obtain the final output vector Z.

[0045]

[0046] In equation (1): Q is the query vector, V is the vector representing the input features, and Q and K are the feature vectors used to calculate the Attention weights, all of which are obtained from the input features. Attention(Q, K, V) is calculated by multiplying V by the corresponding weights based on the level of attention. Its function is to keep the gradient value stable during training.

[0047] The main formula for multi-head attention is as follows: MultiHead(Q, K, V) = Concat(head1, head2, ..., head...) h (2)

[0048]

[0049] In equation (3): WiQ, WiK, and WiV represent the weight matrices of Q, K, and V, respectively.

[0050] like Figure 2 As shown, the structure of the deep coding model (the input layer and some hidden layers form the encoder, and the output layer and some hidden layers form the decoder; the structure of the deep coding model is well known to those skilled in the art and will not be described in detail here) is as follows: For input data, the deep coding model can pass an encoder to obtain a low-dimensional code, and then pass a decoder to reconstruct the original data. The encoder and decoder are trained together.

[0051] Encoding maps the input to a low-dimensional space. This process occurs between the encoding layer and the hidden layer, and its computation is shown in the formula:

[0052] e(x)=f(W T (x+n) (4)

[0053] In equation (4), f represents the activation function, x represents the input of the autoencoder, e(x) is the hidden layer representation of x, which is the low-dimensional vector representation of the input vector x after effective compression, W is the weight matrix of the input layer and the hidden layer, and n is the bias vector representing the hidden layer.

[0054] The decoding process occurs between the coding layer and the hidden layer. The decoder reconstructs the input of the encoder through a symmetrical network structure, and its calculation process is shown in Equation (5):

[0055]

[0056] In equation (5), W represents the output vector of the reconstruction layer. * is the transpose of W, i.e., the mapping matrix from the hidden layer to the output layer, and c represents the bias vector of the output layer. Deep coding models can be implemented by improving the structure and training strategy of autoencoders to meet the needs of different tasks. Its core function is to learn the deep representation meaning of data, reduce the dimensionality of data, and learn the feature representation of data.

[0057] This invention uses existing public training sets to pre-train deep coding models, making them suitable for encoding text vectors.

[0058] II. Construction and Training of the Initial Model

[0059] The initial model used in this invention consists of the pre-trained BERT model and the deep clustering model DAE-CM mentioned above, as follows: Figure 3 As shown, the deep clustering model DAE-CM consists of an encoder and a clustering module (CM) from a pre-trained deep coding model (DAE). It should be noted that the input to the clustering module (CM) can be any layer from the hidden layers of the pre-trained deep coding model (DAE), and the output can be any of those layers. Figure 4As shown, the text clustering model includes the BERT model and the DAE-CM deep clustering model. First, the BERT model is used to initialize the patent text vector representation. BERT's multi-head attention mechanism assigns different weights to different word vectors to enhance attention to certain important words, incorporating contextual semantic information to achieve different understandings of words in different contexts. Then, the trained deep clustering model DAE-CM is used to further perform feature reduction and clustering processing on the patent text representation embedding. The DAE-CM module, with CM embedded in the deep autoencoder module, uses DAE to explicitly learn feature maps to achieve non-linear clustering. First, the initialization vector X of the BERT model is used as the input to DAE, which encodes it multiple times into Z. CM is a single-hidden autoencoder; its input is not directly X, but Z is used as input and encoded through the Y hidden layer, finally outputting a reconstructed structure. DAE outputs a reconstructed version. This structure enables joint optimization of feature embedding and clustering, dynamically adjusting the model to achieve better results. The CM model and the DAE-CM model will be introduced separately below.

[0060] The CM model fits an isotropic Gaussian mixture model with k Gaussian components and a Dirichlet prior on the dataset. The likelihood function of the CM model for complete data can be expressed as:

[0061]

[0062] In equation (6): γ ik =P(z) i =k|x i The denoted ) represents the probability that the selected data point xi comes from the k-th Gaussian component, i.e., the posterior probability. μ represents the probability of selecting the k-th Gaussian component, i.e., the prior probability. k Let α represent the centroid of the k-th cluster. k These are the Dirichlet distribution parameters of the k-th cluster. This represents the average attraction of k clusters.

[0063] Gaussian Mixture Model (GMM) can be viewed as a combination of K individual Gaussian models. Unlike k-means, which performs hard classification by assigning each data point to a specific cluster, GMM provides a probability of each data point being assigned to each cluster. Essentially, it combines several individual Gaussian models to increase model complexity and generate more complex samples.

[0064] The Expectation Maximization (EM) algorithm can calculate the maximum likelihood estimate of a parameter through iteration when the prior information of the parameter to be estimated is unknown and the observation data is incomplete. This algorithm has been successfully developed and applied to the parameter estimation of GMM.

[0065] This paper proposes an alternative, in which equation (6) can be transformed to produce an expression for the reconstruction loss of the autoencoder in the EM algorithm.

[0066] This invention utilizes the EM algorithm to maximize the likelihood function of a Gaussian model, and then uses this to infer the loss function of the subsequent autoencoder. The proof that the parameters of the EM module are similar to those of the GMM is as follows:

[0067] First, the EM algorithm is used to maximize equation (6), then the prior probability is... This can be expressed as the average attraction of k clusters:

[0068]

[0069] According to equation (7), equation (6) can be expressed as:

[0070]

[0071] In equation (8): According to Dirichlet's prior, the first term represents The entropy is a constant and can therefore be omitted. The second term in equation (8) is obtained by simultaneously adding and subtracting the same variable. While keeping the result of the expression unchanged, we can perform the following transformation:

[0072]

[0073] Equation (8) can be further expressed as:

[0074] The probability is a function of xi, which is the dot product of μ.

[0075] F(x; η)=P(Z=k|x)=γ (11)

[0076]

[0077] Since the reconstruction error of an autoencoder is the square of the mean square error, the first term of equation (10) can be interpreted as the reconstruction loss of the autoencoder. The autoencoder consists of an encoder and a decoder, F and G, respectively.

[0078] To better integrate with the autoencoder, this paper goes a step further by directly optimizing equation (10) using the autoencoder and defining CM as a single hidden layer autoencoder with the following F and G encoding and decoding functions:

[0079] F(X) = softmax(XW) enc +B enc )=Γ(13)

[0080]

[0081] In equation (14), X∈RN*d, Γ=γik∈RN*K, reconstruction The encoder's weights and bias parameters are W. enc ∈RK*d, B enc ∈RK, the decoder's weights and bias parameters are W. dec ∈RK*d, B dec ∈Rd.

[0082] Then the loss function of CM is the negative function of equation (10):

[0083]

[0084] Using an autoencoder to optimize equation (10), minimizing E2 is beneficial for sparsifying Y. ik This makes the goal clearer, and ||μk|| 2 The purpose of this approach is to severely penalize large-value weight vectors in the model, minimizing their weights and effectively preventing overfitting and centroid deviation from data points. The Dirichlet prior in E4 effectively balances cluster allocation: without αk being zero, it encourages optimization of all clusters, thus mitigating the bias of E2 towards scattered clusters. This paper chooses K-means++ to initialize centroids. First, a point is randomly selected as the first cluster center. The distances between all samples and the first cluster center are calculated, and the point with the largest distance in the previous step is selected as the second cluster center. Iteration: The distances from all points to their nearest cluster center are calculated, and the point with the largest distance is selected as the new cluster center.

[0085] The following explains the DAE-CM model. CM is a single-hidden-layer autoencoder. The principle behind GMM restricts CM to a linear decoder, allowing only linear partitioning of the input, which is clearly insufficient for clustering requirements. Therefore, this paper embeds CM into DAE, forming a novel deep clustering model, DAE-CM. An implicit feature mapping is used to non-linearly map the input into the feature space of DAE, where clustering is performed. The deep autoencoder (DAE) explicitly learns the feature mapping, thus achieving non-linear clustering.

[0086] The DAE-CM model can simultaneously optimize both CM and DAE. It utilizes a deep autoencoder to find missing parts of the input in the embedding space, guiding CM to perform better optimization. The nested model also allows CM to influence feature extraction in DAE. Using a deep autoencoder architecture prevents feature mapping degradation during optimization, preserving the generative properties of the model.

[0087] If the reconstruction loss of the deep autoencoder is simply added to the CM, the current gradient descent optimizer

[16] often returns a suboptimal solution, which leads to a deviation in the cluster centroids. In order to help find a better optimization point, this paper makes the assumption that the centroids are orthogonal, as shown in Equation (16):

[0088]

[0089] Although equation (16) only involves the learning of μ in CM, it is located in a nested module and indirectly affects the surrounding DAE. This assumption constraint prompts the model to generate an embedding where the centroids can be orthogonal at the same time (orthogonality means that the magnitudes of the vectors are all 1, but the product of the vectors is 0), and minimizes the loss of CM. Thus, the second and third terms in equation (15) are expressed as the following two equations:

[0090]

[0091]

[0092] The assumption in Equation (16) is satisfied for “ideal” clustering

[17] . For the clustering module in this paper, adding this constraint helps guide the clustering towards the ideal clustering direction, and at the same time simplifies the loss function.

[0093] The Lagrange theorem is used to integrate orthogonal constraints, and the final loss (the loss function for each training iteration) is used to update the β parameter and α in the DAE-CM module. k The parameters can be expressed as follows:

[0094]

[0095] In equation (19), L DAE-CM This represents the total loss of the initial model, where β represents the reconstruction loss weight parameter of the encoder. Initially, β = 0.5. For the reconstruction loss of DAE, It is the reconstruction loss of the i-th vector in the input text, where N is the number of all input vectors corresponding to the input text in one training iteration, and x i This represents the i-th input vector in a single training iteration. This represents the vector reconstructed from the i-th input vector in a single training iteration. Let denot , where is the reconstruction loss of the clustering module CM, and ... Let represent the vector reconstructed from the i-th input vector. γ is used to sparsify the parameters of a text clustering model, avoiding the learning of non-salient features in the text. ik This represents the posterior probability, i.e., the probability that the i-th data point comes from the k-th cluster, where K is the final number of clusters generated by text clustering, and k represents the k-th cluster generated by clustering. Dirichlet priors are used to balance the cluster assignments. λ||μ represents the average attractiveness of k clusters, i.e., the average value of each pair of data points in the k clusters. T μ-I k ||1. Use Lagrange's theorem to solve the orthogonal constraint problem, ensuring the sparsity of the parameters in the equation, where λ represents the Lagrange coefficients, μ T I represents the transpose of the cluster center matrix μ. k This represents the identity matrix with eigenvalue k, where k is the kth cluster generated by the clustering process.

[0096] Finally, the initial model obtained in step S1 is trained multiple times using the training set obtained in step S1 until convergence. The pre-trained BERT model, the encoder in the pre-trained deep coding model, and the clustering module in the converged initial model constitute the final text clustering model. The loss function calculated in each iteration is the final loss function mentioned above (i.e., Equation 19).

[0097] III. Text Clustering

[0098] The text clustering method includes: T1, obtaining text data to be clustered; T2, using the above training method to obtain the final text clustering model to perform clustering processing on the text data to be clustered obtained in step T1 to obtain the clustering results.

[0099] IV. Verification Process

[0100] The experimental dataset (i.e., the training set) used Reuters10k (abbreviated as R10K), consisting of 800,000 news articles, which underwent preprocessing. Twenty Newsgroups (abbreviated as TN) collected approximately 20,000 newsgroup documents, evenly divided into 20 different topic (category) sets. Some newsgroups had very similar topics, while others were completely unrelated. PatentsView, supported by the Office of Senior Economists at the United States Patent and Trademark Office (USPTO), contains patent text data such as titles, abstracts, and claims, and has been used for patent similarity tasks.

[0101] The experimental model was set as follows: DAE-CM is the name of the deep embedding clustering model proposed in this paper, consisting of a nested autoencoder DAE and a clustering module CM, where CM is the name of the clustering module constructed in this paper. Both CM and DAE-CM are implemented using TensorFlow. The deep models use the same fully connected autoencoder d-500-500-2000-p-2000-500-500-d, where d and p are the input and feature space dimensions, respectively, and are activated by leaky ReLU. Each layer of the deep autoencoder is activated by leaky ReLU with a slope of 0.2, except for the last layer of the encoder and decoder. CM and DAE-CM were trained using the Adam optimizer (learning rate = 0.001), and Bayesian optimization was used to optimize the hyperparameters. The Dirichlet prior parameter α in the model is related to the Dirichlet prior parameter α of the k-th cluster class. k The performance of CM and DAE-CM was compared by training them with different batch sizes. CM and its benchmark model were trained for 150 epochs, while DAE-CM and its benchmark deep model were trained for 1000 epochs. The main hyperparameter settings of the models are shown in Table 1 below: Table 1

[0102]

[0103]

[0104] In the experiment, P is the feature space dimension of the fully connected autoencoder, α is the Dirichlet prior parameter of the model, B represents the Batch_Size, i.e. the number of samples selected by the model in one training session, β is the reconstruction loss weight parameter of the DAE, and λ is the Lagrange coefficient.

[0105] The clustering performance of the model is evaluated using four commonly used metrics. Cluster accuracy (ACC) is the most common evaluation metric, simply expressed as the number of correctly clustered samples divided by the total number of samples, i.e., the proportion of correctly predicted samples. A higher ACC value indicates better clustering performance. Adjusted Land coefficient (ARI) removes the influence of random labels on the evaluation results; a higher ARI value indicates better clustering performance. Mutual information represents the degree of interdependence between two random variables. Normalized mutual information (NMI) limits the data to a certain range for easier comparison, thus eliminating the influence of outlier data. The F-measure, combining precision and recall, is a weighted harmonic mean of precision and recall. These metrics range from 0 to 1. For clearer comparisons, the values ​​of ACC, ARI, and NMI are multiplied by 100.

[0106] Comparative experimental setup

[0107] To verify the equivalence of the CM module with the Gaussian mixture model, experiments were conducted on the R10K and TN datasets to compare it with other clustering models and Gaussian models. Then, the proposed DAE-CM was compared with traditional deep clustering models. Finally, ablation experiments were incorporated on the PatentsView dataset to verify the importance of each model and the superiority of the proposed model.

[0108] To verify the equivalence of the proposed CM module with the Gaussian mixture model, a comparative experiment was conducted using benchmark models and the proposed CM. The benchmark models included the k-means algorithm, the fully covariance Gaussian model GMM, and the uniformly mixed weighted isotropic Gaussian model iGMM. Each experiment was repeated 20 times, and the average value was recorded as shown in Table 2. The performance of different benchmark clustering models on the dataset was recorded.

[0109] Table 2

[0110]

[0111]

[0112] Experimental results show that the performance of the CM model is comparable to the aforementioned benchmark models. The CM model is equivalent to the GMM, and some metrics are even higher than some other benchmark models. On the R10K dataset, the CM model's ACC, NMI, and ARI are all higher than those of K-means and GMM. Specifically, the CM model's ACC exceeds that of GMM by 15.4% and is also 1.8% higher than K-means, while being slightly lower than iGMM by 1.7%. Overall, it achieves good results.

[0113] To verify the superiority of the proposed DAE-CM deep clustering model, a comparative experiment was conducted with the traditional AE+KM model (which uses an autoencoder for automatic feature extraction and nonlinear dimensionality reduction, and then uses the k-means clustering algorithm for cluster analysis), the DEC model, and the DKM model. Each experiment was performed 20 times, and the average value was recorded as shown in Table 3. The performance of the deep embedding clustering model on the dataset was recorded.

[0114] Table 3

[0115]

[0116] Experimental results show that DAE-CM achieves the highest ACC, NMI, and ARI indices compared to AE+CM, DEC, and DKM models. This is because the DAE-CM model can jointly optimize embedding and clustering. For traditional DAE+KM clustering, it optimizes the shortcomings of clustering adaptation to embedding; compared to DEC, it better optimizes the dependency problem of initial centroids. Specifically, on the NT dataset, DAE-CM's ACC, NMI, and ARI exceed AE+KM by 18.4%, 17.2%, and 17.7%, respectively. On the R10K dataset, DAE-CM's ARI is 6.5% higher than DEC's, and its ARI is 5.4% higher than DEC's.

[0117] To verify the importance of each module and the superiority of the patent clustering method that integrates BERT and an improved deep autoencoder, this paper conducts ablation comparison experiments on the PatentsView dataset using the proposed BERT+DAE-CM, DKM, DAE-CM, and ELMo+DAE-CM. 4000 patents were selected for the experiments, and different numbers of clusters were set. Too many clusters are meaningless, while too few will result in indistinguishable clusters. There is no standard definition for the number of clusters; by manually setting the initial number of clusters, the optimal number of clusters was found to achieve the best clustering effect. The experimental results are as follows. Figure 5 As shown. Figure 5Experimental results show that DKM has the lowest F-means value, indicating that without the proposed DAE-CM model, the clustering performance becomes very poor. While the DAE-CM model can jointly optimize embedding and clustering, achieving good clustering results, the lack of semantic information in patent text data also reduces the clustering effect. ELMo+DAE-CM adds semantic information to the ELMo model to solve the problem of polysemy in patent technical terms, effectively improving its F-means value. However, the proposed BERT+DAE-CM model, which uses BERT to vectorize patent text, has a stronger feature extraction capability of the transformer in BERT than the LSTM feature extractor in ELMo. Its F-means value reaches the highest at 0.9224 when the number of clusters is 10, which is 0.0651 higher than ELMo+DAE-CM and 0.18 higher than DKM, indicating that its optimal number of clusters is 10, at which point the clustering effect is best.

[0118] In summary, this invention proposes a training method for a text clustering model and a corresponding text clustering method, also known as a patent clustering method integrating BERT and an improved deep autoencoder. First, the BERT model is used to initialize the vectors of patents to address the polysemy problem in patent texts. Then, it is theoretically proven that the objective function of the Gaussian Model (GMM) can be transformed to generate an expression for the loss function of a single hidden layer autoencoder, thus forming the CM (Convergent Clustering Model). Experiments verify its equivalence to the Gaussian model. Furthermore, this paper demonstrates how to embed the CM into a deep autoencoder to form a DAE-CM, jointly optimizing the CM and DAE to solve the separation problem between embedding and clustering in traditional deep clustering methods. Experimental results show that the DAE-CM deep clustering model outperforms existing deep clustering models. On a patent dataset, clustering using the proposed patent clustering model shows that the proposed model achieves better clustering results.

[0119] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0120] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0121] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0122] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for a text clustering model, characterized in that, The method includes: S1. Obtain the training set and the initial model. The initial model includes a pre-trained BERT model, a pre-trained deep coding model, and a clustering module. The training set includes text data of multiple categories, and the pre-trained deep coding model includes an encoder, a hidden layer, and a decoder. S2. Using the training set obtained in step S1, iteratively train the initial model obtained in step S1 until convergence. The converged initial model consists of the pre-trained BERT model, the encoder in the pre-trained deep coding model, and the clustering module, forming a text clustering model until convergence, thus obtaining the final text clustering model. The clustering module consists of a single hidden layer and a clustering sub-module with autoencoder properties, and uses the following likelihood function rules: in, For the reconstruction loss of the deep encoding model, For all the input vectors, This represents the i-th input vector. Let represent the vector reconstructed from the i-th input vector. Indicates sparsity To prevent cluster centroids from shifting, Represents the posterior probability. These represent the parameters of the sparsification model. It is a term formed by combining and transforming formulas, where and They mean the same thing: the probability that the i-th data point comes from the l-th cluster. and They represent the same meaning, namely, the center vector of the l-th cluster. Represents the cluster center vector of the kth cluster. The transpose of , The final number of clusters generated by text clustering. This indicates that Dirichlet priors are used to effectively balance the allocation of clusters. These are the Dirichlet prior parameters of the k-th cluster class. For 𝛾 k Take the logarithm to balance the calculation results. The average attraction of the k clusters is represented by K, where K represents the final number of clusters generated by text clustering, and k represents the kth cluster generated by the clustering.

2. The method according to claim 1, characterized in that, In step S1, the pre-trained BERT model is used to encode text data in the training set into text vectors corresponding to the text data; the encoder in the pre-trained deep coding model is used to reduce the dimensionality of the text vectors output by the pre-trained BERT model to obtain dimensionality-reduced text vectors; the clustering module is used to perform autoencoder clustering on the dimensionality-reduced text vectors output by the pre-trained deep coding model; and the hidden layer and decoder reconstruction are used to reconstruct the dimensionality-reduced text vectors output by the pre-trained deep coding model.

3. The method according to claim 2, characterized in that, In the method described above, the initial model is trained in each iteration by calculating the total loss and updating the parameters of the initial model based on the total loss: in, This represents the total loss of the initial model. The reconstruction loss weight parameters of the encoder are initially... β=0.5, For the reconstruction loss of DAE, It is the corresponding number in the input text. The reconstruction loss of each vector. This represents the total number of input vectors corresponding to the input text in one training iteration. In a single training iteration, the th input vectors, In a single training iteration, the th The reconstructed vector from the input vector. This represents the reconstruction loss of the clustering module CM. This indicates the first clustering node in the input clustering module CM. input vectors, Let represent the vector reconstructed from the i-th input vector. The parameters used to sparsify the text clustering model. This represents the posterior probability, that is, the probability that the i-th data point comes from the k-th cluster. The final number of clusters generated by text clustering. This represents the k-th cluster class generated by clustering. Dirichlet priors are used to balance the cluster assignments. This represents the average attractiveness of the k clusters. Let represent the orthogonal constraint of Lagrange's theorem, the problem, and ensure the sparsity of the parameters in the equation, where λ represents the Lagrange coefficients. Represents the cluster center matrix The transpose of the matrix, This represents the identity matrix with eigenvalue k, where k is the kth cluster generated by the clustering process.

4. The method according to claim 1, characterized in that, The clustering module includes a hidden layer and a Gaussian mixture model.

5. A text clustering method, characterized in that, The text clustering method includes: T1. Obtain the text data to be clustered; T2. Using the training method described in any one of claims 1-4, the final text clustering model is obtained by performing clustering processing on the text data to be clustered obtained in step T1 to obtain the clustering results.

6. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 5.

7. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to perform the steps of the method as described in any one of claims 1 to 5.

Images