Theme modeling and text prediction method combining BTM and BiLSTM models

Abstract

The invention provides a theme modeling and text prediction method combining BTM and BiLSTM models, the BiLSTM and BTM models are fused by using a theme attention model, sequence information learned by the BiLSTM and theme information extracted by the BTM are fused together through an attention mechanism of a theme attention layer, and according to current input information and a task target, theme embedding (at, k, k) is automatically adjusted. According to the attention weight between a BiLSTM hidden state (alpha < t >) and a BiLSTM hidden state (ht), the most relevant information is selected for fusion; the dynamic weight distribution enables the model to consider not only the sequence dependency relationship of words, but also the topic distribution in the text when processing the text. The limitation of a single BTM model in the aspect of understanding the deep meaning and the long-term dependency relationship of the text and the limitation of a single BiLSTM model in the aspect of capturing the theme structure in the text are overcome.

Description

Technical Field

[0001] The present invention belongs to the technical fields of deep learning, natural language processing, data mining, and text analysis, and in particular relates to a topic modeling and text prediction method combining a BTM and a Bi_LSTM model. Background Art

[0002] A topic model is a statistical model used to discover abstract topics in a document collection. It is classified into three categories: unsupervised, semi-supervised, and supervised models, and is mainly used in the field of natural language processing. Topic models can discover potential semantic structures from a large amount of text data to better understand the text content, which is helpful for document classification, clustering, and information retrieval. Compared with semi-supervised and supervised topic models, unsupervised topic models do not require labeled data and have the advantages of flexibility, exploratory nature, and wide applicability, making them very suitable for data exploration and preliminary analysis.

[0003] Traditional unsupervised topic model methods are mainly divided into probabilistic generative models and deterministic models, and the main difference lies in their modeling methods. Probabilistic generative models are based on probability theory and assume that data is generated by a potential random process. Such models usually define a generative process that describes how to generate observational data from potential variables. Typical latent Dirichlet allocation and probabilistic latent semantic analysis are probabilistic generative models. For example, in 2022, a team from Beijing Institute of Technology proposed an anomaly detection method based on a conditional diffusion probability generative model. By preprocessing historical time series data, they constructed an offline conditional diffusion probability generative model and used it for real-time anomaly detection, reducing the number of iterations and achieving better anomaly detection results. Deterministic models do not rely on a probabilistic framework, but describe the relationship between data through direct mathematical functions or rules. Such models are usually analytical and do not involve randomness. For example, non-negative matrix factorization is a technique for dimensionality reduction of text data, which decomposes the document-word matrix into the product of two non-negative matrices. One matrix represents the topic distribution of the document, and the other matrix represents the word distribution of the topic. NMF finds these two matrices through an iterative optimization algorithm. Compared with deterministic models, probabilistic generative models have a good theoretical foundation, can handle complex distributions, and can generate new data instances. When dealing with data with uncertainty, probabilistic generative models can provide a natural way to describe and reason about uncertainty.

[0004] However, both deterministic models and probabilistic generative models are unsupervised learning methods. They all reveal the topic structure in the document by analyzing text data, and ignore other relevant information outside the text, including the document's label, classification, rating, release time, geographic location and other information. For example, in a social media comment analysis task, the rating accompanying the comment release is an important contextual information. High-rated documents tend to have positive emotions, and low-rated documents tend to have negative emotions, which can help the model more accurately identify and analyze the topic of the comment. However, unsupervised learning methods can only analyze comment text information, ignoring relevant information outside the text such as ratings. Although supervised topic models can use label data to improve model accuracy, they require a large amount of annotated data, which may be difficult to meet in practical applications. In addition, due to the lack of interpretability, difficulty in processing long-distance dependencies, challenges in hierarchical feature extraction, and data sparsity faced by deep learning models when processing text data, it is difficult to effectively extract high-level structured information, such as the topic structure of the document, directly from the original text. Therefore, the present invention chooses to combine the unsupervised topic model with the supervised deep model to make up for the shortcomings of a single model. The present invention uses the topic information learned by the unsupervised topic model as prior knowledge to help the supervised deep learning model converge faster and reduce the dependence on a large amount of labeled data. The hybrid model enhances the overall topic modeling effect. At the same time, the distribution of topic words analyzed by the topic model can also improve the text prediction accuracy of the deep learning model.

[0005] Deep learning models can be divided into: 1) Feedforward Neural Networks, which refers to a neural network structure with one-way information transmission. It converts input information into output results by transmitting it layer by layer. This network structure is simple and easy to implement, and is widely used in image recognition, speech recognition and other fields. In terms of text prediction, feedforward neural networks can be used for text classification tasks such as sentiment analysis and topic classification. By converting text data into vector representation and then inputting it into the feedforward neural network, the model can learn the complex features in the text data and output the prediction results. For example, the School of Mathematics and Information of South China Agricultural University and the Guangdong Center for Patent Examination Cooperation of the Patent Office of the State Intellectual Property Office proposed a Chinese patent keyword extraction that integrates LSTM and logistic regression in 2022. This invention constructs a new method based on LSTM and logistic regression models to extract keywords from Chinese patents. Experimental results show that the extraction accuracy of this method for all keywords, low-frequency keywords, long-tail keywords, and low-frequency long-tail keywords is 5%, 24%, 11% and 26% higher than other methods, respectively. 2) Generative Models, refers to a class of probabilistic models that can generate new data instances. Unlike discriminative models, generative models can not only classify data, but also generate new data. Among them, the most famous generative models are variational autoencoders (VAE) and generative adversarial networks (GAN). In terms of text prediction, generative models can be used for text generation tasks such as automatic writing and poetry generation. By learning a large amount of text data, generative models can learn the potential distribution of text and predict the next content based on a given part of the text. However, compared with feedforward neural networks, generative models have some shortcomings in text prediction. Generative models need to learn the high-dimensional potential distribution of data, which requires more data and computing resources. Therefore, its training process is usually more complex and time-consuming. Secondly, generative models may face the problem of mode collapse when generating text, resulting in a lack of diversity and creativity in the generated text. In addition, generative models may encounter difficulties when processing long text sequences. Since generative models need to keep modeling the distribution of the entire sequence, this is very challenging computationally. 3) Attention Mechanisms refers to a mechanism that allows a model to dynamically focus on important parts based on contextual information. Its core idea is that for each element in the input sequence, the model assigns a weight to it according to the current context to indicate its importance in the current task. In this way, the model can handle input sequences of different lengths more flexibly and improve the generalization ability of the model. In terms of text prediction, the attention mechanism can be used for machine translation tasks.By dynamically focusing on important parts in the source and target languages, the accuracy of translation is improved. However, compared with feedforward neural networks, the attention mechanism also has obvious disadvantages in text prediction. First, the introduction of the attention mechanism increases the computational burden of the model. It needs to calculate and update the attention weights at each time step, which is particularly time-consuming for long sequences. Secondly, although the attention mechanism improves the model's ability to capture contextual information, it also makes the model more complex, resulting in the model's interpretability not being as intuitive as that of the feedforward neural network. In addition, the attention mechanism may make the model more sensitive to noise in the training data, and sometimes cause the model to pay too much attention to unimportant information when predicting, thereby affecting the accuracy of the prediction. In summary, compared with feedforward neural networks, the shortcomings of the generative model in text prediction are mainly reflected in the difficulty of training, the limited diversity and creativity of the generated text, and the difficulty in processing long text sequences. The shortcomings of the attention mechanism in text prediction include higher computational complexity, lower model interpretability, and sensitivity to data noise.

[0006] Bidirectional long short-term memory network (Bidirectional Long Short-Term Memory, Bi_LSTM) is a special structure of feedforward neural network, which plays an important role in processing tasks that require in-depth understanding of context and long-term dependencies through unique bidirectional structure and powerful sequence modeling ability. However, Bi_LSTM is insufficient in its ability to focus on important information in sequence tasks. In order to solve this problem, the present invention designs a topic attention layer (Attention Layers), which integrates unsupervised BTM (Biterm Topic Model) and supervised Bi_LSTM to complement each other. Specifically, the topic attention layer fuses the sequence information learned by Bi_LSTM and the topic information extracted by BTM through the attention mechanism. This fusion enables the model to not only consider the sequence dependency of words when processing text, but also consider the topic distribution in the text. At the same time, the attention mechanism allows the model to dynamically adjust the weight according to different tasks and contexts, which means that the model can automatically select the most relevant information in different situations, thereby improving the generalization ability and performance of the model. In this way, the topic attention layer enables Bi_LSTM and BTM to work in a complementary manner. This not only enhances the model’s ability to model text sequences, but also improves its ability to identify text topics. In summary, the final hidden state of the document includes both sequential information and topic information, which improves the overall topic modeling effect of the model and the accuracy of text prediction.

[0007] In the field of text data analysis, traditional unsupervised topic modeling methods mainly rely on the text content itself, while ignoring data outside the text, such as labels, classifications, and other information. This limits the accuracy and generalization ability of the model in practical applications, and may not accurately identify the topics and variables in the text, resulting in an impact on subsequent empirical analysis. In addition, the topics generated by these models may be too abstract and difficult to interpret, making it difficult for users to understand the specific meaning of each topic. Although supervised topic models can use labeled data to improve modeling effects, they rely on a large amount of annotated data, which may be difficult to meet in practical applications. Excessive learning of annotated data may lead to overfitting of the model and limited generalization ability.

[0008] On the other hand, although traditional supervised deep learning models are able to utilize label and word vector data in predictive analysis, they have certain limitations in modeling the latent topic structure of text data. First, these models still face difficulties in capturing long-range dependencies and it is difficult to effectively integrate information about related words that appear at a distance in the document, resulting in insufficient understanding of the overall topic of the document. Second, they usually process the text as a bag-of-words model, ignoring higher-level structures in the text, such as paragraphs, sentences, or topics, resulting in the loss of important semantic information. Third, these models mainly focus on local sequence dependencies and have difficulty capturing topic consistency across multiple sentences. Summary of the invention

[0009] To solve the above problems, the present invention provides a topic modeling and text prediction method combining BTM and Bi_LSTM models, which can not only deeply understand the text context, but also effectively utilize topic distribution data, greatly improving the accuracy of topic modeling and the precision of text prediction.

[0010] A topic modeling and text prediction method combining BTM and Bi_LSTM models, comprising the following steps:

[0011] S1: Use the BTM model to obtain the probability distribution of each topic word contained in the document D to be processed, and use the Word2Vec model to convert the probability distribution of each topic word into a topic word embedding vector, and form a topic word embedding matrix V from the topic word embedding vector;

[0012] S2: Use the Word2Vec model to directly convert the document D to be processed into a word embedding sequence, and form a word embedding matrix J from the word embedding sequence;

[0013] S3: Use the Bi_LSTM model to encode each word embedding sequence in the word embedding matrix J, and obtain the encoded document representation H(D) = [h 1 ,h 2 ,...,h t ,…,h M], where h t is the document hidden state at the tth time step in the Bi_LSTM model, where h t =[h t (e 1 ),h t (e 2 ),…,h t (e n ),…,h t (e N )],h t (e n ) is the nth word embedding sequence e in the word embedding matrix J n The encoding representation at the tth time step, M is the total number of time steps;

[0014] S4: Based on the attention mechanism, the attention weight of each topic in the topic word embedding matrix V is obtained at each time step:

[0015] a t,k =softmax([tanh(W t h t +b t )] T ·v k )

[0016] Among them, a t,k is the attention weight of the kth topic at the tth time step, v k is the topic word embedding vector of the kth topic, W t h t The corresponding weight, b t h t Corresponding bias; k = 1, 2, ..., K, K is the total number of topics;

[0017] S5: a corresponding to all topics at the tth time step t,k The sum of the values ​​is taken as the topic relevance α of the document D to be processed at the tth time step t ;

[0018] S6: The topic correlation α of all time steps t With h t Multiply and sum the corresponding data to obtain the final document representation S of the document D to be processed;

[0019] S7: Input the final document representation S into the trained MLP model for text prediction or into the trained BTM model for topic modeling.

[0020] Furthermore, in step S1, the method of using the BTM model to obtain the probability distribution of each subject word contained in the document D to be processed is specifically as follows:

[0021] S11: extract all non-repeated words from the document D to be processed as word pairs, and all word pairs constitute the training data set B of the BTM model;

[0022] S12: Obtain the word probability distribution φ of each topic in the document to be processed D from the Dirichlet distribution z and the probability distribution θ for each topic; where φ z Represents the probability of each word appearing in topic z;

[0023] S13: When processing each word pair biterm(w i ,w j ), firstly, from the probability distribution θ of the topic, the biterm (w i ,w j ) samples a topic z, denoted as z~Multi(θ); then, from the word probability distribution φ z We sampled w i and w j , denoted as w i ,w j ~Mulit(φ z ), where w i and w j Represent the two words in the word pair biterm, Mulit is the abbreviation of multinomial distribution function, each biterm (w i ,w j The calculation method of the joint probability P(b) of ) is as follows:

[0024]

[0025] Among them, θ z represents the probability of topic z appearing in the entire corpus, φ i|z For word w i The conditional probability of being in topic z, φ j|z For word w j The conditional probability in topic z; b is the word pair biterm (w i ,w j )

[0026] S14: The likelihood function P(B) of the training data set B is obtained by summing all word pairs biterm and all possible topics:

[0027]

[0028] S15: Obtain the conditional probability P(z|z -b ,B,α,β) are as follows:

[0029]

[0030] Among them, z -b Indicates that except for the word pair biterm(w i ,w j ) to the subject of the biterm, n z For a biterm(w i ,w j ) is assigned to topic z, n w|z is the number of times any word w in the corpus is assigned to topic z, is biterm(w i ,w j ) i The number of times it is assigned to topic z, is biterm(w i ,w j ) j The number of times it is assigned to topic z; Q represents the vocabulary size; α and β are the probability distribution θ and word probability distribution φ respectively z The Dirichlet prior of

[0031] S16: Get the distribution of any topic word vector as follows:

[0032]

[0033] Among them, φ w|z is the conditional probability of any word w in the corpus in topic z.

[0034] Furthermore, in step S1, the Word2Vec model is used to convert the probability distribution of each topic word into a topic word embedding vector v(T topic ) Specifically:

[0035]

[0036] Among them, L is the theme T topic The total number of words in v(w l ) is the word w l The vector representation output after inputting the Word2Vec model, p(w l |T topic ) as the theme T topic The word w in l The corresponding weight.

[0037] Furthermore, the document indicates that H(D) = [h 1 ,h 2 ,...,h t ,…,h M] is obtained by:

[0038] Embed any word in the word embedding matrix J into the sequence e n Substitute the following Bi_LSTM model for M iterations. The output after each iteration is the encoding result h t (e n ):

[0039] f t =σ(W f ·[h t-1 ,e n ]+b f )

[0040] i t =σ(W i ·[h t-1 ,e n ]+b i )

[0041] o t =σ(W o ·[h t-1 ,e n ]+b o )

[0042]

[0043] Among them, f t is the forget gate output of the tth time step, i t is the input gate output at the tth time step, o t is the output gate output at the tth time step, c t is the cell state update at the tth time step, W f is the weight corresponding to the forget gate, b f is the bias vector corresponding to the forget gate, W i is the weight corresponding to the input gate, b i is the bias vector corresponding to the input gate, W o is the weight corresponding to the output gate, b o is the bias vector corresponding to the output gate, W c Update the corresponding weight for the cell state, b c Update the corresponding bias vector for the cell state, c t-1 is the cell state update at the t-1th time step, h t-1 is the word embedding sequence e n The hidden state at the t-1th time step, h t is the word embedding sequence e n The hidden state at the tth time step, σ is the sigmoid function, tanh is the hyperbolic tangent function, and ° represents element-wise multiplication;

[0044] Embed each word into the sequence e n Encoding result h t (e n ) are combined and concatenated to obtain the document representation H(D) = [h 1 ,h 2 ,...,h t ,…,h M ].

[0045] Furthermore, the input layer of the trained MLP model receives the final document representation S, where the output layer of the MLP model uses a sigmoid function to convert the output into a probability distribution if the prediction task is binary or multi-classification; if the prediction task is regression, the output layer of the MLP model uses a linear activation function to output the predicted value.

[0046] Furthermore, the input layer of the trained BTM model receives the final document representation S, and the output layer outputs the result of topic modeling.

[0047] Beneficial effects:

[0048] 1. The present invention provides a topic modeling and text prediction method combining BTM and Bi_LSTM models, which integrates Bi_LSTM and BTM models by using the topic attention model. The sequence information learned by Bi_LSTM and the topic information extracted by BTM are integrated through the attention mechanism of the topic attention layer. The topic embedding (a) is automatically adjusted according to the current input information and task objectives. t,k , α t ) and Bi_LSTM hidden state (h t ) between attention weights, thereby selecting the most relevant information for fusion; this dynamic weight allocation enables the model to take into account not only the sequence dependencies of words but also the topic distribution in the text when processing text, overcoming the limitations of a single BTM model in understanding the deep meaning and long-term dependencies of the text and the limitations of a single Bi_LSTM model in capturing the topic structure in the text; it can be seen that the present invention can learn the potential topic distribution from the text, which helps to better understand the overall content and semantics of the document, enhance the ability to model the potential topic structure of text data, and capture more complex and subtle topic structures in the text; whether it is necessary to perform document classification, sentiment analysis, information retrieval, or to perform topic mining and trend analysis on large-scale text data, the present invention can provide more accurate and comprehensive results.

[0049] 2. The present invention provides a topic modeling and text prediction method combining BTM and Bi_LSTM models, and generates a topic word embedding vector using the topic word distribution generated by BTM. The word distribution under each topic is regarded as a vector in a high-dimensional space, wherein each dimension of the vector corresponds to a word in the vocabulary, and the value of the dimension corresponds to the weight of the word in the topic. In this way, a word embedding vector is generated for each topic, and the topic word embedding vector provides a low-dimensional, dense vector representation for text data, which not only helps to better understand and analyze text data, but also improves the processing efficiency of text data.

[0050] 3. The present invention provides a topic modeling and text prediction method combining BTM and Bi_LSTM models, which does not require a large amount of manual annotation data and improves the efficiency of text analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A flowchart of a topic modeling and text prediction method combining BTM and Bi_LSTM models provided by the present invention;

[0052] Figure 2 A schematic diagram of a topic word embedding obtained by using the topic word distribution generated by BTM provided by the present invention;

[0053] Figure 3 The present invention provides z Schematic diagram of the internal structure;

[0054] Figure 4 A schematic diagram of obtaining a hidden layer using the Bi_LSTM model provided by the present invention;

[0055] Figure 5 A framework diagram of the topic modeling and text prediction technology combining the BTM and Bi_LSTM models provided by the present invention;

[0056] Figure 6 This is a specific example of the topic modeling and text prediction technology framework diagram combining the BTM and Bi_LSTM models provided by the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0058] In order to solve the problems existing when the two models BTM and Bi_LSTM are used alone, and to improve the accuracy of topic modeling and the precision of text prediction, the present invention proposes a new hybrid model. The model combines unsupervised BTM and supervised Bi_LSTM, and closely connects the two through the topic attention layer. This hybrid model can not only use BTM to automatically learn potential topic information from text, but also use Bi_LSTM to further improve the prediction ability of the model by using topic distribution data. The topic attention layer can allow the model to dynamically adjust weights according to different tasks and contexts, and automatically select the most relevant information, thereby improving the generalization ability and performance of the model.

[0059] Specifically, the design concept of the present invention is to use unsupervised BTM to process large-scale text data and mine potential topic distributions from it. These topic distributions reflect the main topics and concepts in the text data, providing important prior knowledge for subsequent deep learning models. Then, the supervised Bi_LSTM model accepts these topic distributions as additional input features and learns together with the text sequence to better capture the deep meaning and long-term dependencies of the text. The bidirectional structure of Bi_LSTM enables it to take into account the previous and next information in the text at the same time, improving the ability to understand the context.

[0060] However, using the Bi_LSTM model alone may not be able to fully focus on the key information in the text, especially in long texts. In order to solve this problem, the present invention introduces a topic attention layer. The attention layer helps the model automatically identify the most important information for the current task from a large amount of text information by dynamically allocating attention scores and weights, and performs more accurate modeling and prediction. In this model, the attention layer calculates the word embedding sequence of the document generated by Bi_LSTM and the topic embedding generated by BTM+Word2Vec to obtain the "importance" of each word to the current topic, and assigns different weights to select the most important words for subsequent processing, thereby more accurately capturing the topic information of the document and generating a higher quality document representation. The addition of the attention layer enables the model to not only take into account the sequence dependency of words when processing text, but also the topic distribution in the text. This complementary mechanism makes the model more accurate in understanding and predicting text.

[0061] like Figure 1 As shown, a topic modeling and text prediction method combining BTM and Bi_LSTM models includes the following steps:

[0062] S1: Use the BTM model to obtain the probability distribution of each topic word contained in the document D to be processed, and use the Word2Vec model to convert the probability distributions of each topic word into topic word embedding vectors respectively, and form a topic word embedding matrix V from the topic word embedding vectors;

[0063] Among them, the method of using the BTM model to obtain the probability distribution of each topic word contained in the document D to be processed is specifically as follows:

[0064] S11: Extract all non-repeated words from the document D to be processed as word pairs, and all word pairs form the training data set B of the BTM model;

[0065] It should be noted that when extracting word pairs, stop words can be ignored. For example, from the text "Consumed physical strength", ignoring the stop word "了", the extracted biterm (word pair) is "Consumed, physical strength"; finally, combine the biterms extracted from all short texts to form the training data set B of the BTM. Biterm represents an unordered word pair that appears simultaneously in a short context (i.e., an instance of the word co-occurrence pattern).

[0066] S12: Obtain the word probability distribution φ of each topic in the document D to be processed from the Dirichlet distribution z ~Dir(β) and the probability distribution θ of each topic ~Dir(α); where, φ z represents the probability of each word appearing in topic z, and α and β are the Dirichlet priors of θ and φ z respectively;

[0067] For example, φ 1 (Topic 1)=[0.1, 0.2, 0.3, 0.4], which means that under Topic 1, the probability of word 1 is 10%, the probability of word 2 is 20%, the probability of word 3 is 30%, and the probability of word 4 is 40%.

[0068] S13: When processing each word pair biterm(w i , w j ) in the training data set B, first sample a topic z for this biterm(w i , w j ) from the probability distribution θ of the topic, denoted as z ~ Multi(θ); then, sample w z and w i respectively from the word probability distribution φ j , denoted as w i , w j ~ Mulit(φ z ), where, w i and w jRepresent the two words in the word pair biterm, Mulit is the abbreviation of multinomial distribution function, which means the word distribution φ from a certain topic z z Sampling words in, each biterm(w i ,w j The joint probability P(b) of the topic distribution, word distribution and the order of words in the biterm can be calculated as follows:

[0069]

[0070] Among them, θ z represents the probability of topic z appearing in the entire corpus, φ i|z For word w i The conditional probability of being in topic z, φ j|z For word w j The conditional probability in topic z; b is the word pair biterm (w i ,w j )

[0071] S14: The likelihood function P(B) of the training data set B is obtained by summing all word pairs biterm and all possible topics:

[0072]

[0073] S15: Inference using Gibbs sampling. To perform Gibbs sampling, we first randomly select the initial state of the Markov chain, and then calculate the conditional distribution of each biterm b, that is, b = (w i ,w j )’s conditional distribution P(z|z -b ,B,α,β), where z -b represents the topic assignment of all biterms except b, where b is the global biterm set. By applying the chain rule to the joint probability of the entire data, we can obtain the conditional probability p(z|z -b ,B,α,β) are as follows:

[0074]

[0075] Among them, z -b Indicates that except for the word pair biterm(w i ,w j ) to the subject of the biterm, n z For a biterm(w i ,w j ) is assigned to topic z, n w|z is the number of times any word w in the corpus is assigned to topic z, is biterm(w i ,w j ) i The number of times it is assigned to topic z, is biterm(w i ,w j ) j The number of times it is assigned to topic z; Q represents the vocabulary size; α and β are the probability distribution θ and word probability distribution φ respectively z The Dirichlet prior of

[0076] S16: Obtain the probability distribution of any subject word as follows:

[0077]

[0078] Among them, φ w|z is the conditional probability of any word w in the corpus in topic z.

[0079] It should be noted that the topic word distribution data is read from the file to be processed. These data are usually expressed as the weight distribution of each word under each topic. These weights reflect the relevance of the word to the topic. Convert the read text line into a list of floating-point numbers, each list representing the word distribution of a topic. Convert the index of the topic word distribution into a string format. For example, when processing text data to extract the topic word distribution, first read each line of data in the file. Each line of data represents the word weight distribution under a specific topic. The line "Topic 1: 0.5 0.3" indicates that under topic 1, the weight of the first word is 0.5 and the weight of the second word is 0.3. In order to avoid the problem that the slight differences between the weights cannot be captured and the large range and higher precision values ​​cannot be represented, we create a word distribution list for each topic by converting each weight into a floating-point number and storing them in a list. Finally, these lists are organized in a dictionary, where the keys of the dictionary are the topic names, such as "Topic 1", "Topic 2", and "Topic 3", and the values ​​are the corresponding word weight distribution lists.

[0080] Furthermore, for each topic, the weighted average of its word distribution is calculated as the topic embedding, and the Word2Vec model is used to convert the probability distribution of each topic word into a topic word embedding vector v(T topic ) Specifically:

[0081]

[0082] Among them, L is the theme T topic The total number of words in v(w l ) is the word w l The vector representation output after inputting the Word2Vec model, p(wl |T topic ) as the theme T topic The word w in l The corresponding weight.

[0083] It should be noted that the present invention can store the calculated topic embedding vector in an array and return the array.

[0084] In summary, the Word2Vec model learns the word probability distribution by optimizing the objective function, and finally generates an L-dimensional topic word embedding matrix V. Each of its row vectors corresponds to the embedding representation of a word in the vocabulary, which can reflect the topic relevance of the word. For example, in the topic word embedding matrix V, the embedding vectors of words related to the stimulus may be close to each other, while the embedding vectors of words unrelated to the stimulus are relatively distant. In this way, the topic word embedding matrix V can be applied to various downstream tasks, such as text classification, topic modeling, information retrieval, etc., to help the model understand and process text data more effectively.

[0085] S2: Use the Word2Vec model to directly convert the document D to be processed into a word embedding sequence, and form a word embedding matrix J from the word embedding sequence;

[0086] S3: Use the Bi_LSTM model to encode each word embedding sequence in the word embedding matrix J, and obtain the encoded document representation H(D) = [h 1 ,h 2 ,...,h t ,…,h M ], where h t is the document hidden state at the tth time step in the Bi_LSTM model, where h t =[h t (e 1 ),h t (e 2 ),…,h t (e n ),…,h t (e N )],h t (e n ) is the nth word embedding sequence e in the word embedding matrix J n The encoding representation at the tth time step, M is the total number of time steps;

[0087] Specifically, the document indicates that H(D) = [h 1 ,h 2 ,...,h t ,…,h M ] is obtained by:

[0088] Embed any word in the word embedding matrix J into the sequence e n Substitute the following Bi_LSTM model for M iterations. The output after each iteration is the encoding result h t (e n ):

[0089] f t =σ(W f ·[h t-1 ,e n ]+b f )

[0090] i t =σ(W i ·[h t-1 ,e n ]+b i )

[0091] o t =σ(W o ·[h t-1 ,e n ]+b o )

[0092]

[0093] Among them, f t is the forget gate output of the tth time step, i t is the input gate output at the tth time step, o t is the output gate output at the tth time step, c t is the cell state update at the tth time step, W f is the weight corresponding to the forget gate, b f is the bias vector corresponding to the forget gate, W i is the weight corresponding to the input gate, b i is the bias vector corresponding to the input gate, W o is the weight corresponding to the output gate, b o is the bias vector corresponding to the output gate, W c Update the corresponding weight for the cell state, b c Update the corresponding bias vector for the cell state, c t-1 is the cell state update at the t-1th time step, h t-1 is the word embedding sequence e n The hidden state at the t-1th time step, h t is the word embedding sequence e n In the hidden state of the tth time step, σ is the sigmoid function, tanh is the hyperbolic tangent function, tanh(c t ) is the cell state update c tThe value after passing through the hyperbolic tangent function, ° represents element-by-element multiplication (Hadamard product);

[0094] Embed each word into the sequence e n Encoding result h t (e n ) are combined and concatenated to obtain the document representation H(D) = [h 1 ,h 2 ,...,h t ,…,h M ].

[0095] It can be seen that LSTM is a special RNN unit that can effectively capture long-term dependencies. The present invention uses the Bi_LSTM model to learn the contextual relationship between word sequences, thereby capturing the semantic information of the document, encoding the sequence information of the document, and obtaining the hidden layer.

[0096] S4: Based on the attention mechanism, the attention weight of each topic in the topic word embedding matrix V is obtained at each time step:

[0097] a t,k =softmax([tanh(W t h t +b t )] T ·v k )

[0098] Among them, a t,k is the attention weight of the kth topic at the tth time step, decomposing the L-dimensional topic word embedding matrix V into K topic word embedding vectors v 1 ,v 2 ,…,v K , each topic word embedding vector corresponds to a topic, v k is the topic word embedding vector of the kth topic, W t h t The corresponding weight, b t h t Corresponding bias; k = 1, 2, ..., K, K is the total number of topics;

[0099] It should be noted that the present invention designs a topic attention layer, calculates the word embedding sequence of the document generated by Bi_LSTM and the topic embedding generated by BTM+Word2Vec, obtains the "importance" of each word to the current topic, assigns different weights, and selects the most important words for subsequent processing, so that the final document representation contains sequence information and topic information.

[0100] S5: a corresponding to all topics at the tth time step t,kThe sum of the values ​​is taken as the topic relevance α of the document D to be processed at the tth time step t :

[0101]

[0102] It should be noted that α t,k is the attention weight of the LSTM model on topic k at time step t. This formula is to convert the attention weights α of K topics into t,k Perform weighted averaging to obtain the topic mixing ratio α t , α t It is the weighted average of the document D to be processed at time step t for all topics, which reflects the relevance of the document D to be processed at time step t to all topics.

[0103] S6: The topic correlation α of all time steps t With h t Multiply and sum the corresponding values ​​to get the final document representation S of the document D to be processed:

[0104]

[0105] S7: Input the final document representation S into the trained MLP model for text prediction or into the trained BTM model for topic modeling.

[0106] It should be noted that if it is a topic modeling task, the input layer of the MLP model will receive the document S, and the output layer will output the corresponding results according to the prediction task type (classification or regression).

[0107] ① If the prediction task is binary classification, the output layer uses the sigmoid function to convert the output into a probability distribution, as shown below:

[0108]

[0109] in, is the probability of predicting the positive class, σ is the sigmoid function, W o1 and b o1 are trainable weight parameters.

[0110] ②If the prediction task is multi-classification, the output layer uses the softmax function to convert the output into a probability distribution.

[0111]

[0112] in, is the probability of predicting the jth class, W o2 and b o2 is the weight of classifier j, and K is the total number of classifications.

[0113] ③If the prediction task is regression, that is, predicting the value of the auxiliary label, the output layer uses a linear activation function to output the predicted value, as shown below.

[0114] o=W o3 s+b o3

[0115] Where o is the predicted continuous value, W o3 and b o3 are trainable weight parameters.

[0116] Second, if the task is topic modeling, the input layer of the BTM model will receive the document S, and the output layer will output the results of topic modeling.

[0117] Taking the processing of the comment text of "Let's cheer together" as an example, the topic modeling and text prediction method combining the BTM and Bi_LSTM models provided by the present invention is described in detail.

[0118] First, we use the topic word distribution generated by BTM to get a topic word embedding, such as Figure 2 As shown. Extract all non-repeating words from the document as word pairs, for example, extract word pairs [everyone, together] from "Everyone cheers together". For each word pair biterm, extract a topic z from K topics (set by the model) according to the global topic distribution θ. According to the selected topic z and the word distribution φ under the topic z , respectively from φ z Extract two words w from i and w j , forming a word pair set biterm(w i ,w j ). By counting the number of occurrences of each word in each topic and calculating the probability distribution of the word in the topic based on the Dirichlet prior, φ is updated. z ,like Figure 3 As shown in Figure 1, we update θ by counting the number of Biterms under each topic and calculating the probability distribution of each topic based on the Dirichlet prior.

[0119] Second, use Word2Vec to distribute the topic words φ z Converted into topic word embedding vector.

[0120] Third, use Bi_LSTM to calculate the hidden states of all time steps and perform matrix concatenation to obtain the final hidden state H(D), as shown in Figure 4 shown.

[0121] Fourth, the topic attention layer is used to calculate the word embedding sequence of the document generated by Bi_LSTM and the topic embedding generated by BTM+Word2Vec to obtain the final document representation S.

[0122] Fifth, use S as input for a variety of different tasks (classification or regression).

[0123] Combining the above steps, the overall framework diagram of the model proposed in this paper and the specific case description are formed, as shown below: Figure 5 and Figure 6 shown.

[0124] In summary, the present invention provides a topic modeling and text prediction method combining BTM and Bi_LSTM models, which uses the attention mechanism of the topic attention layer to fuse the sequence information learned by Bi_LSTM and the topic information extracted by BTM, so that the model can dynamically adjust the topic embedding according to different tasks and contexts (a t,i , α t ) and Bi_LSTM hidden state (h t ) and automatically selects the most relevant information; that is, the present invention can not only deeply understand the text context, but also effectively utilize the topic distribution data, and at the same time has a hybrid topic modeling and text prediction method with good generalization ability and interpretability. This hybrid model not only enhances the modeling ability of the text sequence, but also improves the recognition ability of the text topic; in practical applications, this hybrid model has higher flexibility and robustness, and can better meet the needs of different scenarios; whether it is necessary to perform document classification, sentiment analysis, information retrieval, or topic mining and trend analysis on large-scale text data, the present invention can provide more accurate and comprehensive results.

[0125] At the same time, the present invention also generates a topic word embedding vector using the topic word probability distribution generated by BTM, and regards the word distribution under each topic as a vector in a high-dimensional space, where each dimension of the vector corresponds to a word in the vocabulary, and the value of the dimension corresponds to the weight of the word in the topic. In this way, a word embedding vector is generated for each topic. The topic word embedding vector provides a low-dimensional, dense vector representation for text data, which not only helps to better understand and analyze text data, but also improves the processing efficiency of text data.

[0126] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may certainly make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A topic modeling and text prediction method combining BTM and Bi_LSTM models, characterized in that: The following steps are involved: S1: Use the BTM model to obtain the probability distribution of each topic word contained in the document D to be processed, and use the Word2Vec model to convert the probability distribution of each topic word into a topic word embedding vector, and form a topic word embedding matrix V from the topic word embedding vector; S2: Use the Word2Vec model to directly convert the document D to be processed into a word embedding sequence, and form a word embedding matrix J from the word embedding sequence; S3: Use the Bi_LSTM model to encode each word embedding sequence in the word embedding matrix J, and obtain the encoded document representation H(D) = [h1,h2,...,h t ,…,h M ], where h t is the document hidden state at the tth time step in the Bi_LSTM model, where h t =[h t (e1),h t (e2),…,h t (e n ),…,h t (e N )],h t (e n ) is the nth word embedding sequence e in the word embedding matrix J n The encoding representation at the tth time step, M is the total number of time steps; S4: Based on the attention mechanism, the attention weight of each topic in the topic word embedding matrix V is obtained at each time step: a t,k =softmax([tanh(W t h t +b t )] T ·v k ) Among them, a t,k is the attention weight of the kth topic at the tth time step, v k is the topic word embedding vector of the kth topic, W t h t The corresponding weight, b t h t Corresponding bias; k = 1, 2, ..., K, K is the total number of topics; S5: a corresponding to all topics at the tth time step t,k The sum of the values ​​is taken as the topic relevance α of the document D to be processed at the tth time step t ; S6: The topic correlation α of all time steps t With h t Multiply and sum the corresponding data to obtain the final document representation S of the document D to be processed; S7: Input the final document representation S into the trained MLP model for text prediction or into the trained BTM model for topic modeling.

2. A topic modeling and text prediction method combining BTM and Bi_LSTM models as described in claim 1, characterized in that: The method of using the BTM model to obtain the probability distribution of each subject word contained in the document D to be processed in step S1 is specifically as follows: S11: extract all non-repeated words from the document D to be processed as word pairs, and all word pairs constitute the training data set B of the BTM model; S12: Obtain the word probability distribution φ of each topic in the document to be processed D from the Dirichlet distribution z and the probability distribution θ for each topic; where φ z Represents the probability of each word appearing in topic z; S13: When processing each word pair biterm(w i ,w j ), firstly, from the probability distribution θ of the topic, the biterm (w i ,w j ) samples a topic z, denoted as z~Multi(θ); then, from the word probability distribution φ z We sampled w i and w j , denoted as w i ,w j ~Mulit(φ z ), where w i and w j Represent the two words in the word pair biterm, Mulit is the abbreviation of multinomial distribution function, each biterm (w i ,w j The calculation method of the joint probability P(b) of ) is as follows: Among them, θ z represents the probability of topic z appearing in the entire corpus, φ i|z For word w i The conditional probability of being in topic z, φ j|z For word w j The conditional probability in topic z; b is the word pair biterm (w i ,w j ) S14: The likelihood function P(B) of the training data set B is obtained by summing all word pairs biterm and all possible topics: S15: Obtain the conditional probability P(z|z -b ,B,α,β) are as follows: Among them, z -b Indicates that except for the word pair biterm(w i ,w j ) to the subject of the biterm, n z For a biterm(w i ,w j ) is assigned to topic z, n w|z is the number of times any word w in the corpus is assigned to topic z, is biterm(w i ,w j ) i The number of times it is assigned to topic z, is biterm(w i ,w j ) j The number of times it is assigned to topic z; Q represents the vocabulary size; α and β are the probability distribution θ and word probability distribution φ respectively z The Dirichlet prior of S16: Get the distribution of any topic word vector as follows: Among them, φ w|z is the conditional probability of any word w in the corpus in topic z.

3. A topic modeling and text prediction method combining BTM and Bi_LSTM models as claimed in claim 1, characterized in that: In step S1, the Word2Vec model is used to convert the probability distribution of each topic word into a topic word embedding vector v(T topic ) specifically: Among them, L is the theme T topic The total number of words in v(w l ) is the word w l The vector representation output after inputting the Word2Vec model, p(w l |T topic ) as the theme T topic The word w in l The corresponding weight.

4. A topic modeling and text prediction method combining BTM and Bi_LSTM models as claimed in claim 1, characterized in that: Document representation H(D) = [h1,h2,...,h t ,…,h M ] is obtained by: Embed any word in the word embedding matrix J into the sequence e n Substitute the following Bi_LSTM model for M iterations. The output after each iteration is the encoding result h t (e n ): f t =σ(W f ·[h t-1 ,e n ]+b f ) I t =σ(W i ·[h t-1 ,e n ]+b i ) the t =σ(W o ·[h t-1 ,e n ]+b o ) Among them, f t is the forget gate output of the tth time step, i t is the input gate output at the tth time step, o t is the output gate output at the tth time step, c t is the cell state update at the tth time step, W f is the weight corresponding to the forget gate, b f is the bias vector corresponding to the forget gate, W i is the weight corresponding to the input gate, b i is the bias vector corresponding to the input gate, W o is the weight corresponding to the output gate, b o is the bias vector corresponding to the output gate, W c Update the corresponding weight for the cell state, b c Update the corresponding bias vector for the cell state, c t-1 is the cell state update at the t-1th time step, h t-1 is the word embedding sequence e n The hidden state at the t-1th time step, h t is the word embedding sequence e n The hidden state at the tth time step, σ is the sigmoid function, tanh is the hyperbolic tangent function, and ° represents element-wise multiplication; Embed each word into the sequence e n Encoding result h t (e n ) are combined and concatenated to obtain the document representation H(D) = [h1,h2,...,h t ,…,h M ].

5. A topic modeling and text prediction method combining BTM and Bi_LSTM models as claimed in claim 1, characterized in that: The input layer of the trained MLP model receives the final document representation S. If the prediction task is binary or multi-classification, the output layer of the MLP model uses the sigmoid function to convert the output into a probability distribution. If the prediction task is regression, the output layer of the MLP model uses a linear activation function to output the predicted value.

6. A topic modeling and text prediction method combining BTM and Bi_LSTM models as claimed in claim 1, characterized in that: The input layer of the trained BTM model receives the final document representation S, and the output layer outputs the results of topic modeling.

Images