Topic detection method and device based on BERT model and seed LDA model

Abstract

The application discloses a topic detection method and device based on a BERT model and a seed LDA model, relates to the field of natural language processing, and obtains an original text set, pre-processes the original text set to obtain a text set; a seed LDA model is constructed, each text in the text set is input into a BERT model to obtain a semantic feature vector of each text, the topics of each text in the text set are subjected to feature extraction through the BERT model and the seed LDA model to obtain a topic feature vector of each text, the semantic feature vector and the topic feature vector of each text are subjected to feature fusion to obtain a fusion feature vector of each text; the fusion feature vectors of all the texts in the text set are input into a K-means algorithm for clustering to obtain a plurality of topic clusters; and topics are extracted based on the plurality of topic clusters by using a TF-IDF algorithm, so that the problems that the topic extraction effect of the LDA model on short texts is not good and the context information of words is neglected are solved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, specifically to a topic detection method and apparatus based on the BERT model and the seed LDA model. Background Technology

[0002] With the rapid development of computer technology and the internet, more and more people are accustomed to using the internet to learn about social hotspots and express their opinions, views, and propositions. The internet has become an important place for people to obtain information, express opinions, and safeguard their rights. Therefore, how to regulate the spread of public opinion events on the internet has become a significant issue with practical implications. Online public opinion is characterized by its rapid spread, significant influence, and high level of participation; netizens' attitudes are easily influenced by the direction of online public opinion dissemination. Topic detection technology has emerged in this context, helping users to quickly extract information on topics of interest from massive amounts of data.

[0003] Topic detection is one of the tasks of topic-based task management (TDT), aiming to detect and extract topics from documents. Current topic detection methods suffer from the following problems:

[0004] 1) Classification-based topic detection requires annotation of the corpus, which is costly in terms of manual labor.

[0005] 2) Cluster-based topic detection is an unsupervised model, but its topic extraction performance is not good enough;

[0006] 3) Topic detection based on topic models only considers the word distribution of a document and does not consider the contextual information of the document;

[0007] 4) The LDA model generates more accurate probabilities for longer training texts, making it more suitable for topic mining of long texts, but it is not effective for topic extraction of short texts. Summary of the Invention

[0008] In view of the aforementioned technical problems, the purpose of the embodiments of this application is to propose a topic detection method and apparatus based on the BERT model and the seed LDA model, so as to solve the technical problems mentioned in the background section above.

[0009] In a first aspect, the present invention provides a topic detection method based on a BERT model and a seed LDA model, comprising the following steps:

[0010] Obtain the original text set, preprocess the original text set, and obtain the text set;

[0011] A seed LDA model is constructed, and each text in the text set is input into the BERT model to obtain the semantic feature vector of each text. The BERT model and the seed LDA model are used to extract the theme features of each text in the text set to obtain the theme feature vector of each text. The semantic feature vector and the theme feature vector of each text are fused to obtain the fused feature vector of each text.

[0012] The fused feature vectors of all texts in the text set are input into the K-means algorithm for clustering, resulting in several topic clusters;

[0013] Topics are extracted based on several topic clusters using the TF-IDF algorithm.

[0014] Preferably, the BERT model and the seed LDA model are used to extract the topic features of each text in the text set, resulting in a topic feature vector for each text, specifically including:

[0015] Input the text set into the seed LDA model, and output a specified number of topics and their corresponding topic words, the probability of each text in the text set corresponding to a different topic, and the probability of different topic words;

[0016] The topic words corresponding to each topic are input into the BERT model to perform feature representation on the topic words. The probability of each topic word is used as a weight to weight the feature representation of the topic words, resulting in a feature vector representation of the topic.

[0017]

[0018] Where topic_vec represents the feature vector representation of the topic, word_vec represents the feature representation of the topic word, topic2word_weight represents the probability of the topic word, i = 1, 2, ..., m, and m is the number of topics;

[0019] By using the probability of different topics for each text in the text set as weights, the feature vectors of the topics are weighted to obtain the topic feature vector for each text:

[0020]

[0021] Where doc_topic_vec represents the topic feature vector of the text, doc2topic_weight represents the probability of the text corresponding to the topic, j = 1, 2, ..., n, and n represents the number of texts.

[0022] Preferably, the semantic feature vector and topic feature vector of each text are fused to obtain the fused feature vector of each text, specifically including:

[0023] The semantic feature vector and topic feature vector of each text are fused using a weighted summation method to obtain the fused feature vector of each text.

[0024] As a preferred option, a seed LDA model is constructed, specifically including:

[0025] Define the text set as D = {d1, d2, ..., d...} M}, where M is the total number of texts in the text set, and each text has N words;

[0026] From the Dirichlet distribution β r Selecting from the generated regular topic word distribution

[0027] From the Dirichlet distribution β s Selecting seed topic word distribution

[0028] The i-th text d is generated from the Dirichlet distribution α. i Topic distribution θ i θ i It is a multinomial distribution;

[0029] From topic distribution θ i The i-th text d is generated by sampling from the middle. i The topic of the j-th word z i,j ;

[0030] From the beta distribution π k Select x from the list; if x is 0, then select from the regular topic distribution. Mid-sampling generates words w; if x is 1, then from the seed topic distribution... Mid-sampling generates words w;

[0031] The relationships between various variables in the seed LDA model are as follows:

[0032]

[0033] in, For the topic z i,j Word distribution, w i,j Words generated from sampling Let p(θ) represent the joint probability distribution of all variables. i |α) represents the topic distribution θ given the Dirichlet distribution α. i The probability, p(z) i,j |θ i ) represents the topic distribution θ i Subject z under conditions i,j The probability distribution of words, Indicating β in the Dirichlet distribution sThe probability distribution of seed topic words under the given conditions Indicates in topic z i,j Words generated under the condition of word distribution w i,j The probability, Indicating β in the Dirichlet distribution r Generating a regular topic word distribution under certain conditions The probability of.

[0034] As a preferred approach, the original text set is preprocessed, specifically including:

[0035] The original text set is cleaned, segmented, and stop words are removed.

[0036] As a preferred approach, the fused feature vectors of all texts in the text set are input into the K-means algorithm for clustering, resulting in several topic clusters, specifically including:

[0037] S31, divide the fused feature vector of all texts in the text set into K parts, and randomly select a sample in each part as the initial cluster center;

[0038] S32, calculate the Euclidean distance between each fusion feature vector of each data set and its corresponding cluster center, and assign it to the cluster with the smallest distance;

[0039] S33, recalculate the cluster center of each cluster, which is the mean of the samples in the cluster;

[0040] S34. Repeat steps S32-S33 until the convergence termination condition is met. The cluster obtained after convergence is the topic cluster.

[0041] As a preferred approach, topics are extracted using the TF-IDF algorithm based on several topic clusters, specifically including:

[0042] Extract the top N words from each topic cluster and calculate the frequency of each word in the text to obtain the word frequency:

[0043]

[0044] Where, n d,w This indicates the number of times word w appears in text d;

[0045] Calculate the inverse document frequency for each word:

[0046]

[0047] Among them, |M w | represents the total number of texts in text set D that contain the word w, and M is the total number of texts in text set D;

[0048] Calculate the product of term frequency and inverse document frequency to obtain the TF-IDF value:

[0049] tfidf(M,d,w)=tf(d,w)×idf(M,w);

[0050] The word corresponding to the maximum TF-IDF value among the first N words in all topic clusters is taken as the topic.

[0051] Secondly, the present invention provides a topic detection device based on a BERT model and a seed LDA model, comprising:

[0052] The preprocessing module is configured to acquire the original text set, preprocess the original text set, and obtain the text set.

[0053] The feature extraction and fusion module is configured to build a seed LDA model, input each text in the text set into the BERT model to obtain the semantic feature vector of each text, extract the theme features of each text in the text set through the BERT model and the seed LDA model to obtain the theme feature vector of each text, and fuse the semantic feature vector and theme feature vector of each text to obtain the fused feature vector of each text.

[0054] The clustering module is configured to input the fused feature vectors of all texts in the text set into the K-means algorithm for clustering, resulting in several topic clusters;

[0055] The topic extraction module is configured to extract topics based on several topic clusters using the TF-IDF algorithm.

[0056] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0057] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] (1) The topic detection method based on BERT model and seed LDA model proposed in this invention can extract the semantic feature vector of each text through BERT model to obtain semantic information and avoid ignoring context information. The topic feature vector of each text in the text set is extracted by BERT model and seed LDA model to obtain topic information. The semantic feature vector and topic feature vector of each text are fused to obtain the fused feature vector of each text to improve the topic detection accuracy of short text. The semantic information and topic information are used for topic detection at the same time, which has a better effect.

[0060] (2) The topic detection method based on BERT model and seed LDA model proposed in this invention guides the LDA model with seed words to construct the seed LDA model, so as to more effectively guide the generation of short text topics and make the generated topics more interpretable. This not only effectively improves the performance of topic detection, but also reduces the cost of manual annotation.

[0061] (3) The topic detection method based on BERT model and seed LDA model proposed in this invention uses K-means algorithm and TF-IDF algorithm for topic clustering and screening, so that the extracted topics are more accurate. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is an exemplary device architecture diagram in which an embodiment of this application can be applied;

[0064] Figure 2 This is a flowchart illustrating the topic detection method based on the BERT model and the seed LDA model, as an embodiment of this application.

[0065] Figure 3 A flowchart illustrating an embodiment of this application based on a BERT model and a seed LDA model;

[0066] Figure 4 A schematic diagram of the seed LDA model in the topic detection method based on the BERT model and the seed LDA model, which is an embodiment of this application;

[0067] Figure 5 This is a schematic diagram illustrating the text semantic feature extraction of the topic detection method based on the BERT model and the seed LDA model, which is an embodiment of this application.

[0068] Figure 6 This is a schematic diagram illustrating the extraction of feature vector representations of topics in a topic detection method based on the BERT model and seed LDA model, as described in an embodiment of this application.

[0069] Figure 7 This is a schematic diagram illustrating the extraction of topic feature vectors from text using a topic detection method based on the BERT model and seed LDA model, as described in an embodiment of this application.

[0070] Figure 8 This is a schematic diagram of a topic detection device based on the BERT model and the seed LDA model, which is an embodiment of this application.

[0071] Figure 9 This is a schematic diagram of the structure of a computer device suitable for implementing the electronic device of the present application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0073] Figure 1 An exemplary device architecture 100 is shown, which can be applied to the topic detection method based on the BERT model and the seed LDA model or the topic detection apparatus based on the BERT model and the seed LDA model of the present application.

[0074] like Figure 1 As shown, the device architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0075] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications, such as data processing applications and file processing applications, can be installed on terminal devices 101, 102, and 103.

[0076] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0077] Server 105 can be a server that provides various services, such as a background data processing server that processes files or data uploaded by terminal devices 101, 102, and 103. The background data processing server can process the acquired files or data and generate processing results.

[0078] It should be noted that the topic detection method based on the BERT model and the seed LDA model provided in this application embodiment can be executed by the server 105 or by the terminal devices 101, 102, and 103. Correspondingly, the topic detection device based on the BERT model and the seed LDA model can be set in the server 105 or in the terminal devices 101, 102, and 103.

[0079] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Any number of terminal devices, networks, and servers can be included depending on implementation needs. If the data being processed does not need to be retrieved remotely, the above architecture may not include a network, requiring only servers or terminal devices.

[0080] Figure 2 This application illustrates an embodiment of a topic detection method based on a BERT model and a seed LDA model, comprising the following steps:

[0081] S1: Obtain the original text set, preprocess the original text set, and obtain the text set.

[0082] In a specific embodiment, the original text set is preprocessed, including:

[0083] The original text set is cleaned, segmented, and stop words are removed.

[0084] For details, please refer to Figure 3The original text set is obtained from the corpus, and the original text set is preprocessed. The data preprocessing methods include data cleaning, word segmentation, and stop word removal. Data preprocessing can remove duplicate data, missing values, and outliers that interfere with data analysis and mining, thereby improving the accuracy and reliability of the data.

[0085] S2. Construct a seed LDA model. Input each text in the text set into the BERT model to obtain the semantic feature vector of each text. Extract the theme features of each text in the text set through the BERT model and the seed LDA model to obtain the theme feature vector of each text. Fuse the semantic feature vector and the theme feature vector of each text to obtain the fused feature vector of each text.

[0086] In a specific embodiment, constructing a seed LDA model specifically includes:

[0087] Define the text set as D = {d1, d2, ..., d...} M}, where M is the total number of texts in the text set, and each text has N words;

[0088] From the Dirichlet distribution β r Selecting from the generated regular topic word distribution

[0089] From the Dirichlet distribution β s Selecting seed topic word distribution

[0090] The i-th text d is generated from the Dirichlet distribution α. i Topic distribution θ i θ i It is a multinomial distribution;

[0091] From topic distribution θ i The i-th text d is generated by sampling from the middle. i The topic of the j-th word z i,j ;

[0092] From the beta distribution π k Select x from the list; if x is 0, then select from the regular topic distribution. Mid-sampling generates words w; if x is 1, then from the seed topic distribution... Mid-sampling generates words w;

[0093] The relationships between various variables in the seed LDA model are as follows:

[0094]

[0095] in, For the topic z i,jWord distribution, w i,j Words generated from sampling Let p(θ) represent the joint probability distribution of all variables. i |α) represents the topic distribution θ given the Dirichlet distribution α. i The probability, p(z) i,j |θ i ) represents the topic distribution θ i Subject z under conditions i,j The probability distribution of words, Indicating β in the Dirichlet distribution s The probability distribution of seed topic words under the given conditions Indicates in topic z i,j Words generated under the condition of word distribution w i,j The probability, Indicating β in the Dirichlet distribution r Generating a regular topic word distribution under certain conditions The probability of...

[0096] Specifically, such as Figure 4 As shown, assume D = {d1, d2, ..., d} M Let M be the total number of texts, and let N be the total number of words in each text. Based on the above process, a seed LDA model is obtained by modeling the text set. One topic can correspond to multiple topic words. This seed LDA model introduces seed words, and by determining the more accurate words from either the regular topic word distribution or the seed topic word distribution based on different parameters selected in the beta distribution, this seed LDA model is an unsupervised model and does not require text annotation, thus reducing manual costs.

[0097] Due to the inherent characteristics of probability distribution models, the longer the training text, the more accurate the generated probabilities. Therefore, LDA models are more suitable for topic mining of long texts, but perform poorly in topic extraction from short texts, which is an area for model optimization. In addition, LDA models use the bag-of-words method, which simplifies the model's complexity, but also ignores the contextual information of words, providing further optimization possibilities for LDA models.

[0098] In a specific embodiment, the BERT model and the seed LDA model are used to extract the topic features of each text in the text set, resulting in a topic feature vector for each text. Specifically, this includes:

[0099] Input the text set into the seed LDA model, and output a specified number of topics and their corresponding topic words, the probability of each text in the text set corresponding to a different topic, and the probability of different topic words;

[0100] The topic words corresponding to each topic are input into the BERT model to perform feature representation on the topic words. The probability of each topic word is used as a weight to weight the feature representation of the topic words, resulting in a feature vector representation of the topic.

[0101]

[0102] Where topic_vec represents the feature vector representation of the topic, word_vec represents the feature representation of the topic word, topic2word_weight represents the probability of the topic word, i = 1, 2, ..., m, and m is the number of topics;

[0103] By using the probability of different topics for each text in the text set as weights, the feature vectors of the topics are weighted to obtain the topic feature vector for each text:

[0104]

[0105] Where doc_topic_vec represents the topic feature vector of the text, doc2topic_weight represents the probability of the text corresponding to the topic, j = 1, 2, ..., n, and n represents the number of texts.

[0106] In a specific embodiment, the semantic feature vector and topic feature vector of each text are fused to obtain the fused feature vector of each text, specifically including:

[0107] The semantic feature vector and topic feature vector of each text are fused using a weighted summation method to obtain the fused feature vector of each text.

[0108] For details, please refer to Figure 5 By inputting text into the BERT model, the semantic feature vector of the text can be obtained, which contains the contextual information of the text. Extracting the topic feature vector of the text requires first using a seed LDA model to obtain the topic and its corresponding topic words, as well as the probability of each text corresponding to a different topic. Then, the BERT model is used to obtain the feature representation of the topic words, and the weighted sum is used to obtain the topic feature vector of the text.

[0109] refer to Figure 6 From the perspective of the text set, the seed LDA model will output a specified number of topics and their corresponding topic words for that text set. The top 10 most probable words for each generated topic are then selected as topic words. These topic words are then used to perform feature representation using the BERT model, with their probabilities weighted to obtain the feature representation vector of the topic.

[0110] refer to Figure 7From the perspective of a single text, the seed LDA model will output the probability of each text corresponding to different topics, and the topic with the highest probability will be taken as the topic of that text. The probabilities of different topics are used as weights to weight the feature vector representation of the topic, resulting in the topic feature vector of each text.

[0111] The topic feature vector better represents the topic information of the current text and is more conducive to topic detection, while the semantic feature vector is used to assist in topic detection. Therefore, the fusion process of the semantic feature vector and the topic feature vector is set as a weighted sum, with corresponding weights of 0.3 and 0.7, respectively.

[0112] S3: Input the fused feature vectors of all texts in the text set into the K-means algorithm for clustering to obtain several topic clusters.

[0113] In a specific embodiment, step S3 specifically includes:

[0114] S31, divide the fused feature vector of all texts in the text set into K parts, and randomly select a sample in each part as the initial cluster center;

[0115] S32, calculate the Euclidean distance between each fusion feature vector of each data set and its corresponding cluster center, and assign it to the cluster with the smallest distance;

[0116] S33, recalculate the cluster center of each cluster, which is the mean of the samples in the cluster;

[0117] S34. Repeat steps S32-S33 until the convergence termination condition is met. The cluster obtained after convergence is the topic cluster.

[0118] Specifically, the fused feature vectors are input into the K-means algorithm for topic clustering. The optimal number of clusters is selected through cluster evaluation to obtain the final topic clusters. The K-means algorithm is a typical Euclidean distance-based clustering algorithm. It does not require manual labeling of the corpus and is a commonly used unsupervised, iterative algorithm. Its main idea is that the closer the Euclidean distance between two objects, the higher their similarity, and they should be clustered into the same class. Clustering is considered convergent when no fused feature vectors are reassigned to other clusters or the cluster centers no longer change, meeting the convergence termination condition. Ultimately, the fused feature vectors of all texts in the text set can be clustered into K topic clusters.

[0119] S4 extracts topics based on several topic clusters using the TF-IDF algorithm.

[0120] In a specific embodiment, step S4 specifically includes:

[0121] Extract the top N words from each topic cluster and calculate the frequency of each word in the text to obtain the word frequency:

[0122]

[0123] Where, n d,w This indicates the number of times word w appears in text d;

[0124] Calculate the inverse document frequency for each word:

[0125]

[0126] Among them, |M w | represents the total number of texts in text set D that contain the word w, and M is the total number of texts in text set D;

[0127] Calculate the product of term frequency and inverse document frequency to obtain the TF-IDF value for each word:

[0128] tfidf(M,d,w)=tf(d,w)×idf(M,w);

[0129] The word corresponding to the maximum TF-IDF value among the first N words in all topic clusters is taken as the topic.

[0130] Specifically, the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm was used for topic word identification. TF-IDF is a simple and efficient word statistics method that can identify words with high frequency in the current document but low frequency in other documents, and considers these words to be representative of the current text. The top ten words of each topic cluster were extracted using the TF-IDF algorithm as topic words, and the final topics were obtained through this selection process.

[0131] The steps S1-S4 above do not represent the order of the steps, but are merely symbolic representations of the steps.

[0132] To evaluate the performance of the topic detection method based on the BERT model and the seed LDA model proposed in the embodiments of this application, the official dataset 20newsgroups and a real public opinion corpus dataset about "a certain sports event" obtained by crawling were selected for experimental comparison.

[0133] The 20newsgroups dataset contains over 18,000 news articles, evenly divided into 20 news groups with different topics, hence the name 20newsgroups text dataset. The 20newsgroups dataset is one of the international standard datasets used for text classification, text mining, and information retrieval research. This invention divides the 20newsgroups dataset into training and testing sets in a 7:3 ratio, with the training set containing 13,192 data points and the testing set containing 5,650 data points.

[0134] The "A Certain Sports Meet" public opinion corpus dataset was obtained by web crawling from relevant Weibo posts from February 1, 2022 to June 30, 2022. Preprocessing methods such as text cleaning, word segmentation, stop word removal, and deduplication were used to filter and ultimately obtain 21,270 Weibo posts. Finally, the dataset was divided into training and test sets in a 7:3 ratio.

[0135] The statistical results of the experimental dataset are shown in Table 1.

[0136] Table 1. 20newsgroups and the "a certain sports meet" dataset

[0137]

[0138] 1) To verify the effectiveness of the embodiments of this application, the following four models were selected as baseline methods for comparison, as follows:

[0139] 1. BERT: The Google team further developed the bidirectional Transformer model, obtaining text context information through pre-training to obtain text vector representations.

[0140] 2. BTM&BERT: A method that combines BTM and BERT models to construct short text feature vectors.

[0141] 3. BERT-LDA: A method that combines BERT and LDA to compensate for the lack of semantic information.

[0142] 4. Word2Vec-LDA: A method that retains the topic features of LDA while using the Word2Vec model to obtain contextual information and obtain a vector representation of the novel review text.

[0143] 2) The evaluation criteria are as follows:

[0144] The evaluation metrics used in the experiment were the silhouette coefficient and the CH value (Calinski-Harabaz).

[0145] The silhouette coefficient measures the quality of clustering results based on the cohesion within clusters and the separation outside clusters. Cohesion reflects how closely a sample point is associated with elements within its cluster; separation reflects how closely a sample point is associated with elements outside its cluster. The silhouette coefficient is calculated as follows:

[0146]

[0147] Where a(i) represents the cohesion of the sample points, i.e.:

[0148]

[0149] The calculation method for b(i) is the same as that for a(i), but it requires traversing other clusters to obtain multiple values ​​{b1(i),…,b m (i)}, select the smallest value from them as the final result.

[0150] The Clustering Hierarchy (CH) metric measures the tightness within a cluster by calculating the sum of squared distances between each point in the cluster and the cluster center, and measures the separation of the dataset by calculating the sum of squared distances between each cluster centroid and the dataset centroid. The CH metric is derived as the ratio of separation to tightness. Therefore, a larger CH value indicates tighter clustering within clusters and greater dispersion between clusters, resulting in better clustering outcomes. The CH metric is calculated as follows:

[0151]

[0152] Where k is the number of clusters, and N is the total number of documents. SS B It is the inter-class variance, SS W It is the within-class variance.

[0153] SS B The calculation expression is as follows:

[0154] SS B =tr(B k );

[0155]

[0156] Among them, c q It is a point mass of type q, c E It is the center point of all data, n q It is the total number of data points of class q.

[0157] SS W The calculation expression is as follows:

[0158] SS W =tr(W k );

[0159]

[0160] Among them, C q It is the collection of all data in class q, c q It is a point mass of the q class.

[0161] 3) The results of the comparative experiment are as follows:

[0162] The best-performing method for each dataset is shown in bold, and the second-best result is shown underlined. Table 2 shows the comparative experimental results of each method on the 20newsgroup dataset.

[0163] Table 2. Comparative experimental results on the 20newsgroup dataset.

[0164]

[0165] The "A Certain Sports Meet" dataset is analyzed month by month, focusing on sub-topics for each month. Table 3 shows the comparative experimental results of various methods on different months of the "A Certain Sports Meet" dataset.

[0166] Table 3 shows the results of the comparative experiment on the dataset of "a certain sports meet".

[0167]

[0168]

[0169] From Tables 2 and 3, the following conclusions can be drawn:

[0170] 1. This invention achieved good results on both datasets, which verifies the effectiveness of the method:

[0171] 1) On the 20newsgroup dataset, best and second-best experimental results were achieved for each metric. The silhouette coefficient metric achieved the best experimental result, improving by 0.07 compared to the second-best result, while the CH value metric achieved the second-best result.

[0172] 2) On the "a certain sports meet" dataset, the silhouette coefficient index achieved the best and second-best experimental results in each month, while the CH value index achieved the best experimental results in April and June, and the second-best experimental results in February and March. For example, the silhouette coefficient index improved by 0.03 compared to the second-best result in April, and the CH value index improved by 40684 compared to the second-best result.

[0173] 2. Compared to the BERT model, this invention significantly improves performance across various datasets. On the 20newsgroup dataset, the silhouette coefficient is improved by 0.58, and the CH value is improved by 23311. On the "A Certain Sports Meet" dataset, it surpasses the BERT model across all months; for example, in February, the silhouette coefficient is improved by 0.74, and the CH value is improved by 9221. This invention obtains contextual information from the text using the BERT model and integrates topic information obtained from the seed LDA model; therefore, its performance is superior to that of a purely pre-trained BERT model.

[0174] 3. Compared to the BERT-LDA method, this invention only lags behind BERT-LDA in three metrics across two datasets, while outperforming it in all other metrics. For example, on the "a certain sports meet" dataset, the silhouette coefficient and CH index for March are improved by 0.04 and 6362 respectively compared to BERT-LDA. This invention further improves the LDA model by adding seed words to guide topic generation, thus making it superior to the BERT-LDA method.

[0175] 4. Compared to the BTM & BERT methods, this invention only lags behind the BTM & BERT models in one metric across both datasets. For example, on the 20newsgroup dataset, this invention achieves comprehensive superiority in both the silhouette coefficient and CH value metrics, improving by 0.16 and 1747 respectively; on the "A Certain Sports Meet" dataset, the silhouette coefficient and CH values ​​for February are improved by 0.03 and 323 respectively compared to the BTM & BERT methods. The BTM model utilizes a Biterm mechanism to extend the text training length, addressing the issue of poor topic extraction performance in short text LDA models. This invention employs a seeded LDA model, guiding the LDA model with seed words to more effectively generate topics for short texts. Compared to the BTM model, the topics generated by this invention are more interpretable, thus its performance is superior to the BTM & BERT methods. Future development of this invention could further leverage the Biterm mechanism to improve performance.

[0176] 5. Compared to the Word2Vec-LDA method, this invention still demonstrates better performance. For example, on the "a certain sports meet" dataset, the silhouette coefficient index for June improved by 0.02, and the CH value index improved by 2154. This invention employs the more advanced and superior BERT model. The BERT model incorporates an attention mechanism and adds word positional features, compensating for the shortcomings of the bag-of-words model and outperforming the Word2Vec pre-trained model.

[0177] 4) Ablation test

[0178] To investigate the contributions of the BERT model and the seed LDA model to the topic detection task, this section sets up ablation experiments, setting the models as four cases:

[0179] 1. BERT: Contains only semantic information of text generated by the BERT model.

[0180] 2. LDA: Contains only the topic information of text generated by the traditional LDA model.

[0181] 3. sLDA: Contains only the topic information of the text generated by the seed LDA model.

[0182] 4. This invention includes semantic information generated by the BERT model and topic information generated by the seed LDA model.

[0183] The best result for each dataset is displayed in bold, and the second best result is displayed underlined.

[0184] The ablation experimental results of each method on the 20newsgroup dataset are shown in Table 4:

[0185] Table 4 Ablation experimental results on the 20newsgroup dataset

[0186]

[0187] The "A Certain Sports Meet" dataset is analyzed by month, with sub-topics for each month being examined separately. Table 5 shows the ablation experiment results of each method on the "A Certain Sports Meet" dataset for different months.

[0188] Table 5 Ablation Experiment Results on the "A Certain Sports Meet" Dataset

[0189]

[0190]

[0191] From Tables 4 and 5, the following conclusions can be drawn:

[0192] (1) Topic information is more helpful for topic detection tasks. The LDA model and the sLDA model outperformed the BERT model in silhouette coefficient and CH value on both datasets. For example, on the 20newsgroup dataset, the LDA model improved the silhouette coefficient and CH value by 0.29 and 1826 respectively compared to the BERT model, while the sLDA model improved the silhouette coefficient and CH value by 0.33 and 1700 respectively compared to the BERT model.

[0193] (2) The seed LDA model outperforms the traditional LDA model. The sLDA model can generate better topic information by guiding the generation of seed words. The sLDA model outperforms the LDA model in almost all two datasets. For example, on the "a certain sports meet" dataset, the silhouette coefficient index for February improved by 0.07, and the CH value index improved by 2365.

[0194] (3) Incorporating semantic information helps with topic detection. This invention outperforms the sLDA model across all metrics in both datasets. For example, in the "a certain sports meet" dataset, the silhouette coefficient metric for April improved by 0.4, and the CH value metric improved by 88769.

[0195] (4) Using semantic and topic information simultaneously for topic detection yields better results. Compared to using only one piece of information, this invention further improves topic detection performance on both datasets. This invention achieves optimal experimental results on all metrics across both datasets. For example, on the "a certain sports meet" dataset, the silhouette coefficient in June is improved by 0.23 compared to the second-best result, and the CH value is improved by 73280 compared to the second-best result.

[0196] In summary, seed words can guide the LDA model to generate better topic information in short texts, and the BERT model can be used to obtain the contextual information of the text. This invention proposes a topic detection method based on the BERT model and the seed LDA model to fuse the contextual information and topic information of the text, so as to improve the topic detection accuracy of short texts.

[0197] Further reference Figure 8 As an implementation of the methods shown in the above figures, this application provides an embodiment of a topic detection device based on the BERT model and the seed LDA model. This device embodiment is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0198] This application provides a topic detection device based on a BERT model and a seed LDA model, including:

[0199] Preprocessing module 1 is configured to acquire the original text set, preprocess the original text set, and obtain the text set.

[0200] Feature extraction and fusion module 2 is configured to build a seed LDA model, input each text in the text set into the BERT model to obtain the semantic feature vector of each text, extract the theme features of each text in the text set through the BERT model and the seed LDA model to obtain the theme feature vector of each text, and fuse the semantic feature vector and theme feature vector of each text to obtain the fused feature vector of each text.

[0201] Clustering module 3 is configured to input the fused feature vectors of all texts in the text set into the K-means algorithm for clustering, resulting in several topic clusters;

[0202] Topic extraction module 4 is configured to extract topics based on several topic clusters using the TF-IDF algorithm.

[0203] The following is for reference. Figure 9 It illustrates an electronic device suitable for implementing embodiments of this application (e.g., Figure 1 The diagram shows the structure of a computer device 900 (a server or terminal device). Figure 9 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0204] like Figure 9 As shown, the computer device 900 includes a central processing unit (CPU) 901 and a graphics processing unit (GPU) 902, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 903 or programs loaded from storage section 909 into random access memory (RAM) 904. The RAM 904 also stores various programs and data required for the operation of the device 900. The CPU 901, GPU 902, ROM 903, and RAM 904 are interconnected via a bus 905. An input / output (I / O) interface 906 is also connected to the bus 905.

[0205] The following components are connected to I / O interface 906: an input section 907 including a keyboard, mouse, etc.; an output section 908 including an LCD, speakers, etc.; a storage section 909 including a hard disk, etc.; and a communication section 910 including a network interface card, such as a LAN card or modem. The communication section 910 performs communication processing via a network such as the Internet. A drive 911 may also be connected to I / O interface 906 as needed. A removable medium 912, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 911 as needed so that computer programs read from it can be installed into storage section 909 as required.

[0206] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 910, and / or installed from removable medium 912. When the computer program is executed by central processing unit (CPU) 901 and graphics processing unit (GPU) 902, the functions defined in the methods of this application are performed.

[0207] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution device, apparatus, or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0208] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0209] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using dedicated hardware-based means to perform the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0210] The modules described in the embodiments of this application can be implemented in software or hardware. These modules can also be located within a processor.

[0211] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire an original text set; preprocess the original text set to obtain a text set; construct a seed LDA model; input each text in the text set into a BERT model to obtain a semantic feature vector for each text; extract the topic features of each text in the text set using the BERT model and the seed LDA model to obtain a topic feature vector for each text; fuse the semantic feature vector and the topic feature vector of each text to obtain a fused feature vector for each text; input the fused feature vector of all texts in the text set into a K-means algorithm for clustering to obtain several topic clusters; and extract topics based on the several topic clusters using the TF-IDF algorithm.

[0212] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A topic detection method based on BERT model and seed LDA model, characterized in that, Includes the following steps: Obtain the original text set, and preprocess the original text set to obtain the text set; Constructing a seed LDA model specifically includes: The text set is defined as , The total number of texts in the text set, each text having One word; From Dirichlet distribution Selecting from the generated regular topic word distribution ; From Dirichlet distribution Selecting seed topic word distribution ; From Dirichlet distribution Select from to generate the first text Theme distribution , It is a multinomial distribution; From the topic distribution Sampling generates the first text The The theme of each word ; From beta distribution Selected from ,if If it is 0, then it is from the regular topic distribution Words generated by mid-sampling ;if If the value is 1, then it is distributed from the seed topic. Words generated by mid-sampling ; The relationships between various variables in the seed LDA model are as follows: ； in, Theme Word distribution, Words generated from sampling This represents the joint probability distribution of all variables. Indicates the distribution in Dirichlet. Theme distribution under conditions The probability, Indicating in topic distribution Subject under conditions The probability distribution of words, Indicates the distribution in Dirichlet. The probability distribution of seed topic words under the given conditions Indicates in the topic Words generated under the condition of word distribution The probability, Indicates the distribution in Dirichlet. Generating a regular topic word distribution under certain conditions The probability; input each text in the text set into the BERT model to obtain the semantic feature vector of each text; extract the topic features of each text in the text set through the BERT model and the seed LDA model to obtain the topic feature vector of each text, specifically including: The text set is input into a seed LDA model, which outputs a specified number of topics and their corresponding keywords, the probability of each text in the text set corresponding to a different topic, and the probability of different keywords. The topic words corresponding to each topic are input into the BERT model, and feature representations are generated for the topic words. The probabilities of the topic words are used as weights to weight the feature representations of the topic words, resulting in a feature vector representation of the topic. ； in, The feature vector represents the topic, word_vec represents the feature representation of the topic word, topic2word_weight represents the probability of the topic word, i=1,2,…,m, where m is the number of topics; Using the probability of different topics for each text in the text set as weights, the feature vectors of the topics are weighted to obtain the topic feature vector for each text: ； in, The semantic feature vector and the topic feature vector of each text are fused to obtain the fused feature vector of each text. The fused feature vectors of all texts in the text set are input into the K-means algorithm for clustering to obtain several topic clusters; Topics are extracted based on the aforementioned topic clusters using the TF-IDF algorithm.

2. The topic detection method based on BERT model and seed LDA model according to claim 1, characterized in that, The process of fusing the semantic feature vector and topic feature vector of each text to obtain a fused feature vector for each text specifically includes: The semantic feature vector and topic feature vector of each text are fused using a weighted summation method to obtain the fused feature vector of each text.

3. The topic detection method based on the BERT model and seed LDA model according to claim 1, characterized in that, The preprocessing of the original text set specifically includes: The original text set is cleaned, segmented, and stop words are removed.

4. The topic detection method based on the BERT model and seed LDA model according to claim 1, characterized in that, The step of inputting the fused feature vectors of all texts in the text set into the K-means algorithm for clustering to obtain several topic clusters specifically includes: S31, divide the fused feature vector of all texts in the text set into K data, and randomly select a sample in each data as the initial cluster center; S32, calculate the Euclidean distance between each fused feature vector of each data and its corresponding cluster center, and assign it to the cluster with the smallest distance; S33, recalculate the cluster center of each cluster, which is the mean of the samples in the cluster; S34. Repeat steps S32-S33 until the convergence termination condition is met. The cluster obtained after convergence is the topic cluster.

5. The topic detection method based on the BERT model and seed LDA model according to claim 1, characterized in that, The extraction of topics based on the aforementioned topic clusters using the TF-IDF algorithm specifically includes: Extract the top N words from each topic cluster and calculate the frequency of each word in the text to obtain the word frequency: ； in, Words Appearing in text The number of times in; Calculate the inverse document frequency for each word: ； in, This indicates that the text set D contains words. The total number of texts, The total number of texts in the text set D; The TF-IDF value is obtained by multiplying the term frequency by the inverse document frequency. ； The word corresponding to the maximum TF-IDF value among the first N words in all topic clusters is taken as the topic.

6. A topic detection device based on the BERT model and the seed LDA model, characterized in that, include: The preprocessing module is configured to acquire the original text set, preprocess the original text set, and obtain a new text set. The feature extraction and fusion module is configured to build a seed LDA model, specifically including: The text set is defined as , The total number of texts in the text set, each text having One word; From Dirichlet distribution Selecting from the generated regular topic word distribution ; From Dirichlet distribution Selecting seed topic word distribution ; From Dirichlet distribution Select from to generate the first text Theme distribution , It is a multinomial distribution; From the topic distribution Sampling to generate the first text The The theme of each word ; From beta distribution Selected from ,if If it is 0, then it is from the regular topic distribution Words generated by mid-sampling ;if If the value is 1, then it is distributed from the seed topic. Words generated by mid-sampling ; The relationships between various variables in the seed LDA model are as follows: ； in, Theme Word distribution, Words generated from sampling This represents the joint probability distribution of all variables. Indicates the distribution in Dirichlet. Theme distribution under conditions The probability, Indicating in topic distribution Subject under conditions The probability distribution of words, Indicates the distribution in Dirichlet. The probability distribution of seed topic words under the given conditions Indicates in the topic Words generated under the condition of word distribution The probability, Indicates the distribution in Dirichlet. Generate regular topic word distribution under certain conditions The probability; input each text in the text set into the BERT model to obtain the semantic feature vector of each text, and extract the topic feature vector of each text in the text set through the BERT model and the seed LDA model, specifically including: The text set is input into a seed LDA model, which outputs a specified number of topics and their corresponding keywords, the probability of each text in the text set corresponding to a different topic, and the probability of different keywords. The topic words corresponding to each topic are input into the BERT model, and feature representations are generated for the topic words. The probabilities of the topic words are used as weights to weight the feature representations of the topic words, resulting in a feature vector representation of the topic. ； in, The feature vector represents the topic, word_vec represents the feature representation of the topic word, topic2word_weight represents the probability of the topic word, i=1,2,…,m, where m is the number of topics; Using the probability of different topics for each text in the text set as weights, the feature vectors of the topics are weighted to obtain the topic feature vector for each text: ； in, The semantic feature vector and the topic feature vector of each text are fused to obtain the fused feature vector of each text. The clustering module is configured to input the fused feature vectors of all texts in the text set into the K-means algorithm for clustering to obtain several topic clusters; The topic extraction module is configured to extract topics based on the aforementioned topic clusters using the TF-IDF algorithm.

7. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images