Hot topic extraction method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a hot topic extraction method and device, electronic equipment and storage medium. The hot topic extraction method comprises: obtaining a target text; obtaining a plurality of topics corresponding to the target text and a word probability distribution vector corresponding to each topic; calculating the topic similarity between each topic in the plurality of topics based on the word probability distribution vector, and calculating the topic consistency corresponding to each topic; based on the topic similarity and the topic consistency, the plurality of topics are merged to obtain at least one topic cluster; the topic popularity and the topic novelty of each topic cluster in the at least one topic cluster are calculated, and the hot topic corresponding to the target text is determined based on the topic popularity and the topic novelty, thereby the semantic and grammatical correlation between each word in the text can be considered when extracting the hot topic, and the accuracy of the extracted hot topic is improved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and storage medium for extracting trending topics. Background Technology

[0002] With the widespread application of internet technology, internet social networking has gradually entered people's lives. People can socialize online and obtain various information from the internet at any time, such as news, current events, and public opinion. Since there are tens of thousands of pieces of information on the internet every day, if users want to obtain the latest hot topics to understand the current social and economic development trends and directions, they need to spend a lot of time identifying and filtering them. Therefore, the extraction of hot topics has become a focus of people's attention.

[0003] In existing research on hot topic extraction, the main method used is the hot topic extraction method based on probabilistic topic models. This method uses probability distribution models such as PLSA and LDA to perform probability statistics on the topics and words corresponding to the text data, and then judges whether the current topic is a hot topic based on the probability of the topics and words.

[0004] However, the hot topic extraction method based on probabilistic topic models ignores the semantic and grammatical relationships between words in the text, resulting in inaccurate hot topic extraction. Summary of the Invention

[0005] To address the aforementioned technical problems, this disclosure provides a method, apparatus, electronic device, and storage medium for extracting trending topics.

[0006] The first aspect of this disclosure provides a method for extracting trending topics, including:

[0007] Obtain the target text;

[0008] Obtain multiple topics corresponding to the target text and the word probability distribution vector for each topic;

[0009] The topic similarity between topics in multiple topics is calculated based on the word probability distribution vector, and the topic consistency corresponding to each topic is calculated. The topic consistency is determined based on the similarity between multiple words corresponding to a topic.

[0010] Based on topic similarity and topic consistency, multiple topics are merged to obtain at least one topic cluster.

[0011] Calculate the topic popularity and topic novelty of each topic cluster in at least one topic cluster. Based on the topic popularity and topic novelty, determine the hot topics corresponding to the target text. The topic popularity is used to characterize the number of times the central topic corresponding to each topic cluster appears in the target text. The topic novelty is used to characterize the degree of association between the central topic corresponding to each topic cluster and other topics other than the central topic. The central topic is the topic with the highest sum of similarity among multiple words corresponding to at least one topic in the topic cluster.

[0012] A second aspect of this disclosure provides a hot topic extraction device, comprising:

[0013] The first acquisition module is used to acquire the target text;

[0014] The second acquisition module is used to acquire multiple topics corresponding to the target text and the word probability distribution vector corresponding to each topic.

[0015] The calculation module is used to calculate the topic similarity between various topics in multiple topics based on the word probability distribution vector, and to calculate the topic consistency corresponding to each topic. The topic consistency is determined based on the similarity between multiple words corresponding to the topic.

[0016] The merging module is used to merge multiple topics based on topic similarity and topic consistency to obtain at least one topic cluster.

[0017] The topic determination module is used to calculate the topic popularity and topic novelty of each topic cluster in at least one topic cluster. Based on the topic popularity and topic novelty, the hot topics corresponding to the target text are determined. The topic popularity is used to represent the number of times the central topic corresponding to each topic cluster appears in the target text. The topic novelty is used to represent the degree of association between the central topic corresponding to each topic cluster and other topics other than the central topic. The central topic is the topic with the highest sum of similarity among multiple words corresponding to at least one topic in the topic cluster.

[0018] A third aspect of this disclosure provides an electronic device, including:

[0019] processor;

[0020] Memory, used to store executable instructions;

[0021] The processor is used to read executable instructions from memory and execute the executable instructions to implement the hot topic extraction method provided in the first aspect above.

[0022] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the hot topic extraction method provided in the first aspect.

[0023] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0024] The hot topic extraction method, apparatus, electronic device, and storage medium provided in this disclosure can acquire target text, obtain multiple topics corresponding to the target text and word probability distribution vectors corresponding to each topic after acquiring the target text, calculate topic similarity among the multiple topics based on the word probability distribution vectors, and calculate topic consistency for each topic. Topic consistency is determined based on the similarity among the multiple words corresponding to the topic. Based on topic similarity and topic consistency, the multiple topics are merged to obtain at least one topic cluster. The topic popularity and topic novelty corresponding to each topic cluster are calculated. Based on topic popularity and topic novelty, the hot topics corresponding to the target text are determined. The topic popularity is used to characterize the hot topics corresponding to each topic cluster. The central theme is the number of times it appears in the target text. Topic novelty is used to characterize the degree of association between the central theme of each topic cluster and other topics besides the central theme. The central theme is the topic with the highest sum of similarity among multiple words in at least one topic corresponding to the topic cluster. Thus, the topic similarity among multiple topics corresponding to the target text and the topic consistency of each topic can be determined. Based on the topic similarity and topic consistency, multiple topics are merged. The topic popularity and topic novelty of at least one topic cluster are calculated. Then, based on the topic popularity and topic novelty, the hot topics corresponding to the target text are determined. The semantic and grammatical associations between words in each topic are considered, which improves the accuracy of the extracted hot topics. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0026] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of a hot topic extraction method provided in an embodiment of this disclosure;

[0028] Figure 2 This is a flowchart of a topic merging processing method provided in an embodiment of this disclosure;

[0029] Figure 3 This is a flowchart illustrating a method for calculating topic popularity and topic novelty provided in an embodiment of this disclosure;

[0030] Figure 4 This is a schematic diagram of the structure of a hot topic extraction device provided in an embodiment of this disclosure;

[0031] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0032] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0033] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0034] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0035] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0036] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0037] In existing research on hot topic extraction, the main method employed is the probabilistic topic model-based method. This method uses probability distribution models such as PLSA and LDA to statistically analyze the probabilities of topics and words in text data, and then determines whether a topic is a hot topic based on the probability of these probabilities. However, this probabilistic topic model-based method ignores the semantic and syntactic relationships between words in the text, leading to inaccurate hot topic extraction. To address this issue, this disclosure provides a hot topic extraction method, which is described below with specific embodiments.

[0038] Figure 1 This is a flowchart of a hot topic extraction method provided in an embodiment of the present disclosure. The method can be executed by a hot topic extraction device, which can be implemented in software and / or hardware. The hot topic extraction device can be configured in an electronic device, such as a server or terminal, wherein the terminal specifically includes a mobile phone, computer or tablet computer, etc.

[0039] like Figure 1 As shown, the hot topic extraction method provided in this embodiment includes the following steps.

[0040] S110, Obtain the target text.

[0041] In this embodiment of the disclosure, the target text is any text for which hot topics need to be extracted.

[0042] In some embodiments of this disclosure, after receiving a hot topic extraction instruction, the electronic device retrieves the text corresponding to the file identifier from a preset database based on the file identifier corresponding to the hot topic extraction instruction, and then obtains the target text.

[0043] In other embodiments of this disclosure, the electronic device can obtain the text to be extracted from the hot topic from a preset database in real time and identify it as the target text.

[0044] S120. Obtain multiple topics corresponding to the target text and the word probability distribution vector corresponding to each topic.

[0045] In this embodiment of the disclosure, the word probability distribution vector is used to characterize the probability distribution vector of each word in each topic, and can also be understood as a vector used to characterize the probability that each word belongs to the topic.

[0046] Specifically, after acquiring the target text, the electronic device inputs the target text into the neural topic model, which outputs multiple topics corresponding to the target text and the word probability distribution vector corresponding to each topic, thereby obtaining multiple topics corresponding to the target text and the word probability distribution vector corresponding to each topic.

[0047] The neural topic model can be an existing topic model, such as the LDA (Latent Dirichlet Allocation) model or the ProdLDA neural topic model. The ProdLDA neural topic model uses the Variational Autoencoder (VAE) framework, which implements topic modeling through an encoder network and a decoder network.

[0048] S130. Calculate the topic similarity between each topic in multiple topics based on the word probability distribution vector, and calculate the topic consistency corresponding to each topic. The topic consistency is determined based on the similarity between multiple words corresponding to the topic.

[0049] Specifically, after acquiring the word probability distribution vector, the electronic device inputs the word probability distribution vector into a preset topic similarity calculation formula for calculation, thereby obtaining the topic similarity between various topics corresponding to the target text. The preset topic similarity calculation formula is as follows:

[0050]

[0051] Where k and k' represent any two topics in the target text; sim(k, k') represents the topic similarity between topics k and k'; Φ k,v Φ represents the probability of word v in topic k, that is, the probability that word v belongs to topic k; k',v V represents the probability of word v in topic k', that is, the probability that word v belongs to topic k'; V represents the total number of topics in the target text.

[0052] Furthermore, the topic consistency for each topic is determined based on the similarity between multiple words contained in each topic. The preset formula for calculating topic consistency is as follows:

[0053]

[0054] Where C(K) represents the topic consistency of topic k; |T k | represents the number of words contained in topic k; sim(v, u) represents the similarity between words v and u.

[0055] The similarity between words v and u can be calculated using cosine similarity, which is similar to existing methods for calculating the similarity between two words, and will not be elaborated upon here.

[0056] S140. Based on topic similarity and topic consistency, multiple topics are merged to obtain at least one topic cluster.

[0057] Specifically, after obtaining the topic similarity between various topics and the topic consistency corresponding to each topic, the electronic device merges multiple topics based on topic similarity and topic consistency to obtain at least one topic cluster.

[0058] Among them, hierarchical clustering algorithms can be used to merge multiple topics, or existing clustering algorithms can be used to merge multiple topics; there is no restriction here.

[0059] S150. Calculate the topic popularity and topic novelty of each topic cluster in at least one topic cluster. Determine the hot topics corresponding to the target text based on the topic popularity and topic novelty. The topic popularity is used to characterize the number of times the central topic corresponding to each topic cluster appears in the target text. The topic novelty is used to characterize the degree of association between the central topic corresponding to each topic cluster and other topics other than the central topic. The central topic is the topic with the highest sum of similarity among multiple words corresponding to at least one topic in the topic cluster.

[0060] In this embodiment of the disclosure, the higher the degree of correlation between the central topic and other topics besides the central topic, the lower the topic novelty, and vice versa.

[0061] Specifically, after merging multiple topics to obtain at least one topic cluster, the electronic device determines the central topic corresponding to each topic cluster. The popularity of each topic cluster is determined based on the number of times the central topic appears in the target text. At the same time, the novelty of each topic cluster is determined based on the similarity between the central topic and other topics and the topic consistency with other topics. After obtaining the topic popularity and topic novelty of each topic cluster, the heat value of each topic cluster is determined based on the topic popularity and topic novelty. The heat values ​​are sorted in descending order, and the obtained heat values ​​are compared with a preset heat value threshold. Target topic clusters with heat values ​​greater than or equal to the preset heat value threshold are determined, and the central topic corresponding to the target topic cluster is determined as the hot topic corresponding to the target text.

[0062] In this embodiment, a target text can be acquired. After acquiring the target text, multiple topics corresponding to the target text and word probability distribution vectors corresponding to each topic are obtained. Based on the word probability distribution vectors, topic similarity among the multiple topics is calculated, and topic consistency for each topic is calculated. Topic consistency is determined based on the similarity among the multiple words corresponding to the topic. Based on topic similarity and topic consistency, the multiple topics are merged to obtain at least one topic cluster. The topic popularity and topic novelty of each topic cluster are calculated. Based on topic popularity and topic novelty, hot topics corresponding to the target text are determined. Topic popularity is used to characterize the central topic corresponding to each topic cluster in the target text. The frequency of occurrence and topic novelty are used to characterize the degree of association between the central topic corresponding to each topic cluster and other topics besides the central topic. The central topic is the topic with the highest sum of similarity among multiple words in at least one topic corresponding to the topic cluster. Thus, the topic similarity among multiple topics corresponding to the target text and the topic consistency of each topic can be determined. Based on the topic similarity and topic consistency, multiple topics are merged. The topic popularity and topic novelty of the obtained at least one topic cluster are calculated. Then, based on the topic popularity and topic novelty, the hot topics corresponding to the target text are determined. The semantic and grammatical associations between words in each topic are considered, which improves the accuracy of the extracted hot topics.

[0063] Based on the above embodiments of this disclosure, obtaining multiple topics corresponding to the target text and the word probability distribution vector corresponding to each topic in S120 can specifically include: inputting the target text into a bag-of-words model, and having the bag-of-words model output multiple words corresponding to the target text and the word frequency corresponding to each word, where the word frequency is used to represent the number of times a word appears in the target text; inputting the target text and multiple words into a semantic analysis model, and having the semantic analysis model output a first vector corresponding to each word among the multiple words corresponding to the target text, and multiplying the first vector by the word frequency corresponding to each word to obtain a second vector corresponding to the target text; inputting the second vector into a neural topic model, and having the neural topic model output multiple topics corresponding to the target text and the word probability distribution vector corresponding to each topic, where the word probability distribution vector is used to represent the probability that each word in each topic belongs to that topic.

[0064] In this embodiment of the disclosure, the bag-of-words model can be an existing bag-of-words model. The specific implementation method for obtaining multiple words corresponding to the target text and the word frequency of each word based on the bag-of-words model is similar to the implementation method of the existing bag-of-words model, and will not be described in detail here.

[0065] The semantic analysis model can be any existing model capable of semantic analysis, such as the BERT model, and there are no restrictions here.

[0066] Specifically, the formula for representing the second vector corresponding to the target text is as follows:

[0067]

[0068] Among them, V d Wv represents the second vector corresponding to the target text; Wv represents the first vector corresponding to the v-th word; X d,v This indicates the number of times the v-th word appears in the target text d.

[0069] In this embodiment of the disclosure, the word probability distribution vector corresponding to each topic can be obtained based on the bag-of-words model, semantic analysis model and neural topic model. The obtained word probability distribution vector takes into account the semantic and grammatical correlation of words in the text, thereby improving the accuracy of the extracted hot topics.

[0070] Furthermore, Figure 2 This is a flowchart of a topic merging processing method provided in an embodiment of this disclosure, such as... Figure 2 As shown, based on topic similarity and topic consistency, multiple topics are merged to obtain at least one topic cluster. This can specifically include the following steps:

[0071] S210. Identify the two primary themes with the highest thematic similarity, and merge the two primary themes to obtain the first cluster.

[0072] Specifically, the electronic device determines the two first topics with the highest topic similarity based on the topic similarity between various topics, and merges the two topics according to the hierarchical clustering algorithm to obtain the first cluster, which includes the two first topics.

[0073] For example, if the target text contains 5 topics, then based on the topic similarity between each topic, the two first topics with the highest topic similarity are determined. For example, topic 4 and topic 5 have the highest topic similarity. In this case, topic 4 and topic 5 are determined as the first topics, and then they are merged to obtain the first cluster.

[0074] S220. Calculate the first topic consistency corresponding to the first cluster, and sum and average the first topic consistency with the topic consistency corresponding to at least one second topic among multiple topics (excluding the two first topics) to obtain the first average value corresponding to the first cluster.

[0075] Specifically, the specific implementation method for calculating the consistency of the first topic corresponding to the first cluster is similar to the implementation method for calculating the consistency of each topic as described above, and will not be repeated here.

[0076] For example, the first topic consistency corresponding to the first cluster obtained by merging topic 4 and topic 5 is added to the topic consistency corresponding to topic 1, topic 2 and topic 3 respectively, and then the average value is calculated to obtain the first average value corresponding to the first cluster.

[0077] S230. Compare the first average value with a preset topic consistency threshold, and determine at least one topic cluster based on the comparison result.

[0078] In this embodiment of the disclosure, the preset topic consistency threshold is a pre-set consistency threshold used to determine whether to continue merging multiple topics.

[0079] Specifically, the first average value is compared with a preset topic consistency threshold, and at least one topic cluster is determined based on the comparison result. When the first average value is less than the preset topic consistency threshold, step S240 is executed. When the first average value is greater than or equal to the preset topic consistency threshold, steps S250-S260 are executed.

[0080] S240. When the first average value is less than a preset topic consistency threshold, the first cluster and at least one second topic are determined as at least one topic cluster.

[0081] For example, when the first average value corresponding to the first cluster is less than the preset topic consistency threshold, the merging process of multiple topics is stopped. At this time, the first cluster, as well as topic 1, topic 2 and topic 3, are determined as the topic clusters obtained after merging multiple topics. At this time, the number of topic clusters is 4.

[0082] S250. When the first average value is greater than or equal to the preset topic consistency threshold, calculate the topic similarity between the first cluster and each topic in at least one second topic, determine the target topic with the highest topic similarity to the first cluster, and merge the first cluster and the target topic to obtain the second cluster.

[0083] In this embodiment of the disclosure, the specific implementation of calculating the topic similarity between the first cluster and each topic in at least one second topic is similar to the implementation of calculating the topic similarity between each topic described above, and will not be repeated here.

[0084] For example, the topic similarity of the first cluster to topic 1 is calculated, the topic similarity to topic 2 is calculated, and the topic similarity to topic 3 is calculated. For example, the topic similarity to topic 3 is the highest. At this time, the first cluster is merged with topic 3 to obtain the second cluster, which includes topic 3, topic 4 and topic 5.

[0085] S260. Calculate the second topic consistency corresponding to the second cluster, determine the second average value corresponding to the second cluster based on the second topic consistency, compare the second average value with the preset topic consistency threshold, repeat the above topic merging process until the target average value of the merged target cluster is less than the preset topic consistency threshold, stop the topic merging process, and determine the target cluster and the topics other than the topic corresponding to the target cluster as at least one topic cluster.

[0086] In this embodiment, the calculation of the second topic consistency corresponding to the second cluster and the determination of the second average value corresponding to the second cluster based on the second topic consistency corresponding to the second cluster are similar to the specific implementation of calculating the first topic consistency corresponding to the first cluster and the first average value corresponding to the first cluster in the above embodiments, and will not be repeated here.

[0087] For example, when the second average value of the second cluster is greater than or equal to a preset topic consistency threshold, the topic similarity 4 between the second cluster and topic 1 and topic similarity 5 between the second cluster and topic 2 are calculated. The topic with the higher similarity value between topic similarity 4 and topic similarity 5 is identified as the topic to be merged with the second cluster. The merging process continues until the target average value corresponding to the merged target cluster is less than the preset topic consistency threshold. The topic merging process is then stopped, and the target cluster and the topics other than the topic corresponding to the target cluster are identified as at least one topic cluster. When the second average value of the second cluster is less than the preset topic consistency threshold, the topic merging process is stopped directly, and the second cluster, topic 1, and topic 2 are identified as at least one topic cluster.

[0088] In this embodiment of the disclosure, multiple topics can be merged, and the final merging method can be selected to obtain at least one topic cluster that is maximized. Then, hot topics are extracted based on at least one topic cluster, which further improves the accuracy of the extracted hot topics.

[0089] In this disclosure, the calculation methods for topic popularity and topic novelty are described in detail, such as... Figure 3 The diagram shows a flowchart for calculating topic popularity and topic novelty. The specific steps are as follows:

[0090] S310. Determine the central theme corresponding to each theme cluster.

[0091] In this embodiment of the disclosure, before calculating the topic popularity and topic novelty corresponding to each topic cluster, the electronic device needs to first determine the central topic corresponding to each topic cluster.

[0092] In some embodiments of this disclosure, when a topic cluster contains a topic, that topic is determined as the central topic corresponding to the topic cluster.

[0093] For example, if the topic cluster contains only topic 1, then topic 1 is directly determined as the central topic.

[0094] In other embodiments of this disclosure, when a topic cluster contains multiple topics, for each topic, the word similarity between multiple words in the topic is calculated, the average value of the word similarity corresponding to each topic is determined based on the word similarity, and the topic consistency corresponding to the topic is determined at the same time; based on the average value of the word similarity corresponding to the topic and the topic consistency corresponding to the topic, the target value corresponding to the topic is obtained, and the topic with the highest target value among the multiple topics is determined as the central topic corresponding to the topic cluster.

[0095] For example, taking the first cluster described above as an example, the first cluster includes topic 4 and topic 5. To determine the central topic of the first cluster, for topic 4 and topic 5, the word similarity between multiple words in topic 4 is calculated, and the average of the word similarity between multiple words is obtained to get the average value of the word similarity corresponding to topic 4. At the same time, the topic consistency corresponding to topic 4 is obtained. The average value of the word similarity corresponding to topic 4 and the topic consistency corresponding to topic 4 are substituted into the preset central topic determination formula to obtain the first value. Similarly, the average value of the word similarity corresponding to topic 5 is calculated, and the topic consistency corresponding to topic 5 is obtained. The average value of the word similarity corresponding to topic 5 and the topic consistency corresponding to topic 5 are substituted into the preset central topic determination formula to obtain the second value. The first value and the second value are compared. If the first value is greater than the second value, then topic 4 is determined as the central topic of the first cluster.

[0096] Furthermore, the pre-defined formula for determining the central theme is as follows:

[0097]

[0098] Where m represents the total number of words in the topic; AvgSim(C i ) represents the topic cluster C i The average similarity of multiple corresponding words, AvgC(C i ) represents cluster C i The corresponding topic consistency; α is a pre-set weight parameter used to balance the influence of the average similarity of multiple words in the topic cluster on topic consistency.

[0099] S320. For each topic cluster, determine the number of times the central topic corresponding to the topic cluster appears in the target text, and determine the number of times as the topic popularity of the topic cluster.

[0100] Specifically, after determining the central topic corresponding to each topic cluster, the electronic device obtains the number of times the central topic corresponding to each topic cluster appears in the target text, and determines the popularity of the topic corresponding to the topic cluster by the number of times the central topic appears in the target text.

[0101] S330. When a topic cluster contains a topic, the topic novelty of the topic cluster is set to 1.

[0102] Specifically, when an electronic device determines that a topic cluster contains a topic, since there are no other topics in the topic cluster besides the central topic, the topic novelty of the topic cluster is set to 1.

[0103] S340. When a topic cluster contains multiple topics, calculate the similarity between the central topic corresponding to the topic cluster and the target topics corresponding to the other topics in the topic cluster, excluding the central topic. Determine the topic novelty corresponding to the topic cluster based on the target topic similarity and the topic consistency corresponding to the other topics, excluding the central topic.

[0104] Specifically, when an electronic device determines that a topic cluster contains multiple topics, it calculates the similarity between the central topic corresponding to the topic cluster and the target topics corresponding to the other topics in the topic cluster, excluding the central topic. The similarity between the target topics and the consistency between the topics corresponding to the other topics, excluding the central topic, are then substituted into a preset topic novelty calculation formula to calculate the topic novelty corresponding to the topic cluster.

[0105] Furthermore, the pre-defined formula for calculating the novelty of a topic is as follows:

[0106]

[0107] Where N(k) represents the topic novelty corresponding to the topic cluster containing the central topic k; sim(k, k') represents the target topic similarity between the central topic k and any other topic k' excluding the central topic k; and C(k') represents the topic consistency corresponding to topic k'.

[0108] It should be noted that there is no restriction on the execution order of steps S320, S330, and S340. Step S320 can be executed simultaneously with steps S330 and S340, or it can be executed after steps S330 and S340.

[0109] In the embodiments of this disclosure, the central topic corresponding to each topic cluster can be determined before calculating topic novelty and topic popularity. Based on the central topic corresponding to each topic cluster and the number of topics corresponding to each topic cluster, the topic popularity and topic novelty corresponding to each topic cluster are calculated respectively, which improves the flexibility and accuracy of calculating the topic popularity and topic novelty corresponding to each topic cluster.

[0110] Based on the above embodiments of this disclosure, determining the hot topics corresponding to the target text based on topic popularity and topic novelty can specifically include: determining the popularity value corresponding to each topic cluster in at least one topic cluster based on topic popularity and topic novelty; comparing the popularity value corresponding to each topic cluster in at least one topic cluster with a preset popularity value threshold, and determining the central topic corresponding to the target topic cluster whose popularity value is greater than or equal to the preset popularity value threshold as the hot topic corresponding to the target text.

[0111] Specifically, after obtaining the popularity and novelty of each topic cluster, the electronic device performs a weighted average of the popularity and novelty to obtain the heat value of each topic cluster. From this, it selects target topic clusters whose heat values ​​are greater than or equal to a preset heat value threshold, and determines the central topic corresponding to the target topic cluster as the hot topic corresponding to the target text.

[0112] Furthermore, the formula for calculating the popularity value corresponding to each topic cluster is as follows:

[0113] H(k)=α′P(k)+(1-α′)N(k)

[0114] Where H(k) represents the popularity value corresponding to the topic cluster; P(k) represents the popularity of the topic cluster containing the central topic k; N(k) represents the novelty of the topic cluster containing the central topic K; ɑ' is a preset weight parameter used to balance the influence of topic novelty and topic popularity. The value range of ɑ' is [0,1], and the specific value can be adjusted according to the specific situation.

[0115] In this embodiment of the disclosure, the hot topics corresponding to the target text can be determined by combining the popularity and novelty of each topic cluster. The degree of correlation between the central topic of each topic cluster and other topics other than the central topic, as well as the similarity between multiple words in at least one topic of the topic cluster, are considered, thereby improving the accuracy of the obtained hot topics.

[0116] Figure 4 This is a schematic diagram of a hot topic extraction device provided in an embodiment of this disclosure.

[0117] In this embodiment, the hot topic extraction device can be installed within an electronic device and is understood as a functional module within the aforementioned electronic device. Specifically, the electronic device can be a server or a terminal, wherein the terminal specifically includes mobile phones, computers, or tablet computers, etc., without limitation.

[0118] like Figure 4As shown, the hot topic extraction device 400 may include a first acquisition module 410, a second acquisition module 420, a calculation module 430, a merging processing module 440, and a topic determination module 450.

[0119] The first acquisition module 410 can be used to acquire target text.

[0120] The second acquisition module 420 can be used to acquire multiple topics corresponding to the target text and the word probability distribution vector corresponding to each topic.

[0121] The calculation module 430 can be used to calculate the topic similarity between various topics in multiple topics based on the word probability distribution vector, and to calculate the topic consistency corresponding to each topic. The topic consistency is determined based on the similarity between multiple words corresponding to the topic.

[0122] The merging processing module 440 can be used to merge multiple topics based on topic similarity and topic consistency to obtain at least one topic cluster.

[0123] The topic determination module 450 can be used to calculate the topic popularity and topic novelty of each topic cluster in at least one topic cluster, and determine the hot topics corresponding to the target text based on the topic popularity and topic novelty. The topic popularity is used to characterize the number of times the central topic corresponding to each topic cluster appears in the target text, and the topic novelty is used to characterize the degree of association between the central topic corresponding to each topic cluster and other topics other than the central topic. The central topic is the topic with the highest sum of similarity among multiple words corresponding to at least one topic in the topic cluster.

[0124] In this embodiment, a target text can be acquired. After acquiring the target text, multiple topics corresponding to the target text and word probability distribution vectors corresponding to each topic are obtained. Based on the word probability distribution vectors, topic similarity among the multiple topics is calculated, and topic consistency for each topic is calculated. Topic consistency is determined based on the similarity among the multiple words corresponding to the topic. Based on topic similarity and topic consistency, the multiple topics are merged to obtain at least one topic cluster. The topic popularity and topic novelty of each topic cluster are calculated. Based on topic popularity and topic novelty, hot topics corresponding to the target text are determined. Topic popularity is used to characterize the central topic corresponding to each topic cluster in the target text. The frequency of occurrence and topic novelty are used to characterize the degree of association between the central topic corresponding to each topic cluster and other topics besides the central topic. The central topic is the topic with the highest sum of similarity among multiple words in at least one topic corresponding to the topic cluster. Thus, the topic similarity among multiple topics corresponding to the target text and the topic consistency of each topic can be determined. Based on the topic similarity and topic consistency, multiple topics are merged. The topic popularity and topic novelty of the obtained at least one topic cluster are calculated. Then, based on the topic popularity and topic novelty, the hot topics corresponding to the target text are determined. The semantic and grammatical associations between words in each topic are considered, which improves the accuracy of the extracted hot topics.

[0125] In some embodiments of this disclosure, the second acquisition module 420 may be specifically used to input the target text into a bag-of-words model, which outputs multiple words corresponding to the target text and the word frequency corresponding to each word, where the word frequency is used to characterize the number of times a word appears in the target text; input the target text and multiple words into a semantic analysis model, which outputs a first vector corresponding to each word among the multiple words corresponding to the target text, and multiplies the first vector by the word frequency corresponding to each word to obtain a second vector corresponding to the target text; input the second vector into a neural topic model, which outputs multiple topics corresponding to the target text and the word probability distribution vector corresponding to each topic, where the word probability distribution vector is used to characterize the probability that each word in each topic belongs to the topic.

[0126] In some embodiments of this disclosure, the merging processing module 440 may be specifically used to determine the two first topics with the highest topic similarity, merge the two first topics to obtain a first cluster; calculate the first topic consistency corresponding to the first cluster, sum and average the first topic consistency with the topic consistency corresponding to at least one second topic among the multiple topics other than the two first topics to obtain a first average value corresponding to the first cluster; compare the first average value with a preset topic consistency threshold, and determine at least one topic cluster based on the comparison result; when the first average value is less than the preset topic consistency threshold, determine the first cluster and at least one second topic as at least one topic cluster; when the first average value is greater than or equal to the preset topic consistency threshold, calculate the topic similarity between the first cluster and each topic among the at least one second topic, determine the target topic with the highest topic similarity to the first cluster, merge the first cluster and the target topic to obtain a second cluster; calculate the second topic consistency corresponding to the second cluster, determine the second average value corresponding to the second cluster based on the second topic consistency corresponding to the second cluster, compare the second average value with the preset topic consistency threshold, and repeat the above topic merging process until the target average value of the merged target cluster is less than the preset topic consistency threshold, then stop the topic merging process, and determine the target cluster and the topics among the multiple topics other than the topic corresponding to the target cluster as at least one topic cluster.

[0127] In some embodiments of this disclosure, the hot topic extraction device 400 may further include a central topic determination module.

[0128] The central topic determination module can be used to determine the central topic corresponding to each topic cluster before calculating the topic popularity and topic novelty corresponding to each topic cluster in at least one topic cluster.

[0129] Furthermore, the central topic determination module can be specifically used for each topic cluster. When a topic cluster contains one topic, the topic is determined as the central topic corresponding to the topic cluster. When a topic cluster contains multiple topics, for each topic, the word similarity between multiple words in the topic is calculated, and the average value of the word similarity corresponding to each topic is determined based on the word similarity. At the same time, the topic consistency corresponding to the topic is determined. Based on the average value of the word similarity corresponding to the topic and the topic consistency corresponding to the topic, the target value corresponding to the topic is obtained, and the topic with the highest target value among multiple topics is determined as the central topic corresponding to the topic cluster.

[0130] In some embodiments of this disclosure, the topic determination module 450 may be specifically used to determine the number of times the central topic corresponding to the topic cluster appears in the target text for each topic cluster, and determine the number of times as the topic popularity of the topic cluster; when the topic cluster contains one topic, the topic novelty of the topic cluster is determined to be 1; when the topic cluster contains multiple topics, the similarity between the central topic corresponding to the topic cluster and the target topics corresponding to the other topics in the topic cluster (excluding the central topic) is calculated, and the topic novelty of the topic cluster is determined based on the target topic similarity and the topic consistency of the other topics (excluding the central topic).

[0131] In some embodiments of this disclosure, the topic determination module 450 may also be specifically used to determine the heat value corresponding to each topic cluster in at least one topic cluster based on topic popularity and topic novelty; compare the heat value corresponding to each topic cluster in at least one topic cluster with a preset heat value threshold respectively, and determine the central topic corresponding to the target topic cluster whose heat value is greater than or equal to the preset heat value threshold as the hot topic corresponding to the target text.

[0132] It should be noted that, Figure 4 The hot topic extraction device 400 shown can execute the various steps in the above method embodiments and achieve the various processes and effects in the above method embodiments, which will not be elaborated here.

[0133] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0134] In this embodiment of the disclosure, Figure 5 The electronic devices shown can be servers or terminals, and terminals specifically include mobile phones, computers, or tablets, etc., without limitation.

[0135] like Figure 5 As shown, the electronic device may include a processor 510 and a memory 520 storing computer program instructions.

[0136] Specifically, the processor 510 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this disclosure.

[0137] Memory 520 may include a large-capacity storage for information or instructions. For example, and not limitingly, memory 520 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 520 may include removable or non-removable (or fixed) media. Where appropriate, memory 520 may be internal or external to the integrated gateway device. In a particular embodiment, memory 520 is a non-volatile solid-state memory. In a particular embodiment, memory 520 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (Electrically Programmable ROM, EPROM), an electrically erasable programmable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0138] The processor 510 reads and executes computer program instructions stored in the memory 520 to perform the steps of the hot topic extraction method provided in this embodiment of the disclosure.

[0139] In one example, the electronic device may also include a transceiver 530 and a bus 540. Wherein, as... Figure 5 As shown, the processor 510, memory 520 and transceiver 530 are connected via bus 540 and communicate with each other.

[0140] Bus 540 may include hardware, software, or both. For example, and not limited to, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 540 may include one or more buses.

[0141] This disclosure also provides a computer-readable storage medium that can store a computer program. When the computer program is executed by a processor, the processor implements the hot topic extraction method provided in this disclosure.

[0142] The aforementioned storage medium may, for example, include a memory 520 containing computer program instructions, which can be executed by a processor 510 of an electronic device to complete the hot topic extraction method provided in this embodiment. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), compact disc-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device.

[0143] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for extracting trending topics, characterized in that, include: Obtain the target text; Obtain multiple topics corresponding to the target text and the word probability distribution vector corresponding to each topic; The topic similarity between each topic in the plurality of topics is calculated based on the word probability distribution vector, and the topic consistency corresponding to each topic is calculated. The topic consistency is determined based on the similarity between multiple words corresponding to the topic. Based on the topic similarity and topic consistency, the multiple topics are merged to obtain at least one topic cluster; Calculate the topic popularity and topic novelty of each topic cluster in the at least one topic cluster, and determine the hot topics corresponding to the target text based on the topic popularity and topic novelty. The topic popularity is used to characterize the number of times the central topic corresponding to each topic cluster appears in the target text, and the topic novelty is used to characterize the degree of association between the central topic corresponding to each topic cluster and other topics other than the central topic. The central topic is the topic with the highest sum of similarity among multiple words corresponding to at least one topic in the topic cluster. The step of obtaining multiple topics corresponding to the target text and the word probability distribution vector corresponding to each topic includes: The target text is input into the bag-of-words model, which outputs multiple words corresponding to the target text and the word frequency of each word. The word frequency is used to represent the number of times the word appears in the target text. The target text and the plurality of words are input into the semantic analysis model, and the semantic analysis model outputs the first vector corresponding to each of the plurality of words in the target text. The first vector is multiplied by the word frequency corresponding to each word to obtain the second vector corresponding to the target text. The second vector is input into the neural topic model, which outputs multiple topics corresponding to the target text and a word probability distribution vector corresponding to each topic. The word probability distribution vector is used to represent the probability that each word in each topic belongs to the topic. Before calculating the topic popularity and topic novelty corresponding to each of the at least one topic cluster, the method further includes: For each topic cluster, when the topic cluster contains a topic, the topic is determined as the central topic corresponding to the topic cluster; When the topic cluster contains multiple topics, for each of the multiple topics, the word similarity between multiple words in the topic is calculated, the average value of the word similarity corresponding to each topic is determined based on the word similarity, and the topic consistency corresponding to the topic is determined at the same time; based on the average value of the word similarity corresponding to the topic and the topic consistency corresponding to the topic, the target value corresponding to the topic is obtained, and the topic with the highest target value among the multiple topics is determined as the central topic corresponding to the topic cluster; Calculating the topic popularity and topic novelty of each topic cluster in the at least one topic cluster includes: For each topic cluster, determine the number of times the central topic corresponding to the topic cluster appears in the target text, and determine the number of times as the topic popularity corresponding to the topic cluster; When the topic cluster contains a topic, the topic novelty of the topic cluster is determined to be 1; When the topic cluster contains multiple topics, the similarity between the central topic corresponding to the topic cluster and the target topics corresponding to the other topics in the topic cluster (excluding the central topic) is calculated. The topic novelty corresponding to the topic cluster is determined based on the target topic similarity and the topic consistency corresponding to the other topics (excluding the central topic).

2. The method according to claim 1, characterized in that, The process of merging the multiple topics based on topic similarity and topic consistency to obtain at least one topic cluster includes: Identify the two first topics with the highest topic similarity, and merge the two first topics to obtain the first cluster; Calculate the first topic consistency corresponding to the first cluster, and sum and average the first topic consistency with the topic consistency corresponding to at least one second topic other than the two first topics among the plurality of topics to obtain the first average value corresponding to the first cluster; The first average value is compared with a preset topic consistency threshold, and the at least one topic cluster is determined based on the comparison result.

3. The method according to claim 2, characterized in that, Determining the at least one topic cluster based on the comparison results includes: When the first average value is less than the preset topic consistency threshold, the first cluster and the at least one second topic are determined as the at least one topic cluster; When the first average value is greater than or equal to the preset topic consistency threshold, the topic similarity between the first cluster and each topic in the at least one second topic is calculated, the target topic with the highest topic similarity to the first cluster is determined, and the first cluster and the target topic are merged to obtain the second cluster; Calculate the second topic consistency corresponding to the second cluster, determine the second average value corresponding to the second cluster based on the second topic consistency, compare the second average value with the preset topic consistency threshold, repeat the above topic merging process until the target average value of the merged target cluster is less than the preset topic consistency threshold, then stop the topic merging process, and determine the target cluster and the topics other than the topic corresponding to the target cluster as the at least one topic cluster.

4. The method according to claim 1, characterized in that, The process of determining the hot topics corresponding to the target text based on the topic popularity and the topic novelty includes: The popularity value of each topic cluster in the at least one topic cluster is determined based on the topic popularity and the topic novelty; The popularity value corresponding to each topic cluster in the at least one topic cluster is compared with a preset popularity value threshold, and the central topic corresponding to the target topic cluster whose popularity value is greater than or equal to the preset popularity value threshold is determined as the hot topic corresponding to the target text.

5. A device for extracting trending topics, characterized in that, include: The first acquisition module is used to acquire the target text; The second acquisition module is used to acquire multiple topics corresponding to the target text and the word probability distribution vector corresponding to each topic; The calculation module is used to calculate the topic similarity between each topic in the plurality of topics based on the word probability distribution vector, and to calculate the topic consistency corresponding to each topic, wherein the topic consistency is determined based on the similarity between multiple words corresponding to the topic; The merging processing module is used to merge the multiple topics based on the topic similarity and the topic consistency to obtain at least one topic cluster; The topic determination module is used to calculate the topic popularity and topic novelty of each topic cluster in the at least one topic cluster, and determine the hot topics corresponding to the target text based on the topic popularity and topic novelty. The topic popularity is used to characterize the number of times the central topic corresponding to each topic cluster appears in the target text, and the topic novelty is used to characterize the degree of association between the central topic corresponding to each topic cluster and other topics other than the central topic. The central topic is the topic with the highest sum of similarity among multiple words corresponding to at least one topic in the topic cluster. The second acquisition module is specifically used to input the target text into the bag-of-words model, and the bag-of-words model outputs multiple words corresponding to the target text and the word frequency of each word. The word frequency is used to represent the number of times the word appears in the target text. The target text and the plurality of words are input into the semantic analysis model, and the semantic analysis model outputs the first vector corresponding to each of the plurality of words in the target text. The first vector is multiplied by the word frequency corresponding to each word to obtain the second vector corresponding to the target text. The second vector is input into the neural topic model, which outputs multiple topics corresponding to the target text and a word probability distribution vector corresponding to each topic. The word probability distribution vector is used to represent the probability that each word in each topic belongs to the topic. The central topic determination module is specifically used to determine the central topic corresponding to the topic cluster when the topic cluster contains a topic, before calculating the topic popularity and topic novelty of each topic cluster in the at least one topic cluster; When the topic cluster contains multiple topics, for each of the multiple topics, the word similarity between multiple words in the topic is calculated, the average value of the word similarity corresponding to each topic is determined based on the word similarity, and the topic consistency corresponding to the topic is determined at the same time; based on the average value of the word similarity corresponding to the topic and the topic consistency corresponding to the topic, the target value corresponding to the topic is obtained, and the topic with the highest target value among the multiple topics is determined as the central topic corresponding to the topic cluster; The topic determination module is specifically used to determine, for each topic cluster, the number of times the central topic corresponding to the topic cluster appears in the target text, and to determine the number of times as the topic popularity of the topic cluster. When the topic cluster contains a topic, the topic novelty of the topic cluster is determined to be 1; When the topic cluster contains multiple topics, the similarity between the central topic corresponding to the topic cluster and the target topics corresponding to the other topics in the topic cluster (excluding the central topic) is calculated. The topic novelty corresponding to the topic cluster is determined based on the target topic similarity and the topic consistency corresponding to the other topics (excluding the central topic).

6. An electronic device, characterized in that, include: processor; Memory, used to store executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the hot topic extraction method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the hot topic extraction method according to any one of claims 1-4.

Images