Hot topic cluster generation method, device and equipment, medium and product

Abstract

The invention discloses a hot topic cluster generation method and device, equipment, a medium and a product. The method comprises the steps that a news data set meeting collection conditions is acquired; performing topic modeling clustering on the news data in the news data set to obtain a topic cluster set to which each piece of news data belongs; splitting and combining the topic cluster set, and determining candidate topic clusters; and determining a final hot topic cluster based on the large model and the candidate topic cluster. According to the method, splitting processing is carried out before the topic clusters are combined, high consistency of news in each topic cluster is ensured, the situation that irrelevant news or only slightly-relevant news is wrongly classified into the same topic cluster is avoided, and therefore the accuracy and purity of topics in the final hot topic cluster are improved, the title of the final hot topic cluster is determined on the basis of a large model, and the accuracy and purity of the topic are improved. And the quality of the title is improved, so that the quality of the final hot topic cluster is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment, medium and product for generating a hot topic cluster. Background Art

[0002] Generating trending topic clusters can significantly increase content exposure and traffic. Leveraging the high attention generated by trending topics, related content is more likely to be recommended by platform algorithms, attracting more clicks and interactions, thereby expanding influence.

[0003] Current methods for automatically generating hot topics primarily use a LDA-like model for topic extraction, followed by clustering using the Kmeans algorithm. To maintain the integrity of the clusters, clusters are merged, and finally, the TextRank algorithm is used to extract a news headline from the cluster as a representation of the hot topic.

[0004] However, these approaches have several issues: First, LDA-like models treat input news headlines as bag-of-words models, making them unable to handle synonyms and other situations. They also ignore the semantic relationships between words and fail to accurately capture the context of words within a sentence. Second, topic cluster merging presupposes that the news within the cluster is highly consistent. Directly merging topics without ensuring consistent representation within the clusters results in a lower purity of the topics within the merged clusters. Summary of the Invention

[0005] The present invention provides a method, device, equipment, medium and product for generating a hot topic cluster, so as to obtain a hot topic cluster with strong consistency and better quality.

[0006] According to a first aspect of the present invention, a method for generating a hot topic cluster is provided, comprising:

[0007] Obtain news datasets that meet collection conditions;

[0008] Performing topic modeling and clustering on the news data in the news data set to obtain a topic cluster to which each piece of news data belongs;

[0009] Splitting and merging the topic clusters to determine candidate topic clusters;

[0010] Based on the large model and the candidate topic clusters, the final hot topic cluster is determined.

[0011] According to a second aspect of the present invention, there is provided a device for generating a hot topic cluster, comprising:

[0012] Data acquisition module, used to obtain news data sets that meet the collection conditions;

[0013] A first generating module is used to perform topic modeling and clustering on the news data in the news data set to obtain a topic cluster to which each piece of news data belongs;

[0014] A second generating module is used to split and merge the topic clusters to determine candidate topic clusters;

[0015] The third generation module is used to determine the final hot topic cluster based on the large model and the candidate topic cluster.

[0016] According to a third aspect of the present invention, there is provided an electronic device, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can execute the hot topic cluster generating method according to any embodiment of the present invention.

[0020] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the hot topic cluster generation method according to any embodiment of the present invention when executed.

[0021] According to a fifth aspect of the present invention, an embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the hot topic cluster generation method of any embodiment of the present invention.

[0022] The technical solution of the embodiment of the present invention obtains a news dataset that meets the collection conditions; performs topic modeling and clustering on the news data in the news dataset to obtain the topic clusters to which each piece of news data belongs; splits and merges the topic clusters to determine candidate topic clusters; and determines the final hot topic cluster based on the large model and the candidate topic clusters. By performing the splitting process before merging the topic clusters, the high consistency of the news within each topic cluster is ensured, preventing unrelated or only weakly related news from being mistakenly classified into the same topic cluster, thereby improving the accuracy and purity of the topics in the final hot topic cluster. The titles of the final hot topic cluster are determined based on the large model, improving the quality of the titles and, in turn, the quality of the final hot topic cluster.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 This is a flow chart of a method for generating a hot topic cluster according to the first embodiment of the present invention;

[0026] Figure 2 This is a flow chart of a method for generating a hot topic cluster according to the second embodiment of the present invention;

[0027] Figure 3 This is a structural diagram of a hot topic cluster generating device provided by Embodiment 3 of the present invention;

[0028] Figure 4 It is a schematic structural diagram of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Example 1

[0032] Figure 1A flowchart of a method for generating a hot topic cluster is provided for the first embodiment of the present invention. This embodiment is applicable to the generation of hot topic clusters. The method can be executed by a hot topic cluster generation device. The hot topic cluster generation device can be implemented in the form of hardware and / or software. The hot topic cluster generation device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0033] S110: Acquire a news dataset that meets collection conditions.

[0034] In this embodiment, the collection condition can be understood as the most recent set number of days. The news data set can be understood as news data collected from multiple data sources, including news titles, news texts, news release time, news release sources, and news link URLs.

[0035] Specifically, the processor may obtain news data that meets the collection conditions and is collected from various data sources to form a news data set.

[0036] S120: Perform topic modeling and clustering on the news data in the news data set to obtain the topic cluster to which each piece of news data belongs.

[0037] In this embodiment, a topic cluster set can be understood as a collection of multiple topic clusters, and each topic cluster also includes multiple news items belonging to the same topic.

[0038] Specifically, the processor may first clean the news data in the news data set to ensure the quality of the news data, and then perform topic modeling and clustering on the news titles in the cleaned news data to obtain the topic clusters to which each piece of news data belongs.

[0039] S130: Split and merge topic clusters to determine candidate topic clusters.

[0040] In this embodiment, the candidate topic cluster can be understood as a topic cluster with high news consistency.

[0041] Specifically, the processor can first split the news headlines of each topic cluster in the topic cluster set, determine whether there are news headlines with inconsistent topics in the topic cluster, split it out as a new topic cluster, and then merge all the topic clusters obtained after the split judgment on all topic clusters, merge the topic clusters that can be merged, and obtain candidate topic clusters.

[0042] S140: Determine the final hot topic cluster based on the large model and the candidate topic clusters.

[0043] In this embodiment, the large model can be understood as a deep learning model with a parameter scale of billions or even trillions. The final hot topic cluster can be understood as a collection of news with titles and sorted by popularity.

[0044] Specifically, the processor can determine the relevance of each news headline in the candidate topic cluster to the topic corresponding to the candidate topic cluster, and input the top several news headlines as the basis for generating the title of the candidate topic cluster into the big model, generate the topic name of the candidate topic cluster through the big model, determine the popularity of the candidate topic cluster by the release time of each news headline in the candidate topic cluster, and obtain the final hot topic cluster. The popularity of multiple final hot topic clusters can be sorted and displayed by popularity.

[0045] The technical solution of the embodiment of the present invention obtains a news dataset that meets the collection conditions; performs topic modeling and clustering on the news data in the news dataset to obtain the topic clusters to which each piece of news data belongs; splits and merges the topic clusters to determine candidate topic clusters; and determines the final hot topic cluster based on the large model and the candidate topic clusters. By performing the splitting process before merging the topic clusters, the high consistency of the news within each topic cluster is ensured, preventing unrelated or only weakly related news from being mistakenly classified into the same topic cluster, thereby improving the accuracy and purity of the topics in the final hot topic cluster. The titles of the final hot topic cluster are determined based on the large model, improving the quality of the titles and, in turn, the quality of the final hot topic cluster.

[0046] Example 2

[0047] Figure 2 This is a flow chart of a method for generating a hot topic cluster provided by the second embodiment of the present invention. This embodiment is a further refinement of the above embodiment. Figure 2 As shown, the method includes:

[0048] S201: Acquire a news dataset that meets collection conditions.

[0049] S202: Preprocess the news data in the news data set to obtain processed news titles and form a news title set.

[0050] In this embodiment, the news headline set can be understood as a set of all processed news headlines, and the news headlines can be understood as a form of summarizing news facts and opinions through concise and eye-catching text.

[0051] Specifically, since the quality of original news data may vary, in order to ensure the quality of subsequent hot topics, the processor can first preprocess the news data in the news dataset, which may include publishing source screening, data normalization, deduplication and deletion of extremely long titles, etc., to obtain processed news titles and form a news title set.

[0052] Furthermore, based on the above embodiment, the steps of preprocessing the news data in the news data set, obtaining processed news headlines and forming a news headline set can be refined as follows:

[0053] Based on a pre-established blacklist, news publishing sources in the news data are filtered to obtain filtered first news data; the format of the first news titles in the first news data is normalized to obtain second news titles; the second news titles are deduplicated to obtain third news titles; the length of the third news titles is normalized to obtain processed news titles and form a news title set.

[0054] In this embodiment, the pre-established blacklist can be understood as a pre-set blacklist of publishing sources. Because the overall quality of news published by some publishing sources is low, a news publishing source blacklist can be pre-established and updated based on historical experience. The first news data can be understood as news data that is not on the pre-established blacklist after screening. The first news title can be understood as the news title in the first news data. The second news title can be understood as the news title after normalization. The third news title can be understood as the news title remaining after deduplication.

[0055] Specifically, the processor can obtain a pre-established blacklist, and for the newly collected news data, first check whether its publishing source is on the blacklist. If so, delete it to obtain the filtered first news data. The processor can convert the English in the first news title in the first news data to lowercase, convert punctuation marks to half-width, and delete extra spaces before and after the title to ensure data normalization. Since the same news may be published on multiple platforms, in order to accurately assess the popularity of the news, the same second news title needs to be deduplicated to obtain a third news title. The processor can statistically calculate the length distribution of the third news title after standardization, delete news titles whose length is less than a set threshold (for example, 0.05%) and greater than a set threshold (for example, 99.5%), and form a news title set.

[0056] S203: Input the news headline set into the text embedding model to determine a first output vector set.

[0057] In this embodiment, the text embedding model can be understood as a model for converting text into vectors. For example, it can be the paraphrase-multilingual-MiniLM-L12-v2 model to accommodate news headlines that are primarily in Chinese and may contain content in other languages. The first output vector set can be understood as a set of vectors.

[0058] Specifically, the processor inputs the news headline set into the text embedding model, and the output result of the text embedding model is used as the first output vector set.

[0059] S204 : Perform dimensionality reduction on each first output vector in the first output vector set to form a second output vector set.

[0060] In this embodiment, the second output vector set can be understood as a vector set whose dimensions meet the requirements.

[0061] Specifically, since the first output vector output by the text embedding model has a large dimension, which is not conducive to subsequent clustering, the processor can reduce the dimension of the first output vector using a dimensionality reduction algorithm to form a second output vector set. For example, the processor can use an algorithm such as UMAP to reduce the dimension of the vector representation of the text.

[0062] S205: Cluster the second output vector set to obtain a clustering result.

[0063] In this embodiment, the clustering result can be understood as the result of grouping titles with similar semantics or themes into the same cluster.

[0064] Specifically, the processor may cluster each second output vector in the second output vector set by using a clustering algorithm to obtain a clustering result. For example, the HDBSCAN algorithm may be used for clustering.

[0065] S206: Determine candidate topic words of the clustering results and combine them with the clustering results as topic clusters to which news titles in each news data belong.

[0066] In this embodiment, the topic candidate words can be understood as words used to characterize the overall topic of each clustering result.

[0067] Specifically, the processor can extract topic candidate words from the clustering results through an algorithm. For example, the topic candidate words can be extracted through the c-TF-IDF algorithm. The processor can use the clustering results as the topic clusters to which the news titles in each news data belong, and use the topic candidate words as the topics of the topic clusters to obtain topic clusters.

[0068] For example, the above-mentioned topic modeling and clustering method can adopt the BERTopic model, which extracts semantically rich topics from text data through three core steps: semantic embedding, dimensionality reduction clustering, and topic representation. Unlike the traditional LDA model based on word frequency, BERTopic uses the context-aware vectors generated by BERT to capture the complex relationships between words, significantly improving the processing capabilities of polysemous words and context-sensitive texts.

[0069] S207 , processing each topic cluster in the topic cluster set by a preset sorting algorithm to obtain an intermediate topic cluster sorted in order of topic relevance.

[0070] In this embodiment, the preset sorting algorithm can be understood as an algorithm for sorting news headlines within a topic cluster according to the degree of topic relevance, for example, it can be a TextRank algorithm. The TextRank algorithm establishes a relationship graph between text units (words or sentences) and uses a voting mechanism to sort important components in the text.

[0071] Specifically, the processor may process each topic cluster in the topic cluster set using a preset sorting algorithm to obtain intermediate topic clusters sorted in order of topic relevance.

[0072] S208: Use the news title ranked first in each intermediate topic cluster as a seed title.

[0073] In this embodiment, the seed title can be understood as the title that best represents the content to be represented by the intermediate topic cluster.

[0074] Specifically, the processor may use the news title ranked first in the topic relevance order in the intermediate topic cluster as the seed title of the intermediate topic cluster.

[0075] S209: Calculate similarity between the seed title and other news titles in the corresponding intermediate topic cluster to obtain similarity values ​​of the other news titles relative to the seed title.

[0076] In this embodiment, other news titles can be understood as news titles in the intermediate topic cluster other than the seed title. The similarity value can be understood as a numerical representation of the degree of similarity between the other news titles and the seed title.

[0077] Specifically, the processor can perform word segmentation and part-of-speech tagging on the seed title, and then accumulate different scores for each word after word segmentation according to special part-of-speech and common part-of-speech to obtain the score of the seed title. The processor can also perform word segmentation and part-of-speech tagging on other news headlines in the intermediate topic cluster to which the seed title belongs, calculate a score based on each word after word segmentation and each word in the seed title, and determine the similarity value of the other news headlines to the seed title based on the two scores.

[0078] Furthermore, based on the above embodiment, the step of calculating the similarity between the seed title and other news titles in the intermediate topic cluster to obtain the similarity value of each other news title relative to the seed title can be refined:

[0079] The seed title is processed through the lexical analysis model to obtain the first participle and the first part-of-speech tagging result of the first participle; for each other news title in the intermediate topic cluster, the other news title is processed through the lexical analysis model to obtain the second participle and the second part-of-speech tagging result of the second participle; each first part-of-speech tagging result and proper noun classification are used to determine the first score of the seed title; if the second participle exists in the first participle, the second score of the other news title is determined based on the proper noun classification and the second part-of-speech tagging result; based on the ratio of the second score to the first score, the similarity value of each other news title relative to the seed title is obtained.

[0080] In this embodiment, the lexical analysis model can be understood as a model for word segmentation and grammatical annotation, for example, it can be a Lexical Analysis of Chinese (LAC) lexical analysis model. The first word segmentation can be understood as the vocabulary contained in the seed title. The first part-of-speech tagging result can be understood as the result after marking the grammatical category to which each first word belongs, for example, it can be a place name, a person's name, a verb, etc. The first score can be understood as being used to characterize the seed title in the form of a score. The second word segmentation can be understood as the vocabulary contained in other news titles. The second part-of-speech tagging result can be understood as the result after marking the grammatical category of each second word. The second score can be understood as other news titles characterized in the form of a score. The proper noun classification can be understood as the part-of-speech category used for division, for example, it can be three types of proper nouns: place names, company names, and person names.

[0081] Specifically, the processor can process the seed title through a lexical analysis model (such as a LAC lexical analysis model) to obtain a first participle and a first part-of-speech tagging result of the first participle. For each other news title in the intermediate topic cluster, the other news title is processed through the lexical analysis model to obtain a second participle and a second part-of-speech tagging result of the second participle. The processor can count the first part-of-speech tagging results that belong to the proper noun category and accumulate a first score, and accumulate the other first part-of-speech tagging results that do not belong to the proper noun category to determine the first score of the seed title. The processor can determine whether the second participle exists in the first participle. If so, the processor can determine whether the existing second part-of-speech tagging result belongs to the proper noun category. If so, the first score is accumulated, and if not, the second score is accumulated to determine the second score of the other news title. The processor can obtain the similarity value of each other news title relative to the seed title based on the ratio of the second score to the first score.

[0082] For example, for the seed title T1 and a news title T2 under the middle topic cluster, the LAC lexical analysis model is used to perform word segmentation and part-of-speech tagging on T1 and T2. Initialize two values ​​a and b, a represents the first score of T1, and b represents the second score of T2. Traverse the part-of-speech tagging results of T1, where a is added 3 points when the part-of-speech tagging result is a proper noun of the three categories of place names, company names and personal names, and a is added 1 point for other part-of-speech tagging results. Traverse the word segmentation results of T2 and determine whether the word segmentation exists in T1. If it does not exist, b will not be added. If it exists, it will be added according to the part-of-speech. When the part-of-speech of the word segmentation is a proper noun of the three categories of place names, company names and personal names, b is added 3 points, and b is added 1 point for other words. b / a is the similarity score.

[0083] S210: Split other news headlines with similarity values ​​less than a set threshold from the intermediate topic cluster to form a new topic cluster.

[0084] In this embodiment, the set threshold can be understood as a similarity threshold set for determining whether topics belong to the same intermediate topic cluster.

[0085] Specifically, the processor can compare the similarity value of each other news headline with a set threshold, split other news headlines with similarity values ​​lower than the set threshold from the intermediate topic cluster to form a new topic cluster, and repeat the similarity calculation process for the new topic cluster until no new headlines are split out.

[0086] S211 , merging the intermediate topic clusters and the new topic cluster based on the similarity of the seed titles to obtain a candidate topic cluster.

[0087] In this embodiment, the candidate topic cluster can be understood as a merged topic cluster.

[0088] Specifically, the processor can determine the seed titles of each intermediate topic cluster after the split, as well as the seed titles of the new topic cluster through a preset sorting algorithm, and determine the similarity between different seed titles through the above-mentioned similarity value determination method. The processor can merge topic clusters whose similarity is higher than the set merge threshold until all topic clusters that meet the merge conditions are merged to obtain candidate topic clusters.

[0089] S212. Extract the top three news titles ranked by topic relevance from the candidate topic clusters using a preset ranking algorithm as candidate topic names.

[0090] In this embodiment, the candidate topic names can be understood as the basis for selecting the final name.

[0091] Specifically, the processor may calculate each candidate topic cluster using a preset sorting algorithm to obtain candidate topic clusters sorted by topic relevance, and the processor may extract the top three news titles sorted by topic relevance as candidate topic names.

[0092] S213: Input the candidate topic names into the large model for simplification to obtain topic cluster titles of the candidate topic clusters.

[0093] In this embodiment, the topic cluster title can be understood as an overall title used to summarize and describe a topic cluster.

[0094] Specifically, the processor may input the three candidate topic names into the large model for simplification to obtain the topic cluster title of the candidate topic cluster.

[0095] For example, the prompt example input into the large model can be: The following are three different news headlines of the same news. Please help me draft a new title for the news based on the content of these three headlines. The title should be refined to about 10 words for easy display, so as to obtain the topic cluster title.

[0096] S214: Screen the publishing sources of the candidate topic clusters to obtain screened topic clusters.

[0097] In this embodiment, the filtered topic clusters can be understood as the result of secondary filtering.

[0098] Specifically, the processor can send the publishing source of each candidate topic in the candidate topic cluster to relevant personnel for manual screening, or conduct secondary screening of the publishing source of the candidate topic cluster through screening through an updated blacklist. For certain publishing sources that continuously publish low-quality news, the processor can add them to the publishing source blacklist.

[0099] S215. Determine the total topic popularity of the filtered topic cluster based on the hour difference between the release time of each news headline in the filtered topic cluster and the current time, and obtain a final hot topic cluster.

[0100] In this embodiment, the release time can be understood as the release time of the news corresponding to the news title. The current time can be understood as the time when the filtered topic cluster is formed. The hour difference is used to represent the time difference between the release time and the current time. The total topic popularity can be understood as representing the total popularity of the entire filtered topic cluster. The final hot topic cluster can be understood as the topic cluster containing the total topic popularity, for example, the final hot topic cluster displayed after sorting each filtered topic cluster according to the total topic popularity.

[0101] Specifically, for each news title under the filtered topic cluster, the processor can calculate the hour difference between its release time and the current time (the minimum value is 1), take the log value of the hour difference for smoothing, and then take the inverse as the heat representation of the news title. The processor can calculate the heat of all news titles under the filtered topic cluster, and accumulate all the heat as the total topic heat of the filtered topic cluster. The processor can sort the candidate topic clusters in descending order of the total topic heat to obtain the final hot topic cluster after sorting.

[0102] The technical solution of the embodiment of the present invention obtains higher quality news data by preprocessing the news data set, thereby ensuring the quality of subsequent hot topics. By performing topic modeling and clustering on the news data in the news data set through the BERTopic model, it is possible to understand the contextual meaning and semantics of words, and provide more accurate extraction of candidate topic words for topic clusters. Splitting the news titles in the topic clusters according to similarity before merging the topic clusters can ensure a high degree of consistency of the news in each topic cluster, avoid erroneously classifying irrelevant or only weakly related news into the same topic cluster, and thus improve the accuracy and purity of the topic. By using a large model to summarize and refine the top three representative news titles in the candidate topic cluster, a more concise and attractive title can be generated, which not only improves the quality of the final displayed news title, but also ensures the uniformity of the title format of different topic clusters, thereby improving the user experience.

[0103] Example 3

[0104] Figure 3 This is a schematic diagram of the structure of a hot topic cluster generation device provided by the third embodiment of the present invention. Figure 3 As shown, the device includes: a data acquisition module 31 , a first generation module 32 , a second generation module 33 and a third generation module 34 .

[0105] Data acquisition module 31, used to obtain news data sets that meet collection conditions;

[0106] A first generating module 32 is configured to perform topic modeling and clustering on the news data in the news data set to obtain a topic cluster to which each piece of news data belongs;

[0107] A second generating module 33 is used to split and merge the topic clusters to determine candidate topic clusters;

[0108] The third generating module 34 is configured to determine a final hot topic cluster based on the large model and the candidate topic clusters.

[0109] The technical solution of the embodiment of the present invention obtains a news dataset that meets the collection conditions; performs topic modeling and clustering on the news data in the news dataset to obtain the topic clusters to which each piece of news data belongs; splits and merges the topic clusters to determine candidate topic clusters; and determines the final hot topic cluster based on the large model and the candidate topic clusters. By performing the splitting process before merging the topic clusters, the high consistency of the news within each topic cluster is ensured, preventing unrelated or only weakly related news from being mistakenly classified into the same topic cluster, thereby improving the accuracy and purity of the topics in the final hot topic cluster. The titles of the final hot topic cluster are determined based on the large model, improving the quality of the titles and, in turn, the quality of the final hot topic cluster.

[0110] Furthermore, the first generating module includes:

[0111] A first generating unit is configured to pre-process the news data in the news data set to obtain processed news headlines and form a news headline set;

[0112] A second generating unit is configured to input the news headline set into a text embedding model to determine a first output vector set;

[0113] a third generating unit, configured to perform dimensionality reduction on each first output vector in the first output vector set to form a second output vector set;

[0114] a first determining unit, configured to cluster the second output vector set to obtain a clustering result;

[0115] The second determining unit is configured to determine the candidate topic words of the clustering result and combine the clustering result as the topic cluster to which the news title in each of the news data belongs.

[0116] The first generating unit is specifically configured to:

[0117] filtering news publishing sources in the news data based on a pre-established blacklist to obtain filtered first news data;

[0118] performing format normalization processing on the first news title in the first news data to obtain a second news title;

[0119] Deduplicating each of the second news headlines to obtain a third news headline;

[0120] The third news headline is length-normalized to obtain processed news headlines and form a news headline set.

[0121] Furthermore, the second generating module 33 includes:

[0122] a third determining unit, configured to process each topic cluster in the topic cluster set using a preset sorting algorithm to obtain intermediate topic clusters sorted in order of topic relevance;

[0123] a fourth determining unit, configured to use the news title ranked first in each of the intermediate topic clusters as a seed title;

[0124] a fifth determining unit, configured to calculate similarity between the seed title and other news titles in the corresponding intermediate topic cluster, and obtain a similarity value between each of the other news titles and the seed title;

[0125] a sixth determining unit, configured to split the other news headlines whose similarity values ​​are less than a set threshold from the intermediate topic cluster to form a new topic cluster;

[0126] The seventh determining unit is configured to merge the intermediate topic clusters and the new topic cluster based on similarities in seed titles to obtain a candidate topic cluster.

[0127] The fifth determining unit is specifically configured to:

[0128] Processing the seed title through a lexical analysis model to obtain a first participle and a first part-of-speech tagging result of the first participle;

[0129] For each other news headline in the corresponding intermediate topic cluster, the other news headline is processed by the lexical analysis model to obtain a second participle and a second part-of-speech tagging result of the second participle;

[0130] Determining a first score for the seed title based on each of the first part-of-speech tagging results and proper noun classification;

[0131] If the second participle exists in the first participle, determining a second score for the other news headline based on the proper noun classification and the second part-of-speech tagging result;

[0132] A similarity value of each of the other news titles relative to the seed title is obtained according to a ratio of the second score to the first score.

[0133] Furthermore, the third generating module 34 is specifically configured to:

[0134] Extracting the top three news titles ranked by topic relevance from the candidate topic cluster using a preset sorting algorithm as candidate topic names;

[0135] Inputting the candidate topic names into the large model for simplification to obtain the topic cluster title of the candidate topic cluster;

[0136] Screening the candidate topic clusters by publishing sources to obtain screened topic clusters;

[0137] The total topic popularity of the filtered topic cluster is determined according to the hour difference between the release time of each news headline in the filtered topic cluster and the current time, thereby obtaining a final hot topic cluster.

[0138] The hot topic cluster generation device provided by the embodiment of the present invention can execute the hot topic cluster generation method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0139] Example 4

[0140] Figure 4 A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0141] like Figure 4 As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., which is communicatively connected to the at least one processor 41. The memory stores a computer program that can be executed by the at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 into the random access memory (RAM) 43. Various programs and data required for the operation of the electronic device 40 can also be stored in the RAM 43. The processor 41, ROM 42, and RAM 43 are connected to each other via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0142] Multiple components in the electronic device 40 are connected to the I / O interface 45, including an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a magnetic disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0143] The processor 41 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 executes the various methods and processes described above, such as the hot topic cluster generation method.

[0144] In some embodiments, the hot topic cluster generation method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the hot topic cluster generation method described above can be performed. Alternatively, in other embodiments, the processor 41 can be configured to execute the hot topic cluster generation method in any other appropriate manner (for example, by means of firmware).

[0145] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0147] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0148] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0149] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0150] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0151] In one embodiment, the present invention further includes a computer program product, which includes a computer program. When the computer program is executed by a processor, the method for generating a hot topic cluster according to any embodiment of the present invention is implemented.

[0152] The computer program product may be implemented by writing computer program code for performing the operations of the present invention in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0153] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0154] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for generating hot topic clusters, characterized in that: include: Obtain news datasets that meet collection conditions; Performing topic modeling and clustering on the news data in the news data set to obtain a topic cluster to which each piece of news data belongs; Splitting and merging the topic clusters to determine candidate topic clusters; Based on the large model and the candidate topic clusters, the final hot topic cluster is determined.

2. The method according to claim 1, characterized in that The topic modeling and clustering of the news data in the news data set to obtain the topic clusters to which each of the news data belongs includes: Preprocessing the news data in the news data set to obtain processed news headlines and form a news headline set; Inputting the news headline set into a text embedding model to determine a first output vector set; performing dimensionality reduction on each first output vector in the first output vector set to form a second output vector set; Clustering the second output vector set to obtain a clustering result; The candidate subject words of the clustering result are determined and combined with the clustering result as the topic cluster to which the news title in each of the news data belongs.

3. The method according to claim 2, characterized in that The preprocessing of the news data in the news data set to obtain processed news headlines and form a news headline set includes: filtering news publishing sources in the news data based on a pre-established blacklist to obtain filtered first news data; performing format normalization processing on the first news title in the first news data to obtain a second news title; Deduplicating each of the second news headlines to obtain a third news headline; The third news headline is length-normalized to obtain processed news headlines and form a news headline set.

4. The method according to claim 1, wherein The splitting and merging of the topic clusters to determine candidate topic clusters includes: Processing each topic cluster in the topic cluster set by a preset sorting algorithm to obtain an intermediate topic cluster sorted in order of topic relevance; Using the first-ranked news headline in each of the intermediate topic clusters as a seed headline; Calculating similarity between the seed title and other news titles in the corresponding intermediate topic cluster to obtain similarity values ​​of each of the other news titles relative to the seed title; Splitting other news headlines whose similarity values ​​are less than a set threshold from the intermediate topic cluster to form a new topic cluster; Based on the similarity of the seed titles between each of the intermediate topic clusters and the new topic cluster, they are merged to obtain a candidate topic cluster.

5. The method according to claim 4, characterized in that The step of calculating the similarity between the seed title and other news titles in the corresponding intermediate topic cluster to obtain a similarity value of each of the other news titles relative to the seed title includes: Processing the seed title through a lexical analysis model to obtain a first participle and a first part-of-speech tagging result of the first participle; For each other news headline in the corresponding intermediate topic cluster, the other news headline is processed by the lexical analysis model to obtain a second participle and a second part-of-speech tagging result of the second participle; Determining a first score for the seed title based on each of the first part-of-speech tagging results and proper noun classification; If the second participle exists in the first participle, determining a second score for the other news headline based on the proper noun classification and the second part-of-speech tagging result; A similarity value of each of the other news titles relative to the seed title is obtained according to a ratio of the second score to the first score.

6. The method according to claim 1, characterized in that The step of determining the final hot topic cluster based on the large model and the candidate topic clusters includes: Extracting the top three news titles ranked by topic relevance from the candidate topic cluster using a preset sorting algorithm as candidate topic names; Inputting the candidate topic names into the large model for simplification to obtain the topic cluster title of the candidate topic cluster; Screening the candidate topic clusters by publishing sources to obtain screened topic clusters; The total topic popularity of the filtered topic cluster is determined according to the hour difference between the release time of each news headline in the filtered topic cluster and the current time, thereby obtaining a final hot topic cluster.

7. A hot topic cluster generation device, characterized in that: include: Data acquisition module, used to obtain news data sets that meet the collection conditions; A first generating module is used to perform topic modeling and clustering on the news data in the news data set to obtain a topic cluster to which each piece of news data belongs; A second generating module is used to split and merge the topic clusters to determine candidate topic clusters; The third generation module is used to determine the final hot topic cluster based on the large model and the candidate topic cluster.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the hot topic cluster generating method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the hot topic cluster generation method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product includes a computer program, which, when executed by a processor, implements the method for generating hot topic clusters according to any one of claims 1 to 6.

Images