Hot topic monitoring method and system based on improved HDBSCAN clustering algorithm

Abstract

The invention discloses a hot topic monitoring method and system based on an improved HDBSCAN clustering algorithm, and the method comprises the steps: capturing news and comment data of a social media platform through an incremental crawler technology, recording a page updating state, and dynamically maintaining the data timeliness; performing word segmentation, stop word filtering and entity recognition preprocessing on the text, and constructing a triple data set; the SBERT model is finely adjusted based on a triple loss function, and the clustering center supervision capability represented by semantic vectors is optimized; dimensionality reduction is performed on the high-dimensional semantic vector, and global and local semantic structures of the text are reserved; combining cosine similarity and a time decay factor to improve HDBSCAN distance measurement, and dynamically adjusting a text similarity weight to realize separation of new and old topics; a popularity analysis model is constructed based on user interaction behaviors and a time window, and topic propagation trend and influence evolution are tracked in real time. According to the method, through semantic enhancement and a dynamic clustering mechanism, the accuracy and real-time performance of hot topic recognition are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of hot topic monitoring, and in particular relates to a hot topic monitoring method and system based on an improved HDBSCAN clustering algorithm. Background Art

[0002] In recent years, hot topic monitoring systems have gradually become a key application area in the field of news information monitoring. Traditional hot topic monitoring relies heavily on manual review, which is plagued by cumbersome processes, slow responses, and high subjectivity. This makes it difficult to adapt to the current demands for timeliness and objectivity in the face of information explosion. To address these challenges, technologies such as natural language processing (NLP), data mining, and machine learning have been widely introduced in recent years. These technologies enable automated screening, classification, sentiment analysis, and trend prediction of hot topic content, significantly improving monitoring efficiency and coverage.

[0003] However, despite the significant success of intelligent algorithms in hot topic monitoring, existing technologies still face numerous shortcomings. First, existing models often suffer from reduced recognition accuracy and insufficient robustness when faced with rapidly changing language expressions, new internet terms, and multimodal content. Second, many systems have a weak grasp of temporal trends, making it difficult to promptly capture the outbreak and decline of hot topics. Furthermore, current mainstream algorithms are often inefficient when processing large-scale social platform data, failing to achieve real-time monitoring and response, severely limiting their practicality. Finally, hot topic clustering methods commonly suffer from technical bottlenecks such as strong parameter dependence and an inability to effectively respond to the dynamic changes of emerging topics.

[0004] In summary, although hot topic monitoring technology has achieved certain results, it still has significant shortcomings in terms of accuracy, timeliness, stability, and dynamic adaptability, and urgently needs to be improved through technological innovation. This provides a clear research direction and technical breakthrough for further optimizing news information monitoring systems. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention proposes a hot topic monitoring method and system based on an improved HDBSCAN clustering algorithm to solve the problems existing in the above-mentioned prior art.

[0006] In a first aspect, to achieve the above-mentioned objectives, the present invention provides a hot topic monitoring method based on an improved HDBSCAN clustering algorithm, comprising the following steps:

[0007] Use an incremental crawler platform to crawl news and comment data from social media, remove redundant content, and form a data pool;

[0008] Preprocessing the text in the data pool, including word segmentation, stop word filtering, and text content extraction, and constructing semantic training triples;

[0009] Load the pre-trained text representation model, fine-tune it based on triple data using the triple loss function, and generate a semantic embedding vector;

[0010] Reducing the semantic embedding vector to a low-dimensional space while preserving the semantic structure;

[0011] Dynamically clustering the vectors after dimensionality reduction is performed using an improved clustering algorithm, wherein the clustering algorithm combines cosine similarity with a time decay factor to adjust the distance between texts;

[0012] Calculate the topic popularity index based on user interaction behavior and time window, and filter and output the hot topic evolution results.

[0013] Optionally, the crawling process of the incremental crawler platform includes:

[0014] Use breadth-first traversal strategy to access target websites and filter valid pages according to set domain name rules and path rules;

[0015] Only crawl content that has been added or updated since the last crawl time, and limit the crawl scope by time window;

[0016] A first-in-first-out queue is used to manage the status of crawled URLs, and the crawling time and update status of each URL are recorded.

[0017] Optionally, the process of preprocessing the text in the data pool includes:

[0018] Use regular expressions to remove HTML tags and noise characters and extract the text content;

[0019] Use a predefined stop word list to remove meaningless words and use a conditional random field model to segment the text.

[0020] A triplet dataset consisting of anchor text, positive samples and negative samples is constructed, where the positive samples are semantically similar to the anchor text and the negative samples are semantically irrelevant to the anchor text.

[0021] Optionally, a pre-trained text representation model is loaded, and fine-tuned based on the triple data using a triple loss function to generate a semantic embedding vector, including:

[0022] Inputting the triplet dataset into a pre-trained sentence encoding model to generate vector representations of anchor texts, positive samples, and negative samples;

[0023] Based on the cosine distance between the anchor text and the positive sample, and the cosine distance between the anchor text and the negative sample, a triplet loss function is constructed to maximize the distance between positive and negative samples;

[0024] The model parameters are updated by minimizing the triplet loss function.

[0025] Optionally, the dimensionality reduction process includes:

[0026] The manifold learning method is used to map high-dimensional semantic vectors to low-dimensional space, preserving the global semantic association and local semantic proximity between texts.

[0027] The vector distribution in the low-dimensional space is optimized by distance weighting, so that the reduced-dimensional vector maintains the semantic structure of the original high-dimensional space.

[0028] Optionally, the improved clustering algorithm includes:

[0029] Calculate the time decay factor based on the time difference of text release and dynamically adjust the weight of cosine similarity;

[0030] Construct a minimum spanning tree based on the mutual reachability distance and generate a hierarchical clustering structure through pruning operations;

[0031] The hierarchical structure is traversed to extract clusters whose stability meets a preset threshold as final hot topics.

[0032] In a second aspect, the present invention further provides a hot topic monitoring system based on an improved HDBSCAN clustering algorithm, which is used to implement a hot topic monitoring method based on the improved HDBSCAN clustering algorithm. The system comprises:

[0033] The data collection module is used to crawl news and comment data from social media through incremental crawlers, eliminate redundant content and form a data pool;

[0034] A preprocessing module, configured to perform word segmentation, stop word filtering, and text content extraction on the text in the data pool, and construct semantic training triples;

[0035] A model fine-tuning module is used to load a pre-trained text representation model, fine-tune it based on the triple data using a triple loss function, and generate a semantic embedding vector;

[0036] A dimensionality reduction processing module, configured to reduce the dimension of the semantic embedding vector to a low-dimensional space while preserving the semantic structure;

[0037] A dynamic clustering module is used to dynamically cluster the vectors after dimensionality reduction using an improved clustering algorithm, wherein the clustering algorithm combines cosine similarity with a time decay factor to adjust the distance between texts;

[0038] The heat analysis module is used to calculate the topic heat index based on user interaction behavior and time window, and filter and output the hot topic evolution results.

[0039] Optionally, the data acquisition module includes:

[0040] Breadth-first crawling unit, used to access the target website using a breadth-first traversal strategy and filter valid pages according to the set domain name rules and path rules;

[0041] Incremental update unit, used to crawl only the content that has been added or updated since the last crawl time, and limit the crawl range by time window;

[0042] The queue management unit is used to manage the status of crawled URLs using a first-in-first-out queue and record the crawling time and update status of each URL.

[0043] In a third aspect, the present invention further provides a computer terminal device, comprising:

[0044] one or more processors;

[0045] a memory, coupled to the processor, for storing one or more programs;

[0046] When the one or more programs are executed by the one or more processors, the one or more processors implement a hot topic monitoring method based on an improved HDBSCAN clustering algorithm.

[0047] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements a hot topic monitoring method based on an improved HDBSCAN clustering algorithm.

[0048] Compared with the prior art, the present invention has the following advantages and technical effects:

[0049] (1) This paper adopts a semantic representation method based on fine-tuning the SBERT model and combines it with Triplet Loss to optimize the semantic distance between texts, making the text representation more precise and effectively improving the accuracy and robustness of clustering. At the same time, UMAP dimensionality reduction is used to reduce the dimensionality of text embeddings, preserving the global and local structures between texts, further improving clustering performance.

[0050] (2) This paper takes into full account the dynamic and temporal characteristics of topics on social platforms and improves the distance metric of the HDBSCAN clustering algorithm. By introducing a time decay factor, the distance between texts is dynamically adjusted to ensure effective distinction between new and old topics. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are provided to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0052] Figure 1Schematic diagram of the framework of a hot topic monitoring system based on an improved HDBSCAN clustering algorithm according to an embodiment of the present invention;

[0053] Figure 2 Schematic diagram of the SBERT fine-tuning strategy based on Triplet Loss in an embodiment of the present invention;

[0054] Figure 3 This is a diagram illustrating the main concept of the improved HDBSCAN clustering algorithm implemented in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0056] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0057] Example 1

[0058] like Figure 1 As shown, this embodiment provides a hot topic monitoring method based on an improved HDBSCAN clustering algorithm, including:

[0059] Use an incremental crawler platform to crawl news and comment data from social media, remove redundant content, and form a data pool;

[0060] Preprocessing the text in the data pool, including word segmentation, stop word filtering, and text content extraction, and constructing semantic training triples;

[0061] Load the pre-trained text representation model, fine-tune it based on triple data using the triple loss function, and generate a semantic embedding vector;

[0062] Reducing the semantic embedding vector to a low-dimensional space while preserving the semantic structure;

[0063] Dynamically clustering the vectors after dimensionality reduction is performed using an improved clustering algorithm, wherein the clustering algorithm combines cosine similarity with a time decay factor to adjust the distance between texts;

[0064] Calculate the topic popularity index based on user interaction behavior and time window, and filter and output the hot topic evolution results.

[0065] Specifically, S1: Using an integrated front-end and back-end incremental crawler platform, combined with an intelligent filtering mechanism, we acquire news reports and commentary data from social platforms (such as Weibo and Twitter). Using a breadth-first traversal approach and following pre-defined crawling rules, we crawl valid pages and remove redundant and invalid content to form a preliminary news data pool.

[0066] S2: Preprocess the acquired text data, including word segmentation and stop word removal. Build a news corpus, extract the text using regular expressions, and construct "anchor text-positive sample-negative sample" (anchor, positive, negative) triples. This provides basic data for model training.

[0067] S3: Load the pre-trained model and fine-tune the model based on the constructed triplet dataset using Triplet Loss as the loss function, so that the model can more accurately extract the semantic information of the text and supervise the clustering center.

[0068] S4: Generate semantic embeddings for the text using the fine-tuned SBERT model. The resulting high-dimensional semantic vectors are reduced in dimension using Unified Mapping (UMAP), preserving the global and local structure of the text and providing higher-quality input for subsequent clustering tasks.

[0069] S5: Dynamically cluster text embedding vectors using the improved HDBSCAN algorithm. Combining cosine similarity and a time decay factor, the distance metric between texts is dynamically adjusted to ensure the distinction between new and old topics.

[0070] S6: Dynamically evaluate topic popularity through user interactions such as comments, likes, and reposts. By combining time windows with a popularity analysis model, we track the lifecycle and spread of topics, generating a topic popularity curve that displays the evolution of hot topics in real time.

[0071] More specifically, S1: Develop crawlers for various platforms (such as Weibo and Twitter), configure output formats, and build a unified crawler platform through front-end and back-end integration. Use incremental crawling technology to record the data status and update time of crawled pages, and perform regular incremental updates to ensure data timeliness and integrity. Furthermore, display crawling results on the web for easy monitoring and management.

[0072] S2: Preprocess the news content acquired by the incremental crawler system, including word segmentation, stop word removal, part-of-speech tagging, and named entity recognition. Regular expressions are then used to extract the text content and construct "anchor text-positive sample-negative sample" triplets to provide basic data for model training.

[0073] S3: Load the pre-trained SBERT model and fine-tune the SBERT model based on the constructed triplet dataset using Triplet Loss as the loss function. This allows the model to more accurately extract the semantic information of the text and supervise the cluster centers.

[0074] S4: Use the fine-tuned SBERT model to convert the text into a high-dimensional embedding vector. Use the UMAP algorithm to reduce the high-dimensional vector to a low-dimensional space to reduce computational complexity and preserve the semantic structure.

[0075] S5: Combining cosine similarity and time decay factors makes the text distance metric more suitable for specific tasks. An improved HDBSCAN algorithm is used to cluster the reduced-dimensional vectors and dynamically update the topic center vector based on the semantic features of the new text.

[0076] S6: Based on user interaction behaviors (such as comments, likes, and reposts) and time factors, a topic popularity analysis model is established to evaluate the popularity level and evolution trend of topics, and measure the influence of hot topics on the Internet in real time.

[0077] As an implementation method of this embodiment, the crawling process of the incremental crawler platform includes:

[0078] Use breadth-first traversal strategy to access target websites and filter valid pages according to set domain name rules and path rules;

[0079] Only crawl content that has been added or updated since the last crawl time, and limit the crawl scope by time window;

[0080] A first-in-first-out queue is used to manage the status of crawled URLs, and the crawling time and update status of each URL are recorded.

[0081] Specifically, step S2 includes:

[0082] S1.1: Use a breadth-first search (BFS) strategy to access the target website and retrieve valid web pages according to the specified crawling rules. Use an incremental crawling method to only crawl content that has been added or updated since the last crawl to avoid redundancy and data duplication.

[0083] S1.2: Define crawling rules including: allowed domain names, path rules and access frequency limits; content screening strategies (based on titles, tags, keywords, etc.); time windows for data crawling (such as the latest published content within 24 hours).

[0084] Assume that the crawling strategy is defined as:

[0085]

[0086] in, It represents the i-th crawling rule. The crawler analyzes and filters the target web pages according to these rules.

[0087] S1.3: Store the extracted URLs of hot topics, news, etc., and record the status of the URLs, including: crawling time T c Crawling result (success / failure), reason for failure, and last update time. Adopt a queue management mechanism based on FIFO (First In First Out) to prioritize the processing of the latest URLs.

[0088] As an implementation manner in this embodiment, the process of preprocessing the text in the data pool includes:

[0089] Use regular expressions to remove HTML tags and noise characters, and extract the text body content;

[0090] Eliminate meaningless words through a predefined stop word list, and use a conditional random field model to segment the text;

[0091] Construct a triple dataset composed of anchor text, positive samples, and negative samples, where the positive samples are semantically similar to the anchor text, and the negative samples are semantically unrelated to the anchor text.

[0092] Specifically, step S2 includes:

[0093] S2.1: After the data collection is completed, it is necessary to screen and clean the text content to ensure the integrity and consistency of the input data. Use regular expressions to extract the text body content, remove HTML tags and noise characters. Use a predefined stop word list to eliminate common words that contribute nothing to semantic understanding (such as "of", "is", "in", etc.) and use a CRF (Conditional Random Field) - based word segmentation model to segment the Chinese text.

[0094] S2.2: Generate semantic embedding representations by fine - tuning the SBERT model, and use Triplet Loss for model fine - tuning. Construct triples (anchor, positive, negative): Anchor: represents the core text of the current topic, Positive: text semantically similar to the Anchor, Negative: text with a large semantic difference from the Anchor.

[0095] More specifically, refer to Figure 2 As shown, step S2 includes:

[0096] S2.1: Perform data preprocessing on the data crawled in S1, delete HTML tags, special symbols, and formatting characters. Unify the text encoding to ensure the consistency of the text format. Use a word segmentation tool (such as Jieba or NLTK) to segment the text. Split the text into independent words or phrases.

[0097] S2.2: Establish a stop word list (such as common words like "de", "shi", "zai", etc.). Delete the stop words that appear in the text to reduce noise interference.

[0098] S2.3: Match common topic structures (such as titles, body texts, time, etc.) through regular expressions. Only retain the valid body content and remove advertisements and irrelevant content.

[0099] S2.4: Construct triples (Anchor, Positive, Negative): Anchor: Select the most representative text from the topic text as the anchor text, Positive: Texts belonging to the same topic as the Anchor, Negative: Texts belonging to different topics from the Anchor. An example of triple construction is as follows:

[0100]

[0101] As an implementation method in this embodiment, the process of loading a pre-trained text representation model and fine-tuning it through the triple loss function based on the triple data to generate semantic embedding vectors includes:

[0102] Input the triple data set into a pre-trained sentence encoding model to generate vector representations of the anchor text, positive samples, and negative samples;

[0103] Based on the cosine distance between the anchor text and positive samples, and the cosine distance between the anchor text and negative samples, construct a triple loss function that maximizes the distance between positive and negative samples;

[0104] Update the model parameters by minimizing the triple loss function.

[0105] Specifically, in step S3:

[0106] S3.1: Use the Triplet Loss function to fine-tune the SBERT model, optimize the semantic expression ability, and supervise the clustering center. The Triplet Loss formula is:

[0107]

[0108] Where: , , respectively represent the embedding representations of the Anchor, Positive, and Negative texts; d(⋅) is the cosine distance; m is the margin (Margin).

[0109] S3.2: Use the fine-tuned SBERT model to map the text into a high-dimensional vector. Let the vector generated by the text X after the SBERT model be:

[0110]

[0111] It is the input text data, and the text is processed by the SBERT (Sentence-BERT) model. To encode, is the output text embedding vector; Represents the vector dimension.

[0112] More specifically, the SBERT model is fine-tuned on the triplet dataset constructed in S2. In step S3:

[0113] S3.1: Load the pre-trained SBERT model, use Sentence-BERT (SBERT) as the base model and use the officially released bert-base-uncased or other models suitable for Chinese.

[0114] S3.2: Build a triplet dataset and format the triplet dataset built in S2 as input. The format example is as follows:

[0115]

[0116] Anchor, Positive, and Negative represent anchor text, similar text, and different topic text respectively.

[0117] S3.3: Introduce the Triplet Loss loss function. The formula of Triplet Loss is as follows:

[0118]

[0119] in, represents the anchor point sample, represents the positive samples of the same type as the anchor point, Represents negative samples that are different from the anchor point; Represents the distance between the anchor text and the positive sample. Represents the distance between the anchor text and the negative sample. is the minimum margin between positive samples and negative samples.

[0120] S3.4: The optimization goal of the loss function is to minimize the distance between the anchor text and the positive sample by minimizing the Triplet Loss, and to maximize the distance between the anchor text and the negative sample.

[0121] S3.5: Use the Adam optimizer for parameter updates. Use batch training (e.g., batch size = 32, where batch size is the batch size). Set the learning rate to 1e-5 or fine-tune it based on your experiments. Monitor the loss during training and use early stopping to prevent overfitting.

[0122] As an implementation method in this embodiment, the dimensionality reduction process includes:

[0123] The manifold learning method is used to map high-dimensional semantic vectors to low-dimensional space, preserving the global semantic association and local semantic proximity between texts.

[0124] The vector distribution in the low-dimensional space is optimized by distance weighting, so that the reduced-dimensional vector maintains the semantic structure of the original high-dimensional space.

[0125] Specifically, in step S4:

[0126] UMAP is used to reduce the high-dimensional embedding vector to a low-dimensional space to maintain the semantic structure. The objective function is:

[0127]

[0128] For samples in high-dimensional space and distance; , is the sample representation in low-dimensional space.

[0129] More specifically, the text embedding vector generated by the SBERT model after fine-tuning in S3 is reduced in dimension. In step S4:

[0130] S4.1: Generate text embedding vectors using the SBERT model. Input the news text into the fine-tuned SBERT model to obtain a high-dimensional text embedding vector. The embedding vector example is as follows:

[0131]

[0132] in, To input text, is the generated embedding vector.

[0133] S4.2: Use UMAP (Uniform Manifold Approximation and Projection) to map the high-dimensional vector to a low-dimensional space. After UMAP dimensionality reduction, the resulting low-dimensional vector is:

[0134]

[0135] The low-dimensional vector can be used for subsequent clustering tasks.

[0136] S4.3: Preserve the relative semantic distance between texts after dimensionality reduction. This reduces computational complexity and improves the clustering efficiency of S5.

[0137] As an implementation method in this embodiment, the improved clustering algorithm includes:

[0138] Calculate the time decay factor based on the time difference of text release and dynamically adjust the weight of cosine similarity;

[0139] Construct a minimum spanning tree based on the mutual reachability distance and generate a hierarchical clustering structure through pruning operations;

[0140] The hierarchical structure is traversed to extract clusters whose stability meets a preset threshold as final hot topics.

[0141] Specifically, in step S5:

[0142] S5.1: After obtaining high-quality text representation, the improved HDBSCAN algorithm is used to dynamically cluster the text data. The semantic similarity between texts is defined as cosine similarity, which is calculated as:

[0143]

[0144] A vector representation of two texts (or text paragraphs), usually a high-dimensional vector generated by SBERT or other text embedding models. Represents the L2 norm (Euclidean length) of a vector.

[0145] S5.2: Considering the decay characteristics of text content in the time dimension, a time decay factor is introduced to dynamically adjust the similarity calculation. For text The time decay function is defined as:

[0146]

[0147] Represents the timestamps (i.e., the time of publication) of the two texts. λ is the time decay coefficient, which controls the effect of time on similarity. A larger λ makes the time difference have a greater impact on similarity, while a smaller λ means that even with a longer time interval, the similarity between the texts remains high.

[0148] The final distance metric formula is:

[0149] This formula integrates the time decay mechanism on the basis of semantic similarity, which reduces the similarity of texts published at long intervals and enhances the timeliness of clustering.

[0150] S5.3: Use the HDBSCAN algorithm to dynamically assign clusters based on text distance.

[0151] Preferably, in step S6:

[0152] S6.1: Calculate the Topic Heat Index (THI), count the number of interactions related to the topic (likes, reposts, comments, favorites), calculate the spread of the topic, and measure its spread on social platforms. Consider the time decay factor to reduce the impact of outdated content. Heat calculation formula:

[0153]

[0154] in is a weight parameter that can be set according to the experiment.

[0155] S6.2: Set a heat threshold Thot. If the THI of a topic exceeds this threshold, it is considered a hot topic.

[0156]

[0157] This threshold can be based on the mean + standard deviation of historical data or set to a fixed value.

[0158] S6.3: Count the THI of all clustered topics and filter out hot topics that exceed the threshold. These hot topics can be further used for recommendation or analysis.

[0159] More specifically, see Figure 3 As shown, in step S5:

[0160] The specific improvement strategy of the HDBSCAN clustering algorithm is:

[0161] S5.1: In the traditional HDBSCAN algorithm, the distance metric is usually Euclidean distance or Manhattan distance. However, in the text clustering task, after using the vector space model (VSM) to represent the text, the semantic similarity between texts is usually measured using cosine similarity.

[0162] In addition, topics have significant temporal characteristics, and earlier content may have less relevance to the current one. Therefore, based on this characteristic, the distance metric is improved as follows:

[0163] Define text With text The similarity measurement formula between them is:

[0164]

[0165] in and Represents text and text Vector representation of (obtained through SBERT); Represents the dot product operation of the vector; ∥⋅∥ represents the norm of the vector. Introducing the time decay factor , which reduces the similarity between texts with longer time intervals, and is defined as:

[0166]

[0167] in and Represents text and The timestamp of ; λ is the time decay coefficient, which is used to control the rate of time decay. The final distance metric is defined as:

[0168]

[0169] That is, a time penalty term is introduced on the basis of similarity to reduce the similarity of texts with longer time intervals.

[0170] S5.2: Calculate the mutual reachability distance between points. In HDBSCAN, the distance between points is defined using the mutual reachability distance to more accurately define the boundaries of clusters when the density changes.

[0171] Definition Point and The core distance is:

[0172]

[0173] Point The maximum distance to its k nearest neighbors.

[0174] The mutual reachability distance is defined as:

[0175]

[0176] That is, the maximum value is taken between the direct distance and the core distance of the two points. By introducing the core distance and direct distance, the boundary problem between sparse areas and dense areas can be better handled.

[0177] S5.3: Construct a minimum spanning tree (MST). Treat the inter-reachable distances between all points as edge weights in a weighted graph. Connect all nodes using the MST to form a complete distance graph. Use Prim's or Kruskal's algorithm to construct a minimum spanning tree, where nodes represent text and edge weights represent the inter-reachable distances between points. This MST ensures graph connectivity while reducing redundant edges and improving clustering efficiency.

[0178] S5.4: Based on the minimum spanning tree, a cluster hierarchy is generated through pruning operations: edges with larger weights are gradually removed from the tree to form clusters with different density levels. This forms a tree-like cluster hierarchy, where each node represents a cluster and each edge represents the relationship between clusters.

[0179] S5.5: Compress the cluster tree. To reduce the complexity of the cluster tree and improve the stability of the clustering results, the cluster tree is compressed. The key is to view cluster splits as persistent clusters that "lose points" rather than simple binary splits. By setting a minimum cluster size, the hierarchy is traversed and the new cluster is determined to be smaller than the threshold. If so, the point is considered to have dropped out of the cluster; otherwise, it is considered a true cluster split. Ultimately, a streamlined tree is generated, recording how cluster size changes with distance, preparing for cluster extraction.

[0180] S5.6: Extract clusters based on cluster stability. In the final clustering tree, select the most stable cluster as the final result. The validity of the cluster is determined by defining cluster stability:

[0181]

[0182]

[0183] Where p represents a point, Defined as when a cluster separates and becomes its own cluster value, Indicates when the point "leaves the cluster" For each cluster, if the sum of the stabilities of its subclusters is greater than the stability of the cluster itself, the stability of the cluster is updated to the sum of the stabilities of the subclusters; conversely, if the stability of the cluster itself is greater, it is marked as the selected cluster, and the selection status of all its descendant clusters is cancelled, and the final clustering result is output.

[0184] Based on this, the embodiment of the present invention provides a hot topic monitoring method based on an improved HDBSCAN clustering algorithm, which can be applied to news platforms such as Sina Weibo and Twitter, and is expected to achieve the following effects:

[0185] (1) Hot Topic Capture and Recommendation Optimization: Through in-depth monitoring and analysis of the generation and distribution mechanisms of hot topics on social platforms, this paper can accurately identify current hot topics and their evolution trends based on the improved HDBSCAN algorithm. Combining semantically enhanced cosine similarity and temporal distance metrics, the system can optimize recommendation strategies to ensure that users receive the hot content that interests them most and is most valuable immediately, thereby significantly improving recommendation accuracy and user satisfaction.

[0186] (2) Deep user interest binding and stickiness enhancement: By carefully analyzing the interaction patterns between users and hot topics, this invention can more accurately capture user interest preferences. With the support of SBERT model fine-tuning and clustering stability enhancement, the platform can dynamically adjust the topic push strategy to provide users with more personalized information that fits their hot interests, thereby effectively extending the user's active time on the platform and significantly enhancing user stickiness.

[0187] (3) Guidance on creating hot content and increasing exposure: By monitoring the distribution efficiency of hot topics, this invention can help the platform identify which types of topic content are more likely to receive high exposure and widespread discussion. By using UMAP dimensionality reduction and HDBSCAN clustering results, the system can provide content creators with detailed data support, helping them understand the laws of content dissemination and create hot content that better meets user preferences and is more likely to become a hit, thereby increasing the overall content exposure and influence.

[0188] (4) Risk warning and topic management optimization: During the hot topic monitoring process, the present invention can timely detect and warn of potential negative topics or risky content (such as rumors, malicious hype, etc.) through time decay factor and cluster evolution trend analysis. Based on these monitoring results, the platform can quickly respond and take effective management measures to ensure the health and safety of platform content and maintain a good community environment.

[0189] Example 2

[0190] In this embodiment, a computer terminal device is provided, including:

[0191] one or more processors;

[0192] a memory, coupled to the processor, for storing one or more programs;

[0193] When the one or more programs are executed by the one or more processors, the one or more processors implement the methods in the above embodiments.

[0194] In this embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the method in the above embodiment is implemented.

[0195] In this embodiment, an electronic device is further provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to execute the method in the above embodiment.

[0196] The above program can be executed in a processor or stored in a memory (or computer-readable medium). Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and can be implemented using any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0197] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more blocks can be implemented by different modules corresponding to different steps.

[0198] This embodiment provides such a device or system. The system is called a hot topic monitoring system based on an improved HDBSCAN clustering algorithm, and includes:

[0199] The data collection module is used to crawl news and comment data from social media through incremental crawlers, eliminate redundant content and form a data pool;

[0200] A preprocessing module, configured to perform word segmentation, stop word filtering, and text content extraction on the text in the data pool, and construct semantic training triples;

[0201] A model fine-tuning module is used to load a pre-trained text representation model, fine-tune it based on the triple data using a triple loss function, and generate a semantic embedding vector;

[0202] A dimensionality reduction processing module, configured to reduce the dimension of the semantic embedding vector to a low-dimensional space while preserving the semantic structure;

[0203] A dynamic clustering module is used to dynamically cluster the vectors after dimensionality reduction using an improved clustering algorithm, wherein the clustering algorithm combines cosine similarity with a time decay factor to adjust the distance between texts;

[0204] The heat analysis module is used to calculate the topic heat index based on user interaction behavior and time window, and filter and output the hot topic evolution results.

[0205] As an implementation method of this embodiment, the data acquisition module includes:

[0206] Breadth-first crawling unit, used to access the target website using a breadth-first traversal strategy and filter valid pages according to the set domain name rules and path rules;

[0207] Incremental update unit, used to crawl only the content that has been added or updated since the last crawl time, and limit the crawl range by time window;

[0208] The queue management unit is used to manage the status of crawled URLs using a first-in-first-out queue and record the crawling time and update status of each URL.

[0209] As an implementation method in this embodiment, the preprocessing module includes:

[0210] The text cleaning unit is used to remove HTML tags and noise characters using regular expressions and extract the text content;

[0211] The word segmentation filtering unit is used to remove meaningless words through a predefined stop word list and segment the text using a conditional random field model;

[0212] The triplet construction unit is used to construct a triplet dataset consisting of anchor text, positive samples and negative samples, where the positive samples are semantically similar to the anchor text and the negative samples are semantically irrelevant to the anchor text.

[0213] As an implementation method in this embodiment, the model fine-tuning module includes:

[0214] A vector generation unit, configured to input the triplet dataset into a pre-trained sentence encoding model to generate vector representations of anchor texts, positive samples, and negative samples;

[0215] A loss calculation unit, used to construct a triplet loss function that maximizes the distance between positive and negative samples based on the cosine distance between the anchor text and the positive sample, and the cosine distance between the anchor text and the negative sample;

[0216] A parameter optimization unit is used to update the model parameters by minimizing the triple loss function.

[0217] As an implementation method in this embodiment, the dimensionality reduction processing module includes:

[0218] Dimensionality reduction execution unit, which uses manifold learning methods to map high-dimensional semantic vectors to low-dimensional space, preserving the global semantic associations and local semantic proximity relationships between texts;

[0219] The structure-preserving unit is used to optimize the vector distribution in the low-dimensional space through distance weighting, so that the vector after dimensionality reduction maintains the semantic structure of the original high-dimensional space.

[0220] As an implementation in this embodiment, the dynamic clustering module includes:

[0221] The time decay unit is used to calculate the time decay factor based on the time difference between text releases and dynamically adjust the weight of the cosine similarity;

[0222] Minimum spanning tree unit, used to construct a minimum spanning tree based on mutual reachability distance, and generate a hierarchical clustering structure through pruning operation;

[0223] The cluster extraction unit is used to traverse the hierarchical structure and extract clusters whose stability meets a preset threshold as final hot topics.

[0224] The system or device is used to implement the functions of the method in the above-mentioned embodiment. Each module in the system or device corresponds to each step in the method, which has been explained in the method and will not be repeated here.

[0225] Through the above implementation, the problem of hot topic monitoring based on the improved HDBSCAN clustering algorithm in the related art is solved, thereby ensuring that the problems existing in the existing technology are solved.

[0226] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A hot topic monitoring method based on an improved HDBSCAN clustering algorithm, characterized in that: The following steps are involved: Use an incremental crawler platform to crawl news and comment data from social media, remove redundant content, and form a data pool; Preprocessing the text in the data pool, including word segmentation, stop word filtering, and text content extraction, and constructing semantic training triples; Load the pre-trained text representation model, fine-tune it based on triple data using the triple loss function, and generate a semantic embedding vector; Reducing the semantic embedding vector to a low-dimensional space while preserving the semantic structure; Dynamically clustering the vectors after dimensionality reduction is performed using an improved clustering algorithm, wherein the clustering algorithm combines cosine similarity with a time decay factor to adjust the distance between texts; Calculate the topic popularity index based on user interaction behavior and time window, and filter and output the hot topic evolution results.

2. The method according to claim 1, characterized in that The crawling process of the incremental crawler platform includes: Use breadth-first traversal strategy to access target websites and filter valid pages according to set domain name rules and path rules; Only crawl content that has been added or updated since the last crawl time, and limit the crawl scope by time window; A first-in-first-out queue is used to manage the status of crawled URLs, and the crawling time and update status of each URL are recorded.

3. The method according to claim 1, characterized in that The process of preprocessing the text in the data pool includes: Use regular expressions to remove HTML tags and noise characters and extract the text content; Use a predefined stop word list to remove meaningless words and use a conditional random field model to segment the text. A triplet dataset consisting of anchor text, positive samples and negative samples is constructed, where the positive samples are semantically similar to the anchor text and the negative samples are semantically irrelevant to the anchor text.

4. The method according to claim 1, wherein The process of loading a pre-trained text representation model and fine-tuning the model using a triplet loss function based on the triplet data to generate a semantic embedding vector includes: Inputting the triplet dataset into a pre-trained sentence encoding model to generate vector representations of anchor texts, positive samples, and negative samples; Based on the cosine distance between the anchor text and the positive sample, and the cosine distance between the anchor text and the negative sample, a triplet loss function is constructed to maximize the distance between positive and negative samples; The model parameters are updated by minimizing the triplet loss function.

5. The method according to claim 1, wherein The dimensionality reduction process includes: The manifold learning method is used to map high-dimensional semantic vectors to low-dimensional space, preserving the global semantic association and local semantic proximity between texts. The vector distribution in the low-dimensional space is optimized by distance weighting, so that the reduced-dimensional vector maintains the semantic structure of the original high-dimensional space.

6. The method according to claim 1, characterized in that The improved clustering algorithm includes: Calculate the time decay factor based on the time difference of text release and dynamically adjust the weight of cosine similarity; Construct a minimum spanning tree based on the mutual reachability distance and generate a hierarchical clustering structure through pruning operations; The hierarchical structure is traversed to extract clusters whose stability meets a preset threshold as final hot topics.

7. A hot topic monitoring system based on an improved HDBSCAN clustering algorithm, characterized in that: The system comprises: The data collection module is used to crawl news and comment data from social media through incremental crawlers, eliminate redundant content and form a data pool; A preprocessing module, configured to perform word segmentation, stop word filtering, and text content extraction on the text in the data pool, and construct semantic training triples; A model fine-tuning module is used to load a pre-trained text representation model, fine-tune it based on the triple data using a triple loss function, and generate a semantic embedding vector; A dimensionality reduction processing module, configured to reduce the dimension of the semantic embedding vector to a low-dimensional space while preserving the semantic structure; A dynamic clustering module is used to dynamically cluster the vectors after dimensionality reduction using an improved clustering algorithm, wherein the clustering algorithm combines cosine similarity with a time decay factor to adjust the distance between texts; The heat analysis module is used to calculate the topic heat index based on user interaction behavior and time window, and filter and output the hot topic evolution results.

8. The system according to claim 7, characterized in that The data acquisition module includes: Breadth-first crawling unit, used to access the target website using a breadth-first traversal strategy and filter valid pages according to the set domain name rules and path rules; Incremental update unit, used to crawl only the content that has been added or updated since the last crawl time, and limit the crawl range by time window; The queue management unit is used to manage the status of crawled URLs using a first-in-first-out queue and record the crawling time and update status of each URL.

9. A computer terminal device, characterized in that: include: one or more processors; a memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the hot topic monitoring method based on the improved HDBSCAN clustering algorithm as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the hot topic monitoring method based on the improved HDBSCAN clustering algorithm as described in any one of claims 1 to 6 is implemented.

Images