A social media public opinion monitoring method and system based on semantic enhanced short text clustering

Abstract

The application provides a social media public opinion monitoring method and system based on semantic enhanced short text clustering, and relates to the technical field of natural language processing. The method first generates a short text semantic embedding vector by using Sentence-BERT, completes initial clustering by dimension reduction through UMAP and density clustering through HDBSCAN, and constructs initial statistics and cluster semantic centroids. Then, Gibbs sampling iteration is carried out according to the initial cluster distribution, asymmetric adaptive word weight is constructed according to the cross-cluster distribution information entropy of words, background words and discriminant words are distinguished, the word prior symmetry is broken, the cluster label inference is optimized, and the clustering parameters are updated synchronously. Finally, the high-similarity cluster pairs are merged into the preset target cluster number through the centroid cosine similarity, and the public opinion result containing labels, text sets and semantic centroids is output. The method can alleviate the problems of short text semantic sparsity and topic fragmentation, and improve the public opinion topic recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a social media sentiment monitoring method and system based on semantically enhanced short text clustering. Background Technology

[0002] With the rapid development of social media channels such as Weibo, WeChat, and short video platforms, the public expresses their opinions in real time through posting and commenting, leading to an explosive growth in online public opinion data. Media organizations need to continuously monitor and analyze trending events, public opinion trends, and potential risks on social media in order to promptly grasp public opinion dynamics and assist in public decision-making and crisis response. The core task of social media public opinion monitoring is to discover topics, identify public opinion themes, and extract representative keywords from massive amounts of noisy, unlabeled short text data. Short text clustering is the basic technical means to achieve the above goals.

[0003] Existing technologies have proposed various technical solutions for social media text clustering and event detection. For example, patent document CN112527960A discloses a method for detecting sudden events based on keyword clustering. This method extracts and clusters keywords in social media texts to form corresponding event themes and identifies sudden events accordingly. This type of method can, to some extent, discover keywords with sudden characteristics and their corresponding events from large-scale social media data. However, it mainly relies on the statistical characteristics of keywords and the correlation between keywords for event detection, and its utilization of the contextual semantics of short texts and the potential semantic relationships between different words remains limited. When different short texts discussing the same event use synonyms, near-synonyms, abbreviations, internet slang, or different narrative styles, the related texts may lack direct keyword overlap, thus easily being classified into different event themes. At the same time, clustering only keywords is insufficient to fully preserve the overall semantic information of the original short text.

[0004] For example, patent document CN106383877A discloses a method for online short text clustering and topic detection on social media. This method preprocesses social media short texts to construct scalable word frequency vectors and performs online clustering based on the cosine similarity between the short texts and existing categories. Simultaneously, it adjusts the clustering results using word indexing, candidate similarity class merging, and a similarity class merging criterion based on information entropy, and identifies popular topics based on the number of texts in each category and text arrival rate. This method can perform incremental short text clustering and alleviates, to some extent, the problems of high-dimensional sparsity of the word vector space and insufficient merging of categories with the same topic. However, the text representation and similarity calculation used in this scheme are still mainly based on word frequency vectors, shared words, and cosine similarity. Its category merging also relies on statistical indicators such as word overlap, similarity threshold, and information entropy. When two short texts express the same or similar semantics but use significantly different specific words, their similarity in the word frequency vector space may still be low, thus forming multiple separate categories. In addition, this method requires setting multiple parameters such as text-category similarity threshold, category merging threshold, and topic popularity threshold, and its clustering results are easily affected by the threshold settings and the order of text input.

[0005] In general, existing model-based short text clustering methods rely on bag-of-words statistics and word co-occurrence for inference. In traditional bag-of-words models or latent Dirichlet (LDD) topic assignment models, the word frequency statistics of short texts are extremely sparse, resulting in a highly sparse document-word co-occurrence matrix. This makes it difficult for the model to obtain stable and reliable word co-occurrence statistics, severely impacting the accuracy of topic identification. Furthermore, during the evolution of breaking events or trending topics, short texts discussing the same macro-topic often exhibit highly diverse linguistic expressions, forming numerous micro-topic fragments with similar semantics but significantly different vocabulary. Traditional clustering algorithms (such as K-Means and LDA) typically pre-determine a fixed number of clusters, making it unable to adaptively handle scenarios where the number of topics is unknown or dynamically changing.

[0006] In view of this, there is an urgent need for a social media sentiment monitoring method and system based on semantically enhanced short text clustering, in order to at least address the above-mentioned shortcomings. Summary of the Invention

[0007] One of the objectives of this invention is to provide a social media sentiment monitoring method and system based on semantically enhanced short text clustering, in order to solve the problems pointed out in the background art.

[0008] This invention provides a social media sentiment monitoring method based on semantically enhanced short text clustering, comprising: Step S1: Obtain the semantic embedding vector of the short text on social media and perform initial clustering. Assign the obtained initial clusters to the initial clustering statistics required for probability inference and the semantic centroid of each cluster. Step S2: Starting from the initial cluster assignment, in the Gibbs sampling iteration, based on the dynamically updated clustering statistics, calculate the information entropy of the word distribution among the clusters; forward normalize the information entropy to map it into asymmetric adaptive word weights to break the prior symmetry of words, so that uniformly distributed background words are given high prior weights to smooth their distribution, and discriminant words clustered in specific clusters are given low prior weights to enhance their sparsity; substitute the asymmetric adaptive word weights into the Gibbs sampling probability formula to resample the cluster labels of the short text, and simultaneously update the clustering statistics and cluster semantic centroids, and obtain the current number of clusters after iteration; Step S3: If the current number of clusters is greater than the preset target number of clusters, then merge the cluster pairs with the highest similarity based on the vector similarity between the semantic centroids of the clusters until the number of clusters is reduced to the target number; the resulting clusters are the public opinion topic clusters, and the output includes the cluster labels, short text sets and semantic centroids of each topic cluster.

[0009] Preferably, the semantic centroid of each cluster is calculated and updated using the following formula: ; in, Cluster The semantic centroid of the cluster; Indicates the current allocation to the cluster. A collection of short texts; Cluster The total number of short texts contained within; Representing short text The semantic embedding vector.

[0010] Preferably, in step S2, the calculation of information entropy and asymmetric adaptive word weights specifically includes: Calculation vocabulary In cluster Distribution ratio in : ; in, Words In cluster The number of times it appears in; Words In cluster The number of times it appears in Words Total number of occurrences across all clusters; Indicates the preset maximum number of clusters; Represents the smoothing constant; Calculate vocabulary based on distribution ratio. Information entropy distributed among clusters : ; Information entropy A forward normalization process is performed to obtain asymmetric adaptive word weights. : ; in, This represents the maximum value of the information entropy of all words in the vocabulary. This indicates an index of all words in the vocabulary. Summation, The sum of the weights of all words. This is an index of words in the vocabulary list.

[0011] Preferably, in step S2, the Gibbs sampling probability formula, in which the asymmetric adaptive word weights are substituted as prior parameters, is as follows: ; in, For document Cluster allocation; This represents the total number of short texts in the corpus. To remove the document Post-cluster The number of documents; To remove the document Postscript In cluster The number of times it appears in; To remove the document Post-cluster Total word count; For words In the document The number of times it appears in; For vocabulary In the document The Middle The index of the next occurrence is taken as a value. to ; For document Total word count; For document The Middle The index of each word position, with a value of to ; These are cluster prior parameters; For vocabulary Asymmetric adaptive word weighting; This is the sum of the weights of all words; This indicates the preset maximum number of clusters.

[0012] Preferably, the periodic update of word weights is performed according to a preset period. That is, after completing the Gibbs sampling operation of τ short texts, the word distribution ratio, information entropy and asymmetric adaptive word weights are recalculated using the latest clustering statistics, and the normalization constant is updated synchronously.

[0013] Preferably, in step S3, the vector similarity is cosine similarity, and its calculation formula is: ; in, Cluster with cluster Cosine similarity; and These represent the semantic centroids of the two clusters, respectively. This represents the L2 norm of a vector.

[0014] Preferably, in step S3, the specific processing steps for merging the cluster pairs with the highest similarity based on the vector similarity between the semantic centroids of the clusters until the number of clusters is reduced to the target number include: Step S31: Using the semantic centroids of all non-empty clusters as input, calculate the cosine similarity between any two non-empty cluster semantic centroids, construct a max-heap priority queue sorted in descending order of cosine similarity; and count the number of all non-empty clusters, denoted as the current cluster number. Step S32: From the max-heap priority queue, pop the cluster pair with the highest cosine similarity. If the current number of clusters is greater than the target number of clusters and the cosine similarity is greater than the merging threshold, then... If the current cluster number is greater than the target cluster number and the cosine similarity is not greater than 1, then merge is performed; Then the threshold The threshold is dynamically reduced or ignored, and the clusters with the highest similarity are forcibly merged to ensure that the final number of clusters can be reduced to the target number of clusters; if the current number of clusters is not greater than the target number of clusters, step S34 is executed directly. Step S33: Based on the semantic embedding vectors of all short texts in the new cluster, recalculate the cluster semantic centroid of the new cluster, and recalculate the cosine similarity between the new cluster and other non-empty clusters based on the cluster semantic centroid of the new cluster. Update the max-heap priority queue using the recalculated cosine similarity, and decrement the current cluster number by one. Step S34: Determine whether the current number of clusters is greater than the preset target number of clusters; if yes, take the updated max-heap priority queue as input and return to step S32; if no, stop merging and output the current clusters and their corresponding semantic centroids as the public opinion monitoring results.

[0015] Preferably, in step S3, the specific steps of the merging operation include: Step S35: Count the number of all non-empty clusters and record it as the current cluster number; calculate the cosine similarity between each pair of non-empty clusters and construct a max-heap priority queue arranged in descending order of cosine similarity; Step S36: Pop the cluster pair with the highest cosine similarity from the max-heap priority queue. Clusters with larger mean amplitudes of semantic embedding vectors are designated as the primary cluster, and the other cluster as the secondary cluster; if the current number of clusters is greater than the target number of clusters and the cosine similarity is greater than the merging threshold... If the number of short texts in the secondary cluster is greater than the target number of clusters and the cosine similarity is not greater than the merging threshold, then the set of short texts in the secondary cluster will be merged into the primary cluster, and step S37 will be executed; if the current number of clusters is greater than the target number of clusters and the cosine similarity is not greater than the merging threshold, then the set of short texts in the secondary cluster will be merged into the primary cluster, and step S37 will be executed If the merging threshold limit is ignored, the short text set of the sub-cluster will still be merged into the main cluster, and step S37 will be executed; if the current cluster number is not greater than the target cluster number, step S38 will be executed directly. Step S37: Based on the merged primary cluster Recalculate the main cluster using the semantic embedding vectors of all short texts within the cluster. The cluster semantic centroid is used to update the max-heap priority queue involving the main cluster. The similarity value was used to delete the secondary clusters. And decrement the current cluster number by one; Step S38: Determine whether the current number of clusters is greater than the preset target number of clusters; if yes, return to step S36; if no, stop merging and output the current clusters and their corresponding semantic centroids as the public opinion monitoring results.

[0016] Preferably, in step S1, the specific method for obtaining the semantic embedding vector and performing initial clustering is as follows: Raw short text data was collected through social media platform interfaces, and after preprocessing, semantic embedding vectors were generated using the Sentence-BERT model. The UMAP algorithm is used to reduce the dimensionality of the semantic embedding vectors, and the HDBSCAN algorithm is used to perform density clustering on the dimensionality-reduced vectors to obtain the initial cluster assignments. If the number of non-empty clusters generated by density clustering is less than the preset maximum number of clusters, the short texts identified as noise are first assigned to the nearest known cluster according to the cosine similarity with the semantic centroids of each known cluster. If the number of non-empty clusters is still less than the preset maximum number of clusters after adding the noisy short texts, the short texts are split into new clusters from the largest cluster according to the cosine similarity between the short text and the semantic centroid of that cluster in ascending order, until the number of non-empty clusters equals the preset maximum number of clusters.

[0017] This invention provides a social media sentiment monitoring system based on semantically enhanced short text clustering, comprising: The initialization module is used to obtain the semantic embedding vectors of short texts from social media and perform initial clustering. The resulting initial clusters are then assigned and mapped to the initial clustering statistics required for probability inference and the semantic centroids of each cluster. The asymmetric inference module is used to calculate the information entropy of word distribution among clusters based on dynamically updated cluster statistics during Gibbs sampling iterations, starting from the initial cluster assignment. The information entropy is positively normalized and mapped to asymmetric adaptive word weights to break the prior symmetry of words. This results in uniformly distributed background words receiving high prior weights to smooth their distribution, while discriminant words clustered in specific clusters receive low prior weights to enhance their sparsity. The asymmetric adaptive word weights are then substituted into the Gibbs sampling probability formula to resample the cluster labels of the short text, and the cluster statistics and cluster semantic centroids are updated synchronously. After iteration, the current number of clusters is obtained. The public opinion monitoring result output module is used to merge the cluster pairs with the highest similarity based on the vector similarity between the semantic centroids of the clusters if the current number of clusters is greater than the preset target number of clusters, until the number of clusters is reduced to the target number of clusters; the resulting clusters are the public opinion topic clusters, and the output includes the cluster labels, short text sets and semantic centroids of each topic cluster.

[0018] The beneficial effects of this invention are as follows: This invention first generates semantic embedding vectors for short social media texts using the Sentence-BERT model, and completes initial cluster assignment by combining UMAP dimensionality reduction and HDBSCAN density clustering. Then, in the Gibbs sampling iteration, the information entropy of word distribution among clusters is dynamically calculated and positively normalized to map it into asymmetric adaptive word weights. This assigns high prior weights to background words to smooth their distribution and low prior weights to discriminant words to enhance their sparsity, thereby breaking the prior symmetry of words and significantly improving the accuracy of cluster label inference for short texts. Finally, through a hierarchical aggregation strategy based on cosine similarity and a max-heap priority queue, fragmented micro-topic clusters are merged into a target number of macro-level public opinion topic clusters, outputting public opinion monitoring results including cluster labels, short text sets, and cluster semantic centroids. This invention effectively solves the dual challenges of semantic sparsity and topic fragmentation in short social media texts, achieving high-precision public opinion topic identification and monitoring.

[0019] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in this application.

[0020] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of a social media sentiment monitoring method based on semantically enhanced short text clustering in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the process of obtaining semantic embedding vectors and performing initial clustering in an embodiment of the present invention; Figure 3 This is a flowchart of the sampling stage in an embodiment of the present invention; Figure 4 This is a flowchart of the semantic cluster centroid merging stage in an embodiment of the present invention; Figure 5 This is a schematic diagram of a social media sentiment monitoring system based on semantically enhanced short text clustering, as described in an embodiment of the present invention. Detailed Implementation

[0022] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0023] This invention provides a social media sentiment monitoring method based on semantically enhanced short text clustering, such as... Figure 1 As shown, it includes: Step S1: Obtain the semantic embedding vector of the short social media text and perform initial clustering. Assign the obtained initial clusters to the initial clustering statistics required for probability inference and the semantic centroid of each cluster.

[0024] The system acquires a batch of short social media texts to be analyzed through the data acquisition module, which are recorded as the corpus. ,in This represents the total number of short texts. In this embodiment, for the real-time public opinion monitoring scenario on the Weibo platform, the system acquires short text data related to specific keywords using a sliding time window (e.g., collecting data every 10 minutes). The batch processing size is typically 1,000 to 50,000 texts.

[0025] After being cleaned by the preprocessing module, the system calls the Sentence-BERT model to encode each short text and generate semantic embedding vectors. During batch encoding, the GPU batch size is set to 128 to fully utilize the GPU's parallel computing capabilities. For text lengths exceeding the model's maximum input length (typically 512 tokens), the system employs a truncation strategy, retaining only the first 512 tokens.

[0026] The 768-dimensional high-dimensional semantic embedding vector suffers from the "curse of dimensionality" under Euclidean distance metrics, leading to poor direct clustering results. To address this, the system first employs the UMAP (Uniform Manifold Approximation and Projection) algorithm to perform nonlinear dimensionality reduction on the semantic embedding vectors, mapping the 768-dimensional vectors to a low-dimensional manifold space of 5 to 20 dimensions. The key hyperparameter of UMAP is set as: the number of nearest neighbors. minimum distance Distance metric .in, The setting helps to make similar text points after dimensionality reduction form a denser cluster in the low-dimensional space, which is beneficial for subsequent density clustering.

[0027] In the low-dimensional space after UMAP dimensionality reduction, the system uses the HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) algorithm to perform density clustering and obtain the initial cluster assignments. HDBSCAN has two major advantages over the traditional K-Means algorithm: first, it does not require pre-specifying the number of clusters. First, it can adaptively discover the natural boundaries of topic clusters from the data density distribution; second, it can identify sparse short texts as noise points, avoiding the forced classification of low-quality edge texts into arbitrary topic clusters. The key hyperparameter of HDBSCAN is set as: minimum cluster size. A topic cluster must contain at least 10 short texts to be considered a valid topic cluster. The threshold for identifying noise points is controlled.

[0028] After HDBSCAN clustering is completed, the system checks the number of non-empty clusters. If the number of non-empty clusters generated by density clustering is equal to or exceeds the preset maximum number of clusters... If the number of non-empty clusters is less than [a certain value], then proceed directly to the next stage. The system will perform the following completion operations in sequence: First, the system performs semantic nearest neighbor assignment on the short texts labeled as noise (labeled -1) by HDBSCAN: calculating the semantic embedding vector of each noisy short text. semantic centroids of each known nonempty cluster The cosine similarity between the two texts is used to assign the noisy short text to the known cluster with the highest cosine similarity, thus making full use of all short text data.

[0029] If, after noise compensation, the number of non-empty clusters is still less than [a certain value], then [the problem is likely The system performs a splitting operation on the largest non-empty cluster (i.e., containing the most short texts): it calculates the cosine similarity between all short texts in the cluster and the semantic centroid of the cluster, sorts them from low to high similarity, and splits the top-ranked short texts (i.e., those most semantically distant from the cluster centroid) into new clusters to ensure that the new clusters have a certain degree of semantic cohesion. This splitting operation is repeated until the number of non-empty clusters equals the preset maximum number of clusters. , Figure 2 This is a flowchart illustrating the process of obtaining semantic embedding vectors and performing initial clustering in this invention.

[0030] After initial clustering, the system calculates the initial clustering statistics required for subsequent probability inference based on the initial cluster assignment results. The main initial clustering statistics include: the number of documents in each cluster. (Initially equal to HDBSCAN assigned to cluster) (Total number of documents); vocabulary-cluster co-occurrence matrix ,in For vocabulary In cluster The initial number of occurrences in the cluster is determined by the cluster. The total number of words in each cluster is obtained by summing the word frequency statistics of all short texts within the cluster. .

[0031] At the same time, the system calculates the initial cluster semantic centroid of each cluster. Specifically, the mean of the semantic embedding vectors of all short texts within the cluster is used: ; in, Assigned to the current cluster A collection of short texts Its base.

[0032] At this point, step S1 is complete, and the system outputs the initial clustering statistics. and the semantic centroid of each cluster Then, pass it to step S2.

[0033] Step S2: Starting from the initial cluster assignment, in the Gibbs sampling iteration, based on the dynamically updated clustering statistics, calculate the information entropy of the word distribution among the clusters; normalize the information entropy to map it into asymmetric adaptive word weights to break the prior symmetry of words, so that uniformly distributed background words are given high prior weights to smooth their distribution, and discriminant words clustered in specific clusters are given low prior weights to enhance their sparsity; substitute the asymmetric adaptive word weights into the Gibbs sampling probability formula to resample the cluster labels of the short text, and update the clustering statistics and cluster semantic centroids simultaneously, and obtain the current number of clusters after iteration.

[0034] Step S2 is the core calculation stage of the technical solution of the present invention, which integrates asymmetric adaptive word weight calculation and topic inference based on Collapsed Gibbs Sampling, and realizes fine-grained semantic correction and optimization for the initial clustering result.

[0035] The first stage of Step S2 is the calculation of information entropy and asymmetric adaptive word weights.

[0036] In each word weight update cycle (that is, after every preset cycle Gibbs sampling operations on short texts), the system obtains from the current word-cluster co-occurrence matrix , the following calculations are performed sequentially:[ For the vocabulary each word in , first calculate the normalized distribution ratio with Laplace smoothing in each cluster :[ ; wherein, the smoothing constant is introduced to prevent the zero-probability problem: when a certain word does not appear in a specific cluster (which is extremely common in short text scenarios), the probability of its distribution ratio will not be zero, so as to avoid the occurrence in information entropy calculation numerical overflow problem. In this embodiment, is set to .

[0037] the denominator of is equal to the total number of occurrences of the word in the entire corpus plus times the smoothing constant, which ensures that for all after summation, there is , that is form a valid probability distribution.

[0038] Based on the normalized distribution ratio, calculate the Shannon information entropy of the word distribution among clusters: ; the value range of . When the word is uniformly distributed in all clusters (such as common stop words "de", "le", etc.) approaches the maximum value , which reflects that the word does not have topic distinguishing ability; when the word When highly concentrated on a specific topic cluster (e.g., "magnitude" is concentrated on the topic of "earthquake disaster") The value approaching 0 reflects the strong topic-specificity of this word.

[0039] By performing positive normalization on the information entropy, we obtain asymmetric adaptive word weights: ; in The maximum value of the information entropy of all words in the vocabulary serves to normalize the denominator and ensure... Further calculate the sum of the weights of all words. , which serves as the normalization constant in the Gibbs sampling formula. Furthermore, to address the potential risk of numerical overflow in information entropy calculation, this system employs a numerical stabilization mechanism to handle logarithmic space operations in information entropy calculation: specifically, the log-sum-exp numerical stabilization algorithm is used, and a minimum lower bound threshold is set to prevent log(0) overflow from causing numerical precision decay. Further, the system executes word weight updates through a multi-threaded concurrent computation framework, sharding the vocabulary by hash value and performing calculations in multiple processing threads. During concurrency, a read-write lock mechanism coordinates global concurrent access, and finally, atomic operations synchronously update the global clustering statistics cache.

[0040] In engineering implementation, all of the above calculations can be performed in batches using vectorization. Specifically, the word-cluster co-occurrence matrix... This can be done directly using NumPy matrix operations. and The full computation time for a single update is typically no more than 100 milliseconds for a vocabulary of tens of thousands, which meets the computation timeliness requirements of real-time monitoring scenarios.

[0041] The second stage of step S2 is an iterative Gibbs sampling driven by asymmetric word weights.

[0042] After initializing the word weights, the system enters the Gibbs sampling iteration loop. In each iteration, the system traverses every short text in the corpus. And perform the following operations in sequence: First, the system removes the document. Contribution to cluster statistics: (i.e., from the document) Current Cluster (Subtract one from the document count) (i.e., from the document) (Minus one from the cluster-word count for each word) and (Subtract the number of documents from the total number of words in the corresponding cluster) The total word count is updated to the statistics buffer in memory. The index here... This indicates that the document has been excluded. The contribution statistics, which is standard practice in collapsed Gibbs sampling (deriving the closed-form derivation after marginal integration of the vocabulary distribution parameters), aim to obtain the cluster assignments of documents given all other documents. The complete conditional posterior distribution of cluster allocation is determined. In this process, to address the issue that frequent random access to large-scale document sets at the system's underlying level can easily lead to cache invalidation, this embodiment employs a document batch processing-based memory access optimization strategy. By pre-allocating incremental statistical buffers to each core processing thread and implementing locally optimized document traversal order during traversal read / write (such as sequential grouping and reading based on initial cluster labels), and finally utilizing an incremental statistical update mechanism in conjunction with underlying memory alignment and prefetch instructions, the random memory access span and invalid memory bandwidth usage are significantly reduced.

[0043] Subsequently, for each cluster in the corpus Calculate documents Belongs to cluster Unnormalized posterior probability: ; The above formula is derived from the standard Dirichlet-polynomial conjugate derivation, and its physical meaning can be decomposed into the product of two factors: First factor The document-cluster distribution factor reflects the cluster distribution. The size of the cluster determines its attractiveness; larger clusters are more likely to attract new documents. The Dirichlet prior parameter for the cluster size (set to in this embodiment) ); The second factor is the vocabulary-cluster distribution factor, which reflects the document's... All words in the cluster The comprehensive likelihood that appears in the equation. The product terms in the molecule. This addresses the issue of the same word appearing multiple times in a document. For vocabulary In the document The Middle (the sequence number of the next occurrence). As asymmetric word prior parameters, it replaces the traditional fixed symmetric prior. Multiplication terms in the denominator As the normalization factor, Traversing documents The sequence number of each word position in the text.

[0044] The system is for all The unnormalized probabilities of each cluster are normalized to obtain a discrete probability distribution for cluster assignment. Sampling is then performed based on this distribution to obtain the document. New cluster label .

[0045] After sampling is completed, the system will send the document. New cluster label The corresponding cluster statistics ( , and Update to the buffer. Figure 3 This is a flowchart of the sampling phase.

[0046] According to the preset cycle The system completes each After Gibbs sampling of each short text, a periodic update of the asymmetric adaptive word weights is triggered: the word distribution ratio, information entropy, and asymmetric adaptive word weights are recalculated using the latest clustering statistics, and the normalization constant is updated synchronously. It should be clarified here that the clustering statistics (number of documents per cluster, word cluster co-occurrence count, and total number of words per cluster) are updated in real time during the Gibbs sampling removal and redistribution steps of each short text, periodically. What is controlled is the frequency of re-deriving word weights based on the latest statistics, rather than the update frequency of the statistics themselves.

[0047] Simultaneously, after each Gibbs sampling iteration, the system synchronously updates the semantic centroids of each cluster based on the updated clustering statistics. This ensures that the semantic centroid of the cluster always accurately reflects the latest semantic distribution of the text within the current cluster.

[0048] The entire Gibbs sampling iteration process continues for a preset number of iteration rounds (set to 100 rounds in this embodiment), or until the change ratio of the clustering results between two adjacent rounds is lower than a preset convergence threshold (set to 1% in this embodiment). After the iteration converges, the system counts the number of all non-empty clusters to obtain the current cluster number, and passes it to step S3.

[0049] Step S3: If the current number of clusters is greater than the preset target number of clusters, then merge the cluster pairs with the highest similarity based on the vector similarity between the semantic centroids of the clusters until the number of clusters is reduced to the target number; the resulting clusters are the public opinion topic clusters, and the output includes the cluster labels, short text sets and semantic centroids of each topic cluster.

[0050] Step S2 involves Gibbs sampling iterations with a preset maximum number of clusters. Using this as an upper bound for inference, the actual number of non-empty clusters after iterative convergence is usually less than [the upper bound]. However, due to The initial setting is often greater than the actual number of topics (this is to ensure that all fine-grained micro-topic fragments can be captured). Even after semantic refinement through Gibbs sampling, some semantically similar micro-topics may still exist in the form of independent clusters, resulting in the current number of clusters being higher than the macro target number of topics expected by the operators.

[0051] Step S3 uses a hierarchical semantic aggregation strategy to further compress the current set of micro-topic clusters to the target number of clusters. (i.e., the number of target clusters) The target clusters are preset by the operations staff based on business needs. For example, for public opinion analysis of a specific emergency, the target clusters can be set to 5 to 10 macro-topics. Step S3 specifically uses cosine similarity for the vector similarity between two clusters. and The calculation formula is as follows: ; in and These are the semantic centroids of the two clusters, This represents the L2 norm of the vector. To significantly accelerate computationally intensive operations between large-scale high-dimensional semantic vectors, the system further optimizes the computation through a vectorized computation framework based on the SIMD instruction set. It calls underlying hardware-level feature instructions such as AVX2 or AVX-512 to perform batch matrix acceleration operations on the cosine similarity calculation of the associated vector dot product and norm-level normalization. Cosine similarity measures the directional consistency of the semantic centroids of two clusters. Compared to Euclidean distance, cosine similarity is insensitive to magnitude in high-dimensional semantic vector spaces, focusing only on directional differences, making it more suitable for measuring the degree of semantic association between topics. A preset merging threshold is used. As a priority criterion for merging: when the cosine similarity of the two clusters exceeds... When the current number of clusters is still greater than the target number of clusters, but the cosine similarity of all cluster pairs does not exceed a certain threshold, the cluster pair is given priority as a merging candidate. To ensure that the number of clusters drops to the target number, forced merging is performed on the cluster pairs with the highest cosine similarity.

[0052] In this invention, the hierarchical semantic aggregation in step S3 provides two equivalent implementation methods: The first implementation of step S3 (steps S31-S34): Execute step S31. The system extracts the semantic centroid of each cluster from all non-empty clusters output in step S2. First, count the total number of non-empty clusters, and denote it as the current cluster number. Then, pair all non-empty clusters together and calculate the cluster number for each pair. ( cosine similarity All cluster pairs and their corresponding cosine similarities are organized into a max-heap data structure, ensuring that the top of the heap always maintains the cluster pair with the highest current cosine similarity. The time complexity of constructing the max-heap is O(log n). ,in This represents the total number of pairings in the non-empty cluster.

[0053] Execute step S32. The system pops the top element from the max-heap to obtain the cluster pair with the highest current cosine similarity.

[0054] The system calculates the current number of clusters and the merging threshold. Based on the comprehensive judgment, the following three-branch logic is executed: if the current cluster number is greater than the target cluster number and the cosine similarity is greater than the merging threshold. If the cluster pair is semantically sufficiently similar, the set of short texts contained in the cluster pair is taken as the union to obtain a new merged cluster, and the process proceeds to step S33; if the current number of clusters is greater than the target number of clusters but the cosine similarity is not greater than the merging threshold. To ensure that the final number of clusters can be reduced to the target number of clusters, the merging threshold limit is ignored, and the set of short texts contained in the cluster pair is still merged to obtain the new cluster after merging, and then proceed to step S33; if the current number of clusters is not greater than the target number of clusters, then directly jump to step S34 to perform the termination judgment.

[0055] Merging threshold A reasonable setting of the similarity criterion has a significant impact on process quality. If the similarity is too high (e.g., 0.95), the priority merging phase becomes more conservative, merging only clusters with almost identical semantics. However, when the similarity of all cluster pairs does not exceed a certain threshold... At this point, the algorithm will automatically ignore the threshold limit and force-merge the clusters with the highest similarity to ensure that the final number of clusters is reduced to the target number. It should be noted that the clusters forcibly merged at this time have relatively large semantic differences, which may affect the cohesion of the topic clusters. Too low a threshold (e.g., 0.5) may force-merge topics with significant semantic differences during the priority merging stage, also impairing the quality of the results. In this embodiment, for Chinese social media corpora, it is recommended to... The initial value is set between 0.75 and 0.85, and operators can make minor adjustments based on the actual business context.

[0056] Execute step S33. After merging, the system uses the semantic embedding vectors of all short texts within the new cluster. The semantic centroid of the new cluster is recalculated according to the formula for the semantic centroid of the cluster. Subsequently, the system... Based on this, the cosine similarity between the new cluster and all other remaining non-empty clusters is recalculated, and the max-heap priority queue is updated with these newly calculated similarity values ​​(i.e., all nodes in the max-heap involving the two original clusters being merged are removed, and the newly calculated similarity pairs are inserted into the max-heap). The time complexity of the max-heap update operation is O(log n). The overall efficiency is far superior to rebuilding the similarity matrix completely after each merge. Furthermore, during this process, the system simultaneously employs a heap maintenance algorithm based on incremental updates: for heap elements to be merged and deleted, lazy deletion is used instead of immediate full heap reconstruction. Forced global max-heap batch reconstruction is triggered only when the proportion of deleted nodes in the heap structure exceeds a preset threshold. Simultaneously, a similarity buffer pool mechanism directly skips and reuses valid similarity calculation results from unchanged historical data, ensuring that the entire semantic cluster merging is not limited by the high overhead of data structure heap maintenance. At the same time, the current cluster number is decremented by one to reflect the fact that a single merge operation reduces the total number of clusters by one.

[0057] Execute step S34. The system determines whether the current cluster number after subtracting one is greater than the preset target cluster number. If yes, then take the updated max-heap priority queue as input and return to step S32 to continue the next round of merging; if no (i.e., the current number of clusters is equal to or less than the target number of clusters), then stop merging and output the current clusters and their corresponding semantic centroids as the final public opinion monitoring results. Figure 4 This is a flowchart of the semantic cluster centroid merging stage.

[0058] The second implementation of step S3 (steps S35-S38): The second implementation method is completely equivalent to the first in its core algorithm logic. The difference lies in that the second implementation method explicitly assigns specific identifiers to the two clusters to be merged. The cluster with the higher cosine similarity is retained as the principal cluster (denoted as...). The clusters with lower cosine similarity are merged as secondary clusters (denoted as...). ), and explicitly delete from cluster after the merge is complete. Preserve and update the primary cluster This explicit identifier design facilitates cluster-level logging and tracing in engineering implementation, making it suitable for public opinion monitoring scenarios that require tracing the merged history.

[0059] Execute step S35. The system counts the number of all non-empty clusters and records it as the current cluster number; calculates the cosine similarity between each pair of non-empty clusters and constructs a max-heap priority queue arranged in descending order of cosine similarity.

[0060] Execute step S36. The system pops the cluster pairs with the highest cosine similarity from the maximum heap priority queue, designates the cluster with more documents (or a larger mean amplitude of semantic embedding vectors) as the primary cluster, and the other cluster as the secondary cluster; the system then merges the clusters based on the current number of clusters and the merging threshold. Based on the comprehensive judgment, the following three-branch logic is executed: if the current number of clusters is greater than the target number of clusters and the cosine similarity is greater than the merging threshold. If the current number of clusters is greater than the target number of clusters but the cosine similarity is not greater than the merging threshold, then all short text sets of the clusters will be merged into the cluster, and the process will proceed to step S37; if the current number of clusters is greater than the target number of clusters but the cosine similarity is not greater than the merging threshold, then... If the number of clusters is reduced to the target number, the merging threshold limit is ignored, and all short text sets of the clusters are merged into the clusters, and the process proceeds to step S37; if the current number of clusters is not greater than the target number of clusters, step S38 is executed directly.

[0061] Execute step S37. The system is based on the merged cluster. All short texts (including the original) and the original The semantic embedding vectors of all short texts are used to recalculate the clusters. Find the cluster semantic centroid and update all involved clusters in the max-heap priority queue. The similarity value; at the same time, the cluster is removed from the current cluster set and the max-heap priority queue. All relevant entries, and decrement the current cluster number by one.

[0062] Execute step S38. The system determines whether the current number of clusters is greater than the preset target number of clusters. If yes, return to step S36; otherwise, stop merging and output the current clusters and their corresponding semantic centroids as the public opinion monitoring results.

[0063] Merging threshold Selection principle: When the similarity is too high (e.g., 0.95), the priority merging stage only merges clusters with almost identical semantics; when the similarity of all remaining cluster pairs does not exceed a certain threshold... At this point, the algorithm will ignore the threshold limit and perform forced merging to ensure that the final number of clusters can be reduced to the target number, preventing the algorithm from exiting prematurely and failing to reach the target number of clusters due to an excessively high threshold. However, the clusters forcibly merged at this time have relatively low semantic relevance, which has a certain impact on the cohesion of topic clusters. If the threshold is too low (e.g., 0.50), topics with significant semantic differences may be forcibly merged in the priority merging stage, which also impairs the quality of the results. In this embodiment, for Chinese social media corpora, it is recommended to... The initial value is set between 0.75 and 0.85; if the target scenario is fine-grained tracking of sudden events (requiring the retention of more sub-topics), it can be appropriately increased. If the target scenario is a general presentation of the overall public opinion situation (requiring the condensation of at least a few main topics), the level can be appropriately reduced. Operations personnel can make minor adjustments based on the actual business context.

[0064] After step S3 is completed, the system will structure each topic cluster into a public opinion monitoring result. The output includes: a unique identifier (cluster tag) for each topic cluster; a collection of all original short texts contained in the cluster (which may include metadata such as the original publication time and user information); the semantic centroid vector of the cluster (which can be used for subsequent topic naming, trend tracking, and similar topic retrieval); and the number and proportion of short texts contained in each topic cluster (used to measure the popularity of each topic).

[0065] The results output module pushes the above public opinion monitoring results to the downstream system in JSON format via API. The downstream system can then perform advanced functions such as automated topic naming (e.g., generating topic keywords by extracting high-weight discriminant words within a cluster), public opinion trend analysis (tracking the centroid trajectory of topics across multiple time windows), and early warning triggering (automatically triggering early warning notifications when the popularity of a specific topic rises sharply in a short period of time).

[0066] This invention provides a social media sentiment monitoring system based on semantically enhanced short text clustering, such as... Figure 5 As shown, it includes: Initialization module 1 is used to obtain the semantic embedding vector of short texts on social media and perform initial clustering, and assign the obtained initial clusters to the initial clustering statistics required for probability inference and the cluster semantic centroid of each cluster; The asymmetric inference module 2 is used to calculate the information entropy of the word distribution among clusters based on dynamically updated cluster statistics during Gibbs sampling iterations, starting from the initial cluster assignment. The information entropy is positively normalized and mapped to asymmetric adaptive word weights to break the prior symmetry of words, so that uniformly distributed background words are given high prior weights to smooth their distribution, and discriminant words clustered in specific clusters are given low prior weights to enhance their sparsity. The asymmetric adaptive word weights are substituted into the Gibbs sampling probability formula to resample the cluster labels of the short text, and the cluster statistics and cluster semantic centroids are updated synchronously. After iteration, the current number of clusters is obtained. The public opinion monitoring result output module 3 is used to merge the cluster pairs with the highest similarity based on the vector similarity between the semantic centroids of the clusters if the current number of clusters is greater than the preset target number of clusters, until the number of clusters is reduced to the target number of clusters; the resulting clusters are the public opinion topic clusters, and the output includes the cluster labels, short text sets and semantic centroids of each topic cluster.

[0067] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A social media sentiment monitoring method based on semantically enhanced short text clustering, characterized in that, include: Step S1: Obtain the semantic embedding vector of the short text on social media and perform initial clustering. Assign the obtained initial clusters to the initial clustering statistics required for probability inference and the semantic centroid of each cluster. Step S2: Starting from the initial cluster assignment, in the Gibbs sampling iteration, based on the dynamically updated clustering statistics, calculate the information entropy of the word distribution among the clusters; forward normalize the information entropy to map it into asymmetric adaptive word weights to break the prior symmetry of words, so that uniformly distributed background words are given high prior weights to smooth their distribution, and discriminant words clustered in specific clusters are given low prior weights to enhance their sparsity; substitute the asymmetric adaptive word weights into the Gibbs sampling probability formula to resample the cluster labels of the short text, and simultaneously update the clustering statistics and cluster semantic centroids, and obtain the current number of clusters after iteration; Step S3: If the current number of clusters is greater than the preset target number of clusters, then merge the cluster pairs with the highest similarity based on the vector similarity between the semantic centroids of the clusters until the number of clusters is reduced to the target number; the resulting clusters are the public opinion topic clusters, and the output includes the cluster labels, short text sets and semantic centroids of each topic cluster.

2. The social media sentiment monitoring method based on semantically enhanced short text clustering as described in claim 1, characterized in that, The semantic centroid of each cluster is calculated and updated using the following formula: ； in, Cluster The semantic centroid of the cluster; Indicates the current allocation to the cluster A collection of short texts; Cluster The total number of short texts contained within; Representing short text The semantic embedding vector.

3. The social media sentiment monitoring method based on semantically enhanced short text clustering as described in claim 1, characterized in that, In step S2, the calculation of information entropy and asymmetric adaptive word weights specifically includes: Calculation vocabulary In cluster Distribution ratio in : ； in, Words In cluster The number of times it appears in; Words In cluster The number of times it appears in Words Total number of occurrences across all clusters; Indicates the preset maximum number of clusters; Represents the smoothing constant; Calculate vocabulary based on distribution ratio. Information entropy distributed among clusters : ； Information entropy A forward normalization process is performed to obtain asymmetric adaptive word weights. : ； in, This represents the maximum value of the information entropy of all words in the vocabulary. This indicates an index of all words in the vocabulary. Summation, The sum of the weights of all words. This is an index of words in the vocabulary list.

4. The social media sentiment monitoring method based on semantically enhanced short text clustering as described in claim 3, characterized in that, In step S2, the Gibbs sampling probability formula, which substitutes the asymmetric adaptive word weights as prior parameters, is as follows: ； in, For document Cluster allocation; This represents the total number of short texts in the corpus. To remove the document Post-cluster The number of documents; To remove the document Postscript In cluster The number of times it appears in; To remove the document Post-cluster Total word count; For words In the document The number of times it appears in; For vocabulary In the document The Middle The index of the next occurrence is taken as a value. to ; For document Total word count; For document The Middle The index of each word position, with a value of to ; These are cluster prior parameters; For vocabulary Asymmetric adaptive word weighting; This is the sum of the weights of all words; This indicates the preset maximum number of clusters.

5. The social media sentiment monitoring method based on semantically enhanced short text clustering as described in claim 4, characterized in that, The periodic updates of word weights are performed according to a preset cycle, that is, every time the word weights are updated... After performing Gibbs sampling on each short text, the word distribution ratio, information entropy, and asymmetric adaptive word weights are recalculated using the latest clustering statistics, and the normalization constant is updated simultaneously.

6. The social media sentiment monitoring method based on semantically enhanced short text clustering as described in claim 1, characterized in that, In step S3, the vector similarity is cosine similarity, and its calculation formula is: ； in, Cluster with cluster Cosine similarity; and These represent the semantic centroids of the two clusters, respectively. This represents the L2 norm of a vector.

7. The social media sentiment monitoring method based on semantically enhanced short text clustering as described in claim 6, characterized in that, In step S3, the specific processing steps for merging the cluster pairs with the highest similarity based on the vector similarity between the semantic centroids of the clusters until the number of clusters is reduced to the target number include: Step S31: Using the semantic centroids of all non-empty clusters as input, calculate the cosine similarity between any two non-empty cluster semantic centroids, construct a max-heap priority queue sorted in descending order of cosine similarity; and count the number of all non-empty clusters, denoted as the current cluster number. Step S32: From the max-heap priority queue, pop the cluster pair with the highest cosine similarity. If the current number of clusters is greater than the target number of clusters and the cosine similarity is greater than the merging threshold, then... If the current cluster number is greater than the target cluster number and the cosine similarity is not greater than 1, then merge is performed; Then the threshold The threshold is dynamically reduced or ignored, and the clusters with the highest similarity are forcibly merged to ensure that the final number of clusters can be reduced to the target number of clusters; if the current number of clusters is not greater than the target number of clusters, step S34 is executed directly. Step S33: Based on the semantic embedding vectors of all short texts in the new cluster, recalculate the cluster semantic centroid of the new cluster, and recalculate the cosine similarity between the new cluster and other non-empty clusters based on the cluster semantic centroid of the new cluster. Update the max-heap priority queue using the recalculated cosine similarity, and decrement the current cluster number by one. Step S34: Determine whether the current number of clusters is greater than the preset target number of clusters; if yes, take the updated max-heap priority queue as input and return to step S32; if no, stop merging and output the current clusters and their corresponding semantic centroids as the public opinion monitoring results.

8. The social media sentiment monitoring method based on semantically enhanced short text clustering as described in claim 6, characterized in that, In step S3, the specific steps of the merging operation include: Step S35: Count the number of all non-empty clusters and record it as the current cluster number; calculate the cosine similarity between each pair of non-empty clusters and construct a max-heap priority queue arranged in descending order of cosine similarity; Step S36: Pop the cluster pair with the highest cosine similarity from the max-heap priority queue. Clusters with larger mean amplitudes of semantic embedding vectors are designated as the primary cluster, and the other cluster as the secondary cluster; if the current number of clusters is greater than the target number of clusters and the cosine similarity is greater than the merging threshold... If the number of subclusters is greater than the target number of clusters and the cosine similarity is not greater than the merging threshold, then the short text set of the subclusters will be merged into the main cluster, and step S37 will be executed; if the current number of clusters is greater than the target number of clusters and the cosine similarity is not greater than the merging threshold, then... If the merging threshold limit is ignored, the short text set of the sub-cluster will still be merged into the main cluster, and step S37 will be executed; if the current cluster number is not greater than the target cluster number, step S38 will be executed directly. Step S37: Based on the merged primary cluster Recalculate the main cluster using the semantic embedding vectors of all short texts within the cluster. The cluster semantic centroid is used to update the max-heap priority queue involving the main cluster. The similarity value was used to delete the secondary clusters. And decrement the current cluster number by one; Step S38: Determine whether the current number of clusters is greater than the preset target number of clusters; if yes, return to step S36; if no, stop merging and output the current clusters and their corresponding semantic centroids as the public opinion monitoring results.

9. The social media sentiment monitoring method based on semantically enhanced short text clustering as described in claim 1, characterized in that, In step S1, the specific method for obtaining the semantic embedding vector and performing initial clustering is as follows: Raw short text data was collected through social media platform interfaces, and after preprocessing, semantic embedding vectors were generated using the Sentence-BERT model. The UMAP algorithm is used to reduce the dimensionality of the semantic embedding vectors, and the HDBSCAN algorithm is used to perform density clustering on the dimensionality-reduced vectors to obtain the initial cluster assignments. If the number of non-empty clusters generated by density clustering is less than the preset maximum number of clusters, the short texts identified as noise are first assigned to the nearest known cluster according to the cosine similarity with the semantic centroids of each known cluster. If the number of non-empty clusters is still less than the preset maximum number of clusters after adding the noisy short texts, the short texts are split into new clusters from the largest cluster according to the cosine similarity between the short text and the semantic centroid of that cluster in ascending order, until the number of non-empty clusters equals the preset maximum number of clusters.

10. A social media sentiment monitoring system based on semantically enhanced short text clustering, characterized in that, include: The initialization module is used to obtain the semantic embedding vectors of short texts from social media and perform initial clustering. The resulting initial clusters are then assigned and mapped to the initial clustering statistics required for probability inference and the semantic centroids of each cluster. The asymmetric inference module is used to calculate the information entropy of word distribution among clusters based on dynamically updated cluster statistics during Gibbs sampling iterations, starting from the initial cluster assignment. The information entropy is positively normalized and mapped to asymmetric adaptive word weights to break the prior symmetry of words. This results in uniformly distributed background words receiving high prior weights to smooth their distribution, while discriminant words clustered in specific clusters receive low prior weights to enhance their sparsity. The asymmetric adaptive word weights are then substituted into the Gibbs sampling probability formula to resample the cluster labels of the short text, and the cluster statistics and cluster semantic centroids are updated synchronously. After iteration, the current number of clusters is obtained. The public opinion monitoring result output module is used to merge the cluster pairs with the highest similarity based on the vector similarity between the semantic centroids of the clusters if the current number of clusters is greater than the preset target number of clusters, until the number of clusters is reduced to the target number of clusters; the resulting clusters are the public opinion topic clusters, and the output includes the cluster labels, short text sets and semantic centroids of each topic cluster.

Images