A big data topic analysis method based on an embedding model

Abstract

This invention relates to a big data topic analysis method based on an embedding model. First, the Sentence-BERT model is used to perform sentence embedding representation on preprocessed Chinese text data. Then, the UMAP projection dimensionality reduction algorithm is used to reduce the dimensionality of the embedded vectors. Next, the HDBSCAN clustering algorithm is used to cluster the dimensionality-reduced vectors. Based on the assignment of each Chinese text in the target Chinese dataset to a corresponding topic class, the Chinese words with the highest c-TF-IDF scores are selected to represent each topic class. Finally, the DSG model is used to perform word embedding representation on the topic words, calculating the similarity between different topic words and between different topic classes, thereby detecting the volatility of newly emerging topic classes. The entire scheme design has higher topic consistency and topic diversity, and can detect new hot topics in a timely and accurate manner, providing early warnings.

Description

Technical Field

[0001] This invention relates to a big data topic analysis method based on an embedding model, belonging to the field of big data topic detection and analysis technology. Background Technology

[0002] Government hotlines are an important channel for interaction between the government and the public, and many cities in China have already established them. Citizens can use these hotlines to report their concerns and issues related to their livelihoods and track the progress of their responses. These hotlines help citizens resolve problems more effectively and improve public satisfaction.

[0003] At the micro level, the application of government service hotlines primarily focuses on basic livelihood issues, absorbing citizens' demands and resolving social problems of concern to them. At the macro level, the citizen service hotline platform serves as an effective bridge for communication between relevant government departments and citizens, playing a significant role in innovating social governance and promoting the construction of a service-oriented government. With the gradual rollout of the hotline system, many cities are handling tens of thousands of work orders daily, with annual work orders reaching millions. Especially in the face of sudden outbreaks like the pandemic, the demands on government service hotlines in handling livelihood issues are even higher, involving efficiency, speed, and effectiveness. By identifying and addressing social hotspots and difficult issues and extracting key themes, the hotline aims to improve its precision in service, gradually achieving the service goal of "proactive intervention before complaints arise."

[0004] Once a citizen's complaint is effectively handled and resolved, the issue resolution work order is considered closed. However, for analysts, these massive amounts of complaints can be organized and analyzed. Problems can be categorized and hot topics analyzed in detail. Furthermore, using map functions and text mining clustering techniques, in-depth analysis and visualization of trending and sensitive words can be achieved. By identifying the thematic focus of these work orders, the direction of public concerns regarding livelihood issues can be revealed. Simultaneously, intelligent alerts can be generated based on the number of work orders falling under different categories, providing better reference and guidance for decision-makers.

[0005] The content of government hotline requests consists of numerous subjective descriptions of issues raised by citizens. Although the text lengths vary, the information contained is relatively rich. However, how to extract the hidden semantic structure information within these texts remains a major research challenge in the fields of natural language processing and text retrieval. Topic modeling is a popular and effective method that mines semantic structure information from text by analyzing high-order co-occurrence patterns of words within a document. Topic modeling techniques have been applied to various research fields with promising results. While LDA (Local Derivative Analysis) topic modeling methods have been widely used in various natural language processing tasks, its effectiveness is often less than ideal when modeling topics in short texts like government hotlines. Furthermore, there is no complete and efficient system for detecting topic volatility after topic extraction. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a big data topic analysis method based on an embedded model, which adopts a novel model structure and control strategy to efficiently and accurately achieve topic-based analysis and hot topic volatility detection of a target Chinese dataset.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention designs a big data topic analysis method based on an embedding model, which realizes topic class analysis of the target Chinese collection according to the following steps A to E;

[0008] Step A. Perform data preprocessing operations on each target Chinese text in the target Chinese set to obtain the corresponding Chinese word segments for each target Chinese text, and then proceed to Step B;

[0009] Step B. Using the Sentence-BERT model, which is fine-tuned based on the pre-trained BERT model, sentence embedding representations are performed for each Chinese word segment corresponding to each target Chinese text to obtain the high-dimensional sentence embedding vectors corresponding to each target Chinese text, and then proceed to step C;

[0010] Step C. Based on the preset neighborhood size information and the preset low-dimensional target space, use the Unified Manifold Approximation and Projection Dimensionality Reduction (UMAP) algorithm to perform dimensionality reduction processing on the high-dimensional sentence embedding vectors of each target Chinese text, and obtain the low-dimensional sentence embedding vectors corresponding to each target Chinese text, and then proceed to step D;

[0011] Step D. Based on the preset minimum generated cluster size, apply the hierarchical and density-based HDBSCAN clustering algorithm to cluster the low-dimensional sentence embedding vectors corresponding to each target Chinese text to obtain the corresponding clusters, which are the topic classes corresponding to the target Chinese set, and then proceed to step E.

[0012] Step E. Based on the low-dimensional sentence embedding vectors of each topic class corresponding to the target Chinese dataset, obtain the Chinese word segments of each target Chinese text included in each topic class. Then, apply the c-TF-IDF algorithm to obtain the c-TF-IDF scores of each Chinese word segment in each topic class, and obtain the preset value of the highest c-TF-IDF score in each topic class. Each Chinese word segment constitutes a topic word in each topic category corresponding to the target Chinese dataset, thereby realizing topic category analysis of the target Chinese dataset.

[0013] As a preferred technical solution of the present invention: it further includes step F, which is performed after step E;

[0014] Step F. Based on the word embedding representation of the topic words using the DSG model, obtain the high-dimensional vectors corresponding to each topic word in each topic class of the target Chinese dataset. Then, combine the historical topic classes in the historical topic class set and the topic words in the historical topic classes, and through similarity calculation, realize the analysis of new topic classes in each topic class of the target Chinese dataset compared with the historical topic class set.

[0015] As a preferred technical solution of the present invention: In step A, firstly, target Chinese texts not expressed by the text expresser in the target Chinese collection are removed; then, fine-grained Chinese word segmentation is performed on each target Chinese text in the target Chinese collection to obtain each Chinese word segment corresponding to the target Chinese text, and Chinese stop words are removed and the Chinese word segments corresponding to the target Chinese text are updated; thus, each Chinese word segment corresponding to each target Chinese text is obtained.

[0016] As a preferred embodiment of the present invention: step B includes steps B1 to B4;

[0017] Step B1. Based on the BERT model which takes Chinese word segmentation as input and the embedding vector corresponding to the Chinese word segmentation as output, connect the average pooling layer to the output of the BERT model to construct a Sentence-BERT model which takes each Chinese word segment corresponding to the Chinese text as input and the high-dimensional sentence embedding vector corresponding to the Chinese text as output, and then proceed to step B2.

[0018] Step B2. Obtain the high-dimensional sentence embedding vectors corresponding to the two sentences in the sentence pair using the Sentence-BERT model. Combine this with the Siamese network classification objective function used to detect whether the meanings of the two high-dimensional sentence embedding vectors are the same, as follows:

[0019]

[0020] Construct a combined network to be trained, and then proceed to step B3; wherein, These represent the high-dimensional sentence embedding vectors corresponding to the two sentences in the sentence pair, respectively. Represents the weight parameters. Indicates detection The classification results indicating whether the meanings are the same;

[0021] Step B3. Based on the sentence pairs in the publicly available text classification dataset, which are known to have the same meaning as each other, fine-tune the training of the combined network to obtain the trained Sentence-BERT model, that is, the Sentence-BERT model fine-tuned based on the BERT model pre-trained model, and then proceed to step B4.

[0022] Step B4. Using the Sentence-BERT model, which is fine-tuned based on the BERT model pre-trained model, sentence embedding representation is performed for each Chinese word segment corresponding to each target Chinese text to obtain the high-dimensional sentence embedding vector corresponding to each target Chinese text, and then proceed to step C.

[0023] As a preferred technical solution of the present invention: step C includes the following steps C1 to C3;

[0024] Step C1. Based on the high-dimensional sentence embedding vectors corresponding to each target Chinese text, obtain the distance between each pair of high-dimensional sentence embedding vectors, and then proceed to step C2;

[0025] Step C2. For each high-dimensional sentence embedding vector, construct a weighted k-nearest neighbor graph corresponding to the high-dimensional sentence embedding vector based on the preset k other high-dimensional sentence embedding vectors that are closest to the high-dimensional sentence embedding vector, thereby obtaining the weighted k-nearest neighbor graphs corresponding to each high-dimensional sentence embedding vector, and then proceed to step C3.

[0026] Step C3. Based on the preset minimum distance in the preset low-dimensional target space, for each high-dimensional sentence embedding vector, with the goal of minimizing the cross-entropy cost function, apply the projection dimensionality reduction UMAP algorithm to project and reduce the weighted k-nearest neighbor graph corresponding to the high-dimensional sentence embedding vector to the preset low-dimensional target space, thereby obtaining the corresponding low-dimensional sentence embedding vector, and then obtaining the low-dimensional sentence embedding vector corresponding to each target Chinese text.

[0027] As a preferred technical solution of the present invention: step D includes the following steps D1 to D6;

[0028] Step D1. Based on the low-dimensional sentence embedding vectors corresponding to each target Chinese text, obtain the distance between each pair of low-dimensional sentence embedding vectors, and then proceed to step D2;

[0029] Step D2. For each low-dimensional sentence embedding vector, based on the low-dimensional sentence embedding vector... Sort the distances between each sentence and its embedding vector in ascending order to obtain the nth position in that sorting. distance This forms a low-dimensional sentence embedding vector. The corresponding core distance is then obtained; subsequently, the core distance corresponding to each low-dimensional sentence embedding vector is obtained, and then proceed to step D3; where, This indicates the first position in the sorting. Other low-dimensional sentence embedding vectors corresponding to each distance. express and Distance function between;

[0030] Step D3. Based on the core distance corresponding to each low-dimensional sentence embedding vector, and combined with the distance between each pair of low-dimensional sentence embedding vectors, the following formula is used:

[0031]

[0032] Obtain the cross-distance between pairwise low-dimensional sentence embedding vectors, and then proceed to step D4; where, These represent the low-dimensional sentence embedding vectors, respectively. The corresponding core distance and low-dimensional sentence embedding vector Corresponding core distance, Represents a low-dimensional sentence embedding vector With low-dimensional sentence embedding vector The distance between them Represents the maximization function. Represents a low-dimensional sentence embedding vector With low-dimensional sentence embedding vector The distance between them;

[0033] Step D4. Using each low-dimensional sentence embedding vector as a vertex, and the distance between any two low-dimensional sentence embedding vectors as the distance between the edges connecting the vertices, construct a minimum spanning tree that connects all vertices with the edges and minimizes the sum of the distances of all connected edges, and then proceed to step D5.

[0034] Step D5. Based on the sorting of all connected edges in the minimum spanning tree in ascending order of their corresponding distances, select each connected edge in sequence according to this sorting, and classify the objects connected to both ends of the connected edge into the same cluster to obtain each cluster, and then proceed to step D6.

[0035] Step D6. Remove all clusters whose size is smaller than the preset minimum generated cluster size, and obtain the remaining clusters, which are the subject classes corresponding to the target Chinese collection.

[0036] As a preferred embodiment of the present invention: step E includes steps E1 to E3;

[0037] Step E1. Based on the low-dimensional sentence embedding vectors of each topic class corresponding to the target Chinese dataset, obtain the Chinese word segments of each target Chinese text included in each topic class. Then, for each topic class, aggregate the Chinese word segments of each target Chinese text in the topic class to form a single document corresponding to the topic class, thereby obtaining the documents corresponding to each topic class of the target Chinese dataset, and then proceed to step E2.

[0038] Step E2. For each topic class corresponding to the target Chinese dataset, further segment each Chinese word in the document corresponding to that topic class using the following formula:

[0039]

[0040]

[0041]

[0042] Obtain the c-TF-IDF scores for each Chinese word segment in the document corresponding to the topic class, and then obtain the c-TF-IDF scores for each Chinese word segment in each topic class corresponding to the target Chinese dataset, and then proceed to step E3; where, Represents the subject class Chinese word segmentation In this topic category The number of times it appears in the document, Represents the subject class The number of Chinese word segments in the document Indicates Chinese word segmentation The number of times a word appears in all document categories, where A represents the average number of Chinese word segments in document categories.

[0043] Step E3. Based on the c-TF-IDF scores of each Chinese word segmentation in each topic class corresponding to the target Chinese dataset, obtain the preset with the highest c-TF-IDF score in each topic class. Each Chinese word segment constitutes a topic word in each topic category corresponding to the target Chinese dataset, thereby realizing topic category analysis of the target Chinese dataset.

[0044] As a preferred embodiment of the present invention: step F includes steps F1 to F3;

[0045] Step F1. Based on the DSG model, the word embedding representation of the topic words is obtained to obtain the high-dimensional vectors of the preset dimensions corresponding to each topic word in each topic class of the target Chinese dataset, and then proceed to step F2;

[0046] Step F2. For each topic class corresponding to the target Chinese set, perform steps F2-1 to F2-4 to obtain the similarity between each topic class corresponding to the target Chinese set and the historical topic class set, which constitutes the classification index of each topic class corresponding to the target Chinese set, and then proceed to step F3.

[0047] Step F2-1. Based on the historical theme categories in the historical theme category set and the keywords in the historical theme categories, perform analysis on the corresponding theme categories of the target Chinese text. The keywords in the text are defined according to the following formula:

[0048]

[0049] Calculate the topic class corresponding to the target Chinese set. The Middle The first keyword and the first The first of the historical theme categories Word similarity between keywords Then proceed to step F2-2; where 1 , This indicates the number of keywords in the subject class corresponding to the target Chinese dataset. , This represents the number of history-themed classes in the history-themed class collection, 1 , This indicates the number of keywords in the historical topic category; Indicates the first The first of the historical theme categories Keywords, Indicates the subject class corresponding to the target Chinese collection. The Middle Keywords, Represent the similarity function, and use Cosine measurement formula;

[0050] Step F2-2. Based on 1 According to the following formula:

[0051]

[0052] Calculate the topic class corresponding to the target Chinese set. The Middle The first keyword and the first Similarity between historical theme categories Then proceed to steps F2-3;

[0053] Step F2-3. Based on 1 According to the following formula:

[0054]

[0055] Calculate the topic class corresponding to the target Chinese text set. With the Similarity between historical theme categories Then proceed to steps F2-4;

[0056] Step F2-4. Based on According to the following formula:

[0057]

[0058] Calculate the topic class corresponding to the target Chinese text set. Similarity with historical theme sets ;

[0059] Step F3. Based on the preset classification threshold within the range of 0 to 1. Based on the classification indicators of each topic category corresponding to the target Chinese collection, if the classification indicators of the topic categories... If the topic category is determined to be a new topic category compared to the historical topic category set, then the topic category and all keywords within it are added to the historical topic category set; if the topic category classification index... If the above conditions are met, then the topic class is determined not to be a new topic class compared to the historical topic class set.

[0060] As a preferred technical solution of the present invention: the distance between the vectors is calculated using any one of sine distance, cosine distance, and Euler distance.

[0061] The big data topic analysis method based on an embedding model described in this invention has the following technical advantages compared with existing technologies:

[0062] (1) The big data topic analysis method based on the embedding model designed in this invention, compared with the unsatisfactory effect of the traditional LDA topic class modeling method when performing topic class analysis on short Chinese text collections, applies word embedding representation technology and considers applying the pre-trained BERT word embedding vector technology to the topic class model to improve the topic class analysis performance.

[0063] (2) In the big data topic analysis method based on the embedding model designed in this invention, when performing BERT word embedding, in view of the fact that BERT is not suitable for semantic similarity search problem and unsupervised task, Sentence-BERT is applied to fine-tune the pre-trained BERT: explore the use of Sentence-BERT (SBERT) model after fine-tuning the pre-trained BERT model through Siamese Network to generate semantic high-dimensional sentence embedding vectors, so that the high-dimensional sentence embedding vectors of semantically similar sentences are closer, so that cosine similarity, Manhattan distance, Euclidean distance, etc. can be used to find semantically similar sentences;

[0064] (3) In the big data topic analysis method based on the embedding model designed in this invention, when reducing the dimension of the high-dimensional sentence embedding vector, the UMAP dimension reduction method is used. Compared with other dimension reduction techniques such as PCA (principal component analysis), the UMAP method maintains the local and global structure of the data during dimension reduction, which is very important for representing the semantics of text data.

[0065] (4) In the big data topic analysis method based on the embedding model designed in this invention, since the UMAP dimensionality reduction method will retain some original high embedding structures, it is very meaningful to use HDBSCAN to find high-density clusters (i.e. popular demand topic classes). The biggest advantage of HDBSCAN compared with DBSCAN is that it does not require manual selection of the neighborhood radius R and MinPoints. Most of the time, it is only necessary to select the smallest generated cluster size. The algorithm can automatically recommend the optimal cluster results.

[0066] (5) In the big data topic analysis method based on the embedding model designed in this invention, after each Chinese text in the target Chinese set is assigned to a cluster class, the next step is to use class-based TF-IDF called c-TF-IDF to obtain the topic class representation. The topic classes extracted by the whole scheme design have higher topic class consistency and topic class diversity.

[0067] (6) In the big data topic analysis method based on the embedding model designed in this invention, a topic class volatility detection method based on word embedding is independently designed. At the same time, the Directional skip-gram (DSG) model is used instead of skip-gram (SG) for word embedding. This can effectively calculate the similarity between different topic words and the similarity between different topic classes. Based on the similarity score, topic class volatility detection is performed, and finally, newly emerging hot topic demand topics can be effectively detected. Attached Figure Description

[0068] Figure 1 This is a flowchart of the big data topic analysis method based on the embedded model designed in this invention;

[0069] Figure 2 This is a schematic diagram of the content data of government hotline requests in the government service platform, which is an application of the present invention.

[0070] Figure 3 This is a schematic diagram of the data preprocessing algorithm in step A of the present invention;

[0071] Figure 4 This is a data display diagram after data preprocessing in step A of the present invention;

[0072] Figure 5 This is the diagram of the fine-tuned twin network structure in step B of the present invention;

[0073] Figure 6 This is a flowchart of the HDBSCAN algorithm in step D of the present invention;

[0074] Figure 7 This is a visualization of the clustering of data after dimensionality reduction to 2D in the implementation of step D of the present invention;

[0075] Figure 8 This involves calculating the c-TF-IDF score map of some clusters in step E of the present invention.

[0076] Figure 9 This means that step E of the present invention is applied to all theme images after a certain day;

[0077] Figure 10 This is a flowchart of the topic volatility detection algorithm in step F of the present invention;

[0078] Figure 11 This is a graph showing the results of thematic volatility detection in the implementation of step F of the present invention. Detailed Implementation

[0079] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0080] This invention presents a big data topic analysis method based on an embedding model. In practical applications, such as for government service hotlines, it obtains the Chinese text of daily hotline requests from citizens and businesses, forming a target Chinese text set. Figure 2 As shown, specifically according to Figure 1 The design shown executes steps A through E to achieve topic class analysis of the target Chinese dataset.

[0081] Step A. Targeting respectively Figure 2 The target Chinese texts in the target Chinese set shown are arranged as follows: Figure 3The data preprocessing operation is shown below. First, regular expressions from Python's `re` module are used to remove target Chinese text from the target Chinese text set that is not expressed by the original author. Then, for each target Chinese text in the target Chinese text set, fine-grained Chinese word segmentation is performed using Python's `jieba` module to obtain the corresponding Chinese word segments, and stop words are removed. The Chinese word segments are then updated. This process yields the individual Chinese word segments for each target Chinese text. Partial results of the data preprocessing are shown below. Figure 4 As shown, proceed to step B.

[0082] Regarding the removal of stop words, the main approach is to use publicly available stop word lists from Harbin Institute of Technology, Baidu, and the Sichuan University Machine Intelligence Laboratory. If a Chinese word segment is listed in these Chinese stop word lists, it is considered a stop word and removed. Otherwise, it is not considered a stop word and is retained without further processing.

[0083] Step B. Using the Sentence-BERT model, which is fine-tuned based on the BERT model pre-trained model, sentence embedding representations are performed for each Chinese word segment corresponding to each target Chinese text to obtain the high-dimensional sentence embedding vectors corresponding to each target Chinese text, and then proceed to step C.

[0084] In practical applications, step B above is performed as follows: Figure 5 As shown, the specific design execution steps are B1 to B4.

[0085] Step B1. Based on the BERT model that takes Chinese word segmentation as input and the embedding vector corresponding to the Chinese word segmentation as output, connect the average pooling layer to the output of the BERT model to construct a Sentence-BERT model that takes each Chinese word segment corresponding to the Chinese text as input and the high-dimensional sentence embedding vector corresponding to the Chinese text as output, and then proceed to step B2.

[0086] The average pooling layer takes the average of the embedding vectors of all Chinese words in the same Chinese text after data preprocessing in each dimension, thereby generating a fixed-size high-dimensional sentence embedding vector for each Chinese text.

[0087] Step B2. Obtain the high-dimensional sentence embedding vectors corresponding to the two sentences in the sentence pair using the Sentence-BERT model. Combine this with the Siamese network classification objective function used to detect whether the meanings of the two high-dimensional sentence embedding vectors are the same, as follows:

[0088]

[0089] Construct a combined network to be trained, and then proceed to step B3; wherein, These represent the high-dimensional sentence embedding vectors corresponding to the two sentences in the sentence pair, respectively. Represents the weight parameters. Indicates detection The classification results determine whether the meanings of the different categories are the same.

[0090] Step B3. Based on the sentence pairs in the publicly available text classification dataset, which are known to have the same meaning as each other, fine-tune the training of the combined network to obtain the trained Sentence-BERT model, that is, the Sentence-BERT model fine-tuned based on the BERT model pre-trained model, and then proceed to step B4.

[0091] Step B4. Using the Sentence-BERT model, which is fine-tuned based on the BERT model pre-trained model, sentence embedding representation is performed for each Chinese word segment corresponding to each target Chinese text to obtain the high-dimensional sentence embedding vector corresponding to each target Chinese text. In this application, each Chinese text is represented as a 384-dimensional high-dimensional sentence embedding vector, and then proceed to step C.

[0092] Step C. Based on the preset neighborhood size information and the preset low-dimensional target space, use the Unified Manifold Approximation and Projection Dimensionality Reduction (UMAP) algorithm to perform dimensionality reduction processing on the high-dimensional sentence embedding vectors of each target Chinese text, and obtain the low-dimensional sentence embedding vectors corresponding to each target Chinese text, and then proceed to step D.

[0093] Among these methods, dimensionality reduction preserves both the local and global structure of the data, effectively representing the semantic information of Chinese text data. The initialization parameters... This indicates that the local neighborhood size of the UMAP projection dimensionality reduction algorithm is 25. These are parameters that control the local and global structure of text data; initialization parameters. This indicates that the target dimension after dimensionality reduction using the UMAP projection algorithm is 10. This refers to the data dimension passed to the clustering model; simultaneously, cosine distance is used when measuring the distance between different sample points, i.e. .

[0094] Using the projection dimensionality reduction UMAP algorithm for dimensionality reduction, compared with other dimensionality reduction techniques such as PCA (Principal Component Analysis), the projection dimensionality reduction UMAP algorithm preserves the local and global structure of the data during dimensionality reduction, which is important for representing the semantics of text data. In practical applications, the above step C is specifically designed to execute the following steps C1 to C3. Before mapping the high-dimensional sentence embedding vector to the low-dimensional sentence embedding vector, the projection dimensionality reduction UMAP algorithm needs to figure out what it looks like in the high-dimensional space, that is, to execute the following steps C1 to C2, and then continue to execute step C3.

[0095] Step C1. Based on the high-dimensional sentence embedding vectors corresponding to each target Chinese text, obtain the distance between each pair of high-dimensional sentence embedding vectors, and then proceed to step C2.

[0096] Step C2. For each high-dimensional sentence embedding vector, construct a weighted k-nearest neighbor graph corresponding to the high-dimensional sentence embedding vector based on the preset k other high-dimensional sentence embedding vectors that are closest to the high-dimensional sentence embedding vector, thereby obtaining the weighted k-nearest neighbor graphs corresponding to each high-dimensional sentence embedding vector, and then proceed to step C3.

[0097] Step C3. Based on the preset minimum distance in the preset low-dimensional target space, for each high-dimensional sentence embedding vector, minimize the cross-entropy cost function as follows:

[0098]

[0099] The UMAP projection dimensionality reduction algorithm is applied to project and reduce the weighted k-nearest neighbor graph corresponding to the high-dimensional sentence embedding vector to a preset low-dimensional target space, thereby obtaining the corresponding low-dimensional sentence embedding vector, and then obtaining the low-dimensional sentence embedding vector corresponding to each target Chinese text.

[0100] Since the UMAP projection dimensionality reduction algorithm preserves some of the original high embedding structures, it is meaningful to use the HDBSCAN clustering algorithm to find high-density clusters (i.e. popular topics). The biggest advantage of the HDBSCAN clustering algorithm compared to DBSCAN is that it does not require manual selection of neighborhood radius R and MinPoints. Most of the time, it is only necessary to select the smallest generated cluster size. The algorithm can automatically recommend the optimal cluster results and defines a new distance measurement method, specifically, let the following step D be executed.

[0101] Step D. Based on the preset minimum generated cluster size, apply the hierarchical and density-based HDBSCAN clustering algorithm, according to... Figure 6 As shown, clustering is performed on the low-dimensional sentence embedding vectors corresponding to each target Chinese text to obtain the corresponding clusters, which are the subject classes corresponding to the target Chinese text set, and then proceed to step E.

[0102] In practical applications, the above step D is specifically designed to be executed as follows: steps D1 to D6.

[0103] Step D1. Based on the low-dimensional sentence embedding vectors corresponding to each target Chinese text, obtain the distance between each pair of low-dimensional sentence embedding vectors, and then proceed to step D2.

[0104] Step D2. For each low-dimensional sentence embedding vector, based on the low-dimensional sentence embedding vector... Sort the distances between each sentence and its embedding vector in ascending order to obtain the nth position in that sorting. distance This forms a low-dimensional sentence embedding vector. The corresponding core distance is then obtained; subsequently, the core distance corresponding to each low-dimensional sentence embedding vector is obtained, and then proceed to step D3; where, This indicates the first position in the sorting. Other low-dimensional sentence embedding vectors corresponding to each distance. express and The distance function between them.

[0105] Step D3. Based on the core distance corresponding to each low-dimensional sentence embedding vector, and combined with the distance between each pair of low-dimensional sentence embedding vectors, the following formula is used:

[0106]

[0107] Obtain the cross-distance between pairwise low-dimensional sentence embedding vectors, and then proceed to step D4; where, These represent the low-dimensional sentence embedding vectors, respectively. The corresponding core distance and low-dimensional sentence embedding vector Corresponding core distance, Represents a low-dimensional sentence embedding vector With low-dimensional sentence embedding vector The distance between them Represents the maximization function. Represents a low-dimensional sentence embedding vector With low-dimensional sentence embedding vector The distance between them.

[0108] The mutual reachability distance is used to represent the distance between the embedding vectors of two low-dimensional sentences after dimensionality reduction. Specifically, the Euclidean metric can be used to determine the distance between different sample points. This ensures that the distance between sample points in dense regions remains unaffected, while the distance between sample points in sparse regions and other sample points is amplified, which increases the robustness of the clustering algorithm to non-dense regions.

[0109] Step D4. Using each low-dimensional sentence embedding vector as a vertex, and the distance between any two low-dimensional sentence embedding vectors as the distance of the connecting edges between vertices, construct a minimum spanning tree that connects all vertices with the smallest sum of the distances of all connecting edges, and then proceed to step D5.

[0110] Regarding minimum spanning trees, in practical applications, the embedding vectors of each low-dimensional sentence are considered vertices, and the weight of an edge between any two vertices represents their mutual reachability. A threshold is set, and this threshold is gradually decreased from high to low. Any edge whose weight exceeds this threshold (i.e., the mutual reachability is too large, and they do not belong to the same cluster) is deleted, and the weighted graph is split. The threshold value is derived from the set of minimum spanning trees (minimum connected subgraphs with the smallest weights) constructed from the weighted graph. Thus, deleting any edge from this set will cause the weighted graph to split.

[0111] Step D5. Based on the sorting of all connected edges in the minimum spanning tree in ascending order of their corresponding distances, select each connected edge in sequence according to this sorting, and classify the objects connected to both ends of the connected edge into the same cluster to obtain each cluster, and then proceed to step D6.

[0112] Step D6. Remove all clusters whose size is smaller than the preset minimum generated cluster size, and obtain the remaining clusters, which are the subject classes corresponding to the target Chinese collection.

[0113] In practical applications, step D6 involves imposing a constraint on the minimum subtree to prevent the generated clusters from becoming too small. First, the minimum generated cluster size parameter is initialized. (Minimum cluster size) = 30 The number of topics can be adjusted; the larger the value, the fewer topics are discovered, and vice versa. Then, the clustering tree is traversed from top to bottom, and when splitting at each node, it is checked whether the number of samples contained in the two sub-clusters is greater than 30. If the number of samples of one of the left or right children is less than 30, the node is deleted directly, and the other child is promoted to the parent node. If the number of samples of both children is less than 30, both are deleted, and the current node does not split further. If the number of samples of both children is greater than or equal to 30, the node continues to split normally downwards.

[0114] In practical applications, steps C to D above reduce the high-dimensional sentence embedding vectors to 2-dimensional low-dimensional sentence embedding vectors before performing clustering. The results are as follows: Figure 7 As shown, proceed to step E.

[0115] Step E. Based on the low-dimensional sentence embedding vectors of each topic class corresponding to the target Chinese dataset, obtain the Chinese word segments of each target Chinese text included in each topic class. Then, apply the c-TF-IDF algorithm to obtain the c-TF-IDF scores of each Chinese word segment in each topic class, and obtain the preset value of the highest c-TF-IDF score in each topic class. Each Chinese word segment is used to construct the subject words in each subject category corresponding to the target Chinese set, thereby realizing the subject category analysis of the target Chinese set, and then proceeding to step F.

[0116] This study uses a class-based TF-IDF algorithm called c-TF-IDF to obtain topic terms, instead of the traditional TF-IDF algorithm. The difference between c-TF-IDF and TF-IDF lies in the term frequency level. In regular TF-IDF, TF measures the term frequency in each document. In c-TF-IDF, TF measures the term frequency within each cluster, and each cluster contains many documents.

[0117] In practical applications, the above step E is specifically designed to be executed as follows: steps E1 to E3.

[0118] Step E1. Based on the low-dimensional sentence embedding vectors of each topic class corresponding to the target Chinese dataset, obtain the Chinese word segments of each target Chinese text included in each topic class. Then, for each topic class, aggregate the Chinese word segments of each target Chinese text in the topic class to form a single document corresponding to the topic class, thereby obtaining the documents corresponding to each topic class of the target Chinese dataset, and then proceed to step E2.

[0119] Step E2. For each topic class corresponding to the target Chinese dataset, further segment each Chinese word in the document corresponding to that topic class using the following formula:

[0120]

[0121]

[0122]

[0123] Obtain the c-TF-IDF scores for each Chinese word segment in the document corresponding to the topic class, and then obtain the c-TF-IDF scores for each Chinese word segment in each topic class corresponding to the target Chinese dataset, and then proceed to step E3; where, Represents the subject class Chinese word segmentation In this topic category The number of times it appears in the document, Represents the subject class The number of Chinese word segments in the document Indicates Chinese word segmentation The number of times a word appears in all document categories, where A represents the average number of Chinese word segments in document categories.

[0124] Step E3. Based on the c-TF-IDF scores of each Chinese word segmentation in each topic class corresponding to the target Chinese dataset, obtain the preset with the highest c-TF-IDF score in each topic class. Each Chinese word segment constitutes a topic word in each topic category corresponding to the target Chinese dataset, thereby realizing topic category analysis of the target Chinese dataset.

[0125] Step F. Based on the DSG model (Directional Skip-Gram) for word embedding representation of topic words, obtain the high-dimensional vectors corresponding to each topic word in each topic class of the target Chinese dataset. Then, combine the historical topic classes in the historical topic class set and the topic words in the historical topic classes, and through similarity calculation, realize the analysis of new topic classes in each topic class of the target Chinese dataset compared with the historical topic class set.

[0126] In practical applications, the c-TF-IDF scores of some clusters are calculated, and the top 20 Chinese word segments with the highest c-TF-IDF scores for each cluster are extracted. Figure 8 As shown, the implementation application will display all theme categories on a certain day, as follows: Figure 9 As shown, proceed to step F.

[0127] In practical applications, step F above is followed... Figure 10 As shown, the specific design is carried out through steps F1 to F3.

[0128] Step F1. Based on the DSG model, the word embedding representation of the topic words is obtained to obtain the high-dimensional vectors of the preset dimensions corresponding to each topic word in each topic class of the target Chinese dataset, and then proceed to step F2.

[0129] Step F2. For each topic class corresponding to the target Chinese set, perform steps F2-1 to F2-4 to obtain the similarity between each topic class corresponding to the target Chinese set and the historical topic class set, which constitutes the classification index of each topic class corresponding to the target Chinese set, and then proceed to step F3.

[0130] Step F2-1. Based on the historical theme categories in the historical theme category set and the keywords in the historical theme categories, perform analysis on the corresponding theme categories of the target Chinese text. The keywords in the text are defined according to the following formula:

[0131]

[0132] Calculate the topic class corresponding to the target Chinese set. The Middle The first keyword and the first The first of the historical theme categories Word similarity between keywords Then proceed to step F2-2; where 1 , This indicates the number of keywords in the subject class corresponding to the target Chinese dataset. , This represents the number of history-themed classes in the history-themed class collection, 1 , This indicates the number of keywords in the historical topic category; Indicates the first The first of the historical theme categories Keywords, Indicates the subject class corresponding to the target Chinese collection. The Middle Keywords, Represent the similarity function, and use Cosine measurement formula.

[0133] Step F2-2. Based on 1 According to the following formula:

[0134]

[0135] Calculate the topic class corresponding to the target Chinese set. The Middle The first keyword and the first Similarity between historical theme categories Then proceed to steps F2-3.

[0136] Step F2-3. Based on 1 According to the following formula:

[0137]

[0138] Calculate the topic class corresponding to the target Chinese text set. With the Similarity between historical theme categories Then proceed to steps F2-4.

[0139] Step F2-4. Based on According to the following formula:

[0140]

[0141] Calculate the topic class corresponding to the target Chinese text set. Similarity with historical theme sets .

[0142] Step F3. Based on the preset classification threshold within the range of 0 to 1. ,initialization Based on the classification indicators of each topic category corresponding to the target Chinese collection, if the classification indicators of the topic categories... If the topic category is determined to be a new topic category compared to the historical topic category set, then the topic category and all keywords within it are added to the historical topic category set; if the topic category classification index... If so, then the topic class is determined not to be a new topic class compared to the historical topic class set. The final result in practical applications is as follows: Figure 11 As shown.

[0143] Step F employs the Directional Skip-gram (DSG) model instead of the Skip-gram (SG) model for word embedding. While the SG model is widely used in various tasks, it relies on word co-occurrence within local contexts for word prediction, neglecting further details such as word order and position. The DSG model, however, considers not only word co-occurrence patterns but also their relative positions, modeled by a special "direction" vector representing whether the predicted word is to the left or right of a given word. The DSG model proposes a novel approach based on the SG model. function:

[0144]

[0145] By each Introduce a new vector To indicate that it is related to The relative direction is used to measure the context words. How to interact with in its left or right context Related.

[0146]

[0147]

[0148] in express function, This represents the learning rate. Is specified Given The target label in the relative direction is defined as

[0149]

[0150] Finally, the formula for the Directional Skip-Gram Model is:

[0151]

[0152]

[0153]

[0154]

[0155] It is necessary to obtain the word embedding representation of each topic word based on the DSG model, and then calculate the similarity between topic words of different topics and the similarity between topic words and topics based on these high-dimensional word embedding representation vectors. In addition, the similarity between different topics is calculated to realize the volatility detection of topics. The specific design and execution are as described in steps F1 to F3 above.

[0156] In practice, the above-mentioned big data topic analysis method based on the embedding model uses any one of the following for calculating the distance between vectors: sine distance, cosine distance, and Eulerian distance.

[0157] The big data topic analysis method based on the embedding model designed in this invention first uses the Sentence-BERT (SBERT) model, which is fine-tuned based on the BERT pre-trained model, to perform sentence embedding representation on each pre-processed Chinese text data. Then, the UMAP algorithm is used to reduce the dimensionality of the embedded vectors. Next, the HDBSCAN clustering algorithm is used to cluster the dimensionality-reduced vectors, assigning each Chinese text in the target Chinese dataset to the corresponding topic class. The topic representation is then obtained using the class-based c-TF-IDF algorithm, and the Chinese words with the highest c-TF-IDF scores are selected to represent each topic class. Finally, the Directional Skip-Gram (DSG) model, which is an improvement on the Skip-Gram (SG) model, is used to perform word embedding representation on the topic words. The similarity between different topic words and between different topic classes is calculated, and then the volatility detection of whether new topic classes have emerged is performed. The theme model extracted from the entire solution design has higher theme consistency and theme diversity compared to the NMF theme model and LDA theme model. Moreover, when performing volatility detection on new theme categories, it can detect new hot demand themes in a timely and accurate manner and provide early warnings.

[0158] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A big data topic analysis method based on an embedding model, characterized in that: Perform the topic category analysis of the target Chinese dataset by following steps A through E; Step A. Perform data preprocessing operations on each target Chinese text in the target Chinese set to obtain the corresponding Chinese word segments for each target Chinese text, and then proceed to Step B; Step B. Using the Sentence-BERT model, which is fine-tuned based on the pre-trained BERT model, sentence embedding representations are performed for each Chinese word segment corresponding to each target Chinese text to obtain the high-dimensional sentence embedding vectors corresponding to each target Chinese text, and then proceed to step C; Step C. Based on the preset neighborhood size information and the preset low-dimensional target space, use the Unified Manifold Approximation and Projection Dimensionality Reduction (UMAP) algorithm to perform dimensionality reduction processing on the high-dimensional sentence embedding vectors of each target Chinese text, and obtain the low-dimensional sentence embedding vectors corresponding to each target Chinese text, and then proceed to step D; Step D. Based on the preset minimum generated cluster size, apply the hierarchical and density-based HDBSCAN clustering algorithm to cluster the low-dimensional sentence embedding vectors corresponding to each target Chinese text to obtain the corresponding clusters, which are the topic classes corresponding to the target Chinese set, and then proceed to step E. Step E. Based on the low-dimensional sentence embedding vectors of each topic class corresponding to the target Chinese dataset, obtain the Chinese word segments of each target Chinese text included in each topic class. Then, apply the c-TF-IDF algorithm to obtain the c-TF-IDF scores of each Chinese word segment in each topic class, and obtain the preset value of the highest c-TF-IDF score in each topic class. Each Chinese word segment is used to construct the topic words in each topic category corresponding to the target Chinese set, thereby realizing the topic category analysis of the target Chinese set, and then proceeding to step F; Step F. Based on the word embedding representation of the topic words using the DSG model, obtain the high-dimensional vectors corresponding to each topic word in each topic class of the target Chinese dataset. Then, combine the historical topic classes in the historical topic class set and the topic words in the historical topic classes, and through similarity calculation, realize the analysis of new topic classes in each topic class of the target Chinese dataset compared with the historical topic class set. Step C above includes steps C1 to C3 as follows; Step C1. Based on the high-dimensional sentence embedding vectors corresponding to each target Chinese text, obtain the distance between each pair of high-dimensional sentence embedding vectors, and then proceed to step C2; Step C2. For each high-dimensional sentence embedding vector, construct a weighted k-nearest neighbor graph corresponding to the high-dimensional sentence embedding vector based on the preset k other high-dimensional sentence embedding vectors that are closest to the high-dimensional sentence embedding vector, thereby obtaining the weighted k-nearest neighbor graphs corresponding to each high-dimensional sentence embedding vector, and then proceed to step C3. Step C3. Based on the preset minimum distance in the preset low-dimensional target space, for each high-dimensional sentence embedding vector, with the goal of minimizing the cross-entropy cost function, apply the projection dimensionality reduction UMAP algorithm to project and reduce the weighted k-nearest neighbor graph corresponding to the high-dimensional sentence embedding vector to the preset low-dimensional target space, thereby obtaining the corresponding low-dimensional sentence embedding vector, and then obtaining the low-dimensional sentence embedding vector corresponding to each target Chinese text.

2. The big data topic analysis method based on an embedding model according to claim 1, characterized in that: In step A, firstly, target Chinese texts not expressed by the original author are removed from the target Chinese text set; then, fine-grained Chinese word segmentation is performed on each target Chinese text in the target Chinese text set to obtain each Chinese word segment corresponding to the target Chinese text, and Chinese stop words are removed and the Chinese word segments corresponding to the target Chinese text are updated; thus, each Chinese word segment corresponding to each target Chinese text is obtained.

3. The big data topic analysis method based on an embedding model according to claim 1, characterized in that: Step B includes steps B1 to B4; Step B1. Based on the BERT model which takes Chinese word segmentation as input and the embedding vector corresponding to the Chinese word segmentation as output, connect the average pooling layer to the output of the BERT model to construct a Sentence-BERT model which takes each Chinese word segment corresponding to the Chinese text as input and the high-dimensional sentence embedding vector corresponding to the Chinese text as output, and then proceed to step B2. Step B2. Obtain the high-dimensional sentence embedding vectors corresponding to the two sentences in the sentence pair using the Sentence-BERT model. Combine this with the Siamese network classification objective function used to detect whether the meanings of the two high-dimensional sentence embedding vectors are the same, and construct a combined training network. The Siamese network classification objective function is as follows: ； Then proceed to step B3; where, These represent the high-dimensional sentence embedding vectors corresponding to the two sentences in the sentence pair, respectively. Represents the weight parameters. Indicates detection The classification results indicating whether the meanings are the same; Step B3. Based on the sentence pairs in the publicly available text classification dataset, which are known to have the same meaning as each other, fine-tune the training of the combined network to obtain the trained Sentence-BERT model, that is, the Sentence-BERT model fine-tuned based on the BERT model pre-trained model, and then proceed to step B4. Step B4. Using the Sentence-BERT model, which is fine-tuned based on the BERT model pre-trained model, sentence embedding representation is performed for each Chinese word segment corresponding to each target Chinese text to obtain the high-dimensional sentence embedding vector corresponding to each target Chinese text, and then proceed to step C.

4. The big data topic analysis method based on an embedding model according to claim 1, characterized in that: Step D includes the following steps D1 to D6; Step D1. Based on the low-dimensional sentence embedding vectors corresponding to each target Chinese text, obtain the distance between each pair of low-dimensional sentence embedding vectors, and then proceed to step D2; Step D2. For each low-dimensional sentence embedding vector, based on the low-dimensional sentence embedding vector... Sort the distances between each sentence and its embedding vector from other low-dimensional sentences in ascending order, and obtain the nth position in that sorting. distance This constitutes a low-dimensional sentence embedding vector. The corresponding core distance is then obtained; subsequently, the core distance corresponding to each low-dimensional sentence embedding vector is obtained, and then proceed to step D3; where, This indicates the first position in the sorting. Other low-dimensional sentence embedding vectors corresponding to each distance. express and Distance function between; Step D3. Based on the core distance corresponding to each low-dimensional sentence embedding vector, and combined with the distance between each pair of low-dimensional sentence embedding vectors, the following formula is used: ； Obtain the cross-distance between pairwise low-dimensional sentence embedding vectors, and then proceed to step D4; where, These represent the low-dimensional sentence embedding vectors, respectively. The corresponding core distance and low-dimensional sentence embedding vector Corresponding core distance, Represents a low-dimensional sentence embedding vector With low-dimensional sentence embedding vector The distance between them Represents the maximization function. Represents a low-dimensional sentence embedding vector With low-dimensional sentence embedding vector The distance between them; Step D4. Using each low-dimensional sentence embedding vector as a vertex, and the distance between any two low-dimensional sentence embedding vectors as the distance between the edges connecting the vertices, construct a minimum spanning tree that connects all vertices with the edges and minimizes the sum of the distances of all connected edges, and then proceed to step D5. Step D5. Based on the sorting of all connected edges in the minimum spanning tree in ascending order of their corresponding distances, select each connected edge in sequence according to this sorting, and classify the objects connected to both ends of the connected edge into the same cluster to obtain each cluster, and then proceed to step D6. Step D6. Remove all clusters whose size is smaller than the preset minimum generated cluster size, and obtain the remaining clusters, which are the subject classes corresponding to the target Chinese collection.

5. The big data topic analysis method based on an embedding model according to claim 1, characterized in that: Step E includes steps E1 to E3; Step E1. Based on the low-dimensional sentence embedding vectors of each topic class corresponding to the target Chinese dataset, obtain the Chinese word segments of each target Chinese text included in each topic class. Then, for each topic class, aggregate the Chinese word segments of each target Chinese text in the topic class to form a single document corresponding to the topic class, thereby obtaining the documents corresponding to each topic class of the target Chinese dataset, and then proceed to step E2. Step E2. For each topic class corresponding to the target Chinese dataset, further segment each Chinese word in the document corresponding to that topic class using the following formula: ； ； ； Obtain the c-TF-IDF scores for each Chinese word segment in the document corresponding to the topic class, and then obtain the c-TF-IDF scores for each Chinese word segment in each topic class of the target Chinese dataset, and then proceed to step E3; where, Represents the subject class Chinese word segmentation In this topic category The number of times it appears in the document, Represents the subject class The number of Chinese word segments in the document Indicates Chinese word segmentation The number of times a word appears in all document categories, where A represents the average number of Chinese word segments in document categories. Step E3. Based on the c-TF-IDF scores of each Chinese word segmentation in each topic class corresponding to the target Chinese dataset, obtain the preset with the highest c-TF-IDF score in each topic class. Each Chinese word segment constitutes a topic word in each topic category corresponding to the target Chinese dataset, thereby realizing topic category analysis of the target Chinese dataset.

6. A big data topic analysis method based on an embedding model according to any one of claims 1 to 5, characterized in that: The distance between the vectors is calculated using any one of the following: sine distance, cosine distance, or Eulerian distance.

Images