A social network short text oriented information hierarchical abstract extraction method

Abstract

The application discloses a kind of social network short text-oriented information hierarchical abstract extraction method, for social network short text data, hierarchical clustering is carried out by constructing multistage clustering index tree, and abstract is extracted to text data based on clustering result.First, the text data set of social network is cleaned and preprocessed;Second, feature extraction can effectively avoid dimension disaster using LDA model;Then, feature data is substituted into RSC algorithm model to generate hierarchical clustering tree, and the multistage clustering index tree is obtained by merging and optimizing the clustering result;Finally, information hierarchical abstract is extracted based on multistage clustering index tree.Because extractive abstract extraction algorithm can extract summarization sentence in short text, therefore the application uses TextRank algorithm to carry out abstract extraction.Using the application, people can quickly and effectively obtain information, which is helpful for data analysis of social network platform and supervision work of social network.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and more specifically, it relates to a method for extracting hierarchical summaries of short text data from social networks. By extracting hierarchical summaries of social network information, it helps people quickly and effectively obtain information, which is beneficial for text data analysis on social network platforms and thus facilitates the supervision of social network platforms. Background Technology

[0002] With the rapid development of science and technology, text data on the internet is growing exponentially. While people can access information more conveniently and quickly, finding and extracting useful information from massive amounts of text data remains extremely difficult. Text clustering technology automatically divides text data into appropriate categories, thereby revealing the distribution of content within seemingly disorganized datasets and narrowing the scope of information retrieval. Abstract extraction technology presents text content concisely and efficiently, while hierarchical abstracting helps people quickly and effectively obtain information, distinguish right from wrong in texts, and make correct value judgments.

[0003] The shortcomings of existing text clustering techniques lie in the following limitations of the clustering algorithms:

[0004] (i) It takes a long time to run and is unable to handle tasks with high timeliness requirements.

[0005] (ii) The clustering results are unclear, making it difficult to extract useful information from them.

[0006] Therefore, existing problems focus on how to effectively improve algorithm efficiency and result certainty to address the data disaster problem in the era of big data. This invention uses the RSC algorithm, which can effectively solve the above problems.

[0007] Regarding the first drawback, the RSC algorithm used in this invention has a time complexity of O(nlogn), compared to the time complexity of traditional clustering algorithms of O(n log n). 2 It takes less time.

[0008] Regarding the second defect, the RSC algorithm used in this invention is a hierarchical clustering algorithm. The result of the algorithm is a clustering tree, which can well show the hierarchical organization of the data and has a good data interpretation capability. Summary of the Invention

[0009] The purpose of this invention is to provide a method for extracting information hierarchical summaries from short texts on social networks, so as to help obtain information quickly and effectively and improve the management efficiency of social network platforms.

[0010] A method for extracting hierarchical information summaries from short texts on social networks includes the following steps:

[0011] 1. Perform data preprocessing on the collected dataset;

[0012] 2. Perform feature extraction on the text data;

[0013] 3. Perform hierarchical clustering on the text data;

[0014] 4. Construct a multi-level clustering index tree based on the clustering results;

[0015] 5. Extract information summaries layer by layer from bottom to top.

[0016] To achieve the above-mentioned objectives, this invention provides a method for extracting hierarchical summaries from short texts on social networks, characterized by:

[0017] For step 1, based on the text characteristics of short texts on social networks, targeted data cleaning operations are required to filter out noisy data, such as web links, emoticons, topics, reposting sources, location addresses, etc. At the same time, it is necessary to delete frequently occurring words and construct a stop word list that is more suitable for this dataset.

[0018] For step 2, to avoid excessively large feature dimensions, the LDA model is used to extract the topic features of the text.

[0019] For step 3, the text hierarchical clustering uses the RSC algorithm to perform hierarchical clustering on short texts from social networks. In the dataset, by detecting mutually nearest neighbor nodes and iteratively creating clustering trees, two main processes are required, including the creation and pruning of clustering subtrees:

[0020] 3.1) Creation of clustering subtrees

[0021] Each piece of text data is treated as a data point, and subtree construction is stopped when two stopping conditions are met through node linking.

[0022] 3.2) Pruning clustered subtrees

[0023] After the clustering subtrees are constructed, pruning is required to avoid elongated subtrees.

[0024] For step 4, constructing a multi-level clustering index tree based on the clustering results includes:

[0025] 4.1) Construct a multi-level clustering index tree

[0026] The clustering result obtained using the RSC algorithm is a clustering tree with scattered nodes. Nodes in the clustering tree are divided into three types: non-leaf nodes with only one child, non-leaf nodes with two or more children, and leaf nodes. The processing methods for these three different node types when constructing the multi-level clustering index tree are as follows:

[0027] (i) If a node is a non-leaf node with only one child, determine the type of its child. If the child is also a non-leaf node, recursively create its child. If its child is a leaf node, create a child node based on the leaf node to display the leaf node's text.

[0028] (ii) If a node is a non-leaf node with two or more children, then traverse its child nodes and repeat step (i).

[0029] (iii) If it is a leaf node, then create a child node based on the leaf node to display the text data corresponding to the leaf node.

[0030] 4.2) Optimize and merge multi-level clustering index trees

[0031] In a multi-level clustering index tree, there are many non-leaf nodes with only one child. These nodes increase the depth of the clustering tree, but their ability to partition the hierarchy is not obvious. They can even be regarded as nodes of the same level. Therefore, the nodes in the multi-level clustering index tree are merged.

[0032] For step 5, constructing a clustering tree with information digests based on the optimized and merged multi-level clustering index tree requires filling the text data corresponding to the index into the clustering tree structure, and generating a digest of the text data corresponding to the child nodes at the non-leaf nodes:

[0033] 5.1) Text Selection and Processing

[0034] Text data used for information summarization needs to be reprocessed to filter out data that may cause reading difficulties, such as internet slang, emoticons, and special symbols.

[0035] 5.2) Prepare word vectors

[0036] In the bottom-up, layer-by-layer information summarization process, the TextRank algorithm is used for summary extraction. This extraction process requires obtaining sentence feature representations. This invention uses the latest Wikipedia Chinese text corpus, segmented by jieba and with stop word removal, to train GloVe word vectors. GloVe is a word representation tool based on global word frequency statistics. Using GloVe for word vectorization allows the vectors to contain as much semantic and syntactic information as possible. Compared to feature creation methods such as TF-IDF and Word2Vec, this avoids excessively large feature dimensions and fully utilizes the corpus.

[0037] 5.3) Extracting the summary

[0038] Finally, the most general and representative sentences are extracted from each level of the clustering tree as information summaries of the text. Attached Figure Description

[0039] Figure 1 This is a flowchart of the specific implementation process. Detailed Implementation

[0040] The following is in conjunction with the appendix Figure 1 Specific embodiments of the present invention will be described to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main points of the invention will be omitted.

[0041] This invention provides a method for hierarchical information summarization of short texts from social networks. It utilizes the RSC algorithm for hierarchical clustering of microblog text data and the TextRank algorithm for summarizing the clustering results. First, the text dataset from the social network is cleaned and preprocessed. Second, features are extracted from the data; the LDA model effectively avoids the curse of dimensionality. Then, the feature data is fed into the RSC algorithm model to generate a hierarchical clustering tree, and the clustering results are merged and optimized to obtain a multi-level clustering index tree. Finally, based on the multi-level clustering index tree, hierarchical information summarization is performed on the text data using the extractive summarization algorithm TextRank.

[0042] A method for extracting hierarchical information summaries from short texts on social networks includes the following steps:

[0043] 1. Perform data preprocessing on the collected dataset;

[0044] 2. Perform feature extraction on the text data;

[0045] 3. Perform hierarchical clustering on the text data;

[0046] 4. Construct a multi-level clustering index tree based on the clustering results;

[0047] 5. Extract information summaries layer by layer from bottom to top.

[0048] This invention provides a method for extracting hierarchical summaries from short texts on social networks, as detailed below:

[0049] 1. Data Preprocessing

[0050] This invention uses the MicroblogPCU dataset, which was initially used to explore spammers on microblogs. This data can also be used to study machine learning methods and conduct research on social networks. Specifically, the user_post.csv data from the MicroblogPCU dataset is used, which contains information about the microblog posts made by users. The data description is shown in Table 1.

[0051] property describe post_id User Weibo ID post_time Weibo posting time content Weibo content poster_id User ID that posted this Weibo poster_url User's personal homepage address repost_num The number of reposts of this Weibo post comment_num The number of comments on this Weibo post reposter_post_id Reposter ID inner_flag Built-in tags

[0052] Table 1

[0053] Since the raw data is unstructured Chinese text, and social network text contains many non-standard expressions, such as internet slang, emoticons, emojis, and special symbols, text cleaning and preprocessing are necessary. Data preprocessing includes four steps:

[0054] ① Filter noisy data: Observe the original data and use regular expressions to delete noisy data, including web links, emoticons, titles, topics, reprint sources, location addresses, etc. At the same time, some redundant and frequently occurring words need to be deleted.

[0055] ② Stop word removal: Based on the existing stop word list, supplement and improve it according to the specific situation to construct a stop word list that is as good as possible. This invention constructs a stop word list that is more suitable for this dataset based on the stop word library of the Machine Intelligence Laboratory of Sichuan University.

[0056] ③ Chinese word segmentation: The processing of Chinese text data requires word segmentation. This invention uses the jieba word segmentation tool for word segmentation.

[0057] ④ Character Count Filtering: Since the data used in this invention covers the period before 2015, all the Weibo text data used are within 140 characters. Because text data that is too short has little value, after word segmentation and removal of stop words, Weibo posts with a text length of less than 10 characters are deleted.

[0058] 2. Feature Extraction

[0059] To avoid the problem of excessively large feature dimensions, this invention uses the LDA topic model for topic feature extraction. Some libraries, such as scikit-learn, sparkMLib, and gensim, can be used to learn LDA topic models, and their principles are basically similar. This invention uses scikit-learn for LDA feature extraction. The latent topic parameter is set to 5, meaning each Weibo post has 5 topic features, resulting in an n×5 LDA feature matrix (n is the number of Weibo posts).

[0060] 3. Text hierarchical clustering

[0061] When performing cluster analysis using the RSC algorithm, the main processes are iteratively constructing cluster subtrees and pruning. The clustering result obtained by the hierarchical clustering algorithm is a hierarchical clustering tree.

[0062] 3.1) Construction of clustering subtrees

[0063] The dataset contains text data posted by Weibo users. Each piece of data (word vector) is considered a data node. First, an empty subtree and a candidate set containing all nodes are initialized. If the candidate set is not empty, a node is randomly selected and linked to its nearest neighbor, i.e., its first-order nearest neighbor. Assuming each node has only one nearest neighbor, the first-order nearest neighbor is further linked to its nearest neighbor. This process generates a linked list. The construction of the subtree satisfies two stopping conditions:

[0064] (i) The k-th nearest neighbor and the (k-2)-th nearest neighbor are a pair of RNN nodes;

[0065] (ii) The k-th nearest neighbor is not in the candidate set.

[0066] When condition (i) is true, a new subtree is constructed, where the (k-2)th nearest neighbor and the (k-1)th nearest neighbor are a pair of RNNs and become the support nodes of the clustering tree. At this point, a root node needs to be manually constructed as the representative node of this subtree and linked to the support node. Finally, all nodes of this subtree are removed from the candidate set. If the candidate set is not empty after deletion, a node is randomly selected from the candidate set, and the above process is repeated. When the candidate set is empty, the construction of the subtree stops. When condition (ii) is true, the linked list is linked to an existing subtree.

[0067] 3.2) Pruning clustered subtrees

[0068] After the subtrees are constructed, pruning is required to avoid elongated subtrees. For any node i, if it belongs to an existing subtree with nodes p and q as support nodes, then the depth of node i is the average of the sum of its shortest path lengths to p and q, and the depth of node i can be considered as its path length to the root node. When pruning each generated subtree, nodes with depths greater than a threshold are pruned based on their path lengths. Each pruned node is linked to a manually set root node, thus forming a special subtree consisting of only one node and one root.

[0069] 3.3) Representative node evaluation indicators

[0070] During the generation of cluster subtrees, when the stopping condition (i) is met, a more representative node needs to be selected from a pair of RNN nodes. The RSC algorithm proposes four metrics for evaluating RNN nodes: node degree, average neighbor degree, step centrality, and distance centrality. The degree and average neighbor degree are relatively easy to understand; in a cluster, a node with a higher degree is more important than a node with a lower degree. If a node's neighbors have higher degrees, then that node is more important, meaning it is located in the center of a dense region of the data. The degree and average neighbor degree of node i are defined as follows:

[0071]

[0072] Where, d i Let i be the degree of node i. Let r be the average degree of the neighbors of node i. ij These are elements in a relation matrix described by a graph structure based on data points. To further measure the centrality of RNNs, two metrics are used: step centrality and distance centrality. The step centrality of node i is defined as follows:

[0073]

[0074] Where |τ| is the total number of nodes in cluster t containing node i, and s ij Let be the length of the shortest path between node i and node j. The distance centrality of node i is defined as follows:

[0075]

[0076] Among them, dist ij Let be the distance between node i and node j. To more fairly and reasonably measure the representativeness of nodes, a hybrid index is proposed, which combines the above four evaluation indicators. This hybrid index is obtained based on the relative importance calculation index and method in complex networks, and its definition is as follows:

[0077]

[0078] If a node has a higher mixture index score, it is considered that the node is more representative.

[0079] 3.4) Clustering Results

[0080] The resulting clustering tree is represented by a dictionary, containing several key-value pairs. Each non-leaf node and its children are represented by a key-value pair. All Weibo posts are considered nodes in the tree, with the node name represented by the row index of the Weibo post. In the resulting clustering tree, the key represents a non-leaf node, and the value of the key represents the child of that non-leaf node, represented by a set.

[0081] Suppose there is a tree T: {1:{0}, 2:{1,3,12}, 3:{4,5}, 5:{6,7}}. The non-leaf nodes in tree T are {1,2,3,5}. Among them, nodes 1 and 3 are children of node 2, and node 5 is a child of node 3. The remaining nodes {0,12,4,6,7} are leaf nodes. Therefore, the root node of tree T is node 2.

[0082] 4. Construct a multi-level clustering index tree

[0083] In a clustering tree with scattered nodes, due to storage structure limitations, the processing of different node types differs during the recursive process of linking nodes to form nested clustering trees. Therefore, it is necessary to identify the node type. Nodes in a clustering tree are classified into the following three types: ① Non-leaf nodes with only one child; ② Non-leaf nodes with two or more children; ③ Leaf nodes. The processing methods for these three different node types when constructing the clustering index tree are as follows:

[0084] (i) If a node is a non-leaf node with only one child, determine the type of its child. If it is also a non-leaf node, recursively create its child. If its child is a leaf node, create a child node based on the leaf node to display the leaf node's text.

[0085] (ii) If a node is a non-leaf node with two or more children, then traverse its child nodes and repeat step (i).

[0086] (iii) If it is a leaf node, then create a child node based on the leaf node to display the text data corresponding to the leaf node.

[0087] In a multi-level index tree, there are many non-leaf nodes with only one child. These nodes increase the depth of the clustering tree, but their ability to partition the hierarchy is not obvious. They can even be regarded as nodes at the same level. Therefore, these nodes in the index tree are merged.

[0088] 5. Abstract Extraction

[0089] Constructing a clustering tree with information digests based on a multi-level clustering index tree simply requires filling the text data corresponding to the index into the clustering tree structure and generating a digest of the text data corresponding to the child nodes at the non-leaf nodes.

[0090] 5.1) Select and process data

[0091] A portion of the text data from the clustering results is selected for summary extraction. Basic text cleaning is required to minimize the impact of noise on the extraction process. However, unlike previous methods, while all punctuation was removed during data preprocessing, the summary extraction process, to quickly extract information, needs to handle data containing internet slang, emoticons, and special symbols that could hinder reading. Some punctuation must also be retained for sentence segmentation.

[0092] 5.2) Prepare word vectors

[0093] This invention uses Chinese Wikipedia text corpus that has been segmented by jieba and had stop words removed to train GloVe word vectors, outputting word vector files. The vectors of all constituent words of each sentence are obtained from the GloVe word vector files, and then the average of these vectors is taken to obtain the combined vector of the sentence as the feature vector of the sentence.

[0094] 5.3) Creating a similarity matrix

[0095] Cosine similarity is used to calculate the similarity between sentences in order to identify similarities. First, an n-order zero matrix is ​​defined (n being the total number of sentences). The cosine similarity between sentences is then calculated to initialize the similarity matrix. The sentence similarity matrix for the experimental data is 238×238 in size.

[0096] 5.4) Extracting the summary

[0097] Traditional extractive summarization methods are typically based on graph models, scoring sentences or phrases by constructing a topological structure, with the aim of extracting the most generalized sentences from the text to form a summary. TextRank is a classic extractive summarization algorithm that uses sentences to represent web pages and similarity scores between sentences to represent web page transition probabilities. The sentence similarity scores are stored in a matrix, similar to the Transformer Matrix in the PageRank algorithm. The main steps for extracting summaries using the TextRank algorithm are as follows:

[0098] ① Merge all documents into a single unit;

[0099] ② Divide the text into several individual sentences;

[0100] ③ Find the word vector representation of each sentence;

[0101] ④ Calculate the similarity between sentences using cosine similarity to obtain a similarity matrix;

[0102] ⑤ Transform the similarity matrix into a graph with sentences as nodes and similarity scores as edge weights;

[0103] ⑥ Calculate the TextRank score, and output a certain number of sentences or phrases to form a summary, ranked from highest to lowest score.

[0104] A graph structure was constructed using the sentence similarity matrix, with sentences as nodes and similarity scores as edge weights. The PageRank algorithm was used to obtain the TextRank values ​​of all sentences. The sentences were then sorted according to their TextRank values, and the top 10 sentences with the highest scores were taken as a summary, as shown in Table 2.

[0105]

[0106] Table 2

[0107] The summary extraction results obtained from Table 2 show that although the text of a microblog is short, there are usually summary sentences in the text data to express the main idea of ​​the microblog. Therefore, the extractive summary extraction algorithm works well on short texts.

[0108] In summary, the hierarchical information summarization method for short texts on social networks presented in this invention has the advantages of high computational efficiency and good summarization results. In a real-world production environment, given short text data from existing social networks, this invention can quickly obtain hierarchical information summarization results, helping to quickly and effectively acquire information and facilitating data analysis and social network supervision on social network platforms.

Claims

1. A method for extracting hierarchical information summaries from short texts on social networks, characterized in that: This paper describes a hierarchical clustering method for extracting information from short texts on social networks. The method extracts summaries from the text data corresponding to the clustering results, resulting in hierarchical summaries. During the clustering process, representative nodes at each level are selected based on four evaluation metrics: The first is the degree of the node. In a cluster, nodes with higher degrees are more important than nodes with lower degrees. A node with a higher degree is located in the center of a dense data region. The second is the average degree of the node's neighbors, which is the average degree of the node's neighbors. The third is step centrality. The fourth is distance centrality. Based on the above four evaluation indicators, a hybrid index is proposed. This hybrid index is calculated according to the relative importance index in complex data networks. The higher the hybrid index score of a node, the more representative the node is.

2. The method for extracting hierarchical information summaries from short texts on social networks according to claim 1, characterized in that, The method includes: The information hierarchical clustering process is divided into three parts: data preprocessing, text feature extraction, and short text hierarchical clustering. For existing public datasets for social networks, the text data is analyzed for features, cleaned and preprocessed to extract the theme features of the text and perform hierarchical clustering, outputting a hierarchical clustering tree. The abstract extraction part requires reprocessing the text data. The abstract score of the text is calculated based on the graph structure, and the summary sentences in the text are extracted as the abstract based on the score.

3. The method for extracting hierarchical information summaries from short texts on social networks according to claim 2, characterized in that, The data cleaning process requires filtering noisy data by observing the characteristics of the original data, and then removing stop words, performing Chinese word segmentation, and filtering by character count.

4. The method for extracting hierarchical information summaries from short texts on social networks according to claim 2, characterized in that, Using the LDA topic model for topic feature extraction can effectively avoid the problem of excessively large feature dimensions and output the topic feature matrix of the text.

5. The method for extracting hierarchical information summaries from short texts on social networks according to claim 2, characterized in that, The process of hierarchical clustering of short texts requires iterative execution of two processes: building cluster subtrees and pruning cluster subtrees. When building cluster subtrees, it is necessary to find two nodes that are each other's nearest neighbors and satisfy two stopping conditions. After the subtree is constructed, it needs to be pruned to avoid the formation of long and thin subtrees.

6. The method for extracting hierarchical information summaries from short texts on social networks according to claim 5, characterized in that, To avoid generating too many long and thin subtrees, the obtained clustering trees are merged to construct a multi-level clustering index tree. In the obtained clustering results, the clustering tree nodes are categorized as follows: non-leaf nodes with only one child, non-leaf nodes with two or more children, and leaf nodes. In the recursive process of linking the nodes to form a nested clustering tree, the processing methods for different types of nodes are different, and finally, a multi-level clustering index tree is obtained.

7. The short text summarization method in the information hierarchy summarization extraction method for short texts on social networks according to claim 2, characterized in that, The method includes: For the reprocessed text data, a multi-level information summarization clustering tree is constructed based on a multi-level index tree. The text data corresponding to the index is filled into the clustering tree structure, and the text summary corresponding to the child node is generated at the non-leaf node. In the process of generating the summary, all texts are first merged into a whole, then divided into several individual sentences, sentence feature vectors are generated and the similarity between sentences is calculated. Finally, the similarity matrix is ​​converted into a graph structure, the summary score is calculated, and the sentence with the highest output score is taken as the summary.

8. The method for extracting hierarchical information summaries from short texts on social networks according to claim 7, characterized in that, Using Chinese text corpora that have undergone jieba word segmentation and stop word removal, Chinese word vectors are trained using the GloVe model, and the output word vector files are used to calculate the feature vectors of sentences.

Images