Automatic group opinion extraction system based on opposing topic analysis

Abstract

A group opinion automatic extraction system based on opposing topic analysis includes: an information collection module, a topic extraction module, a topic hierarchical clustering module, a tendency classification module, an opposing topic analysis module, and a group opinion statistics and output module. The present invention can continuously pay attention to the expression of views and positions of a specific group, identify relatively stable interest points in the group, and hierarchically cluster the topics to automatically form the theme of the group, thereby comprehensively describing the intention of the group from a large aspect and a small aspect, thereby providing a reference for improvement to product designers or managers.

Description

Technical Field

[0001] The present invention relates to a technology in the field of text processing, specifically a system for automatically extracting group opinions from text, analyzing viewpoints and topics, and summarizing and analyzing them. Background Art

[0002] With the development of social networks, individuals form groups and continue to express relatively stable views and opinions on certain topics, which has gradually become a common way of social communication. Whether it is Weibo or WeChat groups. Individuals with similar ideas and interests gather into groups, and the groups develop stable positions on one or more related matters. Such groups continue to attract more similar individuals. Especially when these positions are in line with the trend of the times, the development of this opinion group is more stable. Such a group may be a user of a specific product, the owner of a property area, or the participant of a certain activity. When the opinions of the group are automatically collected in the group, and the consensus of the group on specific topics is identified and summarized, the product developers, community managers and event organizers can improve their design or management accordingly.

[0003] In existing systems, trend analysis systems are often used to complete this type of text analysis work, and most of them use reviews for trend analysis. These reviews are usually for a specific manufacturer or a specific product, such as a restaurant, or a specific model of mobile phone. However, existing systems and methods do not analyze the overall opinion expression of the group, and do not pay attention to the opinion expression within the group. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art and existing systems, the present invention proposes an automatic group opinion extraction system based on opposing topic analysis, which can continuously pay attention to the expression of views and positions of a specific group, identify relatively stable points of interest in the group, and hierarchically cluster the topics to automatically form the theme of the group, thereby comprehensively depicting the intentions of the group from both large and small aspects, thereby providing a reference for improvement to product designers or managers.

[0005] The present invention is achieved through the following technical solutions:

[0006] The present invention comprises: an information collection module, a topic extraction module, a topic hierarchical clustering module, a tendency classification module, an opposing topic analysis module, a group opinion statistics and an output module, wherein: the information collection module collects speech information of individuals in a specific group in a social information publishing platform; the topic extraction module extracts the topic with the highest number of mentions from the speech information; the topic hierarchical clustering module analyzes the distribution of topics in the speech information, and classifies the topics into different categories according to their meanings using a hierarchical clustering algorithm; the tendency classification module uses a tendency classification algorithm to classify the tendencies of topics in the speech information; the opposing topic analysis module summarizes the classification results and analyzes whether there are paired opposing relationships in the same-layer topics and generates an opposing topic word list of all topics; the group opinion statistics are visualized and output by the output module based on the analysis results and the overall tendency of each topic in the opposing topic word list.

[0007] The specific group refers to a group of people who have similar opinions on a certain type of topic, such as all employees of a company or a social group.

[0008] The hierarchical clustering algorithm is to group related topics together as a category of topics. All topics are organized hierarchically, with the upper-level topics having broader conceptual connotations than the lower-level topics, and the lower-level topics are generally a side or component of the upper-level topics.

[0009] The tendency is a digital representation of whether a person supports or opposes a certain topic. There are discrete representations: support is 1, opposition is -1, and objective description or no opinion is 0. There are also continuous representations, where the tendency is represented as a real number in a certain range, such as [1, -1].

[0010] The tendency classification algorithm refers to analyzing a person's tendency towards a certain topic based on his / her text. This algorithm is a two-classification or three-classification algorithm based on natural language understanding.

[0011] The opposition relationship refers to the incompatible tendency relationship between different sub-topics under a topic. That is, in most cases, people who support sub-topic A will oppose sub-topic B at the same time, so A and B exist in an opposition relationship.

[0012] The generation process of the opposing thesaurus is as follows: given two similar topics, when the signs of the tendency means of the two topics in all texts sent by an individual are opposite and greater than a given threshold, the individual is marked as having opposing views on the two topics. The number of people in the group who have opposing views on the two topics is counted. If the proportion of the total number of people (or the number of people who have expressed their attitudes on such topics) is greater than a certain threshold, it means that the two topics are opposing in the group. All topics and independent keyword pairs are saved to form an opposing thesaurus.

[0013] The overall tendency is the overall tendency of a group towards a certain topic, and the calculation process is: the sum of the tendencies of all individuals in the group towards the topic is subtracted from the tendencies of all opposing topics to the topic, and the average value is calculated.

[0014] The information collection module continuously collects information that grows over time. In order to facilitate analysis, the information is summarized and collected by time slices, that is, the information within a certain period of time is saved as a text set. At the beginning of the system, historical information can be saved as a text set, and subsequent text sets are continuous revisions and supplements to the first analysis. The length of the time slice for dividing the text set can be determined according to the number of information releases. The less information is released, the longer the length of the time slice should be set to avoid the system being unable to conduct analysis due to too little data.

[0015] The topic extraction module uses the text collected by the information collection module to extract the keywords with high discussion frequency, and performs certain screening to use them as keywords of group opinions for subsequent analysis.

[0016] The topic hierarchical clustering module hierarchically clusters all the keywords or keyword groups extracted by the topic extraction module, clusters the keywords in the same field together to form a topic, thereby forming a hierarchical thesaurus for the group. Once the segmented thesaurus is formed, the user can manually revise the thesaurus. When a new text set arrives, it is supplemented on the basis of the current hierarchical thesaurus. All keywords under each topic are hierarchically processed, and the lower the level, the more specific the content expressed by the keywords.

[0017] The tendency classification module divides the sentences in the text set collected by the information collection module, classifies the sentences containing the subject words, and records the subject words in the sentences and their tendency scores. Since there is a sentence that expresses both approval of a certain topic and opposition to another topic, it is necessary to separately label the tendency of all the subject words appearing in the text, so as to obtain independent opinion and position scores for each topic. For subsequent analysis, the output of the tendency classification module is the speaker ID, the subject words contained in the speech text, and the tendency score of each subject word.

[0018] The input of the opposing topic analysis module is the subject words and their tendency scores of the tendency classification module and the hierarchical topic forest of the topic hierarchical clustering module. The topic semantic analysis module completes the identification of opposing topics. The topic semantic analysis module analyzes the relationship between keywords in the topic and semantically annotates keywords with opposing views.

[0019] The input of the group opinion statistics and output module is the output of the topic semantic analysis module, namely the topic semantic network, and the output of the tendency classification module, including the tendency classification results of the subject words.

[0020] Technical Effects

[0021] Compared with the prior art, the system of the present invention analyzes the texts published by the group over a long period of time to obtain various subject areas that the group has been interested in for a long time, and analyzes the specific content in each field to obtain the group's positive or negative views on the specific content, and makes a visual presentation. In general, the method first analyzes all the topics of all texts, and analyzes which topics are more common, and analyzes these topics. Then, based on the human habit of expressing opinions, that is, there are positive expressions (agreement) and negative expressions (opposition), some elements in the topic are analyzed for opposition, the opposing elements are found, and the semantic network of the topic is built. Finally, the group's views are expressed based on the semantic network, so as to give the main views of the group as a whole, and the tendencies of each topic. The technical effects of the present invention further include:

[0022] 1. Compared with existing public opinion systems, this system has a longer analysis period, with the aim of grasping long-term discussion topics among groups rather than short-term hot topics.

[0023] 2. Compared with the existing tendency analysis system, this system analyzes the mainstream views based on the texts published by the group, without the need to determine the theme and theme elements in advance.

[0024] 3. Compared with the existing analysis of group behavior based on social networks, this system does not consider the differences between different individuals in the group, nor the influence of different individuals on the group, but only considers the expression content of the group's overall opinions.

[0025] In addition to marking opinions with tendencies, this system organizes keywords that express opinions in the same field in layers and identifies and analyzes opposing words, so that positive and negative expressions in the text can be integrated, so that opinions can be expressed in a unified way. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0027] The automatic extraction and expression system of group opinions implemented in this embodiment processes a microblog text set W, which collects the speeches of 100 bloggers who have mutual attention, make speeches related to current affairs, and have an average monthly number of speeches greater than 60 to simulate the speech of a group. Each text in the text set contains the following features: original blog text, blog ID, number of likes, and publishing time. Figure 1 As shown, this embodiment is divided into an information collection module, a topic extraction module, a topic hierarchical clustering module, a tendency classification module, a topic semantic analysis module, and a group opinion statistics and output module, wherein:

[0028] The information collection module continuously collects the information published by the group. For the convenience of analysis, the information is summarized and collected by time slices, that is, the information within a certain period of time is saved as a text set. In this embodiment, the time is sliced ​​by year. In order to prevent the amount of data from being too small in some time periods, the total texts of the year less than 2,000 will be integrated and saved into the next year's text set, and a text set will be saved every year thereafter.

[0029] In this embodiment, the topic extraction module first uses a stop word list to clean up the text set and remove all stop words. Then extract phrases (segments with a word length of 2) in pairs and remove vocabulary combinations with a frequency higher than 6%. Use the singlepass algorithm to cluster the information in the text set collected by the information collection module, and all texts are clustered into multiple clusters. All clusters are sorted and numbered according to the number of members. The results of each cluster are counted for high-frequency phrases, and the top three high-frequency phrases in each cluster (word length greater than 2 and less than 4) are selected to verify whether the frequency of these high-frequency phrases in the text set is higher than 0.05%, or the number of texts in the cluster is higher than 3 and all texts contain the high-frequency word. Phrases that meet the above requirements will be output as keywords of group opinions in this text set. The keywords in all text sets constitute all the keywords to be processed by this system.

[0030] The topic hierarchical clustering module hierarchically clusters all the keywords or keyword groups extracted by the topic extraction module, and clusters the keywords in the same field together to form a topic, thereby forming a hierarchical topic word list for the group. The hierarchical clustering algorithm adopts a merging algorithm. That is, each keyword is treated as a class at the beginning. Aggregate into high-level classes. The calculation method of the similarity between keywords is as follows: when cocurr>2&&min(tf(word 1), tf(word 2)>2, the similarity is the square of cocurr / (tf(word 1)*tf(word 2)). Otherwise it is 0. Among them, cocurr is the number of co-occurring texts. tf(word) represents the number of texts in which the word has appeared.

[0031] The distance between two clusters is calculated using the complete linkage maximum distance algorithm. The smallest maximum distance is selected as the class distance.

[0032] For easy viewing, the final clustering results will be divided into two layers. The first layer (also called the domain layer) is the core vocabulary of each cluster, also known as the subject word. The selection method is to select the word with the largest average distance within the class. The second layer is the concept layer, which is the non-core vocabulary of each cluster, also known as the sub-subject word.

[0033] Once a word segmentation thesaurus is formed, the user can manually revise the thesaurus. When a new text set arrives, calculate the similarity between all keywords. Supplement the current hierarchical thesaurus. The supplementation process is based on the current hierarchical thesaurus, with the new thesaurus as an auxiliary. Existing keywords retain their original positions. For new words, find the most similar existing keywords in the new thesaurus, and the new words inherit the categories of their neighbors. When the keywords similar to the new word are not in the current hierarchical thesaurus, they can be added to the thesaurus as new categories.

[0034] The tendency classification module is a module of a deep learning algorithm based on natural language. Sample and model processing is to mark the supported subject words in a sentence with a support label pos, and mark the opposed subject words with an opposed label neg. In this way, there are two types of labels in the text, one is the vocabulary supported in the sentence, and the other is the vocabulary opposed in the sentence. According to the sequence labeling method, the subject words appearing in the text are classified to obtain the opinion tendency of each subject word in the sentence (in this embodiment, the positive tendency score of each subject word is 1, the negative tendency score is -1, and the other scores are 0). For subsequent analysis, the output of the tendency classification module is the speaker id, the subject words contained in the speech text, and the average of the tendency scores of each subject word. When a person has not published a text on a certain subject word, the person's tendency score for the subject word is 0.

[0035] The opposing topic analysis module, its input is the subject words and their tendencies of the tendency classification module and the hierarchical subject word table of the subject hierarchical clustering module. The opposing topic analysis module completes the analysis of opposing topics. Opposing topics occur between different sub-topics of the same topic. The existence of opposing topics means that there may be a split in the concept on this topic. There are individual data in the output of the tendency classification module. When the positive and negative signs of the tendency scores of an individual sub-topic word are opposite, and the absolute values ​​of the tendency scores are greater than 0.2, it means that the two sub-topic words in this person's opinion express strong and opposing views. The semantic analysis module will extract all such sub-topic word pairs. And the opposition coefficients of the two sub-topic words are calculated separately by this method: the number of people who strongly oppose the two sub-topic words / the total number of people who have expressed any of the two sub-topic words. If the opposition coefficient is greater than a certain threshold, and the threshold is set to 0.25 in this embodiment, the semantic opposition of the opposing sub-topic word pair is established.

[0036] Some opposing topics are caused by conflicts in their own connotations, but some topics are caused by certain conditions. For example, the opposition of some sub-topics is caused by the groups holding opinions. When the opinions of the groups change, the opposition may change. Therefore, whether the topic is opposing or not is related to time. The opposing topics calculated by the current text set need to be combined with the previous opposing topic data for synthesis.

[0037] The combined word list data will form the current system usage. The subject dictionary can save historical versions to facilitate users to check the changes of the subject.

[0038] The group opinion statistics and output module is inputted with the output of the opposing topic analysis module, i.e. the topic semantic word list and the opposing topic relationship, and the output of the tendency classification module, including the tendency classification results of the topic words. It can show the tendency of individuals on each topic, and can also count the number of people who express opinions on all topics, as well as the overall tendency of opinions. There are differences and consensuses in the group. The overall tendency numerical distribution of topics with consensus is relatively consistent, and the tendency of topics with large differences is relatively dispersed.

[0039] This embodiment has a deep network computing module, so it needs to be configured on a deep learning machine. The processing speed of each Weibo is within 1 second. The opinion analysis module can process 80,000 data in real time. The overall system can process group information with a message volume of about 40,000 messages per day.

[0040] Among the above modules / algorithms / operations, the opposing topic analysis module is original to the present invention, has never been disclosed, and its working method is different from any existing document record: the opposing topic analysis module is, and its technical details with significant improvement compared with the existing conventional technical means are as follows: in a specific subject area, what needs to be analyzed is the difference in details, so the difference in opinions between two people is not enough to explain the difference. Especially for some topics with strong subjectivity, but the semantic analysis of words cannot grasp the clear semantics of the topic in a specific group of people, therefore, the semantic opposition analysis of the topic in a specific group of people has two advantages: one is to clarify the difference in opinions, and the other is to use the independent semantics for semantic expansion.

[0041] After specific actual experiments, under the specific environment settings described in the embodiment, the above-mentioned device / method is started / run with the parameters described in the embodiment, and the experimental data that can be obtained are: the accuracy of the tendency analysis based on the semantic expansion of the opposing thesaurus is 83%, and the topic coverage is increased by 26%.

[0042] Compared with the prior art, the subject coverage of the present invention is improved by 26%, and the display effect of the overall tendency of the group is also improved by the same amount.

[0043] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A group opinion automatic extraction system based on opposing topic analysis, characterized in that: include: Information collection module, topic extraction module, topic hierarchical clustering module, tendency classification module, opposing topic analysis module, group opinion statistics and output module, wherein: the information collection module collects speech information of individuals in a specific group on a social information publishing platform; the topic extraction module extracts the most mentioned topics from the speech information; the topic hierarchical clustering module analyzes the distribution of topics in the speech information, and uses a hierarchical clustering algorithm to classify topics into different categories according to their meanings; the tendency classification module uses a tendency classification algorithm to classify the tendencies of topics in speech information; the opposing topic analysis module summarizes the classification results and analyzes whether there are paired opposing relationships in the same-layer topics and generates an opposing topic word list for all topics; the group opinion statistics are based on the analysis results and the overall tendency of each topic in the opposing topic word list and are visualized and output by the output module; The hierarchical clustering algorithm is to group related topics together as a category of topics, and all topics are organized hierarchically; The tendency classification module divides the sentences in the text set collected by the information collection module, performs tendency classification on the sentences where the subject words appear, and records the subject words in the sentences and their tendency scores. Since there is a sentence that expresses approval of a certain topic and opposition to another topic at the same time, it is necessary to perform tendency marking on all the subject words appearing in the text, so as to obtain independent opinion and position scores for each topic. For subsequent analysis, the output of the tendency classification module is the speaker ID, the subject words contained in the speech text, and the tendency score of each subject word; The generation process of the opposing thesaurus is as follows: given two similar topics, when the signs of the tendency means of the two topics in all texts sent by an individual are opposite and greater than a given threshold, the individual is marked as having opposing views on the two topics, and the number of people in the group who have opposing views on the two topics is counted. If the proportion of the total number of people or the number of people who have expressed their attitudes on such topics is greater than a certain threshold, it means that the two topics are opposing in the group, and all the topics and independent thesaurus pairs are saved to form an opposing thesaurus; The overall tendency is the overall tendency of a group towards a certain topic, and the calculation process is: the sum of the tendencies of all individuals in the group towards the topic is subtracted from the tendencies of all opposing topics to the topic, and the average value is calculated.

2. The automatic group opinion extraction system based on opposing topic analysis according to claim 1 is characterized in that: The information collection module continuously collects information that grows over time. In order to facilitate analysis, the information is summarized and collected by time slices, that is, the information within a certain period of time is saved as a text set. At the beginning of the system, historical information can be saved as a text set, and subsequent text sets are continuous revisions and supplements to the first analysis. The length of the time slice for dividing the text set can be determined according to the amount of information released. The less information is released, the longer the length of the time slice should be set to avoid the system being unable to carry out analysis due to too little data.

3. The automatic group opinion extraction system based on opposing topic analysis according to claim 1 is characterized in that: The topic extraction module uses the text collected by the information collection module to extract the keywords with high discussion frequency, and performs certain screening to use them as keywords of group opinions for subsequent analysis.

4. The automatic group opinion extraction system based on opposing topic analysis according to claim 1 is characterized in that: The topic hierarchical clustering module hierarchically clusters all the keywords or keyword groups extracted by the topic extraction module, and groups the keywords in the same field together to form a topic, thereby forming a hierarchical thesaurus for the group. Once the segmented thesaurus is formed, the user can manually revise the thesaurus. When a new text set arrives, it is supplemented on the basis of the current hierarchical thesaurus. All keywords under each topic are hierarchically processed, and the lower the level, the more specific the content expressed by the keywords.

5. The automatic group opinion extraction system based on opposing topic analysis according to claim 1 is characterized in that: The opposing topic analysis module is inputted with the subject words and their tendency scores of the tendency classification module and the hierarchical topic forest of the topic hierarchical clustering module. The topic semantic analysis module completes the identification of opposing topics. The topic semantic analysis module analyzes the relationship between keywords within the topic and semantically annotates keywords with opposing views.

6. The automatic group opinion extraction system based on opposing topic analysis according to claim 1 is characterized in that: The input of the group opinion statistics and output module is the output of the topic semantic analysis module, namely the topic semantic network, and the output of the tendency classification module, including the tendency classification results of the subject words.

Images