Social public opinion event detection method based on time series and topic similarity analysis

Abstract

The application discloses a social public opinion event detection method based on time series and topic similarity analysis, which comprises the following steps: crawling and preprocessing the published texts related to the required analysis theme in the analysis time period from the social network to obtain the topic time series of each topic, then segmenting the topic time series to obtain the topic time subsequence, extracting the number of heat peak points of the topic time subsequence and the topic time length to preliminarily screen the topics, then obtaining the co-occurrence topic set from the screened effective topics, calculating the peak distance of two topics in each co-occurrence topic pair and screening the co-occurrence topic pairs, calculating the text similarity of two topics in the reserved co-occurrence topic pair, screening the co-occurrence topic pairs with the text similarity greater than a threshold, generating a plurality of graphs according to the screened co-occurrence topic pairs, and each graph representing an event, so as to complete the detection of the social public opinion event. The application can more comprehensively and accurately mine the social public opinion event.

Description

Social media event detection methods based on time series and topic similarity analysis Technical Field

[0001] This invention belongs to the field of social media sentiment analysis technology, and more specifically, it relates to a method for detecting social media sentiment events based on time series and topic similarity analysis. Background Technology

[0002] The main task of social media sentiment awareness is to acquire and identify security elements that can cause changes in the network situation in large-scale social networks, and to extract events to enable intelligent decision-making.

[0003] In the online landscape, public opinion events exhibit different characteristics over time, and these events are highly time-sensitive, resulting in time series that lengthen with the passage of time. However, discussing public opinion events solely from a time series perspective is too simplistic, as text plays a crucial role in content output. A single text message represents the initiator's viewpoint and opinion, and as time passes and public opinion ferments, the cumulative number of text messages increases, forming a public opinion event. By abstracting public opinion heat into a time series, public opinion events can be extracted using time series pattern recognition methods. In a time series, an event exhibits single or few peaks within its contained time series, and its duration is short. An event may also consist of multiple topics; analyzing only one will lead to information omissions. In a series of time series describing the same event, its peaks appear within a certain time range, and their durations overlap significantly. However, multiple events may occur simultaneously, necessitating multimodal event identification using tweet content.

[0004] In summary, existing technologies for detecting social media public opinion events consider relatively singular and ineffective features, making it difficult to detect events comprehensively and accurately. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a social media event detection method based on time series and topic similarity analysis. This method jointly analyzes the relationship between time series, text information, and topics, and describes and filters topics from multiple dimensions, thereby more comprehensively and accurately mining social media events.

[0006] To achieve the above-mentioned objectives, the social media sentiment event detection method based on time series and topic similarity analysis of the present invention includes the following steps:

[0007] S1: Determine the topic to be analyzed based on actual needs, then crawl relevant texts from social networks within the analysis period, and extract tags, timestamps, and text content information for each text. Divide the analysis period into N sub-periods, each sub-period as a time point, and each tag as a topic, with the total number of topics being M. For each topic, count the frequency of the corresponding tag at each time point based on the tag and timestamp, and use this as the popularity value h of the topic at that time point. n,m Let n = 1, 2, ..., N, m = 1, 2, ..., M; for the m-th topic, the time points with a popularity value greater than 0 are considered valid time points, and the number of valid time points for the m-th topic is denoted as D. m Arrange the valid time points in chronological order to obtain the topic time series (t). m,1 ,t m,2 ,…,t m,d ), where t m,d This represents the original sequence number of the d-th time point in the m-th topic within the time period to be analyzed, where d = 1, 2, ..., D. m ;

[0008] S2: Sequentially judge the topic time series (t) m,1 ,t m,2 ,…,t m,d If the time interval between two adjacent valid time points is greater than a preset time threshold, no operation is performed; otherwise, a split point is inserted between these two valid time points. Based on the obtained split point, the topic time series is divided into topic time subsequences. Let K be the number of topic time subsequences obtained for the m-th topic. m The k-th topic time subsequence is denoted as Let K and K represent the start and end time indices of the time subsequence of the m-th topic and the k-th topic, respectively, where k = 1, 2, ..., K. m ;

[0009] For each topic time subsequence T m,k Obtain the popularity value sequence H of the m-th topic within the corresponding time period. m,k Search popularity value sequence H m,k The peak point of heat value in the sequence is denoted as heat value sequence H. m,k The number of intermediate heat peaks is W m,k ;

[0010] Calculate the average topic duration for the m-th topic. τ m,k Represents the topic time subsequence T m,k The length of the m-th topic is then calculated, and the frequency of the peak popularity point of the m-th topic is then calculated. Preset average topic duration threshold T L and the frequency threshold T of peak heat points F If the average topic duration L of the m-th topic m <T L And the frequency of peak heat points is F m <T F If the answer is yes, the topic will be considered valid; otherwise, the topic will be deleted.

[0011] S3: For each time point, filter out valid topics with a popularity value greater than 0 at that time point to obtain the co-occurrence topic set α for that time point. n ;

[0012] S4: For each set of co-occurring topics α n The topics are paired up to obtain co-occurring topic pairs. Then, the peak distance between the two topics in each co-occurring topic pair is calculated using the following method:

[0013] Let P1 and P2 be the two topics in the co-occurring topic pair, and let P be the topic in step S2. i The number of heat peaks obtained is R i Let i = 1, 2, and let t be the time index corresponding to the r-th heat peak. i,r r = 1, 2, ..., R i Topic P i Peak time point sequence Then, the DTW algorithm is used to calculate the distance between the peak time point sequences of topics P1 and P2 as the peak distance between topics P1 and P2.

[0014] A peak distance threshold is preset. If the peak distance between two topics in a co-occurring topic pair is greater than the peak distance threshold, the co-occurring topic pair is deleted; otherwise, the co-occurring topic pair is retained, resulting in a set Y of co-occurring topic pairs.

[0015] S5: Calculate the text similarity between the two topics of each co-occurring topic pair in the set Y. The specific method is as follows:

[0016] Let the two topics in the co-occurring topic pair be Q1 and Q2. Select topic Q from all the published texts crawled in step S1. i The published text set β i The text collection β will be published. i Each published text in the file is preprocessed using preset operations, and the preprocessed published texts are then concatenated into a long text G. i Then for the long text G i Word embedding is performed to obtain the long text G. i The text vector S i; Calculate the similarity between text vector S1 and text vector S2, and use it as the text similarity between topics Q1 and Q2;

[0017] S6: Pre-set a text similarity threshold. If the text similarity between two topics in a co-occurring topic pair set Y is less than the text similarity threshold, delete it; otherwise, retain the co-occurring topic pair to obtain a co-occurring topic pair set Y′. Then, generate a graph with each topic in the co-occurring topic pair set Y′ as a node and the text similarity between co-occurring topic pairs as the edge weight. Each subgraph in the graph represents an event, thereby completing the detection of social media events.

[0018] This invention presents a social media sentiment event detection method based on time series and topic similarity analysis. It crawls and preprocesses published texts related to the desired analysis topic within the time period from social networks to obtain topic time series. These time series are then segmented to obtain topic time subsequences. The number of peak popularity points and the duration of each topic time subsequence are extracted for initial topic filtering. From the filtered valid topics, a set of co-occurring topics is obtained. The peak distance between the two topics in each co-occurring topic pair is calculated, and these pairs are further filtered. The text similarity between the two topics in each co-occurring topic pair is calculated and retained. Co-occurring topic pairs with text similarity greater than a threshold are selected. Several graphs are generated based on the selected co-occurring topic pairs, each graph representing an event, thus completing the detection of social media sentiment events.

[0019] This invention performs joint analysis of the relationship between time series, text information, and topics, describing and filtering topics from multiple dimensions. It can cluster isolated topics extracted from the public opinion space and extract event-related topics and tweets, thereby more comprehensively and accurately mining social public opinion events. Attached Figure Description

[0020] Figure 1 is a flowchart of a specific implementation method for the social media event detection method based on time series and topic similarity analysis of the present invention;

[0021] Figure 2 is an example diagram of the topic time subsequence and popularity peak points in this embodiment;

[0022] Figure 3 is an example diagram of peak distance calculation in this embodiment;

[0023] Figure 4 illustrates the method for obtaining text vectors in this embodiment. Detailed Implementation

[0024] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0025] Example

[0026] Figure 1 is a flowchart illustrating a specific implementation of the social media event detection method based on time series and topic similarity analysis according to the present invention. As shown in Figure 1, the specific steps of the social media event detection method based on time series and topic similarity analysis according to the present invention include:

[0027] S101: Acquire and preprocess social network text:

[0028] The topic to be analyzed is determined based on actual needs. Then, relevant text posts within the analyzed time period are crawled from social networks, and their hashtags, timestamps, and text content are extracted. The analyzed time period is divided into N sub-time periods, each representing a time point. Each hashtag represents a topic, and the total number of topics is M. For each topic, the frequency of the corresponding hashtag at each time point is counted based on the hashtag and timestamp, and this frequency is used as the topic's popularity value h at that time point. n,m Let n = 1, 2, ..., N, and m = 1, 2, ..., M. For the m-th topic, the time points with a popularity value greater than 0 are considered valid time points, and the number of valid time points for the m-th topic is denoted as D. m Arrange the valid time points in chronological order to obtain the topic time series (t). m,1 ,t m,2 ,…,t m,d ), where t m,d This represents the original sequence number of the d-th time point in the m-th topic within the time period to be analyzed, where d = 1, 2, ..., D. m .

[0029] As can be seen, this step completes the acquisition of information about the topic and the generation of the topic's time series.

[0030] S102: Topic Pre-screening:

[0031] Next, the topic time series of each topic needs to be segmented to obtain topic time subsequences. The segmentation method is as follows:

[0032] Judge the topic time series (t) in sequence m,1 ,t m,2 ,…,t m,dIf the time interval between two adjacent valid time points is greater than a preset time threshold, no operation is performed; otherwise, a split point is inserted between these two valid time points. Based on the obtained split point, the topic time series is divided into topic time subsequences. Let K be the number of topic time subsequences obtained for the m-th topic. m The k-th topic time subsequence is denoted as These represent the start and end time points of the time subsequence of the m-th topic and the k-th topic, respectively.

[0033] For each topic time subsequence T m,k Obtain the popularity value sequence H of the m-th topic within the corresponding time period. m,k Search popularity value sequence H m,k The peak point of heat value in the sequence is denoted as heat value sequence H. m,k The number of intermediate heat peaks is W m,k .

[0034] Figure 2 is an example diagram of topic time subsequences and popularity peaks in this embodiment. As shown in Figure 2, topic 1 has 1 topic time subsequence with 1 peak, and topic 2 has 3 topic time subsequences, each with 1 peak.

[0035] Research has shown that social media sentiment events tend to have few peaks and short durations. Therefore, this invention uses this characteristic to pre-screen topics, and the specific method is as follows:

[0036] Calculate the average topic duration for the m-th topic. τ m,k Represents the topic time subsequence T m,k The length of the m-th topic is then calculated, and the frequency of the peak popularity point of the m-th topic is then calculated. Preset average topic duration threshold T L and the frequency threshold T of peak heat points F If the average topic duration L of the m-th topic m <T L And the frequency of peak heat points is F m <T F If the condition is met, the topic will be considered valid; otherwise, it will be deleted.

[0037] S103: Filtering co-occurring topics:

[0038] By identifying the overlapping relationships between topic time series, topics that appear simultaneously at a given point in time are obtained. The aim is to discover the vertical co-occurrence relationships between topics and integrate those topics that exhibit vertical co-occurrence. The specific method is as follows:

[0039] For each time point, valid topics with a popularity value greater than 0 at that time point are selected to obtain the co-occurrence topic set α for that time point. n .

[0040] S104: Topic pair filtering based on peak distance:

[0041] For each set of co-occurring topics α n The topics are paired up to obtain co-occurring topic pairs. Then, the peak distance between the two topics in each co-occurring topic pair is calculated using the following method:

[0042] Let P1 and P2 be the two topics in the co-occurring topic pair, and let P be the topic in step S102. i The number of heat peaks obtained is R i Let i = 1, 2, and let t be the time index corresponding to the r-th heat peak. i,r r = 1, 2, ..., R i Topic P i Peak time point sequence Then, the DTW (Dynamic Time Warping) algorithm is used to calculate the distance between the peak time point sequences of topics P1 and P2 as the peak distance between topics P1 and P2.

[0043] Figure 3 is an example diagram of peak distance calculation in this embodiment. As shown in Figure 3, topic 1 has one popularity peak point [(x 1,1 ,t 1,1 Therefore, the peak time point sequence is obtained as (t) 1,1 Topic 2 has 3 peak popularity points [(x 2,1 ,t 2,1 ),(x 2,2 ,t 2,2 ),(x 2,3 ,t 2,3 Therefore, the peak time point sequence (t) is obtained. 2,1 ,t 2,2 ,t 2,3 Then, the DTW algorithm is used to calculate the peak time point sequence (t). 1,1 ) and peak time point series (t 2,1 ,t 2,2 ,t 2,3 The distance between them is the peak distance between topics 1 and 2.

[0044] Research has found that a greater peak distance indicates a greater time gap between the occurrence of public opinion events. This suggests that the two topics do not describe the same public opinion event. Therefore, topic pairs can be filtered based on peak distance. The specific method is as follows:

[0045] A peak distance threshold is preset. If the peak distance between two topics in a co-occurring topic pair is greater than the peak distance threshold, the co-occurring topic pair is deleted; otherwise, the co-occurring topic pair is retained, resulting in a set Y of co-occurring topic pairs.

[0046] S105: Calculate the text similarity of co-occurring topics:

[0047] The text similarity between the two topics of each co-occurring topic pair in the set Y of co-occurring topic pairs is calculated using the following method:

[0048] Let the two topics in the co-occurring topic pair be Q1 and Q2. Select topic Q from all the published texts crawled in step S101. i The published text set β i The text collection β will be published. i Each published text in the document is preprocessed using preset operations (in this embodiment, this includes link deletion and special character filtering), and the preprocessed published texts are then concatenated into a long text G. i Then for the long text G i Word embedding is performed to obtain the long text G. i The text vector S i Calculate the similarity between text vector S1 and text vector S2, and use this as the text similarity between topics Q1 and Q2.

[0049] Figure 4 illustrates the method for obtaining text vectors in this embodiment. As shown in Figure 4, in this embodiment, text vectors are obtained using the following method:

[0050] The crawled published texts in step S101 are used as the overall corpus. Feature words are extracted from the overall corpus, and the number of feature words is denoted as V. Then, the TF-IDF (term frequency–inverse document frequency) algorithm is used to calculate the frequency of each feature word in the long text G. i TF-IDF value in i,v v = 1, 2, ..., V. TF-IDF is a commonly used weighting technique for information retrieval and data mining. TF stands for Term Frequency, and IDF stands for Inverse Document Frequency. TF-IDF values ​​reflect the importance of a word. A text consists of multiple words, each with a different weight. Combining all weights describes the characteristics of the text. Therefore, in this invention, the long text G... i TF-IDF values ​​of V feature words i,v Constructing the text vector S i=(TF-IDF) i,1 TF-IDF i,2 ,…,TF-IDF i,V ).

[0051] S106: Social Media Public Opinion Event Detection:

[0052] A text similarity threshold is pre-set. If the text similarity between two topics in a co-occurring topic pair set Y is less than the threshold, the co-occurring topic pair is deleted; otherwise, it is retained, resulting in a co-occurring topic pair set Y′. Then, a graph is generated using each topic in the co-occurring topic pair set Y′ as a node and the text similarity between co-occurring topic pairs as the edge weights. Each subgraph in the graph represents an event, thus completing the detection of social media sentiment events.

[0053] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A method for detecting social media sentiment events based on time series and topic similarity analysis, characterized in that, Includes the following steps: S1: Determine the topic to be analyzed based on actual needs, then crawl relevant texts from social networks within the analysis period, and extract tags, timestamps, and text content information for each text. Divide the analysis period into N sub-periods, each sub-period as a time point, and each tag as a topic, with the total number of topics being M. For each topic, count the frequency of the corresponding tag at each time point based on the tag and timestamp, and use this as the popularity value h of the topic at that time point. n,m Let n = 1, 2, ..., N, m = 1, 2, ..., M; for the m-th topic, the time points with a popularity value greater than 0 are considered valid time points, and the number of valid time points for the m-th topic is denoted as D. m Arrange the valid time points in chronological order to obtain the topic time series (t). m,1 ,t m,2 ,…,t m,d ), where t m,d This represents the original sequence number of the d-th time point in the m-th topic within the time period to be analyzed, where d = 1, 2, ..., D. m S2: Sequentially judge the topic time series (t) m,1 ,t m,2 ,…,t m,d If the time interval between two adjacent valid time points is greater than a preset time threshold, no operation is performed; otherwise, a split point is inserted between these two valid time points. Based on the obtained split point, the topic time series is divided into topic time subsequences. Let K be the number of topic time subsequences obtained for the m-th topic. m The k-th topic time subsequence is denoted as Let K and K represent the start and end time indices of the time subsequence of the m-th topic and the k-th topic, respectively, where k = 1, 2, ..., K. m For each topic time subsequence T m,k Obtain the popularity value sequence H of the m-th topic within the corresponding time period. m,k Search popularity value sequence H m,k The peak point of heat value in the sequence is denoted as heat value sequence H. m,k The number of intermediate heat peaks is W m,k ; Calculate the average topic duration for the m-th topic. τ m,k Represents the topic time subsequence T m,k The length of the m-th topic is then calculated, and the frequency of the peak popularity point of the m-th topic is then calculated. Preset average topic duration threshold T L and the frequency threshold T of peak heat points F If the average topic duration L of the m-th topic m <T L And the frequency of peak heat points is F m <T F If the topic is positive, it is considered a valid topic; otherwise, it is deleted. S3: For each time point, select valid topics with a popularity value greater than 0 at that time point to obtain the co-occurring topic set α for that time point. n S4: For each set of co-occurring topics α n The topics are paired up to obtain co-occurring topic pairs. Then, the peak distance between the two topics in each co-occurring topic pair is calculated using the following method: Let the two topics in the co-occurring topic pair be P1 and P2, and let the peak distance between the two topics in step S2 be P1 and P2. i The number of heat peaks obtained is R i Let i = 1, 2, and let t be the time index corresponding to the r-th heat peak. i,r r = 1, 2, ..., R i Topic P i Peak time point sequence Then, the DTW algorithm is used to calculate the distance between the peak time point sequences of topics P1 and P2 as the peak distance between topics P1 and P2. A peak distance threshold is preset. If the peak distance between the two topics in a co-occurring topic pair is greater than the peak distance threshold, the co-occurring topic pair is deleted; otherwise, the co-occurring topic pair is retained, resulting in a set of co-occurring topic pairs Y. S5: Calculate the text similarity between the two topics in each co-occurring topic pair in the set of co-occurring topic pairs Y. The specific method is as follows: Let the two topics in the co-occurring topic pair be Q1 and Q2. Select topic Q from all published texts crawled in step S1. i The published text set β i The text collection β will be published. i Each published text in the file is preprocessed using preset operations, and the preprocessed published texts are then concatenated into a long text G. i； Then, for the long text G... i Word embedding is performed to obtain the long text G. i The text vector S i ; Calculate the similarity between text vectors S1 and S2, and use it as the text similarity between topics Q1 and Q2; S6: Pre-set a text similarity threshold. If the text similarity between two topics in the co-occurring topic pair set Y is less than the text similarity threshold, then delete them; otherwise, retain the co-occurring topic pair to obtain the co-occurring topic pair set Y′; Then, use each topic in the co-occurring topic pair set Y′ as a node and the text similarity between co-occurring topic pairs as the edge weight to generate a graph. Each subgraph in the graph represents an event, thereby completing the detection of social media events.

2. The social media public opinion event detection method according to claim 1, characterized in that, In step S5, the long text G i The text vector S i The following method is used to obtain the corpus: all published texts crawled in step S1 are taken as the overall corpus, and the feature words of the overall corpus are extracted. The number of feature words is denoted as V. Then, the TF-IDF algorithm is used to calculate the feature words of each feature word in the long text G. i TF-IDF value in i,v v = 1, 2, ..., V; The long text G i TF-IDF values ​​of V feature words i,v Constructing the text vector S i =(TF-IDF) i,1 TF-IDF i,2 ,…,TF-IDF i,V ).

Images