Text topic sequence mutation detection method based on semi-Markov topic transition

Abstract

The invention relates to the technical field of intelligent information processing, in particular to a text topic sequence mutation detection method based on semi-Markov topic transition, which comprises the following steps of: outputting topic transition probability and residence time distribution by constructing a topic transition model of which the state duration obeys semi-Markov distribution; and identifying high-frequency transition events in a continuous time window by combining multiple signals such as emotional polarity change and text quantity change so as to realize accurate judgment of text theme sequence mutation points. The method solves the problems that in the prior art, the topic transition time sequence modeling capacity is insufficient, the sudden change point judgment mechanism is single, and text topic sequence sudden change recognition is lagged.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent information processing, and in particular to a text topic sequence mutation detection method based on semi-Markov topic transition. BACKGROUND

[0002] With the rapid development of social media platforms and short text dissemination mechanisms, how to timely identify the sudden text topic sequence event in the mass, dynamic and emotionally variable text information is the focus of the cross-field of natural language processing (NLP), social computing and intelligent decision-making in recent years. The current text topic sequence analysis method is mostly based on sentiment orientation discrimination, topic modeling, entity recognition and text clustering technology, supplemented by time series trend analysis, to realize the monitoring and judgment of the evolution process of the text topic sequence. However, due to the insufficient modeling capability of the traditional method for the dynamic transition process of the text topic sequence, especially in the multi-topic evolution and mutation state recognition, the existing model is difficult to accurately depict the non-Markov nature and the heterogeneity of the state transition time between topics, resulting in the problems of "late detection, high misjudgment rate and poor generalization" in the text topic sequence mutation point detection. The comprehensive judgment of the multiple signals such as emotional polarization, text explosion and topic jump in the mutation event has not yet formed a unified and effective evaluation framework.

[0003] CN111966878B discloses a news event reversal detection method based on machine learning, which constructs a sentiment polarity prediction model through news entities and public comments, and uses an autoregressive model combined with a sentence embedding method to identify the reversal event. This method can accurately identify the sentiment polarity reversal point and the underlying cause, but its modeling idea mainly focuses on event reversal, and lacks the modeling capability for the transition process between text topic sequences, which cannot meet the analysis needs of the transition behavior in the multi-topic alternative evolution process.

[0004] CN112417152A discloses a topic detection method for case-related text topics, which uses the combination of TF-IDF weight and additional weight of case elements, uses a self-encoder for text dimension reduction, and realizes topic detection through spectral clustering. Although it improves the clustering efficiency and accuracy, it is still limited to topic identification in static corpus and lacks the ability to handle topic mutation in dynamic time series, especially lacking the modeling mechanism for topic state persistence and transition probability.

[0005] In summary, the main problems with existing text topic sequence detection technologies are: a lack of modeling mechanisms for topic state transitions, making it difficult to accurately reflect the non-Markovian behavioral characteristics in the parallel evolution of multiple topics; a lack of collaborative confirmation mechanisms for multi-dimensional signals in mutation identification, such as the fusion evaluation of three dimensions: emotional transitions, text burst volume, and topic transition rate; and a reliance on a single indicator for mutation detection, leading to the risk of misjudgment and missed detection. To overcome these problems, this invention provides a text topic sequence mutation detection method based on semi-Markov topic transitions. By constructing a topic transition model whose state duration follows a semi-Markov distribution, it outputs the topic transition probability and dwell time distribution. Combined with multiple signals such as changes in emotional polarity and text quantity, it identifies high-frequency transition events within a continuous time window, achieving accurate determination of text topic sequence mutation points. This method solves the problems of insufficient ability to model the temporal sequence of topic transitions, a single mutation point judgment mechanism, and lagging text topic sequence mutation identification in existing technologies. Summary of the Invention

[0006] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section, as well as in the abstract and title of the present application, to avoid obscuring the purpose of this section, the abstract and title of the invention. Such simplifications or omissions shall not be used to limit the scope of the present invention.

[0007] In view of the aforementioned existing problems, the present invention is proposed.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: acquiring multi-source text data from social media platforms within a preset monitoring range, and constructing a time-series text corpus; The time-series text corpus is preprocessed, including word segmentation, stop word removal, and word vector encoding, to form a time-series topic expression matrix; Based on the time series topic representation matrix, a topic transition model that follows a semi-Markov distribution for state duration is constructed, and the topic transition probability and state dwell time distribution are output. Using the topic transition model, high-frequency topic transition events within a continuous time window are identified and marked as candidate text topic sequence mutation points; For each candidate text topic sequence mutation point, perform intensity assessment and mutation confirmation operations: Calculate the change in the probability of topic transition, the magnitude of the change in sentiment polarity, and the proportion of change in text volume of the candidate text topic sequence mutation point within adjacent time windows. Determine whether the change value of the topic transition probability is greater than a first threshold, whether the change magnitude of the sentiment polarity is greater than a second threshold, and whether the change ratio of the text volume is greater than a third threshold; If at least two of the following criteria are met: the change in topic transition probability, the change in emotional polarity, and the change in text volume, and both exceed the corresponding threshold, then the candidate text topic sequence mutation point is confirmed as a valid text topic sequence mutation point and included in the final text topic sequence mutation detection result; otherwise, the candidate text topic sequence mutation point is removed and marked as normal fluctuation.

[0009] The beneficial effects of this invention are as follows: By sequentially executing the steps of temporal corpus construction, topic expression preprocessing, semi-Markov transition modeling, high-frequency transition detection, and mutation intensity assessment and confirmation, this invention solves the problems of existing text topic sequence mutation detection methods, such as lack of modeling of topic state duration, difficulty in capturing complex topic evolution trends, delayed response to sudden events, and easy misjudgment or omission of mutation identification. Specifically, by constructing a temporal corpus of multi-source text topic sequence data, the continuity and diversity of text topic sequence evolution are ensured; word vectorization preprocessing enhances the accuracy of semantic modeling; the introduction of a semi-Markov model effectively overcomes the memorylessness defect of traditional models, realizing joint modeling of topic transition probability and state dwell time; high-frequency topic transition events are identified based on this model, and potential mutation risks are detected in advance; and mutation confirmation is performed by combining three-dimensional indicators of transition probability, sentiment polarity, and text activity, improving the robustness and accuracy of detection. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating the text topic sequence mutation detection method based on semi-Markov topic transitions as shown in this invention. Figure 2 This is a schematic diagram showing the change in detection accuracy over time for the conventional method and the method of this invention. Figure 3 This is a schematic diagram showing the variation of detection delay with window number in the conventional method and the method of the present invention. Figure 4 This is a schematic diagram showing the change of false alarm rate over time window for the conventional method and the method of this invention. Figure 5 This is a schematic diagram comparing the emotion reversal recognition rates of the traditional method and the method of this invention. Figure 6 This is a schematic diagram comparing the stability changes of the conventional method and the method of the present invention. Detailed Implementation

[0011] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0012] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort should fall within the scope of protection of this invention.

[0013] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0014] According to an embodiment of the present invention, in combination Figure 1 The flowchart shown illustrates a text topic sequence mutation detection method based on semi-Markov topic transitions, which specifically includes the following steps: S1. Obtain multi-source text data from social media platforms within the preset monitoring range and construct a time-series text corpus. Note that the following points should be noted in this step: Based on the real-time requirements of text topic sequence monitoring, the preset time granularity is 15 minutes, 30 minutes, 1 hour, 3 hours or 6 hours. The appropriate granularity can be adaptively selected based on the information release density of social platforms. This embodiment does not impose a unique requirement on it. The acquired raw text data is segmented and aggregated according to its timestamp information and a preset granularity, dividing the entire monitoring period into multiple adjacent and non-overlapping time windows. For example, if 1 hour is selected as the time granularity and the monitoring period is set to 08:00 to 20:00 on a certain day, then 12 time windows are constructed. Each time window ,satisfy:

[0015] in, Indicates time granularity. Indicates the start time of monitoring. Indicates the first The start time of each time window Indicates the first A time window, Indicates the first The end time of each time window.

[0016] Preferably, for each time window Iterate through all the raw text data collected within that time period and extract data that meets the timestamp criteria. The text entries also record their source platforms (such as Weibo, Zhihu, WeChat, and Tieba), and each text entry is tagged with a time tag and a platform tag, forming a tuple:

[0017] in, For text timestamps For platform identifiers (such as WB, ZH, WX). This refers to the text content.

[0018] Preferably, to ensure the quality and accuracy of the text topic sequence corpus, it is necessary to process the original text sets within each time window. Perform the following data cleaning steps: Use semantic vector cosine similarity to screen for duplicate text and remove redundant forwards and copied comments; Use spam text recognition models (such as the BERT classifier) ​​to remove advertising content containing keywords such as promotion, link, and QR code; Content containing only image links, emojis, or no actual text information is also filtered. After cleaning, a semantically valid time window text set is formed. .

[0019] in, Indicates the first The original text collection of each time window, This represents the set of valid text after content deduplication and noise removal.

[0020] Preferably, the cleaned text set under each window is... Based on its window number The texts are stored by number and arranged in numerical order to form a time-series text corpus that increases in chronological order. This structure not only preserves the temporal semantic sequence characteristics, but also provides continuous input for subsequent topic modeling and transition analysis; for example:

[0021] Where C represents the final time-series text corpus arranged in ascending order of time. Indicates the first The first time window The text content is N, where N represents the total number of time windows.

[0022] In the preferred embodiment, to avoid redundant text interfering with the subsequent topic transition modeling results, this embodiment performs a time window analysis. The original text collection within Performing duplicate text removal involves the following steps: Using open-source natural language processing tools (such as jieba) to process collections Each text in The text is segmented and further encoded using the Word2Vec model to obtain the corresponding semantic vector representation. ; Calculate the cosine similarity between any two texts based on semantic vector space. , Semantic similarity:

[0023] when If the timestamp interval between the two text posts is less than a preset time threshold (e.g., 30 minutes), then... Mark as redundant text; in, The similarity threshold (e.g., 0.95). Representation and text Another text is paired for similarity calculation. Representing text The semantic vector representation encoded by a word vector model (such as Word2Vec). Representing text semantic vector representation, Representing text and The cosine similarity value between semantic vectors , Representing semantic vectors respectively , Euclidean length; Remove the text marked as redundant from the set. Remove duplicates from the set to form a preliminary deduplication result set.

[0024] It should be noted that this method effectively identifies and eliminates a large number of copied and pasted comments and similar content that is forwarded in a short period of time, significantly improving the semantic diversity and representativeness of the text set, and providing quality assurance for subsequent topic modeling.

[0025] Preferably, to further improve the quality and accuracy of the time-series text corpus, this embodiment, after screening for duplicate texts, further refines the set... The remaining text in the file is further subjected to noise filtering, specifically including the following steps: Using the BERT binary classification model, which has been trained on large-scale Chinese social texts, for each text To determine whether a text is an advertisement, the model takes the original text after word segmentation as input and outputs the label as either an advertisement or not. If the predicted label is an advertisement, the text is removed. As an example, advertising text includes, but is not limited to: containing marketing-oriented keywords such as "scan to participate," "click the link," or "add WeChat," or containing a large number of non-natural language structures (such as QR code icons, hyperlinks, and combinations of phone numbers). Using regular expressions combined with rule matching mechanisms, identify and remove the following invalid text content: Text containing only image links (such as http: / / xxx.jpg or image structures); Text consisting only of emojis or special symbols, without any valid natural language; Texts shorter than a preset character threshold (e.g., 5 characters) and containing no key emotional words; Finally, all texts that passed the above two rounds of screening are retained as a set. As a time window The corresponding semantically valid text set.

[0026] It should also be noted that this combined approach of ad recognition and invalid structure filtering not only effectively eliminates machine-generated and promotional content, but also improves the accuracy of sentiment and theme analysis, providing more representative input data for subsequent text theme sequence mutation detection models.

[0027] S2. Perform text preprocessing on the time-series text corpus, including word segmentation, stop word removal, and word vector encoding, to form a time-series topic representation matrix. Note that the following points should be noted in this step: For the text set in each time window, a Chinese word segmenter based on the jieba dictionary is used for word segmentation, stop words are removed, and proper nouns are normalized into phrases. The word segmentation results are input into a pre-trained LDA topic modeler to extract topic probability distribution vectors; For the LDA model, set the number of topics K, output the topic distribution vector of each text in each window, and calculate the average topic distribution of all texts in the window to form the topic state vector of the time window; Perform the above steps for all time windows, and combine the topic state vectors corresponding to each time window in chronological order to form a time series topic expression matrix.

[0028] Preferably, in this embodiment, to ensure the accuracy and semantic consistency of subsequent topic modeling, it is necessary to perform Chinese word segmentation and stop word cleaning operations on the original text set within each time window. The specific method is as follows: each text is processed using a Chinese word segmentation tool based on an open dictionary to extract effective words; after word segmentation, the results are filtered using a preset stop word list to remove words that have no actual semantic contribution, including modal particles, auxiliary words, and conjunctions; in addition, for proper nouns consisting of multiple words, phrase recognition rules are used to merge them into unified entity words to avoid them being disassembled during semantic modeling and causing topic recognition bias; after cleaning, effective words with semantic value in each text are retained as input for subsequent modeling.

[0029] Furthermore, after completing text segmentation and cleaning, the processing results are input into a pre-trained topic modeling model for semantic extraction. Specifically, an unsupervised learning topic model can be used to model the segmented text. By learning the co-occurrence relationships of terms in the text, the distribution results of each text on several preset topics are extracted. This modeling process does not require manual annotation and can automatically identify the potential topic structure in the text. The modeling output is the probability distribution of each text among several topics. This probability vector represents the degree of association between the text and each topic. This distribution result serves as the topic feature representation of a single text, providing a foundation for subsequent time window topic state modeling.

[0030] Furthermore, for all texts within each time window, the topic distribution results obtained after topic modeling are extracted, and these topic distribution results are summarized according to the averaging rule to generate an overall topic state representation for that time window. Specifically, the probability values ​​of all texts within that time window on each topic are uniformly weighted and statistically analyzed, and then the statistical results are normalized to form a topic vector representing the semantic state of that time window. This topic vector can be regarded as an aggregate representation of the public's focus or issue structure within that time period, and can effectively reflect the semantic distribution characteristics of the text topic sequence within that time period.

[0031] Preferably, for the full-cycle text topic sequence monitoring data, the topic state vector corresponding to each time window is extracted sequentially in chronological order, and then concatenated sequentially along the time axis to construct a complete time series topic expression matrix. Each row of this matrix represents the topic state of a time window, and each column corresponds to the change in the degree of attention of a topic at different times. The final matrix can be used as the basic input for time series modeling, and provided to the subsequent algorithm model for modeling topic transitions, so as to support the tracking of topic evolution trends and the identification of abrupt change points.

[0032] S3. Based on the time-series topic representation matrix, construct a topic transition model where the state duration follows a semi-Markov distribution, and output the topic transition probability and the state dwell time distribution. Note that the following should be noted in this step: Input the time series topic representation matrix into the semi-Markov modeling framework, and set the initial state set, observation sequence length, and maximum duration threshold; Initialize the state transition probability matrix and the state duration distribution function, and iteratively optimize the model parameters using the expectation-maximization algorithm; In each iteration, the probability of occurrence and the probability of transition path for each topic state under different time slices are estimated by the forward-backward algorithm. Perform a joint loss function minimization process to optimize the following objective function to fit the transition mechanism:

[0033] Output the state transition probabilities after final convergence. The probability distribution of dwell time corresponding to each state is used to characterize the non-Markov transition behavior of the text topic sequence. in, For state Duration exponent coefficient, As a control factor for the intensity of the outbreak, For the smoothness adjustment factor of the transfer, For time delay adjustment parameters, Indicates the first Moments Theme The weight, For topic confidence function, As a word-oriented diversity index, Indicates from state to state The transition probability, where K represents the total number of topic states. and The index numbers represent the topic states, T represents the total length of the time window, and t represents the time slice number, from 1 to T, which are used to construct the dwell probability density of each state at each time step.

[0034] Furthermore, based on the topic transition model, the distribution of topic transition probabilities and state dwell time is output, including: Extract and label topic state sequences from preprocessed text topic sequence data; Based on the topic state sequence, the frequency of transitions between any two topic states is counted. And calculate from state transition to state transition probability The calculation formula is as follows:

[0035] For each state We statistically analyze the continuous dwell time samples in the topic state sequence and construct the probability distribution function of its dwell time. Used to depict state The distribution characteristics of the time lengths that remain constant are calculated using the following formula:

[0036] Then, construct a transition probability matrix from all the state transition probabilities:

[0037] And the set of probability distribution functions for dwell time in all states:

[0038] Together, they serve as the output structure of the semi-Markov theme transition model, used for subsequent high-frequency transition detection and mutation point confirmation operations; in, Indicates the state from state transition to state The transition probability, Indicates from state transition to state The actual number of times, Indicates from state Jump to all states The number of times is counted, where K represents the total number of topic states in the text topic sequence. Representing state The probability density function of dwell time, where t represents the dwell time variable.

[0039] It should be noted that this embodiment constrains the modeling range of text topic sequence time-series data by setting the topic state set, observation sequence length, and maximum dwell time threshold. This effectively solves the problems of high dimensionality, inconsistent time span, and inconsistent state granularity of the original text data, providing structured prior knowledge for subsequent modeling stages and helping to control model complexity. In each iteration, the estimation and correction of model parameters are performed, dynamically improving the accuracy of topic transition paths and state dwell time estimation, effectively avoiding the model from getting trapped in local optima, enhancing the global convergence of parameter estimation, realizing dynamic learning of topic evolution behavior at the probabilistic level, and providing quantifiable prior support for mutation point detection.

[0040] Preferably, by transmitting information from the start and end points of the time series respectively, the marginal probability of the topic state at any time can be accurately estimated. This enables efficient training under incompletely supervised labeling conditions, solving the problem of high dependence on labeling in traditional supervised methods. By embedding multi-dimensional constraints on topic burst intensity, time delay control, transition smoothness, and confidence changes into the objective function, the problem of overfitting and false detection in traditional single-objective modeling when detecting sudden text topic sequence events is solved, ensuring high fitting degree and high generalization ability of the model for the evolution mechanism of real text topic sequences.

[0041] In summary, this step, by introducing a semi-Markov modeling mechanism, accurately characterizes the evolution and transition behavior of text topic sequences under multiple time windows at the structural level, effectively overcoming the deficiency of traditional Markov models that ignore the influence of state dwell time on transitions, and achieving breakthroughs in jointly modeling transition paths and time persistence. By introducing forward-backward algorithms and expectation-maximization optimization strategies, the convergence efficiency and state estimation accuracy of the model are improved. Furthermore, by constructing a loss function with multi-dimensional regulatory factors, the model's ability to identify anomalous topic mutations and its robustness against interference are effectively enhanced.

[0042] S4. Using a topic transition model, identify high-frequency topic transition events within a continuous time window and mark them as candidate text topic sequence mutation points. Note that the following should be noted in this step: Under the set sliding time window parameters, the text topic sequence topic state vector sequence within the target time period is segmented by sliding window to extract the topic state change path within each time window; For each sliding window, count the number of topic transitions within that window and determine whether the number of transitions exceeds a preset transition frequency threshold. If the number of topic transitions within a certain window exceeds the transition frequency threshold, then the average dwell time of each topic state within that window and the average semantic similarity between adjacent topic states are further calculated. If one of the two conditions is met, the time slice corresponding to that window is marked as a candidate text topic sequence mutation point; Two conditions are included: the average dwell time of the topic status within the window is lower than the preset dwell time threshold, indicating that the topic changes frequently and is unstable; the average semantic similarity of adjacent topic statuses within the window is lower than the preset similarity threshold, indicating that there is a lack of continuity between topics and drastic changes occur.

[0043] Specifically, high-frequency transition events include: In two or more consecutive time windows, it was observed that the frequency of topic state transitions was higher than the transition frequency threshold, the state duration was significantly shortened, and the similarity between state vectors decreased rapidly. Within this interval, the total amount of text in the text topic sequence increased by more than 3%, and the sentiment polarity of keywords reversed, together constituting a complex abnormal mutation phenomenon. Among them, polarity reversal includes a sudden change from positive to negative or from negative to positive.

[0044] Preferably, to achieve continuous observation of the trend of text topic state changes within a target time period, a sliding time window parameter can be preset, and a sliding window segmentation operation can be performed on the text topic state vector sequence based on this parameter. Specifically, the preset sliding window parameter includes two elements: window length and sliding step size. The window length is three hours, and the sliding step size is one hour, that is, the window is slid once every hour to extract the topic state sequence within that time period. The topic state vector corresponding to each sliding window constitutes the topic state change path within that window, which facilitates the subsequent identification and analysis of transition events.

[0045] Preferably, to determine whether there is significant topic transition behavior within a certain time period, a transition count operation is performed for each sliding window; specifically, the number of changes in topic state within the window is counted, and if two adjacent topic states are inconsistent, it is counted as one transition; after the count is completed, the transition count is compared with a preset transition frequency threshold; the transition frequency threshold is preferably five times, that is, if the number of topic transitions within a certain time window exceeds five times, it is considered that the topic fluctuations within the window are frequent, and further analysis is needed to determine whether it is a candidate point for a sudden change event.

[0046] If the transition frequency within the window exceeds a preset threshold, preferably, the average dwell time of each topic state within the window and the semantic similarity between adjacent topic states are further calculated. The dwell time is the number of time slices in which the topic is continuously in the same topic state, reflecting the stability of the topic. If the average dwell time is short, usually less than two hours, it indicates that the topic switching is frequent and lacks continuity. In addition, in order to evaluate the semantic differences between topics, the similarity of adjacent topic state vectors needs to be calculated. The preferred method for calculating semantic similarity is to extract the angle relationship between vectors based on the sentence vector model and convert it into a similarity value. If the value is less than 0.65, it is considered that the semantic differences between adjacent topics are large and the topic switching is abrupt.

[0047] As an example, this embodiment uses the following rules to determine the mutation points of candidate text topic sequences: If the topic transition frequency exceeds a set threshold within a certain sliding window, and meets either of the following two conditions, the time point corresponding to the window can be marked as a candidate text topic sequence mutation point: The average residence time of the topic within the window is less than a preset time threshold of two hours, indicating that the topic is highly unstable; The average semantic similarity between adjacent topic states within the window is less than a preset similarity threshold of 0.65, indicating that the topic continuity is broken and drastic changes occur; In addition, if the above phenomena occur in two or more consecutive windows, and the total text volume of the text topic sequence increases by more than 3% within the corresponding time period, or the sentiment polarity of the keywords undergoes a positive-negative reversal, it is comprehensively identified as a composite mutation event and given special attention.

[0048] It should be noted that traditional text topic sequence monitoring methods often focus on single-dimensional analysis of sentiment polarity or keyword frequency, making it difficult to accurately depict the dynamic evolution of public opinion topics over time, especially exhibiting significant lag and false positive rates in identifying abrupt events. Therefore, this invention proposes a text topic sequence abrupt change detection method based on a semi-Markov topic transition model. Specifically, in step S4, a highly sensitive and robust abrupt change point identification mechanism is constructed by introducing a multi-dimensional fusion strategy involving a sliding time window, topic transition frequency, dwell time, and semantic similarity. By setting sliding time window parameters and performing sliding window segmentation on the text topic sequence topic state vector sequence, fine-grained tracking of the topic state evolution process in the time series is achieved. If the topic state changes frequently within the same time window, it may mean external event-driven or internal issue fission. By adopting the sliding window strategy, not only is the response accuracy to event dynamics improved, but the temporal continuity of the analysis process is also guaranteed, effectively avoiding the boundary effects brought about by global statistics.

[0049] By counting the number of topic transitions within each time window and comparing them with a preset transition frequency threshold, a first screening mechanism for the degree of topic disturbance is formed, clearly distinguishing between stable fluctuations and abnormal transitions through threshold setting.

[0050] Furthermore, within the window where the transition frequency exceeds the threshold, average dwell time and semantic similarity are introduced as dual auxiliary criteria to jointly assess the stability and coherence of topics from both temporal and semantic dimensions. Among them, dwell time, as a key indicator reflecting topic stability, captures the topic shift signals hidden in the rapid replacement of topics based on the principle of "short dwell time = high perturbation". Semantic similarity, on the other hand, identifies whether there is a topic gap phenomenon by quantifying the semantic correlation between adjacent topics, that is, the lack of semantic continuity between topics, indicating a drastic reconstruction of information structure.

[0051] Of particular note is that this method constructs a combined judgment rule: when any window simultaneously meets at least one of the following conditions, such as excessively high jump frequency, excessively short dwell time, or excessively low semantic similarity, that time slice is marked as a candidate text topic sequence mutation point; if two or more consecutive windows simultaneously exhibit this phenomenon, accompanied by a significant increase in text volume (more than 3%) and a reversal of keyword sentiment polarity, it is further identified as a composite mutation event; this rule not only integrates the multimodal features of structural change and sentiment evolution, but also reflects the strong correlation recognition capability of this invention in mutation detection mechanism, which can effectively suppress misjudgments caused by single anomalies.

[0052] In summary, this step, based on the semi-Markov topic transition principle, introduces time window sliding segmentation, transition frequency determination, dwell time analysis, and semantic coherence evaluation techniques to achieve high-precision identification of abrupt change points in text topic sequences. This solves the problem of insensitivity to topic abrupt change events in traditional text topic sequence monitoring and significantly improves the timeliness and accuracy of text topic sequence early warning.

[0053] S5. Perform intensity evaluation and mutation confirmation operations on each candidate text topic sequence mutation point. Note that the following points should be noted in this step: Calculate the changes in the probability of topic transitions, the magnitude of sentiment polarity changes, and the proportion of text volume changes of candidate text topic sequence mutation points within adjacent time windows; these include: For each candidate text topic sequence mutation point, the time window t containing that window and its previous window are obtained. Topic transition probability distribution Calculate the total difference to obtain the change in transition probability. :

[0054] Statistical time window t and Sentiment polarity score for each text. The change in the average polarity within the window is calculated to obtain the magnitude of the change in emotional polarity. :

[0055] Get the total amount of text in the window and calculate the percentage change. :

[0056] Where K is the number of topics. Let k be the probability of topic k in window t. For the first The sentiment score of each text. The number of text elements in window t; Determine whether the change in the probability of topic transition is greater than the first threshold, whether the change in sentiment polarity is greater than the second threshold, and whether the change in the proportion of text volume is greater than the third threshold. If at least two of the following criteria are met: the change in topic transition probability, the magnitude of change in sentiment polarity, and the proportion of change in text volume, the candidate text topic sequence mutation point is confirmed as a valid text topic sequence mutation point and included in the final text topic sequence mutation detection result; otherwise, the candidate text topic sequence mutation point is removed and marked as normal fluctuation.

[0057] Furthermore, the methods for selecting the first threshold, the second threshold, and the third threshold include: Obtain a sample set of historical text topic sequence data containing multiple time periods. Each sample data is a change period that has been manually labeled, and the dataset contains at least one hundred time windows. For each time period in the sample set, the corresponding topic transition probability change value, sentiment polarity change magnitude, and text volume change ratio are extracted, and three numerical sequences are constructed respectively. For the sequence representing the probability change of topic transition, arrange them in ascending order, and select the 80th value in the sequence as the first threshold for judging topic transition. For the sequence representing the magnitude of change in emotional polarity, arrange them in ascending order, and select the value ranked 85th in the sequence as the second threshold for judging emotional fluctuations. For a sequence representing the proportion of text volume change, arrange them in ascending order, and select the 90th value in the sequence as the third threshold for judging the change in text density.

[0058] It should be noted that this step uses the principle of cross-time window difference modeling to measure the degree of change in structure, sentiment and density between the candidate time slice and its previous window, and quantifies the strength of the mutation signal. This principle effectively solves the problem of pseudo mutations caused by factors such as regular fluctuations, periodic topic backtracking or short-term emotional surges in text topic sequences, so that the method of this invention not only has high sensitivity, but also good noise resistance and discrimination ability.

[0059] Preferably, the difference in the probability distribution of topic transitions is introduced as an evaluation index to measure the overall shift in the topic distribution within a certain time window compared to the previous time window. Specifically, the difference between the proportions of each topic in the current window and the previous window is aggregated and summed to form the transition change value. This index is based on the state non-memory transition mechanism of a semi-Markov model, which can intuitively reflect whether there are breaks, reconstructions, or jumps in the current topic structure, thereby achieving efficient capture of abrupt changes in the focus of the text topic sequence.

[0060] Furthermore, by calculating the mean difference in the sentiment polarity scores of the text within each time window, the magnitude of the change in sentiment polarity is obtained. This indicator reflects the fluctuation of the public's overall attitude over two consecutive time periods, and is particularly indicative of the reversal of sentiment polarity. When the public's attitude shifts rapidly from positive to negative (or vice versa), it often means that the nature of the event or social interpretation has undergone a key shift, which is an important signal of the sudden trend of the text theme sequence.

[0061] Furthermore, by introducing the relative change ratio of the amount of text within the window, the trend of the dissemination frequency of the text topic sequence information is reflected. This indicator is a direct reflection of the change in the popularity of the text topic sequence and can effectively identify the explosive growth range brought about by the fermentation of the event, media intervention, or collective response of users. Especially before and after a sudden event, there is often a large number of forwards, comments, or concentrated reports in a short period of time. Therefore, this change ratio is an important basis for judging the degree of suddenness of the event.

[0062] Preferably, this step effectively solves the problems of the inability to quantify mutation intensity, high false alarm rate and lack of multi-dimensional fusion in the existing technology, and realizes the screening of core sudden event nodes that truly affect the social public opinion structure and emotional fluctuations from candidate mutation points.

[0063] To verify the advantages of the text topic sequence mutation detection method based on semi-Markov topic transitions provided in this invention compared to existing machine learning-based text topic sequence event reversal detection methods, specifically to verify the improvement in the accuracy of text topic sequence mutation event recognition, as well as the improvements in detection latency, false alarm rate, sentiment reversal recognition rate, and system stability, the following experiments were conducted to compare the performance of the two methods in a real text topic sequence environment simulation: The experimental data comes from a simulated negative event scenario of a brand's product on a major social media platform (Weibo, Zhihu, Tieba). The monitoring period was from November 1 to November 10, 2024. The monitoring scope was: brand name + negative keyword combination. A total of 12,000 original texts were collected, with about 70% from Weibo, about 20% from Zhihu, and about 10% from Tieba. Each text record includes a posting timestamp, source platform, body text, number of likes / replies, and user sentiment markers (automatically preliminarily classified as positive, negative, and neutral).

[0064] The hardware environment is a standard server (Intel Xeon 4 cores, 16GB memory), and the software environment is Python 3.10 for preprocessing and model running, and MATLAB R2023b for data visualization.

[0065] Preprocessing steps: text segmentation, stop word removal, word vector encoding (Word2Vec dimension 100) to construct a time series topic expression matrix; the method of this invention further constructs a semi-Markov topic transition model to output the transition probability and state dwell time distribution; the comparison method uses a machine learning model (feature word vectors + sensitive sense matching + classification model) as the baseline.

[0066] Both methods are trained / initialized on the same corpus. The traditional method uses its publicly available feature extraction and classification process; the method of this invention performs topic modeling, semi-Markov model training, and mutation candidate point identification.

[0067] During the detection phase, each hour is a time window with a sliding step of 30 minutes. Every hour, the window outputs whether a mutation point is determined and records the recognition results of the two methods in each window. The detection status of each method for 6 known manually labeled sudden text topic sequence events (actual events and manually labeled) during the monitoring period is collected: whether they are recognized, the recognition time, and whether they are misjudged.

[0068] The statistical indicators in this embodiment include the following five indicators: Detection accuracy (number of real events identified / 6); Detection latency (average number of seconds from the time the event occurs to the time of recognition); False alarm rate (number of incorrectly marked mutation points / total number of non-event windows); Emotion reversal recognition rate (recognition rate when emotional polarity changes from positive to negative or vice versa during an event). Stability index (an inverse indicator of fluctuations in continuous window judgment results); After both methods are executed, the data is summarized and generated using MATLAB software, as shown below. Figures 2-6 The curve shown is a schematic diagram.

[0069] Reference Figure 2 As can be seen, as the monitoring time window progresses, the detection accuracy of traditional machine learning-based detection methods generally shows a slow upward trend, increasing from 75% to 78.3%, with limited growth and tending towards a plateau. In contrast, the detection accuracy of the method of this invention reaches 88% initially and steadily rises to 91.7%, consistently exceeding the comparison method by more than 10 percentage points throughout. This result indicates that the method of this invention utilizes a semi-Markov model to model the transition and dwell time of the topic state, significantly improving the ability to characterize the abrupt changes in the text topic sequence and effectively solving the shortcomings of traditional methods that rely on static feature classification and are difficult to adapt to abrupt change scenarios.

[0070] Reference Figure 3As can be seen, the detection latency of traditional methods remains between 8.7 and 9.5 seconds, indicating that their response mechanism to sudden events has a certain lag and is difficult to provide rapid feedback. In contrast, the detection latency of the method of this invention gradually decreases from 6 seconds initially to 4.8 seconds, reducing the overall response time by nearly half and demonstrating a significant real-time advantage. This advantage mainly stems from the fact that this invention introduces a joint triggering mechanism of "transition frequency + state duration" in the determination of mutation points, enabling the model to capture key feature changes with higher sensitivity when the topic state undergoes a sudden change, thereby improving detection speed, reducing response latency, and meeting the real-time requirements of high-frequency text topic sequence monitoring.

[0071] Reference Figure 4 As can be seen, the false alarm rate of the traditional method slowly decreased from 18% to 16.7%, but it is still at a relatively high level, indicating that it has a strong tendency to misjudge in non-real event windows. The method of the present invention, on the other hand, decreased from the initial 10% to 8.3%, and the rate of decrease was faster and the stability was stronger. That is, the present invention effectively avoids the misidentification of emotional fluctuations or normal topic changes as sudden events by constructing a joint prior mechanism of state transition probability and dwell time, thereby fundamentally improving the robustness and fault tolerance of the model's judgment and significantly reducing the frequency of false alarms.

[0072] Reference Figure 5 As can be seen, the two methods exhibit differences in their ability to identify emotional shift events (such as a change from positive to negative). The traditional method's emotional reversal recognition rate has remained at 60-64.2% for a long time, with slow improvement and poor recognition effect. In contrast, the present invention initially reached 85%, and subsequently steadily improved to 88.3%, maintaining a significant advantage. This indicates that the present invention, by embedding the emotional polarity change rate and topic transition dynamic indicators, achieves deep perception and accurate locking of potential situational reversal signals. Especially in highly sensitive scenarios where the text topic sequence rapidly shifts, it has stronger response capability and judgment accuracy, solving the problem that the traditional method only relies on word polarity recognition and ignores the topic evolution logic.

[0073] Reference Figure 6 The study demonstrates the consistency and volatility differences between the two methods in determining results within a continuous time window. A higher stability index indicates more stable detection results and less false alarms. The stability index of the traditional method fluctuates between 0.75 and 0.81 throughout the monitoring period, exhibiting some volatility and susceptibility to sudden emotional fluctuations. In contrast, the stability index of the method in this invention steadily increases from 0.88 to 0.93, indicating stronger adaptability to text topic sequence environments and more consistent judgments. This advantage is mainly attributed to the fact that this invention constructs a judgment logic with dual constraints of state transition, dwell time, and threshold, which enhances the model's ability to respond smoothly to input changes, effectively avoids false triggers caused by short-term noise or non-sudden fluctuations, and ensures stable and reliable system operation.

[0074] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A text topic sequence mutation detection method based on semi-Markov topic transitions, characterized in that, include: Acquire multi-source text data from social media platforms within a preset monitoring range and construct a time-series text corpus; The time-series text corpus is preprocessed, including word segmentation, stop word removal, and word vector encoding, to form a time-series topic expression matrix; Based on the time series topic representation matrix, a topic transition model that follows a semi-Markov distribution for state duration is constructed, and the topic transition probability and state dwell time distribution are output. Using the topic transition model, high-frequency topic transition events within a continuous time window are identified and marked as candidate text topic sequence mutation points; For each candidate text topic sequence mutation point, perform intensity assessment and mutation confirmation operations: Calculate the change in the probability of topic transition, the magnitude of the change in sentiment polarity, and the proportion of change in text volume of the candidate text topic sequence mutation point within adjacent time windows. Determine whether the change value of the topic transition probability is greater than a first threshold, whether the change magnitude of the sentiment polarity is greater than a second threshold, and whether the change ratio of the text volume is greater than a third threshold; If at least two of the following criteria are met: the change in topic transition probability, the change in emotional polarity, and the change in text volume, and both exceed the corresponding threshold, then the candidate text topic sequence mutation point is confirmed as a valid text topic sequence mutation point and included in the final text topic sequence mutation detection result; otherwise, the candidate text topic sequence mutation point is removed and marked as normal fluctuation.

2. The text topic sequence mutation detection method based on semi-Markov topic transition as described in claim 1, characterized in that, The construction of the time-series text corpus includes: Based on the preset time granularity, the collected original text data of social media is segmented and aggregated according to the timestamp, and divided into multiple adjacent and non-overlapping time windows to construct a time window set; For each time window, extract all raw text data within that time period and mark it with platform tags to obtain a set of raw text with time and platform tags; Perform content deduplication on the original text set and remove invalid noise content such as advertisements and image links; The cleaned text is stored according to window number to form a multi-window text sequence arranged in ascending order of time, thus obtaining the time-series text corpus.

3. The text topic sequence mutation detection method based on semi-Markov topic transitions according to claim 1 or 2, characterized in that, The time-series text corpus is preprocessed to form a time sequence. Sequence topic representation matrix, including: For the text set in each time window, a Chinese word segmenter based on the jieba dictionary is used for word segmentation, stop words are removed, and proper nouns are normalized into phrases. The word segmentation results are input into a pre-trained LDA topic modeler to extract topic probability distribution vectors; For the LDA model, set the number of topics K, output the topic distribution vector of each text in each window, and calculate the average topic distribution of all texts in the window to form the topic state vector of the time window; Perform the above steps for all time windows, and combine the topic state vectors corresponding to each time window in chronological order to form a time series topic expression matrix.

4. The text topic sequence mutation detection method based on semi-Markov topic transitions according to claim 3, characterized in that, Based on the aforementioned time-series topic representation, a topic transition model is constructed where the state duration follows a semi-Markov distribution, including: The time series topic representation matrix is ​​input into the semi-Markov modeling framework, and the initial state set, observation sequence length, and maximum duration threshold are set. Initialize the state transition probability matrix and the state duration distribution function, and iteratively optimize the model parameters using the expectation-maximization algorithm; In each iteration, the probability of occurrence and the probability of transition path for each topic state under different time slices are estimated by the forward-backward algorithm. Perform a joint loss function minimization process to optimize the following objective function to fit the transition mechanism: Output the state transition probabilities after final convergence. The probability distribution of dwell time corresponding to each state is used to characterize the non-Markov transition behavior of the text topic sequence. in, For state Duration exponent coefficient, As a control factor for the intensity of the outbreak, For the smoothness adjustment factor of the transfer, For time delay adjustment parameters, Indicates the first Moments Theme The weight, For topic confidence function, As a word-oriented diversity index, Indicates from state to state The transition probability, where K represents the total number of topic states. and The index numbers represent the topic states, T represents the total length of the time window, and t represents the time slice number, from 1 to T, which are used to construct the dwell probability density of each state at each time step.

5. The text topic sequence mutation detection method based on semi-Markov topic transitions according to claim 4, characterized in that, The topic transition model outputs the topic transition probability and state dwell time distribution, including: Extract and label topic state sequences from preprocessed text topic sequence data; Based on the topic state sequence, the jump frequency between any two topic states is counted. And calculate from state transition to state transition probability The calculation formula is as follows: For each state We statistically analyze the continuous dwell time samples in the topic state sequence and construct the probability distribution function of its dwell time. Used to depict state The distribution characteristics of the time lengths that remain constant are calculated using the following formula: Then, construct a transition probability matrix from all the state transition probabilities: And the set of probability distribution functions for dwell time in all states: Together, they serve as the output structure of the semi-Markov theme transition model, used for subsequent high-frequency transition detection and mutation point confirmation operations; in, Indicates the state from state transition to state The transition probability, Indicates from state transition to state The actual number of times, Indicates from state Jump to all states The number of times is counted, where K represents the total number of topic states in the text topic sequence. Representing state The probability density function of dwell time, where t represents the dwell time variable.

6. The text topic sequence mutation detection method based on semi-Markov topic transitions according to claim 4, characterized in that, Using the aforementioned topic transition model, high-frequency transition events are identified and candidate mutation points are labeled, including: Under the set sliding time window parameters, the text topic sequence topic state vector sequence within the target time period is segmented by sliding window to extract the topic state change path within each time window; For each sliding window, count the number of topic transitions within that window and determine whether the number of transitions exceeds a preset transition frequency threshold. If the number of topic transitions within a certain window exceeds the transition frequency threshold, then the average dwell time of each topic state within that window and the average semantic similarity between adjacent topic states are further calculated. If one of the two conditions is met, the time slice corresponding to that window is marked as a candidate text topic sequence mutation point; The two conditions include: the average dwell time of the topic state within the window is lower than a preset dwell time threshold, indicating that the topic changes frequently and is unstable; and the average semantic similarity of adjacent topic states within the window is lower than a preset similarity threshold, indicating that there is a lack of continuity between topics and drastic changes occur.

7. The text topic sequence mutation detection method based on semi-Markov topic transitions according to claim 6, characterized in that, The high-frequency transition events include: In two or more consecutive time windows, it was observed that the frequency of topic state transitions was higher than the transition frequency threshold, the state duration was significantly shortened, and the similarity between state vectors decreased rapidly. Within this interval, the total amount of text in the text topic sequence increased by more than 3%, and the sentiment polarity of keywords reversed, together constituting a complex abnormal mutation phenomenon. The polarity reversal includes a sudden change from positive to negative or a sudden change from negative to positive.

8. The text topic sequence mutation detection method based on semi-Markov topic transitions according to claim 1, characterized in that, The calculation of the change in the probability of topic transition, the magnitude of the change in sentiment polarity, and the proportion of change in text volume of the candidate text topic sequence mutation point within adjacent time windows includes: For each candidate text topic sequence mutation point, the time window t containing that window and its previous window are obtained. Topic transition probability distribution Calculate the total difference to obtain the change in transition probability. : Statistical time window t and Sentiment polarity score for each text. The change in the average polarity within the window is calculated to obtain the magnitude of the change in emotional polarity. : Get the total amount of text in the window and calculate the percentage change. : Where K is the number of topics. Let k be the probability of topic k in window t. For the first The sentiment score of each text. The number of text elements in window t.

9. The text topic sequence mutation detection method based on semi-Markov topic transitions according to claim 1, characterized in that, The method for selecting the first threshold, the second threshold, and the third threshold includes: Obtain a sample set of historical text topic sequence data containing multiple time periods. Each sample data is a change period that has been manually labeled, and the dataset contains at least one hundred time windows. For each time period in the sample set, the corresponding topic transition probability change value, sentiment polarity change magnitude, and text volume change ratio are extracted to construct three numerical sequences respectively. For the sequence representing the probability change of topic transition, arrange them in ascending order, and select the 80th value in the sequence as the first threshold for judging topic transition. For the sequence representing the magnitude of change in emotional polarity, arrange them in ascending order, and select the value ranked 85th in the sequence as the second threshold for judging emotional fluctuations. For a sequence representing the proportion of text volume change, arrange them in ascending order, and select the 90th value in the sequence as the third threshold for judging the change in text density.

Images