Public opinion popularity anomaly detection method and system based on information denoising and clustering

Abstract

The invention discloses a public opinion popularity anomaly detection method and system based on information denoising and clustering, and the method comprises the steps: obtaining a to-be-detected public opinion data set, and obtaining a target data set after denoising; the average power, the sample entropy and the standard deviation of the target data set are calculated to serve as feature information, clustering is conducted on elements in the target data set according to the feature information, and the class with the small number of elements serves as a first abnormal data set; according to a preset time window length and a time window moving step length, sequentially intercepting and sorting data from the initial position of the target data set to obtain a process data set, inputting the process data set into the auto-encoder, and extracting elements with deviation of an input signal and an output signal exceeding a limit to construct a second abnormal data set; and inputting the first abnormal data set and the second abnormal data set into a public opinion popularity anomaly detection model to obtain an anomaly classification result of the to-be-detected public opinion data. The method can adapt to complex scenes and has efficient processing capacity, and the performance of public opinion popularity anomaly detection is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a method and system for detecting public opinion heat anomaly based on information denoising and clustering. Background Art

[0002] With the rapid development of informatization, the Internet has become the main channel for information dissemination, and online public opinion has increasingly become the focus of attention from all walks of life. The heat of public opinion reflects the degree of attention a certain event has received on the Internet, and its abnormal fluctuations usually indicate potential social risks, market changes or brand crises. Therefore, the anomaly detection technology of public opinion heat has important application value in the fields of public opinion management, crisis response and brand maintenance.

[0003] Existing methods for monitoring the heat of public opinion and detecting anomalies often have several major problems: First, online public opinion data contains a large amount of noise information, which interferes with the heat analysis of specific events and reduces the detection accuracy; second, most existing methods are based on fixed thresholds for anomaly detection. This static threshold is difficult to adapt to the complex and changeable network environment, resulting in poor generalization; third, when faced with large-scale data, the existing anomaly detection algorithms are inefficient and difficult to meet real-time requirements. Summary of the invention

[0004] In order to solve the problems existing in the above-mentioned prior art, the present invention aims to provide a method and system for detecting anomalies in public opinion heat that can effectively remove noise, adapt to complex scenarios, and have efficient processing capabilities. By improving the denoising technology, optimizing the discrimination method of anomaly detection, and improving the efficiency of the algorithm, the performance of anomaly detection of public opinion heat can be significantly improved, providing users with more reliable crisis warning and decision support tools.

[0005] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention includes:

[0006] The method for detecting abnormal public opinion heat based on information denoising and clustering includes the following steps:

[0007] Obtain the public opinion dataset to be detected, and obtain the target dataset after denoising;

[0008] The average power, sample entropy and standard deviation of the target data set are respectively calculated as feature information, the elements in the target data set are clustered according to the feature information, and a class with a smaller number of elements is taken as a first abnormal data set;

[0009] According to the preset time window length and time window moving step, data is intercepted and sorted in sequence from the starting position of the target data set to obtain a process data set, which is input into the trained autoencoder, and elements with excessive deviations between the input signal and the output signal are extracted to construct a second abnormal data set;

[0010] The first abnormal data set and the second abnormal data set are input into a trained public opinion heat anomaly detection model built based on a classification neural network to obtain anomaly classification results of the public opinion data to be detected.

[0011] Preferably, the training method of the autoencoder includes:

[0012] Obtain the historical public opinion data set and obtain the first training set after denoising;

[0013] respectively calculating the average power, sample entropy and standard deviation of the first training set as feature information, clustering the elements in the first training set according to the feature information, and taking the class with a larger number of elements as a normal data set;

[0014] According to a preset time window length and a time window moving step length, data is intercepted and sorted in sequence starting from the starting position of the normal data set to obtain a second training set;

[0015] Minimizing the deviation between the input signal and the output signal of the autoencoder is used as the objective function, and the second training set is used to complete the training of the autoencoder.

[0016] Preferably, the training method of the public opinion heat anomaly detection model includes:

[0017] Obtain the historical public opinion data set and obtain the third training set after denoising;

[0018] respectively calculating the average power, sample entropy and standard deviation of the third training set as feature information, clustering the elements in the third training set according to the feature information, taking the class with more elements as a normal data set, and taking the class with fewer elements as a third abnormal data set;

[0019] According to the preset time window length and time window moving step, data is intercepted and sorted in sequence from the starting position of the third training set to obtain a process data set, which is input into the trained autoencoder, and elements with excessive deviations between the input signal and the output signal are extracted to construct a fourth abnormal data set;

[0020] A public opinion heat anomaly detection model is constructed based on a classification neural network, and the public opinion heat anomaly detection model is trained using the third anomaly data set and the fourth anomaly data set.

[0021] Preferably, the denoising method includes: decomposing the data set to be denoised using a variational mode decomposition method to obtain a number of sub-modes with different energy ratios, selecting three sub-modes with the highest energy ratios as effective sub-modes for element reconstruction, and obtaining a denoised data set.

[0022] Preferably, the denoising method includes: decomposing the data set to be denoised using an empirical wavelet transform method to obtain a number of frequency sub-layers with different frequencies, and selecting a number of frequency sub-layers lower than a preset frequency as elements of the data set after denoising.

[0023] The present invention also provides a public opinion heat anomaly detection system based on information denoising and clustering, including:

[0024] Data collection module, used to obtain the public opinion data set to be tested;

[0025] A denoising module, connected to the data collection module, for denoising the public opinion data set to obtain a target data set;

[0026] A feature calculation module, connected to the denoising module, for respectively calculating the average power, sample entropy and standard deviation of the target data set as feature information;

[0027] A clustering module, connected to the feature calculation module, for clustering the target data set according to the feature information, and taking a class with a smaller number of elements after clustering as a first abnormal data set;

[0028] An autoencoder module, which has a built-in trained autoencoder and is connected to the denoising module, is used to intercept and sort data in sequence from the starting position of the target data set according to a preset time window length and a time window moving step, obtain a process data set and input it into the trained autoencoder, extract elements with excessive deviations between the input signal and the output signal to construct a second abnormal data set;

[0029] The classification detection module has a built-in trained public opinion heat anomaly detection model constructed based on a classification neural network, which is connected to the clustering module and the autoencoder module respectively, and is used to input the first anomaly data set and the second anomaly data set into the public opinion heat anomaly detection model to obtain the anomaly classification results of the public opinion data to be detected.

[0030] Preferably, the training method of the autoencoder built into the autoencoder module includes:

[0031] Obtain the historical public opinion data set and obtain the first training set after denoising;

[0032] respectively calculating the average power, sample entropy and standard deviation of the first training set as feature information, clustering the elements in the first training set according to the feature information, and taking the class with a larger number of elements as a normal data set;

[0033] According to a preset time window length and a time window moving step length, data is intercepted and sorted in sequence starting from the starting position of the normal data set to obtain a second training set;

[0034] Minimizing the deviation between the input signal and the output signal of the autoencoder is used as the objective function, and the second training set is used to complete the training of the autoencoder.

[0035] Preferably, the training method of the public opinion heat anomaly detection model built into the classification detection module includes:

[0036] Obtain the historical public opinion data set and obtain the third training set after denoising;

[0037] respectively calculating the average power, sample entropy and standard deviation of the third training set as feature information, clustering the elements in the third training set according to the feature information, taking the class with more elements as a normal data set, and taking the class with fewer elements as a third abnormal data set;

[0038] According to the preset time window length and time window moving step, data is intercepted and sorted in sequence from the starting position of the third training set to obtain a process data set, which is input into the trained autoencoder, and elements with excessive deviations between the input signal and the output signal are extracted to construct a fourth abnormal data set;

[0039] A public opinion heat anomaly detection model is constructed based on a classification neural network, and the public opinion heat anomaly detection model is trained using the third anomaly data set and the fourth anomaly data set.

[0040] Preferably, the denoising method includes: decomposing the data set to be denoised using a variational mode decomposition method to obtain a number of sub-modes with different energy ratios, selecting three sub-modes with the highest energy ratios as effective sub-modes for element reconstruction, and obtaining a denoised data set.

[0041] Preferably, the denoising method includes: decomposing the data set to be denoised using an empirical wavelet transform method to obtain a number of frequency sub-layers with different frequencies, and selecting a number of frequency sub-layers lower than a preset frequency as elements of the data set after denoising.

[0042] Beneficial Effects

[0043] 1. Intelligent anomaly detection: The present invention uses an unsupervised clustering method to classify the denoised public opinion data into anomalies, avoiding the limitation of traditional anomaly detection methods that rely on fixed thresholds, so that the system can adaptively identify abnormal data in different scenarios, thereby improving the generalization ability and accuracy of the detection results.

[0044] 2. Efficient processing of large-scale data: The present invention uses an autoencoder model for data processing, which can efficiently screen and filter large-scale public opinion data. During the anomaly detection process, the system can quickly filter out normal data and only judge abnormal data, greatly improving the processing efficiency of public opinion heat anomaly detection.

[0045] 3. Accurate classification: The present invention constructs a public opinion heat anomaly classification model based on a classification neural network. After detecting abnormal data, it can further classify and identify it, providing decision makers with detailed abnormal type information to assist in crisis response and brand maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic flow chart of a method for detecting abnormal public opinion heat based on information denoising and clustering provided in a preferred embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the structure of a public opinion heat anomaly detection system based on information denoising and clustering provided in another preferred embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings. In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention.

[0049] Embodiment 1

[0050] like Figure 1 As shown, this embodiment provides a method for detecting abnormal public opinion heat based on information denoising and clustering, including the steps of:

[0051] S1. Obtain the public opinion dataset to be detected and obtain the target dataset after denoising.

[0052] The public opinion data in this invention refers to the collection of information such as public sentiment, opinions, discussions and attention related to a certain event, topic or subject on the Internet. These data usually come from social media, news platforms, forums, blogs and other online channels, and are presented in the form of time series, reflecting the public's attention and reaction intensity to a certain event in a specific time period. It includes but is not limited to:

[0053] 1. Public attention data: changes in attention to an event on social platforms or public opinion monitoring platforms. These data are usually expressed as the number of searches, readings, reposts, and comments on a topic or event within a specific time period.

[0054] 2. Sentiment analysis data: emotional attitudes in text information related to events, such as positive, negative or neutral emotional expressions. This data can be analyzed and extracted using natural language processing technology.

[0055] 3. Heat change data: The heat trend of a public opinion event refers to the change in the public's attention to the event over time. This data is usually presented in the form of time series data, which is manifested as the fluctuation of the heat value of the event at different times.

[0056] This type of data can be collected through public opinion monitoring platforms such as Zhiwei Events, Baidu Index, and Sina Public Opinion. Among these data, there are data that conform to the normal trend of public opinion heat, and there are also data with abnormal public opinion heat trend.

[0057] Public opinion data comes from various channels such as Internet social platforms and news media. The data types are complex and easily interfered by irrelevant information, such as advertisements, repeated information, invalid comments, etc. These noise information will affect the accuracy of public opinion heat analysis and the reliability of anomaly detection. Therefore, the present invention considers filtering out irrelevant, invalid or interfering data through denoising, thereby retaining the core public opinion information related to the target event and improving the accuracy of data analysis.

[0058] Those skilled in the art will know that, since the characteristics of public opinion data are different, the composition of their noise information is also different, resulting in different denoising methods producing different denoising effects. In some preferred embodiments, some analyses of different noise characteristics and denoising methods adapted thereto are provided, specifically including:

[0059] For public opinion data with obvious frequency components or frequency mixing phenomena, they often have long-term periodicity or long-term trends, and there are multiple different frequency components. For example, for the public opinion data of an event, the public opinion heat may fluctuate periodically over time. For such noise containing multiple mixed modes or uneven frequencies, you can consider using variational mode decomposition (VMD) to process it. The specific method is as follows:

[0060] The variational mode decomposition method is used to decompose the data set to be denoised, and several sub-modes with different energy ratios are obtained. When selecting effective sub-modes from the self-modal signal decomposed by VMD, the energy of the effective signal is usually greater than the energy of the noise. Therefore, sub-components with higher energy ratios are more likely to have effective signal components. This embodiment considers selecting three sub-modes with the highest energy ratio as effective sub-modes for element reconstruction to obtain the denoised data set.

[0061] For nonlinear, non-stationary and drastically changing public opinion data, the frequency components are complex and the time span is short. For example, in the case of a sudden event in a short period of time, the public opinion heat may rise or fall sharply, and the frequency components in the data will change rapidly. For such sudden changes in trends or drastic and irregular noise, the empirical wavelet transform (EWT) can be considered for processing. The specific method is as follows:

[0062] The empirical wavelet transform method is used to decompose the data set to be denoised, and several frequency sub-layers with different frequencies are obtained. Among the different frequency signals obtained by EWT, the low-frequency signal can well reflect the changing trend of the original data, while the high-frequency signal has little effect on the changing trend of the time series. Therefore, among the signals of different frequency sub-layers obtained by EWT, several frequency sub-layers below the preset frequency are selected as the elements of the denoised data set.

[0063] S2. Calculate the average power, sample entropy and standard deviation of the target data set as feature information respectively, cluster the elements in the target data set according to the feature information, and take a class with fewer elements as a first abnormal data set.

[0064] The technical considerations for selecting feature information in the present invention are described below:

[0065] 1. In the public opinion heat data, a higher average power may indicate that the public opinion heat fluctuates greatly during the time period, which means that the event has a higher level of public attention; while a lower average power means that the public opinion heat changes more slowly and the public attention is relatively low. Therefore, consider using average power to distinguish between high-intensity and low-intensity public opinion events to help determine whether there are significant public opinion fluctuations within a certain period of time. For time series data, average power reflects the intensity of the signal over time, and it can measure the overall fluctuations in the entire data set.

[0066] 2. In the data of public opinion heat, higher sample entropy means that the data is more irregular, and the change of public opinion heat is more random, reflecting the suddenness of the event or the diversity of public opinion; lower sample entropy means that the data changes are more regular and predictable, and the public opinion heat changes regularly over time. Therefore, consider using sample entropy to help identify complex patterns or abnormal changes in public opinion data, especially when identifying whether there are sudden public opinion events in the short term. The level of sample entropy can provide a strong basis for detection. It measures the complexity of the pattern in the time series and the similarity between sequences in the data.

[0067] 3. A higher standard deviation indicates that the data fluctuates greatly, and the public opinion heat changes significantly at different times, indicating that the public's attention to the event fluctuates more violently; a lower standard deviation indicates that the data is relatively concentrated, the public opinion heat changes less, and the public's attention to the event is relatively stable. Therefore, consider using the standard deviation to measure the fluctuation range of public opinion heat to help identify whether public opinion fluctuates violently or remains stable over a period of time. It reflects the degree of deviation between each sample value and the average value in the public opinion data.

[0068] It should be noted that the calculation formula of the above characteristic information belongs to the prior art in this field and is not the focus of the present invention, so it will not be described here in detail.

[0069] Clustering is an unsupervised machine learning method that aims to group elements in a data set according to their similarities or features, so that elements in the same group (cluster) are similar to each other, while elements in different groups are quite different. In the present invention, clustering is to analyze the public opinion data after denoising and divide the data into several categories according to its characteristic information. Specifically, the clustering algorithm groups the data according to the similarity measurement between the data (such as Euclidean distance or other measurement methods). In the present invention, it is considered that most of the public opinion data belongs to normal public opinion heat information, and only a small part of the data may show abnormal fluctuations. Therefore, it is considered to distinguish normal public opinion data from abnormal public opinion data through a clustering algorithm. It should be noted that the traditional public opinion heat anomaly detection method usually relies on a fixed threshold set by humans to judge abnormal situations, which is difficult to adapt to complex public opinion data in different scenarios. The present invention uses clustering, without relying on a fixed threshold, but adaptively divides the data into different categories according to its inherent characteristics, avoiding the problem of inaccurate threshold setting or inability to adapt to different scenarios. And clustering can screen out data that may be abnormal in advance, thereby reducing the amount of data to be processed later. This helps to improve the processing efficiency of the entire detection system, especially when facing large-scale public opinion data, it can quickly identify abnormal data that needs further analysis and reduce the consumption of computing resources. It should be noted that the present invention is not intended to limit the specific clustering algorithm. Those skilled in the art can reasonably choose any one of the existing clustering algorithms, such as K-means, hierarchical clustering, DBSCAN, Gaussian Mixture Model, etc., according to actual conditions.

[0070] S3. According to the preset time window length and time window moving step, data is intercepted and sorted in sequence from the starting position of the target data set to obtain a process data set, which is input into the trained autoencoder, and elements with excessive deviations between the input signal and the output signal are extracted to construct a second abnormal data set.

[0071] Since public opinion data is usually a dynamic process that changes over time, this step considers using time windows to decompose large-scale continuous time series data into several small segments. In each time window, the changes in public opinion data are more concentrated and refined, which facilitates local analysis of the trend of public opinion heat changes in the time period. This can better capture short-term public opinion fluctuations, especially for those public opinion data with short-term abnormal fluctuations. The use of time windows can help discover these local changes.

[0072] The autoencoder (AE) is a neural network model for unsupervised learning, which is mainly used to effectively encode and reconstruct input data. Its characteristic point is that the difference (deviation) between input and output is minimized through continuous optimization, so that the autoencoder can learn the main features of the data and filter out irrelevant details or noise. The technical considerations for selecting the autoencoder in the present invention are as follows:

[0073] First, the autoencoder is an unsupervised learning model that can learn and extract features from data without labels. This is very suitable for public opinion data scenarios, because public opinion data is usually not clearly labeled, especially in the absence of a large amount of abnormal data. The autoencoder can learn patterns from normal data and find abnormal data that deviates greatly from the normal pattern during the detection process.

[0074] Secondly, through the encoder part of the autoencoder, high-dimensional public opinion data can be compressed into low-dimensional feature representations to extract the main information in the data. This helps the model focus on the key features of public opinion data and avoid interference from irrelevant information. For example, some noise or subtle random fluctuations in public opinion data will be ignored by the autoencoder, making it easier for the model to capture the real changes in public opinion heat.

[0075] Again, when reconstructing data, the autoencoder can reconstruct the input data that matches the normal pattern very well, but it cannot reconstruct the abnormal data well, resulting in a large reconstruction error. The present invention uses this feature to judge abnormal situations by calculating the difference between the input data and the output data of the autoencoder. If the reconstruction error of a certain section of public opinion data exceeds a preset threshold, the data will be marked as abnormal data. This method can effectively detect those abnormal fluctuations in public opinion data and help identify potential crises or emergencies.

[0076] Finally, the characteristics of public opinion data may be different in different scenarios and time periods. The use of autoencoders can automatically adapt to different public opinion backgrounds and data changes by adaptively learning the patterns of normal public opinion data. As the model continues to receive new data for updated training, the system can dynamically adjust the anomaly detection criteria to better adapt to complex and changing public opinion environments.

[0077] In some preferred embodiments, a method for training an autoencoder is provided, which specifically includes:

[0078] S301, obtaining a historical public opinion data set, and obtaining a first training set after denoising;

[0079] S302, respectively calculating the average power, sample entropy and standard deviation of the first training set as feature information, clustering the elements in the first training set according to the feature information, and taking the class with a larger number of elements as a normal data set;

[0080] S303, according to a preset time window length and a time window moving step, sequentially intercept and sort data from the starting position of the normal data set to obtain a second training set;

[0081] S304: Minimizing the deviation between the input signal and the output signal of the autoencoder is used as the objective function, and completing the training of the autoencoder using the second training set.

[0082] In this embodiment, the purpose of using a normal data set obtained after clustering instead of the entire historical public opinion data set or other data to train the autoencoder is:

[0083] 1. As mentioned earlier, the main goal of the autoencoder is to minimize the error between the input data and the output data. During the training phase, if the input public opinion data is all normal data, the autoencoder will learn the main features and patterns of these data, so that it can reconstruct the normal data well. When the trained autoencoder processes normal public opinion data, since the model has learned the features of these data, it can produce an output that is very close to the input, with a small reconstruction error.

[0084] 2. Similarly, in practical applications, the key to anomaly detection lies in the error between the input data and the data reconstructed by the autoencoder. If the model has been trained with normal data, it will be able to reconstruct these normal data well. When abnormal data is input, since the patterns of these data are significantly different from the training data (i.e. normal data), the autoencoder cannot effectively reconstruct such data, resulting in a large reconstruction error. By setting an error threshold, when the error exceeds the threshold, the input data can be judged as abnormal.

[0085] 3. In the public opinion heat data, the vast majority of data usually falls within the normal range, and there are relatively few abnormal cases. Therefore, the number of normal data sets is large and easier to obtain. Using normal data for autoencoder training can ensure that the model obtains sufficient data to learn normal patterns, thereby improving the training effect and the generalization ability of the model.

[0086] S4. Input the first abnormal data set and the second abnormal data set into a trained public opinion heat anomaly detection model built based on a classification neural network to obtain anomaly classification results of the public opinion data to be detected.

[0087] The classification neural network (CNN) is a supervised learning artificial neural network model, which is mainly used to classify input data. It establishes a mapping relationship between input data and its corresponding category by learning a large amount of labeled training data, and then classifies and predicts unseen data. The present invention has no clear requirements for the specific type of reference classification neural network, and those skilled in the art can choose in the prior art as needed. The focus of the present invention is on its training process. Consider that the first abnormal data set is extracted from the target data set by a clustering algorithm, representing a small number of categories in the cluster. This type of abnormal data is usually manifested as discrete points or pattern anomalies that are significantly different from normal data. And it also contains feature information, which characterizes the heat type of abnormal public opinion. The second abnormal data set is extracted by an autoencoder model, representing data that the autoencoder cannot effectively reconstruct and has large deviations. This type of abnormal data may appear as sudden fluctuations or unpredictable anomalies that deviate from the normal pattern. Therefore, this step considers combining these two types of abnormal data for training the classification neural network. The purpose is, on the one hand, to enable the model to learn more different types of abnormal patterns, so as to better distinguish the diversified abnormal data in the detection process, improve the accuracy and precision of classification, and improve the classification accuracy and generalization ability of the model. On the other hand, considering that abnormal data usually accounts for a small part of the data set, the samples are scarce and uneven, the first abnormal data set and the second abnormal data set are used in combination, which helps to increase the sample size of abnormal data. By increasing the diversity and quantity of abnormal data, the model can more comprehensively cover possible abnormal scenarios and reduce the probability of missed reports and false positives. Further, if only a single type of abnormal data set (such as the first abnormal data set or the second abnormal data set) is used for training, the classification neural network may only learn a certain type of abnormal features, which makes the model easy to overfit to the training data in practical applications and cannot effectively deal with other types of abnormal data. By using the first and second abnormal data sets for training together, the model can learn more abnormal features, avoid the model from over-relying on a certain type of data features, and thus perform more robustly in practical applications and reduce the misjudgment of unseen data.

[0088] In some preferred embodiments, a training method for the public opinion heat anomaly detection model is provided, which specifically includes:

[0089] S401. Obtain a historical public opinion data set, and obtain a third training set after denoising.

[0090] S402, respectively calculating the average power, sample entropy and standard deviation of the third training set as feature information, clustering the elements in the third training set according to the feature information, taking the class with more elements as a normal data set, and taking the class with fewer elements as a third abnormal data set.

[0091] S403, according to the preset time window length and time window moving step, starting from the starting position of the third training set, intercepting and sorting the data in sequence, obtaining a process data set, inputting it into the trained autoencoder, extracting the elements where the deviation between the input signal and the output signal exceeds the limit to construct a fourth abnormal data set.

[0092] S404: construct a public opinion heat anomaly detection model based on a classification neural network, and use the third abnormal data set and the fourth abnormal data set to complete the training of the public opinion heat anomaly detection model.

[0093] Embodiment 2

[0094] like Figure 2 As shown, this embodiment provides a public opinion heat anomaly detection system based on information denoising and clustering, including:

[0095] Data collection module, used to obtain the public opinion data set to be tested;

[0096] A denoising module, connected to the data collection module, for denoising the public opinion data set to obtain a target data set;

[0097] A feature calculation module, connected to the denoising module, for respectively calculating the average power, sample entropy and standard deviation of the target data set as feature information;

[0098] A clustering module, connected to the feature calculation module, for clustering the target data set according to the feature information, and taking a class with a smaller number of elements after clustering as a first abnormal data set;

[0099] An autoencoder module, which has a built-in trained autoencoder and is connected to the denoising module, is used to intercept and sort data in sequence from the starting position of the target data set according to a preset time window length and a time window moving step, obtain a process data set and input it into the trained autoencoder, extract elements with excessive deviations between the input signal and the output signal to construct a second abnormal data set;

[0100] The classification detection module has a built-in trained public opinion heat anomaly detection model constructed based on a classification neural network, which is connected to the clustering module and the autoencoder module respectively, and is used to input the first anomaly data set and the second anomaly data set into the public opinion heat anomaly detection model to obtain the anomaly classification results of the public opinion data to be detected.

[0101] In some preferred embodiments, the training method of the autoencoder built into the autoencoder module includes:

[0102] Obtain the historical public opinion data set and obtain the first training set after denoising;

[0103] respectively calculating the average power, sample entropy and standard deviation of the first training set as feature information, clustering the elements in the first training set according to the feature information, and taking the class with a larger number of elements as a normal data set;

[0104] According to a preset time window length and a time window moving step length, data is intercepted and sorted in sequence starting from the starting position of the normal data set to obtain a second training set;

[0105] Minimizing the deviation between the input signal and the output signal of the autoencoder is used as the objective function, and the second training set is used to complete the training of the autoencoder.

[0106] In some preferred embodiments, the training method of the public opinion heat anomaly detection model built into the classification detection module includes:

[0107] Obtain the historical public opinion data set and obtain the third training set after denoising;

[0108] respectively calculating the average power, sample entropy and standard deviation of the third training set as feature information, clustering the elements in the third training set according to the feature information, taking the class with more elements as a normal data set, and taking the class with fewer elements as a third abnormal data set;

[0109] According to the preset time window length and time window moving step, data is intercepted and sorted in sequence from the starting position of the third training set to obtain a process data set, which is input into the trained autoencoder, and elements with excessive deviations between the input signal and the output signal are extracted to construct a fourth abnormal data set;

[0110] A public opinion heat anomaly detection model is constructed based on a classification neural network, and the public opinion heat anomaly detection model is trained using the third anomaly data set and the fourth anomaly data set.

[0111] In some preferred embodiments, it is characterized in that the denoising method includes: using variational mode decomposition to decompose the data set to be denoised, obtaining several sub-modes with different energy ratios, selecting three sub-modes with the highest energy ratios as valid sub-modes for element reconstruction, and obtaining the denoised data set.

[0112] In some preferred embodiments, the denoising method includes: using empirical wavelet transform to decompose the data set to be denoised to obtain several frequency sublayers with different frequencies, and selecting several frequency sublayers lower than a preset frequency as elements of the denoised data set.

[0113] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. The method for detecting abnormal public opinion heat based on information denoising and clustering is characterized by: Includes steps: Obtain the public opinion dataset to be detected, and obtain the target dataset after denoising; The average power, sample entropy and standard deviation of the target data set are respectively calculated as feature information, the elements in the target data set are clustered according to the feature information, and a class with a smaller number of elements is taken as a first abnormal data set; According to the preset time window length and time window moving step, data is intercepted and sorted in sequence from the starting position of the target data set to obtain a process data set, which is input into the trained autoencoder, and elements with excessive deviations between the input signal and the output signal are extracted to construct a second abnormal data set; The first abnormal data set and the second abnormal data set are input into a trained public opinion heat anomaly detection model built based on a classification neural network to obtain anomaly classification results of the public opinion data to be detected.

2. The method for detecting abnormal public opinion heat based on information denoising and clustering as claimed in claim 1, characterized in that: The training method of the autoencoder includes: Obtain the historical public opinion data set and obtain the first training set after denoising; respectively calculating the average power, sample entropy and standard deviation of the first training set as feature information, clustering the elements in the first training set according to the feature information, and taking the class with a larger number of elements as a normal data set; According to a preset time window length and a time window moving step length, data is intercepted and sorted in sequence starting from the starting position of the normal data set to obtain a second training set; Minimizing the deviation between the input signal and the output signal of the autoencoder is used as the objective function, and the second training set is used to complete the training of the autoencoder.

3. The method for detecting abnormal public opinion heat based on information denoising and clustering as claimed in claim 1, characterized in that: The training method of the public opinion heat anomaly detection model includes: Obtain the historical public opinion data set and obtain the third training set after denoising; respectively calculating the average power, sample entropy and standard deviation of the third training set as feature information, clustering the elements in the third training set according to the feature information, taking the class with more elements as a normal data set, and taking the class with fewer elements as a third abnormal data set; According to the preset time window length and time window moving step, data is intercepted and sorted in sequence from the starting position of the third training set to obtain a process data set, which is input into the trained autoencoder, and elements with excessive deviations between the input signal and the output signal are extracted to construct a fourth abnormal data set; A public opinion heat anomaly detection model is constructed based on a classification neural network, and the public opinion heat anomaly detection model is trained using the third anomaly data set and the fourth anomaly data set.

4. The method for detecting abnormal public opinion heat based on information denoising and clustering as claimed in claim 1, 2 or 3, characterized in that: The denoising method comprises: decomposing the data set to be denoised by using a variational mode decomposition method to obtain a plurality of sub-modes with different energy ratios, selecting three sub-modes with the highest energy ratios as effective sub-modes for element reconstruction, and obtaining the denoised data set.

5. The method for detecting abnormal public opinion heat based on information denoising and clustering as claimed in claim 1, 2 or 3, characterized in that: The denoising method comprises: decomposing the data set to be denoised by using an empirical wavelet transform method to obtain a number of frequency sublayers with different frequencies, and selecting a number of frequency sublayers lower than a preset frequency as elements of the data set after denoising.

6. The public opinion heat anomaly detection system based on information denoising and clustering is characterized by: include: Data collection module, used to obtain the public opinion data set to be tested; A denoising module, connected to the data collection module, for denoising the public opinion data set to obtain a target data set; A feature calculation module, connected to the denoising module, for respectively calculating the average power, sample entropy and standard deviation of the target data set as feature information; A clustering module, connected to the feature calculation module, for clustering the target data set according to the feature information, and taking a class with a smaller number of elements after clustering as a first abnormal data set; An autoencoder module, which has a built-in trained autoencoder and is connected to the denoising module, is used to intercept and sort data in sequence from the starting position of the target data set according to a preset time window length and a time window moving step, obtain a process data set and input it into the trained autoencoder, extract elements with excessive deviations between the input signal and the output signal to construct a second abnormal data set; The classification detection module has a built-in trained public opinion heat anomaly detection model constructed based on a classification neural network, which is connected to the clustering module and the autoencoder module respectively, and is used to input the first anomaly data set and the second anomaly data set into the public opinion heat anomaly detection model to obtain the anomaly classification results of the public opinion data to be detected.

7. The public opinion heat anomaly detection system based on information denoising and clustering as claimed in claim 6, characterized in that: The training method of the autoencoder built into the autoencoder module includes: Obtain the historical public opinion data set and obtain the first training set after denoising; respectively calculating the average power, sample entropy and standard deviation of the first training set as feature information, clustering the elements in the first training set according to the feature information, and taking the class with a larger number of elements as a normal data set; According to a preset time window length and a time window moving step length, data is intercepted and sorted in sequence starting from the starting position of the normal data set to obtain a second training set; Minimizing the deviation between the input signal and the output signal of the autoencoder is used as the objective function, and the second training set is used to complete the training of the autoencoder.

8. The public opinion heat anomaly detection system based on information denoising and clustering as claimed in claim 6, characterized in that: The training method of the public opinion heat anomaly detection model built into the classification detection module includes: Obtain the historical public opinion data set and obtain the third training set after denoising; respectively calculating the average power, sample entropy and standard deviation of the third training set as feature information, clustering the elements in the third training set according to the feature information, taking the class with more elements as a normal data set, and taking the class with fewer elements as a third abnormal data set; According to the preset time window length and time window moving step, data is intercepted and sorted in sequence from the starting position of the third training set to obtain a process data set, which is input into the trained autoencoder, and elements with excessive deviations between the input signal and the output signal are extracted to construct a fourth abnormal data set; A public opinion heat anomaly detection model is constructed based on a classification neural network, and the public opinion heat anomaly detection model is trained using the third anomaly data set and the fourth anomaly data set.

9. The public opinion heat anomaly detection system based on information denoising and clustering as described in claim 6, 7 or 8, characterized in that: The denoising method comprises: decomposing the data set to be denoised by using a variational mode decomposition method to obtain a plurality of sub-modes with different energy ratios, selecting three sub-modes with the highest energy ratios as effective sub-modes for element reconstruction, and obtaining the denoised data set.

10. The public opinion heat anomaly detection system based on information denoising and clustering as claimed in claim 6, 7 or 8, characterized in that: The denoising method comprises: decomposing the data set to be denoised by using an empirical wavelet transform method to obtain a number of frequency sublayers with different frequencies, and selecting a number of frequency sublayers lower than a preset frequency as elements of the data set after denoising.

Images