A method for predicting the public opinion trend of major emergencies based on multi-modal data

Abstract

The present invention discloses a method for predicting the public opinion trend of major emergencies based on multi-modal data, which relates to the field of network technology and includes the following steps: collecting the public opinion data of major emergencies and preprocessing it to obtain the preprocessed public opinion data; extracting features according to the types of the preprocessed public opinion data to obtain the sentiment vectors of various types of public opinion data; performing weighted fusion on the sentiment vectors of various types of public opinion data and the numerical communication features to obtain the comprehensive popularity of public opinion; using a TCN time series prediction model to predict the comprehensive popularity of public opinion, and completing the prediction of the public opinion trend of major emergencies based on multi-modal data. This method solves the problem in the prior art that it is difficult to take into account the use of multi-modal data and the processing of long sequence information during the prediction of public opinion trends, resulting in low prediction accuracy.

Description

Technical Field

[0001] The present invention relates to the field of network technology, and particularly to a method for predicting the public opinion trend of major emergencies based on multi-modal data. Background Art

[0002] With the accelerating development of informatization in the all-media era, the scale of Internet users has gradually expanded, the speed and scope of information dissemination have been continuously expanding, and the public opinion information of various themed events on social media presents various modal forms such as text, pictures, videos, and numerical values. Major emergencies usually refer to events that are extremely destructive and occur suddenly with uncertainty, affecting national security and social stability. The frequent occurrence of major emergencies has brought a huge threat to people's lives and property and social stability. The superposition effect of the risk society and the information society makes the evolution of public opinion on major emergencies highly uncertain and complex, and it is very likely to amplify, transfer, or create new risks.

[0003] The existing technologies mainly consider single communication characteristics or text characteristics when predicting public opinion trends, and there are the following problems: 1) The lack of comprehensive utilization of multi-modal data makes it impossible to measure the popularity of public opinion on major emergencies from a comprehensive perspective; 2) In text analysis, most traditional natural language processing libraries are oriented to English corpora and have poor applicability to Chinese; 3) In the analysis of public opinion on major emergencies, the utilization of video modal data is less; 4) The traditional ARIMA model for predicting public opinion trend time series has high requirements for the stationarity of time series data, and the deep learning model RNN has problems of gradient explosion or disappearance. Although the LSTM and GRU models alleviate the problems of RNN, when the amount of time series data is large, the information memory ability and calculation efficiency of LSTM for long sequences still need to be improved.

[0004] Facing the massive multi-modal public opinion information, constructing an efficient method to make full use of multi-modal public opinion data to perceive the public opinion trend of major emergencies has become an urgent problem to be solved. Summary of the Invention

[0005] Aiming at the above deficiencies in the existing technologies, a method for predicting the public opinion trend of major emergencies based on multi-modal data provided by the present invention solves the problem that it is difficult to balance the use of multi-modal data and the processing of long-sequence information in the existing technologies during public opinion trend prediction, resulting in low prediction accuracy.

[0006] To achieve the above invention purpose, the technical solution adopted by the present invention is as follows:

[0007] Provide a method for predicting the public opinion trend of major emergencies based on multi-modal data, which includes the following steps:

[0008] S1. Collect the public opinion data of major emergencies and preprocess it to obtain the preprocessed public opinion data. The public opinion data includes text, pictures, videos, and numerical communication features. The numerical communication features include reposts, comments, and likes.

[0009] S2. Extract features according to the types of the preprocessed public opinion data to obtain the sentiment vectors of various types of public opinion data.

[0010] S3. Perform weighted fusion on the sentiment vectors of various types of public opinion data and the numerical communication features to obtain the comprehensive popularity of public opinion.

[0011] S4. Use the TCN time series prediction model to predict the comprehensive popularity of public opinion, and complete the prediction of the public opinion trend of major emergencies based on multi-modal data.

[0012] Further, the specific method for preprocessing the public opinion data in step S1 includes the following sub-steps:

[0013] S1-1. Convert the public opinion video data in different formats into files in the unified MP4 format, and remove invalid and error videos. Split the pictures and audio of the public opinion video data, and convert the split audio files into text format to form the public opinion text data corresponding to the audio. Extract the key frames of the public opinion video according to the inter-frame difference method, and convert them into single-frame images in JPG format.

[0014] S1-2. Through the jieba natural language processing technology, perform word segmentation, remove stop words, and remove punctuation marks on the public opinion text data.

[0015] S1-3. Obtain the missing values of the communication feature data, and fill them with the mean value of the sum of the two items before and after the missing value to obtain the preprocessed public opinion data.

[0016] Further, the specific method of step S2 includes the following sub-steps:

[0017] S2-1. Perform sentiment analysis on the public opinion text data through the SnowNLP library of natural language processing tools to obtain the text sentiment vector.

[0018] S2-2. Extract features from the public opinion pictures and single-frame images through the deep learning model ResNeXt to obtain the image features.

[0019] S2-3. Perform image sentiment classification on the image features through the softmax classifier to obtain the image sentiment vector.

[0020] Further, the specific method of step S3 includes the following sub-steps:

[0021] S3-1. According to the formula:

[0022]

[0023] Normalize each modal data; where u st represents the value of the t-th modality of the s-th sample data after normalization; u st represents the value of the t-th modality of the s-th sample data; n is the total number of sample data; the modal data includes text sentiment vectors, image sentiment vectors, and propagation features;

[0024] S3-2. According to the formula:

[0025]

[0026] Obtain the proportion h of the t-th modality st ;

[0027] S3-3. According to the formula:

[0028]

[0029] Obtain the weight W″ of the t-th modality j ; where ln(·) represents the natural logarithm with the constant e as the base; m is the total number of modalities, m = 3;

[0030] S3-4. According to the formula:

[0031] U = W1” * P + W2” * E_pic + W3” * Pr

[0032] Obtain the comprehensive popularity U of public opinion; where P is the text sentiment vector; E_pic is the image sentiment vector; Pr is the sum of reposts, comments, and likes in the numerical propagation features.

[0033] The beneficial effects of the present invention are as follows: This method performs audio-visual splitting operations on video data, converts audio into text format, extracts key frames from the video to convert it into images, respectively obtains the sentiment vectors of various types of public opinion data, and uses the method of weighted fusion of the sentiment vectors of various types of public opinion data with numerical propagation features to utilize the sentiment vectors of various types of public opinion data, obtains the comprehensive popularity of public opinion, and then uses the TCN time series prediction model to predict the trend of public opinion. The present invention not only comprehensively considers various modal data affecting the trend of public opinion, but also solves the performance problems existing in the traditional public opinion prediction process. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow diagram of this method;

[0035] Figure 2 is a flow diagram of the public opinion data collection process in the embodiment;

[0036] Figure 3 Schematic diagram of the multi-modal data preprocessing process in the embodiment;

[0037] Figure 4 Schematic diagram of the text sentiment analysis process in the embodiment;

[0038] Figure 5 Schematic diagram of the video sentiment analysis process in the embodiment. Detailed implementation manners

[0039] The following describes the detailed implementation manners of the present invention to facilitate those skilled in the art of the present technology to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the detailed implementation manners. For those of ordinary skill in the art of the present technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are within the scope of protection.

[0040] As Figure 1 shown, the method for predicting the public opinion trend of major emergencies based on multi-modal data includes the following steps:

[0041] S1. Collect the public opinion data of major emergencies and preprocess it to obtain the preprocessed public opinion data; the public opinion data includes text, pictures, videos, and numerical communication features, and the numerical communication features include reposting, commenting, and liking;

[0042] S2. Extract features according to the types of the preprocessed public opinion data to obtain the sentiment vectors of each type of public opinion data;

[0043] S3. Perform weighted fusion on the sentiment vectors of each type of public opinion data and the numerical communication features to obtain the comprehensive popularity of public opinion;

[0044] S4. Use the TCN time series prediction model to predict the comprehensive popularity of public opinion to complete the prediction of the public opinion trend of major emergencies based on multi-modal data.

[0045] In an embodiment of the present invention, the process of collecting public opinion data is as Figure 2 shown, specifically as follows:

[0046] S11. Define the storage structures of different modal data, including a list for storing the text published by social network users on the social media platform regarding major emergencies, and define a dictionary to store the numerical public opinion communication features. The addresses of picture and video files can be directly set as local folders;

[0047] S12. Select the social media platforms for the source of public opinion data, which can be Weibo, WeChat, and forums. Obtain the URL addresses of relevant pages according to the search box and advanced options, and use them to set the requests and request parameters of the crawler program;

[0048] S13. The fully automatic crawler program continuously determines whether the URL address table is empty. If it is empty, the crawler task ends; otherwise, it obtains the URL page information, parses the page content, obtains the content of the required defined fields, and stores them;

[0049] S14. The same URL address may contain multiple pages. To prevent missing pages, it is necessary to repeatedly check whether there is a next page until all page data has been parsed and stored.

[0050] After obtaining various types of public opinion data, it needs to be preprocessed before subsequent operations. For example, Figure 3 As shown, the specific process of multi-modal data preprocessing is as follows:

[0051] S1-1. Use the jieba natural language processing technology to perform word segmentation, stop word removal, and punctuation removal operations on the text data.

[0052] S1-2. Uniformly convert video data in different formats into the same MP4 file format and remove invalid and incorrect videos. Videos often play a supplementary role to the text. The voice and picture information in the videos will affect the emotions of viewers, thereby affecting the popularity of event public opinion. For the convenience of analysis, the present invention can use the open-source video processing software 123APPS to split the audio and video of the video file, and convert the split audio data into a text format file. Then, extract the key frames of the video based on the inter-frame difference method and convert them into single-frame images in JPG format. The inter-frame difference method performs a difference operation on two adjacent frame images when the change between them is relatively obvious, obtains the absolute value of the pixel value difference of the image object position, and takes the value greater than the threshold Th as the key frame. The difference image is:

[0053]

[0054] where x and y respectively represent the horizontal and vertical pixel sizes, and f k represents the k-th frame image, and Th can take a value of 10.

[0055] S1-3. For numerical type propagation feature time series data, the crawled data may have missing values. When there are missing values, fill them with the mean of the sum of the two items before and after the missing value.

[0056] For example, Figure 4As shown in the figure, when performing text sentiment analysis, the SnowNLP library, a natural language processing tool, can be used to perform sentiment analysis on text data and calculate the text sentiment vector. The SnowNLP library can implement operations such as text sentiment analysis, text classification, and feature extraction through the Naive Bayes principle, and is well applicable to the task analysis of Chinese corpora. The calculation principle of SnowNLP in the text sentiment analysis task is as follows:

[0057] Suppose that during the occurrence of major emergencies, the public sentiment is manifested in two categories, E1 and E2, and W1, W2,..., W r are r mutually independent feature words in the text data. Then the calculation process of the sentiment index of category E1 is as follows:

[0058]

[0059] According to the Naive Bayes total probability formula, we know that:

[0060] P(W1, W2,..., W r ) = P(W1, W2,..., W r |E1)*P(E1) + P(W1, W2,..., W r |E2)*P(E2)

[0061] When the SnowNLP library performs text sentiment analysis, the specific expansion formula is:

[0062]

[0063] It can be further simplified to:

[0064]

[0065] As Figure 5 shown, for the image data of major emergencies, the deep learning model ResNeXt can be used to extract image features. ResNeXt combines the ResNet and Inception structures, uses a group of convolutional kernels with different receptive field sizes, and realizes the extraction of image features through the split-transform-merge strategy. The strategy transformation process is as follows:

[0066]

[0067] Among them, y_pic represents the output, that is, the image feature; x_pic represents the input, and Γ() represents any function that can project x_pic into a low-dimensional subspace and perform transformation. C represents the number of groups for grouped convolution.

[0068] Then, the image features are input into a softmax classifier for image sentiment classification. The softmax normalizes the probability of the image feature vector, maps the probability value of the sentiment analysis output result to between 0 and 1, and calculates the image sentiment vector \(E_{pic}\) as follows:

[0069]

[0070] where \(q\) represents the number of sentiment categories in image sentiment classification, and \(W'\) represents the scientific weight parameter between hidden layers in the softmax classifier.

[0071] It can be seen that this method includes three modalities of data: text, image, and numerical propagation features. The three modalities of numerical values are combined into a set of comprehensive public opinion popularity data. Since the contribution degrees of different modality data to the popularity trend of major emergencies are different, they cannot be simply added directly. Therefore, this method calculates the weight ratio of each modality data to obtain the comprehensive popularity of major emergency public opinion. The specific process is as follows:

[0072] S3-1. According to the formula:

[0073]

[0074] Perform standardization processing on each modality data; where represents the value of the \(t\)-th modality of the \(s\)-th sample data after standardization; \(u\) st represents the value of the \(t\)-th modality of the \(s\)-th sample data; \(n\) is the total number of sample data; the modality data includes text sentiment vector, image sentiment vector, and propagation features;

[0075] S3-2. According to the formula:

[0076]

[0077] Obtain the ratio \(h\) of the \(t\)-th modality st ;

[0078] S3-3. According to the formula:

[0079]

[0080] Obtain the weight \(W''\) of the \(t\)-th modality j ; where \(\ln(\cdot)\) represents the natural logarithm with the constant \(e\) as the base; \(m\) is the total number of modalities, \(m = 3\);

[0081] S3-4. According to the formula:

[0082] \(U = W1''*P + W2''*E_{pic} + W3''*Pr\)

[0083] Obtain the comprehensive popularity U of public opinion; where P is the text sentiment vector; E_pic is the image sentiment vector; Pr is the cumulative sum of reposts, comments, and likes in the numerical communication features.

[0084] In step S4, the TCN time series prediction model includes dilated convolution, causal convolution, and residual modules. Introducing the residual module can improve the problem of gradient explosion and disappearance existing in the current time series model. By changing the size of the dilation rate d in the TCN time series prediction model, the convolutional kernel can perceive a longer time range, thus better capturing the long-term dependence relationship of time series data. For the sequence Y=(y1, y2,..., y t ) at the given time step t, with the filter G=(g1, g2,..., g l ), the calculation principle of dilated convolution is:

[0085]

[0086] Causal convolution strictly adheres to time constraints, ensuring that the output at a certain moment is only related to the input at the current moment and before, and has nothing to do with the input at future moments. It not only effectively prevents the problem of information leakage, but also ensures the sequentiality of the model. The calculation principle of causal convolution is:

[0087]

[0088] In summary, the present invention performs operations of separating audio and video on video data, converts the audio into text format, extracts key frames from the video to convert it into images, respectively obtains the sentiment vectors of various types of public opinion data, and uses the method of weighted fusion of the sentiment vectors of various types of public opinion data with numerical communication features to utilize the sentiment vectors of various types of public opinion data, obtains the comprehensive popularity of public opinion, and then uses the TCN time series prediction model to predict the trend of public opinion. The present invention not only comprehensively considers various modal data affecting the trend of public opinion, but also solves the performance problems existing in the traditional public opinion prediction process.

Claims

1. A method for predicting the public opinion trend of major emergencies based on multi-modal data, characterized in that, It includes the following steps: S1. Collect the public opinion data of major emergencies and preprocess it to obtain the preprocessed public opinion data; the public opinion data includes text, pictures, videos and numerical communication features, and the numerical communication features include forwarding, commenting and liking; S2. Extract features according to the types of the preprocessed public opinion data to obtain the sentiment vectors of each type of public opinion data; S3. Perform weighted fusion on the sentiment vectors of each type of public opinion data and the numerical communication features to obtain the comprehensive popularity of public opinion; S4. Use the TCN time series prediction model to predict the comprehensive popularity of public opinion to complete the prediction of the public opinion trend of major emergencies based on multi-modal data; The specific method for preprocessing the public opinion data in step S1 includes the following sub-steps: S1-1. Convert the public opinion video data in different formats into files in the unified MP4 format, and remove invalid and incorrect videos; split the pictures and audio of the public opinion video data, and convert the split audio files into text format to form the public opinion text data corresponding to the audio; extract the key frames of the public opinion video according to the inter-frame difference method and convert them into single-frame images in JPG format; S1-2. Through the jieba natural language processing technology, perform word segmentation, stop word removal and punctuation removal on the public opinion text data; S1-3. Obtain the missing values of the communication feature data, and fill them with the mean of the sum of the two items before and after the missing values to obtain the preprocessed public opinion data; The specific method of step S2 includes the following sub-steps: S2-1. Perform sentiment analysis on the public opinion text data through the SnowNLP library of natural language processing tools to obtain the text sentiment vector; S2-2. Extract features from the public opinion pictures and single-frame images through the deep learning model ResNeXt to obtain the image features; S2-3. Perform image sentiment classification on the image features through a softmax classifier to obtain the image sentiment vector; The specific method of step S3 includes the following sub-steps: S3-1. According to the formula: Perform standardization processing on each modal data; where represents the value of the t-th modality of the s-th sample data after normalization; represents the value of the t-th modality of the s-th sample data; n is the total number of sample data; the modality data includes text sentiment vectors, image sentiment vectors, and propagation features; S3-2. According to the formula: Obtain the proportion of the t-th modality ; S3-3. According to the formula: Obtain the weight of the t-th modality ; where represents the natural logarithm with base e; m is the total number of modalities, m = 3; S3-4. According to the formula: Obtain the comprehensive popularity of public opinion ; where is the text sentiment vector; is the image sentiment vector; is the cumulative sum of reposts, comments, and likes in the numerical communication features.

Images