A short video rumor detection method based on multi-modal consistency

Abstract

A short video rumor detection method based on multi-modal consistency belongs to the field of natural language processing. In view of the problem that the theme is distorted and the key frame is tampered in the rumor video, a short video rumor classification model TKCM based on theme and key frame is designed. TKCM uses an aggregation network to obtain the theme features of the video, uses an attention network to obtain the key frame features of the video, inputs the two information into a modal adjustment mechanism to adjust the importance of the three modalities, splices the features for short video rumor detection. In view of the inconsistency of rumor videos, a method of learning inconsistent information is proposed, and the method of integrating theme, key frame and inconsistency information ICIM is used for rumor detection. The experimental results show that the F1 value on the short video rumor data set is 4% to 7% higher than that of the commonly used video classification model. Compared with the model before integrating consistency information, the F1 value is improved by 2.3%.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of natural language processing, and particularly relates to a short video rumor detection method based on multi-modal consistency. BACKGROUND

[0002] With the rapid development of short video platforms in recent years, short videos are becoming more and more popular, and the audience is also growing. According to the report published by the data analysis network QuestMobile platform in January 2021: in June 2020, the monthly active user number of Douyin reached 51336 million, and the monthly active user number of Kuaishou reached 42975 million; the monthly average use time of Douyin reached 1569.5 minutes, and that of Kuaishou was 1162.6 minutes. Everyone can express their opinions and upload videos on short video platforms at any time and anywhere, which makes short video platforms become the position of public opinion.

[0003] At the same time, some people also create rumor videos by fabricating content, cutting videos and combining content to achieve some bad purposes, such as attracting attention, guiding and spreading, etc. This makes it difficult for people to distinguish between rumors and real situations, and makes some people unintentionally become spreaders of rumors. Statistics from the School of Journalism and Communication of Beijing Normal University show that nearly one-fifth of online rumors now have short videos. Due to the wide audience, fast spread and strong incitement, online rumors can easily cause public anxiety and panic, leading to various group events and seriously threatening social security. In the era of rapid development of short videos, various short video platforms have become a breeding ground for rumor spread. Because videos are more deceptive than pure text or graphics, and their sources and authenticity are difficult to verify, video online rumors are more harmful. Therefore, accurately and timely detecting short video rumors has important practical significance for maintaining the stability of public opinion on social media platforms, protecting the country's network discourse power and ensuring the smooth development of social order.

[0004] In existing works, the rumor detection of text information usually uses several coding technologies commonly used in natural language processing, including RNN, CNN, adversarial networks, graph structure and automatic coding, etc.

[0005] Rumor data often contains not only text information, but also visual information such as pictures and videos. Traditional statistical-based methods use additional image quantity, image popularity and image type to detect rumors. However, these statistical-based features cannot explain the semantic features of the image. With the development of deep learning, scholars propose to use neural networks to extract features from images and use the extracted features for rumor detection. However, existing image falsification techniques can change the semantic information of the image, and the convolutional network model can only extract pixel-level image information and cannot determine whether the image is falsified.

[0006] Research on text and visual information shows that text and picture information prove to be effective in rumor detection tasks. Effectively combining text and visual information for rumor detection can also improve the effect of rumor detection.

[0007] Currently, there are two main problems in short video rumor detection:

[0008] (1) Short video rumor multi-modal information fusion. The classic multi-modal fusion more considers multiple modalities starting from the bottom layer features, and gradually aggregates into cross-modal features through deep learning, but the rumor video contains means such as distortion of the theme and tampering of the key frame, therefore, the multi-modal fusion of short video rumors needs to learn rumor features from the theme and key frame.

[0009] (2) Inconsistency between modalities. The process of artificially creating a rumor video is usually through splicing, editing and fabricating the video's picture, title and audio. This leads to inconsistencies in the content, scene and time described between the three modalities. The classic multi-modal video classification more extracts features of multiple modalities, and fuses them into the overall features of the video for classification, and cannot learn the inconsistent information between modalities of the rumor video. Therefore, the traditional multi-modal video classification method cannot be well applied to rumor video detection, and short video rumor detection needs to learn the inconsistent information between modalities. SUMMARY

[0010] The purpose of the present application is to provide a short video rumor detection method based on multi-modal consistency, which is used for short video rumor-busting. The generation means of short video rumors is to distort the theme and tamper with the key frame, and the artificially synthesized rumor video will have inconsistent information between modalities. The first step of the present application is to fuse multi-modal rumor features from the theme and key frame. The second step is to guide the fusion between multi-modalities by learning the inconsistent information between modalities.

[0011] The present application comprises the following steps:

[0012] 1) Establish a short video rumor classification model TKCM based on theme and key frame;

[0013] 2) using a pre-trained modal feature extractor to extract single modal features, using an aggregation network to obtain the topic features of the video, using an attention network to obtain the key frame features of the video, and inputting the three modalities into the short video rumor classification model TKCM;

[0014] 3) inputting the topic features and key frame features of each modality into the modal adjustment mechanism adjustment network to adjust the importance attention of the three modalities, splicing the topic features and key frame features, and fusing multi-modal information;

[0015] 4) constructing a consistency dataset to train an inter-modal consistency detection model to obtain consistency information;

[0016] 5) fusing the topic features, key frame features and consistency information to serve as a short video rumor detection means.

[0017] In step 1), the short video rumor classification model TKCM includes a rumor part and a non-rumor part, rumor videos are obtained from a short video platform, and rumor parts existing in the rumor videos are manually intercepted to establish a short video rumor dataset; the non-rumor part is obtained from videos on the short video platform about knowledge popularization and having a large number of followers.

[0018] In step 2), the three modalities include a text modality, a visual modality and an audio modality; the specific steps of using a pre-trained modal feature extractor to extract single modal features can be: obtaining a single frame feature vector of each modality through a pre-trained model, aggregating single frames of each modality to form overall topic features; at the same time, using an attention method to screen to obtain key frames; using two aggregation networks, NeXtVLAD network and Attention Cluster network, to obtain topic features and key frame features; the key frame feature extraction uses an attention mechanism network Attention Cluster, which can assign more weights to relatively important frames, and is mainly used to select key frames in the video and obtain key frame representations under the three modalities; the topic feature extraction uses a NeXtVLAD network to select a cluster center, and then encodes vision, text and audio to obtain topic feature representations under each modality.

[0019] In step 3), the importance of the three modalities is different, before splicing the three modalities, the feature vectors of the three modalities are each multiplied by a hyperparameter less than or equal to 1, the values of the feature vectors of each modality are appropriately reduced or relatively enlarged, and the dominant role of the three modalities in the final classifier is changed in advance.

[0020] In step 4), the consistent data set is obtained by combining different modalities in different videos, and for inconsistent data, each modality in different videos in the data set is randomly combined; for the data consistent between modalities, the corresponding modalities in the non-rumor data are combined.

[0021] In step 5), the text-visual and text-audio consistency models are embedded into the overall rumor classification framework, and the obtained consistent features are respectively passed through a fully connected layer, and then spliced with the theme key frame features of each modality obtained in TKCM. Finally, the linear layer is mapped to two-dimensional space, and the rumor detection binary task is performed.

[0022] The application establishes two models, a short video rumor detection model based on theme and key frame, and a modal consistency detection model, and the multi-modal fusion features obtained by the two models are used together for rumor detection.

[0023] Compared with the prior art, the application has the following advantages and effects:

[0024] (1) The application enhances the video rumor detection effect by fusing the information of two modalities, and each of the three modalities has its unique information difference and can improve the video rumor classification effect through fusion.

[0025] (2) There are means such as distortion of the theme and tampering of the key frame in the rumor video, and the multi-modal fusion of the application learns the rumor features from the theme and key frame of the short video rumor.

[0026] (3) For modal inconsistent rumor videos, the application proposes to use BERT to learn the information of whether the video text-visual and text-audio modalities are consistent, and then embed the trained model into the overall framework, and fuse the learned consistency information with the rumor feature information to assist the model in detecting rumors.

[0027] (4) The F1 value of the application on the short video rumor data set is improved by 4% to 7% compared with the commonly used video classification model. Compared with the model before the consistency information is integrated, the F1 value is improved by 2.3%. DETAILED DESCRIPTION

[0028] Figure 1 It is a structure diagram of a short video rumor detection model based on theme and key frame.

[0029] Figure 2 It is a detailed structure of the NeXtVLAD network used by the theme feature extraction module.

[0030] Figure 3 It is an Attention Cluster model structure used by the key frame feature extraction module.

[0031] Figure 4 A text-video consistency classification model is provided.

[0032] Figure 5 A whole structure diagram of short video rumor detection based on multi-modal consistency provided by the present application is shown. DETAILED DESCRIPTION

[0033] The following embodiments will further illustrate the present application in conjunction with the accompanying drawings. The embodiments of the present application establish two models, a short video rumor detection model based on theme and key frame, and an inter-modal consistency detection model, and the multi-modal fusion features obtained by the two models are used together for rumor detection.

[0034] 1. Short video rumor detection model based on theme and key frame

[0035] First, the single-modal features are extracted using the pre-trained modal feature extractors, then the theme features of the video are obtained using the aggregation network, the key frame features of the video are obtained using the attention network, finally the information of the two is input into the modal adjustment mechanism adjustment network to adjust the importance attention of the three modalities, and the features are spliced for short video rumor detection. The specific model structure is shown in Figure 1 .

[0036] The video contains three modalities, namely text modality, visual modality and audio modality. The pre-trained feature extractors under each modality are used for feature extraction.

[0037] The text comes from the title of the video. After data preprocessing, the output of the last layer is obtained by the pre-trained model BERT, and the text feature vector corresponding to the context H t ={h t1 , h t2 …h tn} is obtained, where H t ∈R tn*768 , and tn is the token length of the text.

[0038] For audio, a pre-trained model is adopted to obtain audio feature vectors. The audio is resampled to 16 kHz mono audio, and a 25 ms Hann time window is used to perform a short-time Fourier transform on the audio with a 10 ms frame shift to obtain a spectrogram, which is then mapped to a sixty-four order mel filter bank to calculate a mel spectrum, and log(mel-spectrum + 0.01) is calculated to obtain a stable mel spectrum. The bias value of 0.01 added is to avoid taking the logarithm of 0. These features are framed with a time length of 0.96 s, and there is no overlap between frames, each frame contains 64 mel frequency bands, and the time length is 10 ms (i.e. a total of 96 frames). The model output data format is [nums_frames, 128], where nums_frames is the frame length, and nums_frames represents the audio time length / 0.96. Through these operations, the feature encoding representation H of the audio is obtained a = {h a1 , h a2 … h an}, where H a ∈ R an*768 , and an is the frame length of the audio.

[0039] After frame extraction, a plurality of images are obtained, and a video pre-trained model is used to extract features from the extracted images. In order to maintain synchronization with the audio, a frame is extracted every 0.96 seconds, and the frame is saved; the extracted frame in the video is sequentially input into the pre-trained ResNet-50 model to obtain the corresponding encoding representation of each frame. The feature vector representation of the sequence frame in a video is H p = {h p1 , h p2 … h pn}, where H p ∈ R pn*768 , and pn is the number of frames extracted from each video.

[0040] The subject feature extraction adopts a NeXtVLAD network to select a clustering center, and encodes vision, text and audio to obtain subject characteristics in each modality. The complete structure of NeXtVLAD is shown in Figure 2 , and the simplified structure of the subject feature extraction module is shown in Figure 1 . The feature information of each modality is decomposed into a feature vector at a relatively low latitude using an attention method before aggregation and encoding. The input of the NeXtVLAD network is defined as M frames, each frame is N-dimensional, and G is the grouping size. First, the input vector x is passed through a fully connected layer to upgrade the N-dimensional features of the input vector to λN-dimensional features, denoted as where λ is usually set to 2, i.e. (M, N) is converted to (M, 2*N). Then, through three branches, the first branch converts to (M, λN) will soon be transformed into Then Each with k dimensions Cluster center c k The difference, i.e., in the formula below The second branch road will The input is fed into a fully connected layer and then passes through the activation function sofimax, as shown in the formula below. This indicates that the feature of the g-th group in the i-th frame is at the k-th cluster center c. k The proportion. The third branch road will The input is fed into a fully connected layer and then passes through a sigmoid function, as shown in the formula below. Let represent the weight of the i-th frame in the g-th group, which is equivalent to the attention size; finally, each frame is divided into g lower-dimensional feature vector spaces, and each video frame can be represented as:

[0041]

[0042] The features of multiple frames are aggregated and summed in both the frame and group dimensions to obtain the aggregated feature Y. j k, i.e., the following formula:

[0043]

[0044] The obtained aggregated features are then dimensionality reduced. Y is processed through a fully connected layer. jk Dimensionality reduction is performed to obtain the output of NeXtVLAD.

[0045] Keyframe feature extraction employs an Attention Cluster network. It assigns more weights to relatively important frames, primarily used to select keyframes in the video and obtain keyframe representations across three modalities. The model structure of Attention Cluster is as follows: Figure 3 As shown, the entire model can be divided into two parts: local feature integration and obtaining global keyframe feature vectors.

[0046] Local feature integration yields attention-based global features. The attention output is essentially equivalent to a weighted average, v = aX. v is the global feature derived from the attention unit, and a is the weight vector composed of two fully connected layers, a = softmax(w2 tanh(W1X)). T +b1)+b2). ​​In actual implementation, v is generated by shift operations. where, a and b are learnable scalars. By adding an independent learnable linear transformation to the output of each attention unit and performing L2 regularization, each attention unit tends to learn different distribution features, which makes the attention cluster better learn data from different distributions and improves the learning representation of the entire network. Since the attention cluster is used, the outputs of each attention unit are combined to obtain multiple global features g = [v1, v2, …, v N ] where N is the number of clustering clusters.

[0047] Global key frame feature vectors are obtained. After splicing multiple global features, a fully connected layer is used to reduce the dimension, and the obtained vector features are used as the output of the model. By using the Attention Cluster network to aggregate each modality feature vector extracted in the above steps, the key frame feature vectors under the text, video and vision are obtained, respectively denoted as H t-AC ∈R 1*768 , H a-AC ∈R 1 *768 , H p-AC ∈R 1*768 .

[0048] The theme feature vectors and key frame feature vectors under each modality extracted by the two aggregation networks are spliced respectively, and the formula is as follows, to obtain the video-level feature vector corresponding to each modality. Denoted as:

[0049]

[0050]

[0051]

[0052] Each modality feature vector is multiplied by a hyperparameter in the range of (0, 1], and the values of the feature vectors of each modality are appropriately reduced or relatively enlarged, to change the dominant role of the three modalities in the final classifier in advance. The invention refers to this as a modality adjustment mechanism. The three modalities are converted according to the following formula before splicing, where a, b and g are artificial set hyperparameters.

[0053]

[0054]

[0055]

[0056] The adjusted multi-modal features are respectively subjected to linear transformation by a fully connected layer to obtain H t , H a , H p, and then the H v , and finally linearly mapped to two-dimensional space for binary classification task, as follows:

[0057] H t = H t-out W t

[0058] H a = H a-out W a

[0059] H p = H p-out W p

[0060] H v = [H t , H a , H p ]

[0061] logits = W T H v + b

[0062] p(y|logits) = softmax(logits)

[0063] Where W and b are the weights and bias of the final linear layer, respectively. To train the model, the cross-entropy loss function is minimized.

[0064] 2. Short video rumor detection model based on inter-modal consistency

[0065] Artificial rumor videos often cause inconsistencies in the content, scene, and time described between the three modalities. For example, the title is "elephants can fly", but the video shows other animals flying, which is called inter-modal inconsistency. This part is used to detect inter-modal inconsistency.

[0066] First, separate the video title, visual sequence and audio sequence of the rumor video dataset to obtain the dataset of text, vision and audio modalities in each video. The consistency dataset is obtained by combining different modalities in different videos.

[0067] For data with negative labels, i.e. inconsistent datasets between modalities, each modality in different videos in the dataset is randomly combined. For example, the visual modality of video A and the text modality of video B are combined into a piece of data. The rumor dataset has 584 pieces, and the non-rumor dataset has 625 pieces. Through random splicing, 1209 inconsistent data can be obtained.

[0068] For the data labeled as positive, that is, the inter-modal is consistent, the modal in the non-rumor data is consistent by default. Therefore, the corresponding modal in the non-rumor data is combined, for example, in the rumor data set, the visual modal of video A and the text modal of video A are combined into one piece of data. The non-rumor data set has 625 pieces, so 625 pieces of consistency data can be obtained.

[0069] After the data is constructed, a consistency classification model is designed, and the model structure is as shown in Figure 4 illustrated by text-video consistency data. Since BERT and its various variants have been proven to be able to better process text, visual and audio data. Moreover, the effectiveness of BERT in encoding text and the text-video consistency data set and the text-audio consistency data set all require deep understanding of text, so the present application selects BERT to learn consistency information.

[0070] For the text classification task, a [CLS] symbol is inserted at the input start position of BERT, and the output vector corresponding to the symbol is used as the semantic representation of the entire text for text classification. [CLS] can be understood as: compared with other words or words in the text, the introduction of this additional symbol represents no obvious semantic information, which can more fairly integrate the semantic information of each word or word in the text as the feature representation of the entire text. The [SEP] symbol is used as a separator and is placed in the middle of two sentences. In the present application, this idea is adopted and modified to adapt to the input of multi-modal. Taking the text-video consistency classification model as an example, by splicing "[CLS] text representation [SEP] visual representation" as the input of BERT, [SEP] is used to separate the feature representation between different modalities.

[0071] In the consistency model, the output corresponding to [CLS] is used as the input for the final classification. In this way, the consistency information learned by the model can finally be saved in the feature vector of the [CLS] character, so that the subsequent embedding model uses [CLS] to extract the learned consistency information.

[0072] By training BERT on the text-video consistency data set and the text-audio consistency data set respectively, text-video BERT and text-audio BERT containing consistency information are obtained, represented by bert-p and bert-a respectively.

[0073] 3. Fusion of theme key frame and consistency classification model

[0074] The characteristics of rumors often not only lie in the inconsistency between modalities, but also in the misinterpretation of the theme and the tampering of key frames to be consistent with the description content. Therefore, a theme key frame and consistency classification model need to be fused together to detect short video rumors, and the overall structure of the fusion model is as shown in FIG. 1. Figure 5

[0075] Two consistency features H [CLS]-p and H [CLS]-a and three modal video-level features, i.e., H and H are spliced to obtain H o , as shown in the following formula:

[0076]

[0077] The obtained video representation H o is mapped to a two-dimensional space through a linear layer for a binary classification task. The obtained two-dimensional vector is subjected to a softmax operation to obtain the probability of the output category y, and the specific process is as shown in the following formula. The loss function adopts a cross-entropy loss:

[0078] logits=W T H o +b

[0079] p(y|logits)=softmax(logits)

[0080] The following specific embodiments include three parts: detecting short video rumors based on themes and key frames, consistency classification, and fusing theme key frames and consistency information to debunk rumors. The part of detecting short video rumors based on themes and key frames, hereinafter referred to as TKCM, includes four embodiments: TKCM modal combination analysis, TKCM performance evaluation, TKCM module evaluation, and TKCM case analysis. The part of consistency classification includes one embodiment: experimental evaluation of consistency classification. The part of fusing theme key frames and consistency information to debunk short video rumors, hereinafter referred to as ICIM, includes three embodiments: ICIM performance evaluation, ICIM module evaluation, and ICIM case analysis.

[0081] 1. Part of the experiment of detecting short video rumors based on themes and key frames

[0082] The present application constructs a short video rumor dataset, obtains debunking short videos from the Douyin platform, and these debunking videos contain rumor parts. The rumor parts are artificially cropped as short video rumors. For non-rumor short videos, obtain the videos of popular knowledge and bloggers with a large number of user attentions from the Douyin platform. The two parts together serve as a short video rumor dataset, and the size of the dataset is shown in Table 1.

[0083] Table 1 Rumor dataset​

[0084] Rumor dataset Non-rumor Rumor Sum Training set 500 468 968 Test set 125 116 241 Sum 625 584 1209

[0085] In the experiment, the dimension of the text feature vector extracted by BERT is 768, the dimension of the audio feature vector extracted by the audio pre-training model is 128, and the dimension of the visual feature vector extracted by ResNet is 1024. When training the model, the optimization function is ADAM, the initial learning rate is 0.0002, and L2 regularization and Dropout are used to prevent model overfitting.

[0086] In the modal adjustment mechanism, the settings of the hyperparameters are α=0.8, β=0.7, and γ=1, which are found to be the best after experiments.

[0087] I. TKCM modal combination analysis

[0088] In order to verify the effect of various modalities and their combinations in short video rumor detection, single modalities and multiple modal combinations are set up for experiments. In the experiment, in order to be fair, the modal adjustment mechanism is deleted, and Table 2 shows the experimental results. The accuracy, precision, recall, and F1 value are used as evaluation indicators.

[0089] Table 2: Experimental results of modal combination

[0090] Modality Accuracy Precision Recall F1-score Text 69.7％ 71.3％ 69.9％ 70.4％ Vision 72.1％ 72.6％ 72.8％ 73.4％ Audio 68.9％ 70.5％ 68.8％ 69.6％ Text+Vision 75.9％ 77.2％ 76.0％ 76.6％ Text+Audio 72.6％ 74.0％ 72.8％ 76.6％ Vision+Audio 74.6％ 76.2％ 74.4％ 75.3％ Text+Vision+Audio 75.5％ 76.7％ 76.0％ 76.3％

[0091] Comparing the experimental results of the three single modalities, it can be found that using visual modality alone is better than using other modalities alone. Using visual modality has two more percentage points of accuracy than using text modality, and three more percentage points of accuracy than using audio modality. This shows that visual modality plays a more important role in rumor video detection, followed by text modality. This also indicates the size of the hyperparameter setting for the modal adjustment mechanism proposed in the invention.

[0092] Comparing single modalities and their corresponding double modal combinations, such as comparing the experimental results of using only text modality and text + visual, text + audio double modal combinations, it can be found that whether the visual modality or the audio modality is integrated, the effect has been improved, which shows that the information contained in the text modality and the visual modality and the audio modality is different, and the information of the two modalities can be fused to enhance the effect of video rumor detection.

[0093] Comparing the experimental results of double modal combinations and three modalities, the result of the best combination in double modalities (i.e. text + visual) is higher than that of three modal combinations in accuracy, which shows that it is necessary to add the modal adjustment mechanism.

[0094] In summary, the visual modality plays a more important role in short video rumor detection, followed by the text modality. The information contained in the modalities is different, and the fusion of the information of the two modalities can enhance the video rumor detection effect. Each of the three modalities has its unique information difference and can enhance the video rumor classification effect through fusion.

[0095] II. Performance evaluation of TKCM

[0096] The experimental results of the short video rumor detection model based on theme and key frame are compared with those of other models, and Table 3 shows the experimental results, where the bold part is the optimal result. By comparing the effects of the model in this paper and other video classification models, it can be found that the model performs best on the short video rumor dataset, with an accuracy of 1.7% to 4.6% higher than other models, which shows the effectiveness of feature extraction based on theme and key frame, and also shows that modeling of theme and key frame is more suitable for short video rumor detection tasks.

[0097] Table 3 Comparison of experimental results of TKCM and different models

[0098] Model Accuracy Precision Recall F1-score VideoLSTM 73.9％ 75.4％ 73.6％ 74.5％ FSTCN 72.6％ 74.4％ 72.2％ 73.2％ ActionVLAD 75.1％ 76.4％ 75.2％ 75.8％ TPN 75.1％ 76.9％ 74.4％ 75.6％ att-RNN 72.2％ 73.8％ 72.0％ 72.9％ MVT 75.5％ 77.0％ 75.2％ 76.1％ TKCM 77.2％ 78.2％ 77.6％ 77.9％

[0099] In Table 3, VideoLSTM, FSTCN, ActionVLAD and TPN are single-modal-based video classification, while att-RNN, MVT, TKCM and are multi-modal-based, from which it can be seen that multi-modal-based methods generally have higher accuracy than single-modal-based methods, indicating that using multiple modalities is helpful for video rumor detection. Among them, att-RNN uses visual and text modalities, but has lower accuracy. att-RNN uses relatively primitive LSTM and VGG for feature extraction in each modality, while MVT uses models such as ERNIE for text to obtain high-level features. By comparing with MVT, it can be found that the reason why att-RNN performs poorly in rumor video detection is that the feature extraction model of att-RNN is weak, which cannot obtain high-level features of visual and text, and at the same time verifies the importance of each modality feature extractor.

[0100] Through this experiment, it can be known that modeling of theme and key frame is suitable for rumor video task detection. Using multiple modalities is helpful for video rumor detection, and the feature extractor of each modality affects the rumor video detection effect.

[0101] III. Evaluation of each module of TKCM

[0102] Table 4 shows the comparison of experimental results based on the topic and key frame detection short video rumor model and the removal of each module, where the bold part is the optimal result. Through the table, it can be seen that TKCM-NV and TKCM-AC can improve the classification effect by using only the theme features and key frame features, respectively. Combining theme features and key frame features can more effectively improve the classification effect of rumor videos. This result shows that by combining the two kinds of information on the three modalities of the video, the classification effect of the model can be effectively improved. Secondly, the clustering theme extraction and key frame extraction can mine different information in rumor detection, forming an information complementary advantage.

[0103] Table 4

[0104] Model Accuracy Precision Recall F1-score TKCM-NV 75.9％ 77.2％ 76.0％ 76.6％ TKCM-AC 75.5％ 77.0％ 75.2％ 76.1％ TKCM-MAM 75.5％ 76.7％ 76.0％ 76.3％ TKCM 77.2％ 78.2％ 77.6％ 77.9％

[0105] In the table, AC represents AttentionCluster network, NV represents NeXtVLAD network, MAM represents modal adjustment mechanism, and "-" represents removing the module from the model.

[0106] To explore the effect of modal adjustment mechanism, the modal adjustment mechanism is removed, and the experiment is used to verify the effectiveness of the modal adjustment mechanism. The experiment shows that after removing the modal adjustment mechanism, the accuracy rate decreases. This shows that the importance of the three modalities is different and that the modal adjustment mechanism can help the model better improve the classification effect. At the same time, through the weight size of the modal adjustment mechanism of the three modalities (visual 1.0, text 0.8, audio 0.7), it can be found that the visual modality plays the most important role in short video rumor detection.

[0107] The attention mechanism needs sufficient amount of data for learning, and due to the small amount of data, the network training is not sufficient, and it cannot allocate enough attention to each modality, and the importance difference between modalities cannot be reflected, resulting in poor classification effect. To solve this problem, a modal adjustment mechanism is used. In order to verify the feasibility and effectiveness of the modal adjustment mechanism, the modal adjustment mechanism is replaced by a layer of self-attention mechanism, and the experiment is compared.

[0108] Table 5 Comparison of experimental results of modal adjustment mechanism

[0109] Model Accuracy Precision Recall F1-score Attention 76.3％ 77.9％ 76.0％ 76.9％ MAM 77.2％ 78.2％ 77.6％ 77.9％

[0110] In the table, MAM represents the modal adjustment mechanism, and Attention represents the attention mechanism.

[0111] Table 5 shows the experimental results of the modal adjustment mechanism. By comparing the experimental results, the modal adjustment mechanism has a one percentage point improvement in F1 value than the attention mechanism, and the modal adjustment mechanism performs better than the attention mechanism in the case of small amount of data through the setting method of hyperparameters, which illustrates the effectiveness of the modal adjustment mechanism proposed in this paper.

[0112] At the same time, it also illustrates that the attention mechanism cannot fully play the relationship between these inputs when trained on a small amount of data, and cannot guide the model to pay sufficient attention to important features, while the modal adjustment mechanism guides the model to pay more attention to important modalities through artificial setting, thereby solving the problem that attention cannot be fully trained in the case of small amount of data.

[0113] Embodiment 4: TKCM case analysis

[0114] Several samples are selected for experimental testing. Table 6 selects two examples for illustration. Table 7 is the prediction results of each model in the above two examples. For convenience of viewing, the true label is bolded and underlined, and the prediction correct is bolded.

[0115] Table 6 rumor case

[0116]

[0117] By comparing TKCM-NV, TKCM-AC and TKCM, it can be found that TKCM-NV is wrong for example 1 and correct for example 2, TKCM-AC is correct for example 1 and wrong for example 2, and when the two networks are combined, the prediction of the two examples is correct. It illustrates that information fusion of two networks can complement and enhance information, thereby helping the short video rumor detection task.

[0118] The prediction results of TKCM on the two examples are correct, VideoLSTM, FSTCN, att-RNN, ActionVLAD, TPN and MVT all have errors in prediction, which shows that TKCM has better detection ability in some examples and can capture effective information that other models have not obtained. It has better detection ability than other models.

[0119] Table 7 prediction results of different models on samples

[0120] Model Example 1 Example 2 True label Rumor ]]> ​ Rumor ]]> ​ VideoLSTM Non-rumor Non-rumor FSTCN Non-rumor Non-rumor ActionVLAD Rumor Non-rumor TPN Non-rumor Non-rumor att-RNN Non-rumor Rumor MVT Non-rumor Rumor TKCM-NV Non-rumor Rumor TKCM-AC Rumor Non-rumor TKCM Rumor Rumor

[0121] 2. Consistency classification part

[0122] Embodiment 5: consistency classification experiment evaluation

[0123] The consistent classification model is a version of BERT pre-trained on Chinese corpus, provided by BERT-base-chinese version of Hugging face, and the vector dimension of each token is 768. In the training process, the optimization function used is ADAM, the initial learning rate is 0.0002, and L2 regularization and Dropout are used to prevent model overfitting.

[0124] Two data sets are constructed in this embodiment, the first is a text-visual consistency data set; the second is a text-audio consistency data set. That is, text-visual consistency data and inconsistent data, text-audio consistency data and inconsistent data are constructed respectively. Table 8 shows the detailed data of the two data sets.

[0125] Table 8 Consistency data set

[0126] Consistent data amount Inconsistent data amount Sum Text-vision consistency 625 1209 1834 Text-audio consistency 625 1209 1834

[0127] In order to better understand the consistency data set constructed by the present application, Table 9 shows examples of positive and negative samples in the text-audio data set.

[0128] Table 9 Text-visual consistency data examples

[0129]

[0130] Since the audio information in the video is a series of audio signals, in order to show the audio information in the video, the content of the speech in the video is recorded as audio information display, and the actual operation is to process the audio signal in wav format.

[0131] In order to verify the expression effect of the consistency classification model on the consistency information characteristics, the inter-modal consistency classification experiment is carried out using the constructed consistency data set, and the experimental results are shown in Table 10.

[0132] Table 10 Consistency classification experiment results

[0133] Model Accuracy Precision Recall F1-score BERT-p 88.8％ 80.4％ 88.8％ 84.4％ BERT-a 86.6％ 77.1％ 86.4％ 81.5％

[0134] In the table, BERT-p and BERT-a represent text-visual and text-audio consistency classification models respectively.

[0135] From the experimental results in the table, it can be found intuitively that the accuracy of text-visual consistency classification reaches 88.8%, and the accuracy of text-audio also reaches 86.6%. It is shown that using the pre-trained model BERT and fine-tuning can obtain better classification effect, and can efficiently identify the consistency or not between modalities. This better consistency classification effect shows that the model has learned the consistency information between modalities, and can mine the consistency information of new data after embedding the whole model.

[0136] The model is guided by the consistency label. In the downstream task of the model, “[CLS]” is input into the last linear classifier, and good experimental results are achieved. At the same time, the higher classification effect also proves that the representation corresponding to “[CLS]” contains the consistency information learned by the model, and the consistency information can be obtained by using the “[CLS]” representation in the embedded overall model, so as to assist the model to detect short video rumors.

[0137] 3. Short video rumor-busting part fusing theme key frame and consistency information

[0138] Embodiment 6: ICIM performance evaluation

[0139] The consistency rumor detection experiment is as follows. When training the model, the bert-a and bert-p are frozen and no longer trained, the optimization function is ADAM, the initial learning rate is 0.0002, and L2 regularization and Dropout are used to prevent model overfitting. The total parameter quantity of the model is about 600M.

[0140] As described above, in the modal adjustment mechanism, the hyperparameters are set as α = 0.8, β = 0.7, and γ = 1, that is, the text modality is multiplied by 0.8, the audio modality is multiplied by 0.7, and the visual modality is multiplied by 1.0 before splicing.

[0141] In the classifier, a one-dimensional vector containing the same number of probability values as the number of categories is output, which represents the probability value of the video belonging to each category. The category corresponding to the maximum probability value is taken as the prediction result. Table 11 shows the comparison experimental results, and the bold part is the optimal result. By comparing the effects of the model in this paper and other video classification models, it can be found that the short video rumor dataset in this paper shows the best effect.

[0142] After fusing the text-visual and text-audio consistency information, the accuracy and precision are the highest among all comparison models, which shows that the text-visual and text-audio consistency information is beneficial to the automatic rumor detection of the model and improves the effect of rumor detection. ICIM increases the consistency information difference on the basis of TKCM, and the classification effect is improved, which shows that there is a consistency information difference between different modalities in the short video rumor, and this information difference helps rumor classification and improves the classification effect.

[0143] Table 11 Comparison of experimental results of ICIM and different models

[0144] Model Accuracy Precision Recall F1-score VideoLSTM 73.9％ 75.4％ 73.6％ 74.5％ FSTCN 72.6％ 74.4％ 72.2％ 73.2％ ActionVLAD 75.1％ 76.4％ 75.2％ 75.8％ TPN 75.1％ 76.9％ 74.4％ 75.6％ att-RNN 72.2％ 73.8％ 72.0％ 72.9％ MVT 75.5％ 77.0％ 75.2％ 76.1％ TKCM 77.2％ 78.2％ 77.6％ 77.9％ ICIM 79.7％ 80.6％ 80.0％ 80.2％

[0145] Embodiment 7: Evaluation of each module of ICIM

[0146] The consistency information between different modalities may affect the detection effect. To study the influence degree of the difference of the consistency information between modalities on rumor classification, text-visual consistency and text-audio consistency information are deleted respectively in this embodiment for experiment, and the experimental results are shown in Table 12.

[0147] Table 12 ICIM different modal consistency influence results

[0148] Model Accuracy Precision Recall F1-score ICIM-(bert-a) 79.3％ 81.0％ 78.4％ 79.7％ ICIM-(bert-p) 77.6％ 78.9％ 77.6％ 78.2％ TKCM 77.2％ 78.2％ 77.6％ 77.9％ ICIM 79.7％ 80.6％ 80.0％ 80.2％

[0149] In Table 12, bert-p and bert-a respectively represent text-visual consistency BERT and text-audio consistency BERT, and “ICIM-” indicates that the ICIM module is removed.

[0150] In comparison with only one kind of consistency information, that is, the first and second rows in the table, only text-visual features are integrated, the effect is optimal, which shows that the text-visual consistency information plays a more important role in rumor detection. Since the recall rate represents the proportion of correctly predicted positive samples, it shows that the text-visual consistency information can better classify the positive samples. The precision rate decreases after integrating both information, and part of the positive samples do not have obvious information in text-audio consistency, and the integration of text-audio consistency information brings noise instead.

[0151] In order to verify the classification results of the ICIM model on different types of rumor videos, we more accurately divided the rumor dataset according to the type of rumor, and then counted the classification results of each type of rumor. The number of division in the test set and the statistical results are shown in Table 13. Then, compare the classification results of TKCM and ICIM on different types of rumors, see Table 14.

[0152] Text type rumor short video refers to the content of the rumor presented in the form of text in the video. For example, the content is presented in the form of chat records in the video, and the corresponding voiceover and BGM are constructed in the video.

[0153] Knowledge type rumor short video refers to the discussion content in the video involving a professional field, which requires specific professional knowledge to identify whether it is a rumor; fabricated message type rumor short video refers to a false message artificially fabricated to achieve a certain purpose. In the video, the explanation of this message makes this type of rumor short video appear real; distorted fact type rumor short video refers to distorting the actual situation in the video, and integrating the distorted video content in the short video to attract the attention of others.

[0154] Table 13 ICIM experimental results on different types of rumor videos

[0155]

[0156] By comparing TKCM and ICIM models, it can be found that the detection effect of each type except the text type short video rumor is improved after adding consistency information. Especially in the distortion of the truth class and the fabricated message class, it shows that consistency information helps the most in the detection of these two types of short video rumors. Compared with simply learning the theme and key frame features of the video, consistency information can compare the information between modalities, so that the inconsistency between modalities of the distortion of the truth class is found, thereby improving the classification effect of the model on the distortion of the truth class and the fabricated message class.

[0157] Table 14 TKCM and ICIM experimental comparison of different types of rumor short videos

[0158] Type TKCM ICIM Text type 50.0％ 50.0％ Knowledge type 76.3％ 78.9％ Fabricated message 75.0％ 83.3％ Distorted fact 80.0％ 90.0％ Total 75.0％ 79.3％

[0159] Example 8: ICIM case analysis

[0160] In this example, several samples are selected for experimental testing. Table 15 selects two examples to illustrate. Table 16 shows the prediction results of each model in the above two examples. In order to facilitate viewing, the correct prediction is bolded in the table.

[0161] Table 15 ICIM rumor cases

[0162]

[0163] Example 1 shows that penguins are animals that cannot fly, and the animal in the video is a bird, which is inconsistent with the penguin in the title.

[0164] Example 2 shows that the artificial egg in the video is a toy, and the artificial egg in the title is inconsistent with the toy in the video.

[0165] Table 16 Different model case prediction results

[0166] Model Example 1 Example 2 VideoLSTM Non-rumor Non-rumor FSTCN Non-rumor Non-rumor att-RNN Rumor Non-rumor ActionVLAD Non-rumor Non-rumor TPN Non-rumor Non-rumor MVT Rumor Non-rumor ICIM Rumor Rumor TKCM Non-rumor Non-rumor ICIM-(bert-p) Rumor Non-rumor ICIM-(bert-a) Rumor Rumor

[0167] From the table, it can be found that the ICIM and ICIM-(bert-a) models can effectively identify the inconsistent rumor videos, while the other models make mistakes in predicting the above two examples.

[0168] By comparing the TKCM model and the ICIM model, it shows that integrating a module with consistency information can help the model identify modal inconsistency in rumor videos, and proves that adding consistency in video rumor classification can effectively assist the model in automatic rumor detection.

[0169] By comparing the ICIM-(bert-a) model and the ICIM-(bert-p) model, the two prediction results of the ICIM-(bert-a) are correct, and the prediction of the ICIM-(bert-p) model for example 2 is wrong, which shows that in some examples, the text-visual consistency information is more important and can more effectively assist the model to automatically detect short video rumors.

[0170] By comparing the TKCM model and the ICIM-(bert-p) model, the prediction results of the TKCM model in two examples are wrong, and after the text-audio consistency information is integrated, the prediction result for example 1 is correct, which shows that the text-audio consistency information plays a role and can also assist the model to automatically detect short video rumors.

[0171] For VideoLSTM, FSTCN, att-RNN, ActionVLAD and TPN, the prediction results are all wrong, among which att-RNN uses visual and text modal features, and other benchmark models only use visual features, which shows that simply using part of the modal information is insufficient for rumor video detection. att-RNN and MVT obtain correct prediction results on example 1, which may be due to obtaining more title text information than other benchmark models.

[0172] Experiments and results show that the present application has the following advantages and effects compared with the prior art:

[0173] (1) From the example analysis, it can be seen that the information contained in the modalities is different, and the video rumor detection effect can be enhanced by fusing the information of the two modalities. Each of the three modalities has its own information difference and can improve the video rumor classification effect through fusion.

[0174] (2) The classic multi-modal fusion more considers multiple modalities from the bottom features, and gradually aggregates into cross-modal features through deep learning. However, there are means such as distortion of the theme and tampering of the key frame in the rumor video, and the present application considers that multi-modal fusion needs to learn rumor features from the theme and key frame of the short video rumor.

[0175] (3) For modal inconsistent rumor videos, the present application proposes to use BERT to learn whether the text-visual and text-audio modalities of the video are consistent. Then embed the trained model into the overall framework, fuse the learned consistency information with the rumor feature information, and assist the model to detect rumors.

[0176] (4) The F1 value of the application on the short video rumor data set is 4%-7% higher than that of the commonly used video classification model. Compared with the model before the consistency information is integrated, the F1 value is improved by 2.3%.

Claims

1. A short video rumor detection method based on multi-modal consistency, characterized in that The method comprises the following steps: 1) establishing a short video rumor classification model TKCM based on a theme and key frames; 2) using a pre-trained modality feature extractor to extract single modality features, using an aggregation network to obtain theme features of the video, using an attention network to obtain key frame features of the video, and inputting the three modalities into the short video rumor classification model TKCM; The specific steps of using a pre-trained modality feature extractor to extract single modality features are: obtaining single-frame feature vectors of each modality through a pre-trained model, and aggregating the single frames of each modality to form overall theme features; meanwhile, an attention mechanism is used to select key frames; two aggregation networks, namely a NeXtVLAD network and an AttentionCluster network, are used to obtain theme features and key frame features, respectively; The key frame feature extraction adopts an attention mechanism network Attention Cluster, which can assign more weights to relatively important frames, is mainly used to select key frames in the video, and obtain key frame representations under three modalities; The theme feature extraction adopts a NeXtVLAD network to select a cluster center, and then encodes the vision, text and audio to obtain theme feature representations under each modality; 3) inputting the theme features and the key frame features of each modality into a modality adjustment mechanism adjustment network to adjust the importance attention of the three modalities, splicing the theme features and the key frame features, and fusing multi-modality information; The importance of the three modalities is different, and before splicing the three modalities, the feature vectors of the three modalities are respectively multiplied by a hyperparameter less than or equal to 1, so that the values of the feature vectors of each modality are appropriately reduced or relatively amplified, and the dominant role of the three modalities in the final classifier is changed in advance; 4) constructing a consistency dataset and training an inter-modality consistency detection model to obtain consistency information; 5) fusing the theme features, the key frame features and the consistency information to serve as a short video rumor detection means; The specific steps of fusing the theme features, the key frame features and the consistency information are: embedding the text-vision and text-audio consistency models into the overall rumor classification framework, passing the obtained consistency features through a fully connected layer, and then splicing the modality theme key frame features obtained in the TKCM; mapping to a two-dimensional space through a linear layer, and performing rumor detection binary tasks.

2. The short video rumor detection method based on multi-modal consistency according to claim 1, characterized in that In step 1), the short video rumor classification model TKCM includes a rumor part and a non-rumor part, the rumor part is established by artificially intercepting a rumor part in a debunking video obtained from a short video platform to establish a short video rumor dataset; the non-rumor part is obtained from a video about knowledge popularization and a blogger with a large number of followers on a short video platform.

3. The short video rumor detection method based on multi-modal consistency according to claim 1, characterized in that In step 2), the three modalities include a text modality, a visual modality and an audio modality.

4. The short video rumor detection method based on multi-modal consistency according to claim 1, characterized in that In step 4), the consistency dataset is obtained by combining modalities in different videos, for inconsistent data, each modality in different videos in the dataset is randomly combined; for data consistent between modalities, the corresponding modalities in non-rumor data are combined.

Images