Multi-modal false news monitoring method based on comparative learning

Abstract

The invention discloses a multi-modal false news monitoring method based on comparative learning, and belongs to the technical field of false short video news detection. Comprising the following steps: designing a VisualTheme module to perform visual and text feature extraction based on a CLIP text multi-view contrast learning feature extractor, and realizing extraction and fusion of two features of a key frame and a frame theme; features of titles, continuous frames and music in short video news are extracted based on text, vision and audio pre-training feature extractors, and extraction of other modal features of non-key frames is achieved; and designing a multi-modal fusion and classifier by utilizing the extracted multiple modal features, and generating a classification result by taking the multi-modal features as input. And a text multi-view contrast learning mechanism module is designed to optimize parameters. According to the method, mutual information and semantic consistency of multi-text features are improved through the text multiple views, generalization features of multiple text views are learned, and fusion of the text features and other modalities is promoted.

Description

Technical Field

[0001] The present invention relates to the field of fake short video news detection, specifically a multimodal short video fake news monitoring method based on supervised text multi-view comparative learning. Background Art

[0002] With the widespread popularity of short video platforms, short videos have become an important way of spreading news. Whether it is official accounts, news media, or ordinary users, they can edit various events into news videos. This highly free content production method provides a basis for the generation of false short video news. Because video methods have a stronger effect in spreading false news, they will further aggravate the impact of false news. Therefore, studying fake news in the form of videos is crucial for timely detection and intervention in the spread of false news.

[0003] Fake news videos are composed of video entities and titles as well as social comments and publisher information. Different from traditional fake news detection based on images and texts, short video fake news detection has the following characteristics: First, short video fake news detection needs to analyze multiple modalities contained in the video to obtain more information. It is necessary to extract and integrate complex new feature information from the same modality, and to analyze multiple perspectives under the same modality to assist in understanding the news content and detecting fake news. Second, short video platforms provide a quick video editing function that allows users to easily edit and tamper with the original news video. Users can easily introduce artificial subtitles, which not only introduces a new text information but also destroys the integrity of the video. Finally, short video platforms also have rich social information. How to reasonably use user information, comments and other modal information also poses challenges to short video fake news detection.

[0004] The work based on fake news detection is mainly divided into two parts according to the different news carriers. Traditional image and text fake news detection and the latest short video fake news detection. Traditional image and text multimodal fake news detection often only involves the feature fusion between the two modalities. Researchers usually use pre-trained networks to extract information about the two features of images and text, and then promote the alignment of text and visual features through attention or cross-attention mechanisms, thereby exploring the common consistency semantics between text and image content to improve detection capabilities. FND-CLIP, a multimodal fake news detection method based on CLIP learning, uses the CLIP pre-trained model to measure cross-modal similarity and guide feature mapping and fusion. Although the traditional image and text fake news detection model has achieved good results on image and text datasets. However, in the more complex short video fake news detection task, it still has many shortcomings such as insufficient use of modal information.

[0005] In the detection of short video fake news, previous studies often use the heterogeneous multimodal information of news videos for verification. They often focus on how to fuse the features of the extracted different heterogeneous information to explore semantic consistency. However, the heterogeneity between cross-modal information leads to less similarity of feature information. Therefore, the quality of feature extraction and modality relevance in different modalities will have a direct impact on the effect of fake news detection models. Especially in short-video fake news, since forged and homemade fake news often use video content as the background, eye-catching subtitles are used in the picture to introduce the main content of the fake news. Therefore, text information and titles are still the main content of short video news, and the pictures in the video also contain key information such as occupations, places, and objects. According to the characteristics of short video fake news, how to reasonably extract the features of a series of key frames (pictures) and text information among many modalities so that the features of the two different modalities have more similarities is still the focus of research. In addition, fake news videos usually contain a lot of homogeneous information in addition to heterogeneous information. The main type of homogeneous information is text information, including video titles, video comments, publisher information, subtitles that appear in the video, and other multi-perspective text information. These text modalities all contain important features for short video fake news detection. SV-FEND verifies that text information is still the most important modality that affects the effect of the multimodal fake news short video detection model. However, because text features in different text modalities contain both similar repetitive information and noise, this will lead to a decline in the quality of text features. Therefore, it is also an important challenge to analyze and balance text information from multiple perspectives in a single modality to provide stable positive features. Summary of the invention

[0006] The purpose of the present invention is to realize a multimodal short video fake news monitoring system based on supervised text multi-view comparative learning, so as to realize automatic detection and classification of fake short video news and solve the problems mentioned in the above background technology.

[0007] To achieve the above purpose, the method of the present invention designs the following technical method: a multimodal short video fake news monitoring system with supervised text multi-view comparative learning, comprising the following steps:

[0008] Step S1: Based on the feature extractor of CLIP text multi-view contrastive learning, a VisualTheme module is designed to extract visual and text features, and the extraction and fusion of key frame and frame theme features are realized;

[0009] Step S2: extracting features of titles, continuous frames, and music in short video news based on text, vision, and audio pre-trained feature extractors, and extracting other modal features of non-key frames;

[0010] Step S3: Design a multimodal fusion and classifier using the multimodal features extracted in steps S1 and S2, and use the multimodal features as input to generate classification results.

[0011] Step S4: Design a text multi-view contrastive learning mechanism module, calculate the contrastive learning loss of the frame theme features extracted in step S1 and the title features and comment features extracted in step S2, and use them together with the classification loss generated by the classification module in step S3 to optimize the system parameters.

[0012] Figure 1 VisualTheme is a text and visual feature encoder designed based on CLIP. In order to better integrate visual features and text features, the VisualTheme-VT frame theme feature extraction module includes a keyframe-theme extraction module, a keyframe-theme fusion module, and a short video hot word library.

[0013] In the frame theme feature extraction stage, the visual extractor based on the CLIP model of multimodal contrastive learning is used to extract the key frame F i ∈{F1,F2,F3,...,F n}. Then, the pre-trained CLIP text extraction module is used to extract the selected keywords W from the short video news vocabulary. i ∈{W1,W2,W3,...,W N} (Here N represents the number of extracted keywords). In this way, a series of key frame visual features H are obtained. frame ={{H f1 ,H f2 ,H f3 ,...,H fn}} (here n represents the number of key frames extracted from a short video news) and the text features H of the first N keywords words ={H w1 ,H w2 ,H w3 ,...,H wN}.

[0014] H fi =CV(F i )

[0015] H wi =CT(W i )

[0016] The extracted key frame features H of each frame are transformed into fi ∈H frames And all keyword text features H wordsCompare. Use CLIP comparison module to output the confidence scores of N keywords corresponding to each frame. It is stipulated that when the confidence score is greater than 0.5. The frame is considered to have a frame keyword. The frame keyword is a keyword with a confidence greater than 0.5. Finally, frame keywords are extracted for each frame. These frame keywords are spliced ​​together and sent to the pre-trained BERT feature extraction model to obtain the frame theme text feature H vt .

[0017] Scores = CL(H words ,H fi ))=(S1,S2,S3,...,S N )

[0018]

[0019] Through a cross-modal attention and fusion network, frame features and topic features are enhanced. frames+ . And H frames+ and H frames The feedforward network is used again to enhance the fitting capability and adjust the feature dimension. The query of the frame topic attention mechanism is Q t =H vt W Q , key is K v =H frames W K ,value is V v+ =H frames+ W V The query of the keyframe attention mechanism is Q v =H frames W Q , value and key are K t =H vt W K ,V t =H vt W V .

[0020] Q t =H vt W Q ,K t =H vt W K ,V t =H vt W V

[0021] Q v =H frames W Q ,K v =H frames W K ,Vv =H frames W V

[0022] Q v+ =H frames+ W Q ,K v+ =H frames+ W K ,V v+ =H frames+ W V

[0023]

[0024] In this way, the key frame features and frame theme features are obtained after mutual enhancement.

[0025] Figure 2 In the lower left part of the figure, feature extraction of other modalities is performed. The text is input into the pre-trained BERT, and then the text feature H is extracted through the feedforward neural network layer. t Then the pre-trained VGGish model is used to extract audio features, and the extracted features are adjusted by the feedforward neural network to obtain the audio feature H a The motion features H are extracted from the consecutive frames using the pre-trained C3D model

[45] c3d . Use the pre-trained BERT to extract the features H of each review c and author information feature H u .

[0026] Figure 2 In the lower right part, a frame-theme video fusion module and a non-keyframe feature extraction module are designed for feature extraction. Through steps S1 and S2, 7 modalities are obtained to describe the news short video, among which the keyframe and frame theme have been learned through a cross-modal transformer to learn the correlation between the two modalities. After that, the self-attention mechanism

[44] is used to model the correlation of these 7 features, and finally the fused features are obtained. Specifically, the obtained 7 features are connected to a feature sequence X m7 And input it into a classic transformer layer, and finally get the multimodal fusion feature X m .

[0027] Q x =X m7 W Q ,K x =X m7 W K ,V m7 =X m7 W V

[0028]

[0029] Using a fully connected layer with softmax activation, the multimodal feature vector X m Projecting into the target space of two categories, real and fake news videos, we get the probability distribution:

[0030] P = softmax(W b X m +b b )

[0031] Where p = [p0, p1] is the predicted probability vector, p0 and p1 represent the predicted probabilities of the label as 0 (real news video) and 1 (fake news video), respectively. Wb is the weight matrix, and b is the bias term. Therefore, for each news article, the goal is to minimize the binary cross entropy loss function, as follows:

[0032] L p =-[(1-y)logp0+ylogp1]

[0033] where y∈{0,1} represents the ground truth label.

[0034] like Figure 2 As shown in the upper right part, a text multi-view contrastive learning module is designed. In order to bring the view features with the same label in the text multi-view modality in the short video closer in space (i.e., the features describing real news), and to pull away the many text modality view features that are not real news, so that the expression of multiple text views is more consistent and fully utilizes their mutual information, a supervised text multi-view contrastive learning loss is designed. The upstream feature extraction network will input the extracted four modalities into the contrastive learning network:

[0035] They are visual text features Title features Publisher Characteristics Comment Features (where Nb is the batch size). Because these four features are extracted by pre-trained BERT, their feature dimensions are consistent. Assume that hi is the anchor point for contrastive learning. When h i ,h j j∈β and y i =y j (Here y i and j Yes i and h j Label) here constitutes a positive sample pair. Similarly, when h i ,hk ∈β and y i ≠y k , which constitutes a negative sample pair. Then calculate the contrast loss of all positive pairs, including all (h i ,h j j) and (h j ,h i )

[0036]

[0037] f(u,v) is to calculate the cosine similarity of u and v vectors in the feature space;

[0038] The learning goal of the proposed system is to train the model through the loss function generated by supervised text multi-view contrastive learning and the minimization of the binary cross entropy loss function generated by the step S3 classifier. The total loss L is represented by the sum of the two losses:

[0039] L=γL mv +μL p +λ||θ|| 2

[0040] Among them, γ and μ are tuned hyperparameters. Θ represents all trainable parameters of the model, and λ represents the l2-regularization coefficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is the flowchart of VisualTheme text visual feature extraction and fusion module.

[0042] Figure 2 It is the overall architecture of the designed network. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the present invention to clearly and completely describe the technical solutions and implementation details in the specific implementation process of the present invention. Obviously, the described implementation examples are only part of the implementation examples of the present invention, not all the embodiments; based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The implementation plan of this experiment is as follows:

[0045] Step S1: Build dataset: The system is trained and run on the FakeSV dataset, which is the only fake news video dataset that provides rich event and debunking samples. This dataset collects news videos from popular short video platforms in China, such as Douyin, and uses manual annotations. FakeSV consists of 1,827 fake news videos, 1,827 real news videos, and 1,884 videos debunked under 738 events. For each news video, this dataset provides the video, title, metadata, comments, and user profile.

[0046] Step S2: The 3654 short videos in the dataset are segmented and processed. First, the audio of each video is extracted independently into WAV format through the audio extraction file. Then, the audio file is extracted-1 layer output as the feature extraction result through the VGGilsh audio feature extractor. Multimodal features are extracted using pre-trained bert-base-Chinese, VGGish and C3D models. In the Co-attention module, 4 heads are used, the hidden layer size is 128, and the number of heads of Self-AttentionTransformer is 2. The number of audio frames is set to 50 respectively. The key frames are sorted using the frame difference method, and the frames with the largest frame difference in the first 30 frames are extracted and arranged in chronological order. Based on the FakeSV dataset, word frequency statistics were obtained from a database of 70 keywords, with a maximum number of 23 comments.

[0047] Step S3: Network training. The batch size is 128. In , the learning rate is set to 1×10-3 and 5×10-5 respectively. All experiments are performed on NVIDIA Tesla V100 16GB GPU using PyTorch. The model is trained with 20 early-stops to prevent overfitting. ReLU is used as the nonlinear activation function, and the Adam algorithm is used to optimize the loss function. In addition to feature extraction, end-to-end model training is also performed.

[0048] Step S4: Model testing. In order to mitigate the performance bias caused by the randomness of data segmentation, five-fold cross validation (Acc.) is used for evaluation, with macro accuracy (Advanced Accuracy), macro recall (Recall) and macro F1 score (F1) as evaluation indicators. For each fold, the dataset is divided into a training set and a test set with a sample ratio of 4:1 at the event level. This ensures that there is no event overlap between different sets, thereby avoiding the model from detecting fake news videos by memorizing event information. In the experiment, the model is trained in the 5-category fold in turn, and the training weights are saved. Finally, the experimental results are tested according to the 5 split methods in the prediction stage.

[0049] like Figure 1As shown in Table 1, the comparison results are shown in Table 1. As shown in Table 1, SV-CMD has the highest accuracy, F1 score, precision and recall on the FakeSV dataset, exceeding the state-of-the-art methods by 1.5%, 1.57%, 1.28% and 1.52% respectively. The performance of SV-CMD is better than most state-of-the-art methods, mainly due to the following reasons. First, the pre-trained CLIP encoder in the frame topic feature extraction and fusion module in SV-CMD can generate semantically rich text and image features in the same semantic space, ensuring that the fused features correctly reflect the correlation between numerous keywords and keyframes, and provide complementary information for the two modal peaks. The supervised multi-view contrastive learning loss allows the four modalities of text views to be mapped to the same space, and the loss is used to optimize the distance between different positive sample views in the semantic space to learn the generalization features of text views. Further enhance the accuracy of the classification model.

[0050] The main innovation of the present invention is to propose a multimodal short video fake news detection framework SV-CMD based on multi-view contrastive learning. The framework includes text multi-view supervised contrastive learning, which improves the mutual information and semantic consistency of multi-text features through text multi-view, learns the generalization features of multiple text views, and promotes the fusion of text features and other modalities. In addition, a cross-modal visual text feature extraction module named VisualTheme is proposed. This method is based on the CLIP pre-training model and the multimodal feature fusion method. Under the guidance of the short video news keyword library, it can stably extract the visual modal features and keyword modal text features of key frames in short videos. Through comprehensive actual tests on the FAKE-SV event dataset and time dataset, the proposed SV-CMD has better results than before, proving that the fake short video news detection system designed by the present invention has better performance.

[0051] Table 1 is a comparison of the effects of the model of the present invention and other models.

[0052]

[0053]

[0054] It is obvious to those skilled in the art that the present invention is not limited to the details of the above-mentioned exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention; therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, and therefore it is intended to include all changes within the meaning and scope of the equivalent elements of the claims within the present invention; any figure mark in the claims should not be regarded as limiting the claims involved.

[0055] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A multimodal fake news monitoring method based on contrastive learning, characterized by: The following steps are involved: Step S1: Based on the feature extractor of CLIP text multi-view contrastive learning, a VisualTheme module is designed to extract visual and text features, and realize the extraction and fusion of key frame and frame theme features; Step S2: extracting features of titles, continuous frames, and music in short video news based on text, vision, and audio pre-trained feature extractors, and extracting other modal features of non-key frames; Step S3: Design a multimodal fusion and classifier using the multimodal features extracted in step S1 and step S2, and use the multimodal features as input to generate classification results; Step S4: Design a text multi-view contrastive learning mechanism module, calculate the contrastive learning loss of the frame theme features extracted in step S1 and the title features and comment features extracted in step S2, and use them together with the classification loss generated by the classification module in step S3 to optimize the system parameters.

2. The multimodal fake news monitoring method based on contrastive learning according to claim 1 is characterized in that: The VisualTheme module is a text and visual feature encoder designed based on CLIP; The VisualTheme-VT frame theme feature extraction module consists of a keyframe-theme extraction module, a keyframe-theme fusion module, and a short video hot word library.

3. The multimodal fake news monitoring method based on contrastive learning according to claim 2 is characterized in that: In the frame theme feature extraction stage, the visual extractor based on the CLIP model of multimodal contrastive learning is used to extract the key frame F i ∈{F1,F2,F3,...,F n }; then, the pre-trained CLIP text extraction module is used to extract the selected keywords W from the short video news vocabulary i ∈{W1,W2,W3,...,W N }, N represents the number of extracted keywords; thus, a series of key frame visual features H are obtained frame ={{H f1 , H f2 , H f3 , ..., H fn }} and the text features H of the first N keywords words ={H w1 ,H w2 ,H w3 ,...,H wN }; n represents the number of key frames extracted from a short video news; H fi =CV(F i ) H wi =CT(W i ) The extracted key frame features H of each frame are transformed into fi ∈H frames And all keyword text features H words Compare the frames; use the CLIP comparison module to output the confidence scores of the N keywords corresponding to each frame; stipulate that when the confidence score is greater than 0.5, the frame is considered to have a frame keyword, which is a keyword with a confidence greater than 0.5; finally, frame keywords are extracted for each frame; these frame keywords are spliced ​​together and sent to the pre-trained BERT feature extraction model to obtain the frame theme text feature H vt ; Scores=CL(H words ,H fi ))=(S1,S2,S3,...,S N ) 4. The multimodal fake news monitoring method based on contrastive learning according to claim 3 is characterized in that: Enhance frame features and topic features through a cross-modal attention and fusion network; use the residual network to H frames+ ; and H frames+ and H frames Again, the feature dimension is adjusted by enhancing the fitting ability through the feedforward network; The query of the frame topic attention mechanism is Q t =H vt W Q , key is K v =H frames W K ,value is V v+ =H frames+ W V ; The query of the keyframe attention mechanism is Q v =H frames W Q , value and key are K t =H vt W K , V t =H vt W V ; Q t =H vt W Q ,K t =H vt W K ,V t =H vt W V Q v =H frames W Q ,K v =H frames W K ,V v =H frames W V Q v+ =H frames+ W Q ,K v+ =H frames+ W K ,V v+ = frames+ W V The key frame features and frame theme features are obtained after mutual enhancement.

5. The multimodal fake news monitoring method based on contrastive learning according to claim 1 is characterized in that: The text is input into the pre-trained BERT, and then the text features H are extracted through the feed-forward neural network layer. t ; Then use the pre-trained VGGish model to extract audio features, and adjust the extracted features through the feedforward neural network to obtain the audio feature H a ; Use the pre-trained C3D model [45] to extract motion features H from consecutive frames c3d ; Use pre-trained BERT to extract features H for each review c and author information feature H u .

6. The multimodal fake news monitoring method based on contrastive learning according to claim 1 is characterized in that: A feature extraction module for frame-topic video fusion and a feature extraction module for non-keyframes are designed. Through steps S1 and S2, seven modalities are obtained to describe the news short video, in which the keyframe and frame topic have been learned through a cross-modal transformer to learn the correlation between the two modalities. After that, the self-attention mechanism [44] is used to model the correlation of the seven features, and finally the fused features are obtained. Specifically, the seven features are connected to a feature sequence X m7 And input it into a classic transformer layer, and finally get the multimodal fusion feature X m ; Q x =X m7 W Q ,Kx=X m7 W K ,V m7 =X m7 W V Using a fully connected layer with softmax activation, the multimodal feature vector X m Projecting into the target space of two categories, real and fake news videos, we get the probability distribution: P=softmax(W b X m +b b ) Where p = [p0, p1] is the predicted probability vector, p0 and p1 represent the predicted probability of the label as 0, i.e., a real news video, and 1, i.e., a fake news video; Wb is the weight matrix, and b is the bias term; for each news article, the goal is to minimize the binary cross entropy loss function, as follows: L p =-[(1-y)log p0+y log p1] where y∈{0, 1} represents the ground truth label.

7. The multimodal fake news monitoring method based on contrastive learning according to claim 1, characterized in that: Design a text multi-view contrastive learning module. The upstream feature extraction network will input the extracted four modalities into the contrastive learning network: They are visual text features Title features Publisher Characteristics Comment Features Where Nb is the batch size; Define hi as the anchor point for contrastive learning. When h i ,h j j∈β and y i =y j Constitute a positive sample pair, y i and j Yes i and h j The label; when h i ,h k ∈β and y i ≠y k , which constitutes a negative sample pair; then calculate the contrast loss of all positive pairs, including all (h i ,h j j) and (h j ,h i ): f(u,v) calculates the cosine similarity of u and v vectors in the feature space.

8. The multimodal fake news monitoring method based on contrastive learning according to claim 1, characterized in that: The model is trained by the loss function generated by supervised text multi-view contrastive learning and the minimization of the binary cross entropy loss function generated by the step S3 classifier; the total loss L is represented by the sum of the two losses: L=γL mv +μL p +λ||θ|| 2 Among them, γ and μ are tuned hyperparameters; Θ represents all trainable parameters of the model, and λ represents the l2-regularization coefficient.

Images