False news video detection method based on knowledge enhanced graph attention network

Abstract

The invention provides a false news video detection method based on a knowledge enhancement graph attention network, and the method comprises the steps: carrying out the extraction through a pre-training language model based on complete text information, and obtaining the semantic features of a text; extracting the voice information by using a pre-trained voice model to obtain emotional features of the voice; extracting the key frame by using a pre-trained sliding window model to obtain visual features; obtaining global comment features of the video by using a pre-trained vertical field detection model and combining the vertical field features of the video and the weight values of the comments; obtaining social features of the user by using the social features of the video publisher; obtaining time sequence scene graph features through the scene graph sequence; fusing the multi-modal features to obtain a multi-modal feature representation; and obtaining a prediction result based on the multi-modal feature representation. The invention provides a comprehensive and efficient false news video detection scheme, and the detection precision and the practical application value are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and natural language processing, and in particular to a false news video detection method based on a knowledge-enhanced graph attention network. Background Art

[0002] With the rapid development of the Internet and social media, the spread of fake news has become more and more serious, especially fake news in video content. As a powerful communication medium, the visual and auditory effects of video make fake news more deceptive. Traditional fake news detection methods mainly rely on text analysis, usually focusing on text information such as news titles, article content or comments, but this method cannot fully utilize the visual, voice and time sequence information in the video. Therefore, the existing technology has many limitations in video fake news detection and cannot fully capture the false features in the video content.

[0003] At present, the technology of video fake news detection can be divided into methods based on visual information, voice / subtitle information, and text comment analysis. However, these methods are usually single-modal and cannot comprehensively analyze various types of information in the video. For example, visual content may not be able to independently determine whether a video is fake news, and voice or subtitle analysis is also difficult to fully reflect the semantic information of the video as a whole. In addition, the video itself has time series characteristics, and the changes and development of information are dynamic. Many existing methods fail to effectively process this time series information, resulting in deficiencies in early detection of fake news and change capture. Summary of the invention

[0004] In view of the above situation, the main purpose of this invention is to propose a fake news video detection method based on knowledge-enhanced graph attention network to solve the above technical problems.

[0005] The present invention proposes a fake news video detection method based on knowledge-enhanced graph attention network, which comprises the following steps: Step 1: Use optical character recognition (OCR) technology to extract subtitles from the video, and then splice the subtitles with the video title to obtain complete text information; Based on the complete text information, the semantic features of the text are extracted through the pre-trained language model (BERT); Step 2: Use the open source video editing toolkit (MoviePy) to extract voice information from the video, and then use the pre-trained voice model (HuBERT) to extract the voice information to obtain the emotional features of the voice; Step 3: Extract key frames in the video by calculating the difference between adjacent frames, and then use the pre-trained sliding window model (Swin Transformer) to extract the key frames to obtain visual features; Step 4: Use the pre-trained stance detection model (StanceBERTa) to extract the stance features of the video comments, and then combine the stance features of the video and the weight of the comments to obtain the global comment features of the video; Step 5: extract the social features of the video publisher, and then use the social features of the video publisher to obtain the social features of the user; Step 6: Use an unbiased scene graph generation method to convert the key frame sequence of the video into a scene graph sequence, extract the scene graph sequence through a scene graph attention network, and obtain the temporal scene graph features; Step 7: Fuse the semantic features of the text, the emotional features of the speech, the visual features, the global comment features of the video, the social features of the user, and the temporal scene graph features to obtain a global multimodal feature representation; Step 8: Input the global multimodal feature representation into the multi-layer perceptron to obtain the prediction result.

[0006] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention improves the accuracy and robustness of fake news video detection by integrating text, voice, vision, comments and social features; 2. The present invention effectively improves the semantic understanding ability of scene graphs by introducing external knowledge graphs for knowledge enhancement; 3. The present invention captures the temporal features in the video through a dynamic scene graph attention network, thereby improving the ability of early identification of fake news.

[0007] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 This is a flow chart of the fake news video detection method based on knowledge-enhanced graph attention network proposed by the present invention; Figure 2 This is a model architecture diagram of the fake news video detection method based on knowledge-enhanced graph attention network proposed in this invention. DETAILED DESCRIPTION

[0009] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0010] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0011] See also Figure 1 , an embodiment of the present invention proposes a false news video detection method based on knowledge-enhanced graph attention network, the method comprising the following steps: Step 1: Use optical character recognition technology to extract subtitles from the video, and then splice the subtitles with the video title to obtain complete text information; Based on the complete text information, the semantic features of the text are extracted through the pre-trained language model; In step 1, based on the complete text information, the semantic features of the text are extracted through the pre-trained language model. The corresponding relationship in the process is: ; in, Represents the semantic features of the text, Represents complete text information. Indicates that it has been extracted by a pre-trained language model.

[0012] Step 2: Use an open source toolkit for video editing to extract voice information from the video, and then use a pre-trained voice model to extract the voice information to obtain the emotional features of the voice; In step 2, an open source toolkit for video editing is used to extract voice information from the video, and then a pre-trained voice model is used to extract the voice information to obtain the emotional features of the voice. The specific steps are as follows: The voice information is subjected to noise reduction processing to obtain the noise-reduced voice information. The relationship between the corresponding process is: ; in, Indicates the voice information after noise reduction. It means that it has been processed by the noise reduction function. Indicates voice information; The denoised speech information is normalized to obtain the normalized speech information. The corresponding relationship is: ; in, represents the normalized voice information, Indicates that it has been processed by the normalization function; The normalized speech information is input into the pre-trained speech model to obtain the emotional characteristics of the speech. The corresponding relationship in the process is: ; in, Indicates the emotional characteristics of speech. Represents pre-trained speech model processing.

[0013] Step 3: extract key frames in the video by calculating the difference between adjacent frames, and then use the pre-trained sliding window model to extract the key frames to obtain visual features; In step 3, the key frames in the video are extracted by calculating the difference between adjacent frames, and then the pre-trained sliding window model is used to extract the key frames to obtain visual features. The specific steps are as follows: The frame difference calculation is performed on the video frame sequence to obtain the difference value between adjacent frames. The relationship between the corresponding process is: ; in, Indicates Frame and The difference between frames, Indicates that after being processed by the inter-frame difference calculation function, Indicates the video frame, Indicates the video frame; According to the difference value, the key frames are screened by a set difference value threshold to obtain a key frame sequence; The key frame sequence is input into the pre-trained sliding window model to obtain the visual features of the key frames. The relationship between the corresponding process is: ; in, Indicates The visual features of key frames, represents the pre-trained sliding window model processing, Indicates keyframes; Collect the visual features of all key frames to obtain a visual feature set of the key frames; The visual feature set of the key frame is processed through the self-attention mechanism to obtain the visual feature. The relationship between the corresponding process is: ; in, Represents visual features, It means that it has been processed by the self-attention mechanism. A collection of visual features representing a keyframe.

[0014] Step 4: Use the pre-trained stance detection model to extract the stance features of the video comments, and then combine the stance features of the video and the weight of the comments to obtain the global comment features of the video; In step 4, the pre-trained stance detection model is used to extract the stance features of the video comments, and then the global comment features of the video are obtained by combining the stance features of the video and the weight values ​​of the comments. The specific steps are as follows: Collect a comment dataset related to the video. Each comment in the comment dataset consists of the text content of the comment and the number of likes. The pre-trained stance detection model is used to encode the text content of each comment to obtain stance features. The corresponding relationship is: ; in, Indicates the characteristics of the position, It means processed by the pre-trained stance detection model. Indicates The text content of the comment; The weight of the comment is calculated based on the number of likes for each comment. The corresponding relationship is: ; in, Indicates The weight of the comments, Indicates Number of likes for comments, Indicates the total number of comments on the video. Indicates Number of likes for a comment; Combine the weight values ​​of the comments to perform weighted summation of all stance features to obtain the global comment features of the video. The corresponding relationship is: ; in, Represents the global comment features of the video.

[0015] Step 5: extract the social features of the video publisher, and then use the social features of the video publisher to obtain the social features of the user; In step 5, the social features of the video publisher are extracted, and then the social features of the user are obtained by using the social features of the video publisher. The specific steps are as follows: The publisher's related attributes are obtained from the video's metadata, and then the user's initial feature vector is constructed based on the publisher's related attributes. The corresponding process has the following relationship: ; in, represents the user's initial feature vector, Indicates the number of fans, Indicates the number of friends, Indicates the number of published videos. Represents the interactive characteristics of the publisher; The user's initial feature vector is normalized to obtain the user's social features. The corresponding relationship is: ; in, Represents the social characteristics of the user.

[0016] Step 6: Use an unbiased scene graph generation method to convert the key frame sequence of the video into a scene graph sequence, extract the scene graph sequence through a scene graph attention network, and obtain the temporal scene graph features; In step 6, the key frame sequence of the video is converted into a scene graph sequence using an unbiased scene graph generation method, and the scene graph sequence is extracted through a scene graph attention network to obtain the temporal scene graph features. The specific steps are as follows: Apply the pre-trained object detection model to the keyframe sequence to detect the visual objects in each keyframe, where each visual object includes a category label and a bounding box. The relationship between the corresponding process is: ; in, Indicates visual objects in keyframes, Represents the detection process of the pre-trained object detection model; Modeling the spatial and semantic relationships between visual objects to generate a preliminary relationship graph between visual objects; Further optimize the preliminary relationship graph between visual objects to generate a scene graph corresponding to the key frames; Arrange the scene graphs corresponding to all key frames in chronological order to form a scene graph sequence; Based on the nodes and relationships in the scene graph sequence, a set of query keywords is obtained, and the relationship between the corresponding process is: ; in, Represents a set of query keywords. All represent query keywords. Represents a node, To express a relationship; Use the query keyword set in the external knowledge graph to retrieve knowledge triples; Filter the knowledge triples to retain the knowledge related to the objects and relations in the scene graph; The knowledge related to the objects and relations in the scene graph is added to the scene graph sequence as a supplementary node to obtain the scene graph sequence after knowledge expansion; The scene graph attention network is constructed using the nodes in the scene graph sequence. The relationship of the construction process is: ; in, represents the scene graph attention network, Representation Node Aggregate the updated features, represents the number of multi-head attention, Represents a splicing operation, Representation Node The set of adjacent nodes of Indicated in Adjacent nodes in attention type For Node The attention weight, , and They all represent parameter matrices, Representation Node The characteristic representation of Representation Node The characteristic representation of Represents edge The characteristic representation of It means that after normalization operation, It means that it has been processed by the activation function. represents the transpose of the attention weights; Based on the structure of recurrent neural network, the output of the scene graph attention network at the previous moment is used as the input of the scene graph attention network at the next moment to construct a dynamic scene graph attention network; The scene graph sequence after knowledge expansion is input into the dynamic scene graph attention network to obtain the temporal scene graph features. The relationship between the corresponding process is: ; in, Represents the temporal scene graph features, It means that it is processed by the self-attention mechanism. It means that it has been processed by the dynamic scene graph attention network. Both represent scene graphs after knowledge expansion.

[0017] Step 7: Fuse the semantic features of the text, the emotional features of the speech, the visual features, the global comment features of the video, the social features of the user, and the temporal scene graph features to obtain a global multimodal feature representation; In step 7, the semantic features of the text, the emotional features of the speech, the visual features, the global comment features of the video, the social features of the user, and the temporal scene graph features are fused to obtain a global multimodal feature representation. The specific steps are as follows: A fully connected neural network is used to align the semantic features of text, emotional features of speech, visual features, global comment features of video, social features of users, and temporal scene graph features into the same feature space to obtain the aligned feature representation. The relationship between the corresponding processes is: ; in, represents the aligned feature representation, Indicates processing by a fully connected neural network; Represents the semantic features of text , emotional characteristics of speech , visual features , global comment features of the video , social characteristics of users and temporal scene graph features Any of the following: Based on the aligned feature representation, the modal weight is obtained through the self-attention mechanism. The corresponding relationship is: ; in, represents the modal weight, It means that it has been processed by evaluating the importance function of modal features; According to the modal weights, the aligned feature representations are weighted and summed to obtain the global multimodal feature representation. The corresponding process has the following relationship: ; in, Represents global multimodal feature representation.

[0018] Specifically, in this step, the evaluation of the modal feature importance function is achieved through self-attention.

[0019] Step 8: Input the global multimodal feature representation into the multilayer perceptron to obtain the prediction result; In step 8, the global multimodal feature representation is input into the multilayer perceptron to obtain the prediction result. The relationship between the corresponding process is: ; in, Represents the prediction result, It means processed by multi-layer perceptron. represents the linear layer parameters, Represents the bias term.

[0020] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0021] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0022] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A fake news video detection method based on knowledge-enhanced graph attention network, characterized in that: The method comprises the following steps: Step 1: Use optical character recognition technology to extract subtitles from the video, and then splice the subtitles with the video title to obtain complete text information; Based on the complete text information, the semantic features of the text are extracted through the pre-trained language model; Step 2: Use an open source toolkit for video editing to extract voice information from the video, and then use a pre-trained voice model to extract the voice information to obtain the emotional features of the voice; Step 3: extract key frames in the video by calculating the difference between adjacent frames, and then use the pre-trained sliding window model to extract the key frames to obtain visual features; Step 4: Use the pre-trained stance detection model to extract the stance features of the video comments, and then combine the stance features of the video and the weight of the comments to obtain the global comment features of the video; Step 5: extract the social features of the video publisher, and then use the social features of the video publisher to obtain the social features of the user; Step 6: Use an unbiased scene graph generation method to convert the key frame sequence of the video into a scene graph sequence, extract the scene graph sequence through a scene graph attention network, and obtain the temporal scene graph features; Step 7: Fuse the semantic features of the text, the emotional features of the speech, the visual features, the global comment features of the video, the social features of the user, and the temporal scene graph features to obtain a global multimodal feature representation; Step 8: Input the global multimodal feature representation into the multi-layer perceptron to obtain the prediction result.

2. The method for detecting fake news videos based on knowledge-enhanced graph attention network according to claim 1 is characterized in that: In step 1, based on the complete text information, the semantic features of the text are extracted through the pre-trained language model, and the relationship between the corresponding process is: ； in, Represents the semantic features of the text, Represents complete text information. Indicates that it has been extracted by a pre-trained language model.

3. The method for detecting fake news videos based on knowledge-enhanced graph attention network according to claim 2 is characterized in that: In step 2, an open source toolkit for video editing is used to extract voice information from the video, and then a pre-trained voice model is used to extract the voice information to obtain the emotional features of the voice. The specific steps are as follows: The voice information is subjected to noise reduction processing to obtain the noise-reduced voice information. The relationship between the corresponding process is: ； in, Indicates the voice information after noise reduction. It means that it has been processed by the noise reduction function. Indicates voice information; The denoised speech information is normalized to obtain the normalized speech information. The corresponding relationship is: ； in, represents the normalized speech information, Indicates that it has been processed by the normalization function; The normalized speech information is input into the pre-trained speech model to obtain the emotional characteristics of the speech. The corresponding relationship in the process is: ； in, Indicates the emotional characteristics of speech. Represents pre-trained speech model processing.

4. The method for detecting fake news videos based on knowledge-enhanced graph attention network according to claim 3 is characterized in that: In step 3, the key frames in the video are extracted by calculating the difference between adjacent frames, and then the key frames are extracted using a pre-trained sliding window model to obtain visual features. The specific steps are as follows: The frame difference calculation is performed on the video frame sequence to obtain the difference value between adjacent frames. The relationship between the corresponding process is: ； in, Indicates Frame and The difference between frames, Indicates that after being processed by the inter-frame difference calculation function, Indicates the video frame, Indicates the video frame; According to the difference value, the key frames are screened by a set difference value threshold to obtain a key frame sequence; The key frame sequence is input into the pre-trained sliding window model to obtain the visual features of the key frames. The relationship between the corresponding process is: ； in, Indicates The visual features of key frames, represents the pre-trained sliding window model processing, Indicates keyframes; Collect the visual features of all key frames to obtain a visual feature set of the key frames; The visual feature set of the key frame is processed through the self-attention mechanism to obtain the visual feature. The relationship between the corresponding process is: ； in, Represents visual features, It means that it has been processed by the self-attention mechanism. A collection of visual features representing a keyframe.

5. The method for detecting fake news videos based on knowledge-enhanced graph attention network according to claim 4 is characterized in that: In step 4, the stance features of the video comments are extracted using the pre-trained stance detection model, and then the global comment features of the video are obtained by combining the stance features of the video and the weight values ​​of the comments. The specific steps are as follows: Collect a comment dataset related to the video. Each comment in the comment dataset consists of the text content of the comment and the number of likes. The pre-trained stance detection model is used to encode the text content of each comment to obtain stance features. The corresponding relationship is: ； in, Indicates the characteristics of the position, It means processed by the pre-trained stance detection model. Indicates The text content of the comment; The weight of the comment is calculated based on the number of likes for each comment. The corresponding relationship is: ； in, Indicates The weight of the comments, Indicates Number of likes for comments, Indicates the total number of comments on the video. Indicates Number of likes for a comment; Combine the weight values ​​of the comments to perform weighted summation of all stance features to obtain the global comment features of the video. The corresponding relationship is: ； in, Represents the global comment features of the video.

6. The method for detecting fake news videos based on knowledge-enhanced graph attention network according to claim 5 is characterized in that: In step 5, the social features of the video publisher are extracted, and then the social features of the user are obtained by using the social features of the video publisher. The specific steps are as follows: The publisher's related attributes are obtained from the video's metadata, and then the user's initial feature vector is constructed based on the publisher's related attributes. The corresponding process has the following relationship: ； in, represents the user's initial feature vector, Indicates the number of fans, Indicates the number of friends, Indicates the number of published videos. Represents the interactive characteristics of the publisher; The user's initial feature vector is normalized to obtain the user's social features. The corresponding relationship is: ； in, Represents the social characteristics of the user.

7. The method for detecting fake news videos based on knowledge-enhanced graph attention network according to claim 6 is characterized in that: In step 6, the key frame sequence of the video is converted into a scene graph sequence using an unbiased scene graph generation method, and the scene graph sequence is extracted through a scene graph attention network to obtain a temporal scene graph feature. The specific steps are: as follows: Apply the pre-trained object detection model to the keyframe sequence to detect the visual objects in each keyframe, where each visual object includes a category label and a bounding box. The relationship between the corresponding process is: ； in, Indicates visual objects in keyframes, Represents the detection process of the pre-trained object detection model; Modeling the spatial and semantic relationships between visual objects to generate a preliminary relationship graph between visual objects; Further optimize the preliminary relationship graph between visual objects to generate a scene graph corresponding to the key frames; Arrange the scene graphs corresponding to all key frames in chronological order to form a scene graph sequence; Based on the nodes and relationships in the scene graph sequence, a set of query keywords is obtained, and the relationship between the corresponding process is: ； in, Represents a set of query keywords. All represent query keywords. Represents a node, To express a relationship; Use the query keyword set in the external knowledge graph to retrieve knowledge triples; Filter the knowledge triples to retain the knowledge related to the objects and relations in the scene graph; The knowledge related to the objects and relations in the scene graph is added to the scene graph sequence as a supplementary node to obtain the scene graph sequence after knowledge expansion; The scene graph attention network is constructed using the nodes in the scene graph sequence. The relationship of the construction process is: ； in, represents the scene graph attention network, Representation Node Aggregate the updated features, represents the number of multi-head attention, Represents a splicing operation, Representation Node The set of adjacent nodes of Indicated in Adjacent nodes in attention type For Node The attention weight, , and They all represent parameter matrices, Representation Node The characteristic representation of Representation Node The characteristic representation of Represents edge The characteristic representation of It means that after normalization operation, It means that it has been processed by the activation function. represents the transpose of the attention weights; Based on the structure of recurrent neural network, the output of the scene graph attention network at the previous moment is used as the input of the scene graph attention network at the next moment to construct a dynamic scene graph attention network; The scene graph sequence after knowledge expansion is input into the dynamic scene graph attention network to obtain the temporal scene graph features. The relationship between the corresponding process is: ； in, Represents the temporal scene graph features, It means that it is processed by the self-attention mechanism. It means that it has been processed by the dynamic scene graph attention network. Both represent scene graphs after knowledge expansion.

8. The method for detecting fake news videos based on knowledge-enhanced graph attention network according to claim 7 is characterized in that: In step 7, the semantic features of the text, the emotional features of the speech, the visual features, the global comment features of the video, the social features of the user and the temporal scene graph features are integrated to obtain a global multimodal feature representation. The specific steps are as follows: A fully connected neural network is used to align the semantic features of text, emotional features of speech, visual features, global comment features of video, social features of users, and temporal scene graph features into the same feature space to obtain the aligned feature representation. The relationship between the corresponding processes is: ； in, represents the aligned feature representation, Indicates processing by a fully connected neural network; Represents the semantic features of text , emotional characteristics of speech , visual features , global comment features of the video , social characteristics of users and temporal scene graph features Any of the following: Based on the aligned feature representation, the modal weight is obtained through the self-attention mechanism. The corresponding relationship is: ； in, represents the modal weight, It means that it has been processed by evaluating the importance function of modal features; According to the modal weights, the aligned feature representations are weighted and summed to obtain the global multimodal feature representation. The corresponding process has the following relationship: ； in, Represents global multimodal feature representation.

9. The method for detecting fake news videos based on knowledge-enhanced graph attention network according to claim 8, characterized in that: In step 8, the global multimodal feature representation is input into the multilayer perceptron to obtain the prediction result. The relationship between the corresponding process is: ； in, Represents the prediction result, It means processed by multi-layer perceptron. represents the linear layer parameters, Represents the bias term.

Images