False news detection method and device based on scene graph semantic enhancement

Abstract

The invention relates to a false news detection method and device based on scene graph semantic enhancement. The method comprises the steps of constructing a semantic enhancement frame, performing reverse training on the semantic enhancement frame according to a preset news information sample and a loss function to obtain a trained semantic enhancement frame, inputting news information to be detected into the trained semantic enhancement frame, and outputting a false information detection result of the news information to be detected. By adopting the method, the false news detection precision can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for detecting fake news based on scene graph semantic enhancement. Background Art

[0002] With the rapid development of social media, the cost of information production and dissemination has significantly decreased. However, this has also led to the proliferation of fake news, which has had serious negative impacts on society, the economy, and politics. Faced with a massive amount of news and carefully disguised false information, manual identification of fake news is inefficient and difficult, making fake news detection a research hotspot in recent years. In the task of fake news detection, cross-modal semantic consistency is a key criterion. Therefore, how to effectively extract the semantic information contained in visual content has become a core challenge in improving the effectiveness of multimodal fake news detection.

[0003] Currently, there are emerging approaches to extracting visual entities from images to obtain semantic information, and to achieve semantic alignment by simultaneously extracting entities from images and text. These empirical studies collectively demonstrate that extracting semantic information through visual object understanding can effectively promote model optimization in this field. However, such approaches deconstruct image semantics into discrete sets of entities and fail to capture the interactive relationships between entities. As complex semantic carriers, the deep semantics of news images essentially stems from the structured integration of spatial configurations, action-oriented dependencies, and logical interdependencies between visual entities. Summary of the Invention

[0004] Based on this, it is necessary to provide a fake news detection method and device based on scene graph semantic enhancement to address the above technical problems.

[0005] A method for detecting fake news based on scene graph semantic enhancement, the method comprising:

[0006] Construct a semantic enhancement framework; the input of the semantic enhancement framework is news information; the news information consists of text and image, and feature extraction is performed on the text to obtain fine-grained text features; the image is represented by target entities and relationships, and feature embedding is performed on the target entity using a pre-trained visual model to obtain entity features, and feature encoding is performed on the relationship using a pre-trained encoding model to obtain relationship features, and based on the fine-grained text features, the entity features and the relationship features, a multi-head attention mechanism is used to enhance the entity features and the relationship features to obtain enhanced relationship features and enhanced entity features; the fine-grained text features, the enhanced entity features and the enhanced relationship features are spliced to obtain a fine-grained feature representation; the coarse-grained features of the text and the image are spliced to obtain a coarse-grained feature representation, and the fine-grained feature representation and the coarse-grained feature representation are fused and input into a classifier to classify the news information;

[0007] Reverse training the semantic enhancement framework according to a preset news information sample and loss function to obtain a trained semantic enhancement framework;

[0008] The news information to be detected is input into the trained semantic enhancement framework, and the false information detection result of the news information to be detected is output.

[0009] In one embodiment, the method further includes: using a BERT model to extract features from the text to obtain fine-grained text features.

[0010] In one embodiment, the method further includes: extracting a visual feature vector of the region corresponding to the target entity using a pre-trained ResNet50 model, and obtaining the entity feature as follows:

[0011]

[0012] in, Represents the feature embedding of the target entity i, and the target entity is represented by o i , represents the position coordinates of the i-th target entity, m is the maximum number of target entities in the image, s represents the number of target entities in the image, d r It is the feature dimension of the last layer output of the ResNet50 model.

[0013] In one embodiment, the further step includes: inputting the relationship into a BERT model for feature encoding, and obtaining the relationship feature as follows:

[0014]

[0015] Among them, t r Represents the relationship feature, r jrepresents the jth relation, p represents the number of relations in the image, n is the maximum number of relations in the image, d t It is the feature dimension of the last layer output of the BERT model.

[0016] In one embodiment, the method further includes: constructing a query matrix, a key matrix and a value matrix of a multi-head attention mechanism based on the fine-grained text features, the entity features and the relationship features, and respectively h =r p W Q , K h =t p W K 、V r =v p W V , W Q 、W K 、W V Represents learning parameters; the entity features are enhanced according to the query matrix, key matrix and value matrix, and the enhanced entity features and enhanced relationship features are obtained respectively:

[0017]

[0018] Among them, v O represents the enhanced entity feature, t R represents the enhanced relational features, H represents the number of attention heads, q is the embedding dimension, and f(·) is a fully connected layer.

[0019] In one embodiment, the method further includes: concatenating the fine-grained text features, the enhanced entity features, and the enhanced relationship features to obtain a fine-grained feature representation as follows:

[0020]

[0021] Among them, S d Represents fine-grained features.

[0022] In one embodiment, the method further includes: concatenating the coarse-grained features of the text and the image to obtain a coarse-grained feature representation as follows:

[0023]

[0024] Among them, S w represents the coarse-grained feature, t c represents the coarse-grained text representation, v c represents a coarse-grained image representation.

[0025] In one embodiment, the method further includes: using cosine similarity to measure the coarse-grained text representation and the coarse-grained image representation in the coarse-grained features, and obtaining the coarse-grained similarity as follows:

[0026]

[0027] According to the coarse-grained similarity, the fusion weight is calculated as:

[0028] weight = 1-Linear(sim)

[0029] Among them, weight represents the fusion weight, and Linear(·) represents the linear network layer;

[0030] According to the fusion weight, the fine-grained feature representation and the coarse-grained feature representation are fused as follows:

[0031]

[0032] Among them, S represents the fusion feature.

[0033] In one embodiment, the loss function is:

[0034] L=-∑[ylog(p)+(1-y)log(1-p)]

[0035] y represents the label distribution of the news information sample, and p represents the probability that the news information sample is predicted to be fake news.

[0036] A fake news detection device based on scene graph semantic enhancement, the device comprising:

[0037] A model construction module is used to construct a semantic enhancement framework; the input of the semantic enhancement framework is news information; the news information consists of text and images, and features are extracted from the text to obtain fine-grained text features; the image is represented by a target entity and a relationship, and a pre-trained visual model is used to embed features of the target entity to obtain entity features, and a pre-trained encoding model is used to encode features of the relationship to obtain relationship features, and a multi-head attention mechanism is used to enhance the entity features and the relationship features based on the fine-grained text features, the entity features, and the relationship features to obtain enhanced relationship features and enhanced entity features; the fine-grained text features, the enhanced entity features, and the enhanced relationship features are spliced to obtain a fine-grained feature representation; the coarse-grained features of the text and the image are spliced to obtain a coarse-grained feature representation, and the fine-grained feature representation and the coarse-grained feature representation are fused and input into a classifier to classify the news information;

[0038] A training module, configured to perform reverse training on the semantic enhancement framework based on a preset news information sample and loss function to obtain a trained semantic enhancement framework;

[0039] The detection module is used to input the news information to be detected into the trained semantic enhancement framework and output the false information detection result of the news information to be detected.

[0040] The above-mentioned fake news detection method and device based on scene graph semantic enhancement introduces scene graphs into the fake news detection model, captures entity relationship information by generating entity relationship descriptions, deeply understands the deep semantic information of images, eliminates the semantic gap between images and text, and then achieves relationship-level alignment between images and text based on the fine-grained feature fusion method of the attention mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 1 is a flow chart of a method for detecting fake news based on scene graph semantic enhancement in one embodiment;

[0042] Figure 2 A schematic diagram of a semantic enhancement framework structure in one embodiment;

[0043] Figure 3 Schematic diagram showing the effectiveness of the SGSE relationship in a microblog dataset in another embodiment; wherein (a) SESE-w / o-rel distribution, (b) SGSE distribution;

[0044] Figure 4 2 is a structural block diagram of a fake news detection device based on scene graph semantic enhancement in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0046] In one embodiment, Figure 1 As shown, a fake news detection method based on scene graph semantic enhancement is provided, comprising the following steps:

[0047] Step 102: construct a semantic enhancement framework.

[0048] In this framework, the input is news information; news information consists of text and images, and features are extracted from the text to obtain fine-grained text features; images are represented by target entities and relationships, and a pre-trained visual model is used to embed features of the target entity to obtain entity features, and a pre-trained encoding model is used to encode features of the relationship to obtain relationship features. Based on the fine-grained text features, entity features, and relationship features, a multi-head attention mechanism is used to enhance the entity features and relationship features to obtain enhanced relationship features and enhanced entity features; the fine-grained text features, enhanced entity features, and enhanced relationship features are spliced to obtain a fine-grained feature representation; the coarse-grained features of the text and image are spliced to obtain a coarse-grained feature representation, and the fine-grained feature representation and the coarse-grained feature representation are fused and input into the classifier to classify the news information.

[0049] It is worth noting that the image in this step is generally a scene graph, which can provide richer semantic information than traditional image representation. The scene graph is defined as G = {O, R}, where O = [o1, o2, ..., o s ] represents the detected target entity, and oi includes the position coordinates of the i-th entity. In addition, s represents the number of entities in the scene graph. Similarly, R = [r1, r2, ..., r p ] is a set of relations, where each r i represents a relation formed by entity-relation-entity triples, such as “a man is walking on the sidewalk.” Meanwhile, p represents the number of relations in the scene graph.

[0050] Step 104 , reverse training is performed on the semantic enhancement framework according to the preset news information samples and loss function to obtain a trained semantic enhancement framework.

[0051] In this step, the news information sample includes news information and corresponding labels. Since this application is for fake news detection, the labels can be "fake news" or "not fake news," and can also be set according to actual needs. The loss function can be a cross-entropy loss function. The semantic enhancement framework is trained through reverse training to enable it to accurately classify news information.

[0052] Step 106: input the news information to be detected into the trained semantic enhancement framework, and output the false information detection result of the news information to be detected.

[0053] In the above-mentioned fake news detection method based on scene graph semantic enhancement, the scene graph is introduced into the fake news detection model. By generating entity relationship descriptions, entity relationship information is captured, the deep semantic information of the image is deeply understood, and the semantic gap between image and text is eliminated. Then, a fine-grained feature fusion method based on the attention mechanism is used to achieve relationship-level alignment between image and text.

[0054] In one embodiment, the BERT model is used to extract features from the text to obtain fine-grained text features. In this embodiment, a piece of news is defined as n = {t, v}, where t represents text and v represents image. Unlike traditional unidirectional models, the BERT model can fully consider contextual information to more accurately predict and understand the meaning of words. After processing by the BERT encoder, the final output is a fine-grained text feature vector t p .

[0055] In one embodiment, the scene graph is represented by relations and entities. For entities, a pre-trained ResNet50 model is used to extract the visual feature vector of the corresponding area. Compared with other visual models (such as VGG19 and Transformer), ResNet50 performs better in stability and efficiency due to its residual structure and fewer parameters. The visual area corresponding to each entity is input into the model. Since the number of visual entities in different images is different, the dimensions are aligned by zero padding, and the pre-trained ResNet50 model is used to extract the visual feature vector of the area corresponding to the target entity, and the entity features are obtained as follows:

[0056]

[0057] in, Represents the feature embedding of the target entity i, and the target entity is represented by o i , represents the position coordinates of the i-th target entity, m is the maximum number of target entities in the image, s represents the number of target entities in the image, d r It is the feature dimension of the last layer output of the ResNet50 model, and the corresponding entity image area is represented as Vision(o i ).

[0058] For the relationship features, the relationship is input into the BERT model for feature encoding, and the relationship features are obtained as follows:

[0059]

[0060] Among them, t r Represents the relationship feature, r j represents the jth relation, p represents the number of relations in the image, n is the maximum number of relations in the image, d t It is the feature dimension of the last layer output of the BERT model.

[0061] Through fine-grained feature extraction, fine-grained features have been constructed for visual and textual data. As the subject of news, text contains more important information in the news. Not all entities and relations are highly relevant to the text. Therefore, a cross-modal attention mechanism is introduced to screen out entities and relations that are more important to the text. Specifically, three projection heads with the same structure are first used to map the visual, textual and relational features into a unified space. Each projection head includes a fully connected layer, a normalization layer, a ReLU activation function and a Dropout layer. In this way, a low-dimensional entity feature v is obtained. p , text features t p and relational features r p The cross-attention mechanism consists of four layers: a multi-head attention layer, a fully connected layer, and two residual connections and normalization layers.

[0062] In another embodiment, the query matrix, key matrix and value matrix of the multi-head attention mechanism are constructed based on the fine-grained text features, entity features and relationship features, respectively. h =r p W Q , K h =t p W K 、V r =v p W V , W Q 、W K 、W V Represents learning parameters; according to the query matrix, key matrix and value matrix, the entity features are enhanced, and the enhanced entity features and enhanced relationship features are obtained respectively:

[0063]

[0064] Among them, v O represents the enhanced entity feature, t R represents the enhanced relational features, H represents the number of attention heads, q is the embedding dimension, and f(·) is a fully connected layer.

[0065] In one embodiment, the fine-grained text features, enhanced entity features, and enhanced relationship features are concatenated to obtain a fine-grained feature representation as follows:

[0066]

[0067] Among them, S d Represents fine-grained features.

[0068] In this embodiment, feature stitching can be achieved by stitching through average pooling layers.

[0069] In one embodiment, a pre-trained model CLIP is used for coarse-grained feature extraction. CLIP maps images and texts to a shared feature space through contrastive learning, allowing different modal information to be correlated with each other, freeing the model from dependence on details and possessing strong generalization capabilities. The coarse-grained features of text and images extracted by CLIP are respectively represented by t c and v c express.

[0070] Specifically, with the help of CLIP's multimodal capabilities, the coarse-grained features of text and image are spliced together to obtain the coarse-grained feature representation:

[0071]

[0072] Among them, S w represents the coarse-grained feature, t c represents the coarse-grained text representation, v c represents a coarse-grained image representation.

[0073] In order to more effectively fuse features of different granularities, a projection layer consisting of two fully connected neural network layers is designed. After processing by the projection layer, the joint representation of coarse-grained and fine-grained features is mapped to a unified space. Considering the different importance of coarse-grained and fine-grained features in different news articles, cosine similarity is used to measure the fusion weight. When t c and v c When the consistency is high, it indicates that the coarse-grained features are more important.

[0074] The cosine similarity is used to measure the coarse-grained text representation and coarse-grained image representation in the coarse-grained features, and the coarse-grained similarity is obtained as follows:

[0075]

[0076] Because fine-grained module networks are more complex, models often emphasize fine-grained cross-modal representations. To achieve a balanced fusion of coarse-grained and fine-grained multimodal features, the cosine similarity results are passed through a linear network layer and then converted into fine-grained weights. This approach adaptively adjusts the multi-grained fusion weights based on coarse-grained similarity.

[0077] Specifically, the fusion weight is calculated as:

[0078] weight = 1-Linear(sim)

[0079] Among them, weight represents the fusion weight, and Linear(·) represents the linear network layer. Finally, according to the fusion weight, the fine-grained feature representation and the coarse-grained feature representation are fused as follows:

[0080]

[0081] Among them, S represents the fusion feature.

[0082] In one embodiment, the obtained multimodal joint representation S is input into a classifier to obtain the predicted probability of news authenticity. The classifier consists of a fully connected layer and a softmax layer. Its loss function is cross entropy:

[0083] L=-∑[ylog(p)+(1-y)log(1-p)]

[0084] y represents the label distribution of the news information sample, and p represents the probability that the news information sample is predicted to be fake news.

[0085] In summary, the semantic enhancement framework is as follows Figure 2 As shown, Figure 2 The upper part uses CLIP to perform coarse-grained extraction and fusion of text and images, and the lower part contains three core modules: fine-grained image feature extraction based on scene graph, fine-grained semantic fusion based on text filtering, and multi-granularity feature integration module.

[0086] based on Figure 2 The semantic enhancement framework presented here is evaluated using three public datasets: the Weibo dataset, which contains Chinese news from the Sina Weibo platform, whose authenticity is officially verified by Sina Weibo; the GossipCop and PolitiFact datasets, which are English datasets containing entertainment news and political news, respectively. Detailed distribution of each dataset is shown in Table 1: Descriptive Statistics of the Dataset.

[0087] Table 1 Descriptive statistics of the dataset

[0088]

[0089] Accuracy, precision, recall, and F1 value are selected as evaluation indicators of the model. Each dataset is divided into a training set and a test set. When generating the scene graph, we only consider the relations between the subject, object, and relation words with a probability greater than 0.1. For the Chinese dataset, we select the BERT-base-chinese and Chinese-CLIP models and limit the text length to 200 characters; for the English dataset, we use the BERT-base-uncased and CLIP models and limit the text length to 395 characters. In the experiments, we keep the parameters of the pre-trained BERT, ResNet, RelTR, and CLIP models fixed.

[0090] In order to verify the performance of the SGSE model, eight state-of-the-art baseline methods were introduced as benchmark comparisons in the experimental evaluation. These methods include:

[0091] MVAE: This method applies variational autoencoders for fake news detection.

[0092] SAFE: This method studies the correlation between different modalities and applies it to fake news detection.

[0093] CAFE: This method improves detection accuracy by adaptively aggregating unimodal features and cross-modal correlations, and quantifies the ambiguity between text and image modalities using KL divergence.

[0094] ●CFFN: This method performs fine-grained fusion of cross-modal information based on word correlation;

[0095] MRAN: This method uses a relation-aware attention network to generate high-level fusion features of text and images.

[0096] DistilBertt: This method uses the relevance of user-generated content and user-shared content as a clue to improve the accuracy of false information detection.

[0097] QMFND: This method proposes a quantum convolutional neural network to reduce the time sensitivity of the data, thereby improving the detection accuracy of the model;

[0098] ●CAF-ONDD: This method introduces complementary attention fusion to enhance the model’s ability to detect subtle differences and inconsistencies.

[0099] The experimental results are shown in Table 2, where the highest performance metrics are highlighted in bold. "-" in the table indicates missing data points not provided in the reference. Among all models, the semantic enhancement framework achieved the best detection performance. It achieved accuracies of 91.1%, 91.8%, and 88.1%, respectively, representing improvements of 1.0%, 0.3%, and 4.4% over the state-of-the-art models. It also ranked among the top in the other three metrics. This result can be explained from three aspects: First, the semantic enhancement framework utilizes image entity relationships as supplementary cues to generate a scene graph, enhancing semantic information and enabling the model to more comprehensively extract image information. Second, through a fine-grained fusion mechanism based on text filtering, we achieve semantic alignment between text and image at the relational level. Finally, we model text and image from a multi-granular perspective, resulting in a more comprehensive feature representation. The experimental results demonstrate the effectiveness of the semantic enhancement framework in multimodal fake news detection.

[0100] Table 2 Comparative experiments on three datasets

[0101]

[0102] In addition, we conducted five sets of ablation experiments to verify the effectiveness of different components in the SGSE model. The specific experimental settings are shown in Table 3.

[0103] The experimental results show that whenever any component in the semantic enhancement framework is removed, the accuracy of the model decreases, thus verifying the effectiveness of each module. In addition, fine-grained multimodal features can more effectively improve the news detection performance of the semantic enhancement framework than coarse-grained multimodal features. On the Gossipcop dataset, the decrease in model accuracy after removing the relationship features is relatively small. This may be because most of the images in the Gossipcop dataset contain fewer image entities and more single relationships. In order to further analyze the effectiveness of the relationship, we performed a visual analysis on the Weibo dataset. Figure 3 As shown in Figure 3, the inclusion of relations can more clearly distinguish fake news and real news samples, thereby improving the classification performance.

[0104] SGSE-w / o-rel: removes relational features and uses only the visual regions corresponding to entities as fine-grained image features; SGSE stands for Semantic Enhancement Framework.

[0105] SGSE-w / o-Att: removes the cross-attention mechanism and only fuses fine-grained features into a common space through the projection head;

[0106] SGSE-w / o-sim: omits weights when fusing fine-grained features with coarse-grained features, and instead uses direct concatenation.

[0107] SGSE-F-only: Removes the coarse-grained module and relies only on fine-grained features for fake news detection;

[0108] SGSE-C-only: Removes the fine-grained module and uses only coarse-grained features for fake news detection.

[0109] Table 3. Ablation experiment results of SGSE on three datasets

[0110]

[0111] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0112] In one embodiment, Figure 4 As shown, a fake news detection device based on scene graph semantic enhancement is provided, including: a model building module 402, a training module 404 and a detection module 406, wherein:

[0113] The model construction module 402 is used to construct a semantic enhancement framework; the input of the semantic enhancement framework is news information; the news information is composed of text and images, and features are extracted from the text to obtain fine-grained text features; the image is represented by a target entity and a relationship, and features of the target entity are embedded using a pre-trained visual model to obtain entity features, and features of the relationship are encoded using a pre-trained encoding model to obtain relationship features. Based on the fine-grained text features, the entity features, and the relationship features, a multi-head attention mechanism is used to enhance the entity features and the relationship features to obtain enhanced relationship features and enhanced entity features; the fine-grained text features, the enhanced entity features, and the enhanced relationship features are spliced to obtain a fine-grained feature representation; the coarse-grained features of the text and the image are spliced to obtain a coarse-grained feature representation, and the fine-grained feature representation and the coarse-grained feature representation are fused and input into a classifier to classify the news information;

[0114] The training module 404 is used to perform reverse training on the semantic enhancement framework according to a preset news information sample and loss function to obtain a trained semantic enhancement framework;

[0115] The detection module 406 is used to input the news information to be detected into the trained semantic enhancement framework and output the false information detection result of the news information to be detected.

[0116] In one embodiment, the model building module 402 is further configured to

[0117] In one embodiment, the model building module 402 is further configured to use a BERT model to perform feature extraction on the text to obtain fine-grained text features.

[0118] In one embodiment, the model building module 402 is further configured to use a pre-trained ResNet50 model to extract a visual feature vector of the region corresponding to the target entity, and obtain the entity feature as follows:

[0119]

[0120] in, Represents the feature embedding of the target entity i, and the target entity is represented by o i , represents the position coordinates of the i-th target entity, m is the maximum number of target entities in the image, s represents the number of target entities in the image, d r It is the feature dimension of the last layer output of the ResNet50 model.

[0121] In one embodiment, the model building module 402 is further configured to input the relationship into the BERT model for feature encoding, and obtain the relationship feature as follows:

[0122]

[0123] Among them, t r Represents the relationship feature, r j represents the jth relation, p represents the number of relations in the image, n is the maximum number of relations in the image, d t It is the feature dimension of the last layer output of the BERT model.

[0124] In one embodiment, the model building module 402 is further configured to construct a query matrix, a key matrix, and a value matrix of a multi-head attention mechanism based on the fine-grained text features, the entity features, and the relationship features, respectively. h =r p W Q , K h =t p W K 、V r =v p W V , W Q 、W K 、W V Represents learning parameters; the entity features are enhanced according to the query matrix, key matrix and value matrix, and the enhanced entity features and enhanced relationship features are obtained respectively:

[0125]

[0126] Among them, v O represents the enhanced entity feature, t R represents the enhanced relational features, H represents the number of attention heads, q is the embedding dimension, and f(·) is a fully connected layer.

[0127] In one embodiment, the model building module 402 is further configured to concatenate the fine-grained text features, the enhanced entity features, and the enhanced relationship features to obtain a fine-grained feature representation as follows:

[0128]

[0129] Among them, S d Represents fine-grained features.

[0130] In one embodiment, the model building module 402 is further configured to concatenate the coarse-grained features of the text and the image to obtain a coarse-grained feature representation as follows:

[0131]

[0132] Among them, S w represents the coarse-grained feature, t c represents the coarse-grained text representation, v c represents a coarse-grained image representation.

[0133] In one embodiment, the model building module 402 is further configured to use cosine similarity to measure the coarse-grained text representation and the coarse-grained image representation in the coarse-grained features, and obtain the coarse-grained similarity as:

[0134]

[0135] According to the coarse-grained similarity, the fusion weight is calculated as:

[0136] weight = 1-Linear(sim)

[0137] Among them, weight represents the fusion weight, and Linear(·) represents the linear network layer;

[0138] According to the fusion weight, the fine-grained feature representation and the coarse-grained feature representation are fused as follows:

[0139]

[0140] in, S Indicates fusion features.

[0141] In one embodiment, the loss function is:

[0142] L=-∑[ylog(p)+(1-y)log(1-p)]

[0143] y represents the label distribution of the news information sample, and p represents the probability that the news information sample is predicted to be fake news.

[0144] Regarding the specific limitations of the fake news detection device based on scene graph semantic enhancement, please refer to the limitations of the fake news detection method based on scene graph semantic enhancement above, which will not be repeated here. The various modules in the above-mentioned fake news detection device based on scene graph semantic enhancement can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0145] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0146] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and such modifications and improvements are intended to fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A fake news detection method based on scene graph semantic enhancement, characterized in that: The method comprises: Construct a semantic enhancement framework; the input of the semantic enhancement framework is news information; the news information consists of text and image, and feature extraction is performed on the text to obtain fine-grained text features; the image is represented by target entities and relationships, and feature embedding is performed on the target entity using a pre-trained visual model to obtain entity features, and feature encoding is performed on the relationship using a pre-trained encoding model to obtain relationship features, and based on the fine-grained text features, the entity features and the relationship features, a multi-head attention mechanism is used to enhance the entity features and the relationship features to obtain enhanced relationship features and enhanced entity features; the fine-grained text features, the enhanced entity features and the enhanced relationship features are spliced to obtain a fine-grained feature representation; the coarse-grained features of the text and the image are spliced to obtain a coarse-grained feature representation, and the fine-grained feature representation and the coarse-grained feature representation are fused and input into a classifier to classify the news information; Reverse training the semantic enhancement framework according to a preset news information sample and loss function to obtain a trained semantic enhancement framework; The news information to be detected is input into the trained semantic enhancement framework, and the false information detection result of the news information to be detected is output.

2. The method according to claim 1, characterized in that Feature extraction is performed on the text to obtain fine-grained text features including: The BERT model is used to extract features from the text to obtain fine-grained text features.

3. The method according to claim 1, characterized in that The pre-trained visual model is used to embed features of the target entity to obtain entity features, including: The pre-trained ResNet50 model is used to extract the visual feature vector of the target entity corresponding area, and the entity features are obtained as follows: in, Represents the feature embedding of the target entity i, and the target entity is represented by o i , represents the position coordinates of the i-th target entity, m is the maximum number of target entities in the image, s represents the number of target entities in the image, d r It is the feature dimension of the last layer output of the ResNet50 model.

4. The method according to claim 1, wherein The relationship is feature-encoded using a pre-trained encoding model to obtain relationship features, including: The relationship is input into the BERT model for feature encoding, and the relationship features are obtained as follows: Among them, t r Represents the relationship feature, r j represents the jth relation, p represents the number of relations in the image, n is the maximum number of relations in the image, d t It is the feature dimension of the last layer output of the BERT model.

5. The method according to any one of claims 1 to 4, characterized in that According to the fine-grained text features, the entity features, and the relationship features, a multi-head attention mechanism is adopted to enhance the entity features and the relationship features, including: According to the fine-grained text features, the entity features and the relationship features, the query matrix, key matrix and value matrix of the multi-head attention mechanism are constructed as Q h =r p W Q , K h =t p W K 、V r =v p W V , W Q 、W K 、W V Represents learning parameters; the entity features are enhanced according to the query matrix, key matrix and value matrix, and the enhanced entity features and enhanced relationship features are obtained respectively: Among them, v O represents the enhanced entity feature, t R represents the enhanced relational features, H represents the number of attention heads, q is the embedding dimension, and f(·) is a fully connected layer.

6. The method according to claim 5, characterized in that The fine-grained text features, the enhanced entity features, and the enhanced relationship features are concatenated to obtain a fine-grained feature representation, including: The fine-grained text features, the enhanced entity features, and the enhanced relationship features are concatenated to obtain a fine-grained feature representation: Among them, S d Represents fine-grained features.

7. The method according to any one of claims 1 to 4, characterized in that The coarse-grained features of the text and the image are concatenated to obtain a coarse-grained feature representation, including: The coarse-grained features of the text and the image are concatenated to obtain the coarse-grained features represented as: Among them, S w represents the coarse-grained feature, t c represents the coarse-grained text representation, v c represents a coarse-grained image representation.

8. The method according to claim 7, characterized in that Fusing the fine-grained feature representation and the coarse-grained feature representation, including: The cosine similarity is used to measure the coarse-grained text representation and coarse-grained image representation in the coarse-grained features, and the coarse-grained similarity is obtained as follows: According to the coarse-grained similarity, the fusion weight is calculated as: weight = 1-Linear(sim) Among them, weight represents the fusion weight, and Linear(·) represents the linear network layer; According to the fusion weight, the fine-grained feature representation and the coarse-grained feature representation are fused as follows: Among them, S represents the fusion feature.

9. The method according to any one of claims 1 to 4, characterized in that The loss function is: L=-∑[y log(p)+(1-y)log(1-p)] y represents the label distribution of the news information sample, and p represents the probability that the news information sample is predicted to be fake news.

10. A fake news detection device based on scene graph semantic enhancement, characterized in that: The device comprises: A model construction module is used to construct a semantic enhancement framework; the input of the semantic enhancement framework is news information; the news information consists of text and images, and features are extracted from the text to obtain fine-grained text features; the image is represented by a target entity and a relationship, and a pre-trained visual model is used to embed features of the target entity to obtain entity features, and a pre-trained encoding model is used to encode features of the relationship to obtain relationship features, and a multi-head attention mechanism is used to enhance the entity features and the relationship features based on the fine-grained text features, the entity features, and the relationship features to obtain enhanced relationship features and enhanced entity features; the fine-grained text features, the enhanced entity features, and the enhanced relationship features are spliced to obtain a fine-grained feature representation; the coarse-grained features of the text and the image are spliced to obtain a coarse-grained feature representation, and the fine-grained feature representation and the coarse-grained feature representation are fused and input into a classifier to classify the news information; A training module, configured to perform reverse training on the semantic enhancement framework based on a preset news information sample and loss function to obtain a trained semantic enhancement framework; The detection module is used to input the news information to be detected into the trained semantic enhancement framework and output the false information detection result of the news information to be detected.

Images