A method for out-of-context false information detection based on global information enhancement

Abstract

The invention discloses a method for detecting false information out of context based on global information enhancement, which relates to the field of artificial intelligence technology and comprises the following steps: acquiring a triple composed of an image and two titles, inputting the triple into a multimodal language reasoning model, inputting the triple with a neutral category index into a visual semantic reasoning model, judging the triple with an implied category index as true information that has not been separated from the context, and judging the triple with a contradictory category index as false information out of the context; outputting the sum of the relationship scores between the image and the two titles based on the visual semantic reasoning model; calculating the similarity between the two titles based on a pre-trained language model, and judging the triple with the sum both greater than a preset image-text threshold and less than a preset text threshold as false information out of the context; the false information detection method can better detect false information out of the context with high detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for detecting out-of-context false information based on global information enhancement. Background Art

[0002] Against the backdrop of the rapid development of Internet technology, the widespread dissemination of self-media and multimedia has brought about a large amount of data growth. However, this has also triggered a series of content security issues.

[0003] False information out of context, called out-of-context or cheapfake, poses a major challenge to multimodal fact-checking. Image and title pairing is used to support false descriptions. Detecting this out-of-context forged information requires combining both image and title modalities. How to effectively utilize the complex context of images and texts, and detect false information as accurately as possible with less auxiliary information, is a problem worth exploring.

[0004] Faced with the negative impact of false information, current detection methods mostly rely on public reporting and manual verification, which is obviously insufficient in terms of timeliness. Given the intentional nature of false information, as well as its wide range of subject coverage and varied styles, single-modal analysis is obviously not enough for today's multimedia online communication. Cheapfake involves the use of non-AI methods to manipulate information, such as image editing tools, mixing images into text with false contexts, and so on. These techniques are simple, fast, easy to implement, and do not require a lot of resources. As a result, Cheapfake is widely used, generating a large amount of misleading information that is difficult to detect from users, making it a serious problem that needs attention. Summary of the invention

[0005] Based on the technical problems existing in the background technology, the present invention proposes a method for detecting false information out of context based on global information enhancement, which can better detect false information out of context with high detection accuracy.

[0006] The present invention proposes a method for detecting false information out of context based on global information enhancement, comprising:

[0007] Get a triplet of an image and two titles, swap the positions of the two titles and form two sets of input features with the image respectively, input the two sets of input features into the trained multimodal language reasoning model respectively, obtain the fused features based on the bidirectional reasoning process, and take the category index with the highest score in the fused features;

[0008] Input the triples with neutral category index into the trained visual semantic reasoning model, judge the triples with implied category index as true information that has not been separated from the context, and judge the triples with contradictory category index as false information that has been separated from the context;

[0009] Perform global semantic reasoning on the triples based on the visual semantic reasoning model and output the relationship scores between the image and the two titles. and ;

[0010] Calculate the similarity between two titles based on the pre-trained language model ,Will and Both are greater than the preset image and text threshold and Less than the preset text threshold The triples of are judged as false information out of context.

[0011] Furthermore, the multimodal language reasoning model, the visual semantic reasoning model, and the pre-trained language model constitute an information detection model. The training process of the information detection model is as follows:

[0012] Obtain images and captions to construct a training set, where each sample in the training set includes an image and one or more captions associated with the image, and all captions in the training set are true information without being separated from the context;

[0013] Use the residual learning module to convolve the image into image patch features of P hidden sizes;

[0014] Apply byte pair encoding to the title in the sample and convert it into two subword sequences. The two subword sequences are embedded in the image block features respectively to obtain the input features corresponding to the title.

[0015] The multimodal language reasoning model uses Transformer as the backbone network, and inputs different input features into the pre-trained multimodal language reasoning model to obtain the prediction category of whether it is false information;

[0016] Input the triples with neutral category index into the pre-trained visual semantic reasoning model, use the top-down attention mechanism on the image to generate the encoding feature group; use the graph convolutional neural network to reason on the encoding feature group, and perform global semantic reasoning on the obtained relationship enhancement features to obtain the image representation features, align the image representation features with the title and output the relationship scores between the image and different titles;

[0017] Construct a total loss function to train the multimodal language reasoning model and the pre-trained visual semantic reasoning model.

[0018] Furthermore, the pre-training process of the visual semantic reasoning model is as follows:

[0019] Get triplets multiple times to build a pre-training set;

[0020] Based on the top-down attention mechanism, each image generates a set of encoded features, and each encoded feature in the encoded feature group represents a region of the image;

[0021] Calculate the affinity between any two encoding features in the encoding feature group, build a graph structure with encoding features as nodes and affinity as edges, add each node to the corresponding edge based on graph convolutional network reasoning to obtain enhanced nodes, and combine all enhanced nodes to obtain the final predicted representation of the image;

[0022] Input the title into the GRU encoder, output the text features in the same feature space as the image features, and then align the text features with the image features;

[0023] Construct a pre-training loss function to pre-train the visual semantic reasoning model.

[0024] Furthermore, the pre-training loss function The details are as follows:

[0025] ;

[0026] ;

[0027] ;

[0028] in, is the triple ranking loss, Generate loss for title, is the marginal parameter, yes and The similarity function of yes and The similarity function of yes and The similarity function of for , and are positive sample pairs, and In small batches and The hardest negative sample in To enhance the collection of nodes, is a trainable parameter, is the length of the output word sequence, For the given previous word , some contextual information And the model parameters Under the condition of The probability of is the index of the word in the word sequence.

[0029] Furthermore, the trained information detection model is tested based on the test set, where each sample in the test set includes an image and two titles. During the information detection model testing and execution process, the two titles are checked for forgery claims, as follows:

[0030] Judgment rule 1: By checking the opposition relationship between the two titles of "whether they are true or not", determine whether the two titles are self-contradictory or contradictory;

[0031] Judgment rule two: check whether the two titles have similarities with keywords related to forgery, and determine whether the two titles are false information out of context.

[0032] Furthermore, the judgment rule 1 is as follows:

[0033] The two titles are Title 1 and Title 2;

[0034] Add title 1 as a true marker and a false marker respectively, and generate two new titles: positive probe and negative probe ;

[0035] Count positive probes The semantic similarity between the two titles and , is the semantic similarity, For Title 1;

[0036] Counting negative probes The semantic similarity between the two titles, and , For Title 2;

[0037] like Less than and At The difference is greater than the preset threshold, or Less than and Difference If the value is greater than a preset threshold, the corresponding triplet will be marked as false information out of context.

[0038] Furthermore, the second judgment rule is as follows:

[0039] Set additional offensive words and generate test titles;

[0040] The semantic similarity between the two titles and the test title is calculated respectively. If the difference between the two similarities is higher than the preset similarity threshold, it means that the title with high similarity is forged false information, and the corresponding triplet is marked as false information out of context.

[0041] Furthermore, the multimodal language reasoning model includes an encoder and a decoder. The encoder consists of a self-attention mechanism 1 and a feedforward network 1. The decoder consists of a self-attention mechanism 2, a feedforward network 2 and a cross-attention mechanism. Head scaling is added to the self-attention mechanism 1, a normalization layer is added after the self-attention mechanism 1, and a normalization layer is added to the first layer of the feedforward network 1.

[0042] Furthermore, in the training process of the information detection model, the pre-trained language model does not participate in the training and is directly used in the execution process of the information detection model.

[0043] Furthermore, after all enhanced nodes are obtained, global reasoning is performed on the enhanced nodes to filter out irrelevant information and obtain the final predicted representation of the image.

[0044] The advantages of the out-of-context false information detection method based on global information enhancement provided by the present invention are: detecting the target triplet, considering the two aspects of image and title, title and title, optimizing the understanding of the global context, using multi-step discrimination, and better detecting out-of-context false information; the accuracy of the detection method is 3.4% higher than the existing method, and it also has advantages over other improved methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a structural schematic diagram of the present invention;

[0046] Figure 2 It is a schematic diagram of the structure of the multimodal language reasoning model;

[0047] Figure 3 Schematic diagram of the structure of the visual semantic reasoning model. DETAILED DESCRIPTION

[0048] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0049] like Figures 1 to 3 As shown, the present invention proposes a method for detecting false information out of context based on global information enhancement, comprising the following steps:

[0050] An image and two titles are obtained to form a triplet. The positions of the two titles are swapped and they are combined with the image to form two sets of input features. The two sets of input features are respectively input into the trained multimodal language reasoning model. Fusion features are obtained based on the bidirectional reasoning process. The category index with the highest score in the fusion features is taken. The triplet with a neutral category index is input into the trained visual semantic reasoning model. The triplet with an implied category index is judged as true information, and the triplet with a contradictory category index is judged as false information out of context.

[0051] Perform global semantic reasoning on the triples based on the visual semantic reasoning model and output the relationship scores between the image and the two titles. and ;

[0052] Calculate the similarity between two titles based on the pre-trained language model ,Will and Both are greater than the preset image and text threshold and Less than the preset text threshold The triples of are judged as false information out of context.

[0053] This embodiment mainly records the process of the method for detecting false information out of context, that is, whether the (image, title 1, title 2) triple is false information out of context (OOC). The main purpose is to detect whether there is a conflict between them and narrow the scope for subsequent fact verification. In the same triple, title 1 and title 2 are different titles corresponding to the same image. This embodiment detects the target triple, considers the image and title, and the title and title, optimizes the understanding of the global context, and uses multi-step discrimination to better detect false information out of context. The accuracy of the detection method in this embodiment is 3.4% higher than the existing method, and it also has advantages over other improved methods.

[0054] In this embodiment, the multimodal language reasoning model, the visual semantic reasoning model, and the pre-trained language model constitute an information detection model. The training process of the information detection model is as follows:

[0055] Obtain images and captions to construct a training set, where each sample in the training set includes an image and one or more captions associated with the image, and all captions in the training set are true information without being separated from the context;

[0056] Use the residual learning module to convolve the image into image patch features of P hidden sizes;

[0057] Apply byte pair encoding to the title in the sample and convert it into two subword sequences. The two subword sequences are embedded in the image block features respectively to obtain the input features corresponding to the title.

[0058] The multimodal language reasoning model uses Transformer as the backbone network, and inputs different input features into the pre-trained multimodal language reasoning model to obtain the prediction category of whether it is false information;

[0059] Input the triples with neutral category index into the pre-trained visual semantic reasoning model, use the top-down attention mechanism on the image to generate the encoding feature group; use the graph convolutional neural network to reason on the encoding feature group, and perform global semantic reasoning on the obtained relationship enhancement features to obtain the image representation features, align the image representation features with the title and output the relationship scores between the image and different titles;

[0060] Construct a total loss function to train the multimodal language reasoning model and the pre-trained visual semantic reasoning model.

[0061] The training process of the information detection model is as follows:

[0062] Step 1: Get images and titles to build a training set;

[0063] This example will introduce two datasets used. One is the dataset in the COSMOS baseline, and the other is the public dataset in the Grand Challenge on Detecting Cheapfakes (GCDC) competition. The training / validation sets of the two datasets are the same, but the test sets are different. The test set COSMOS_test has 1700 samples, and the test set GCDC_test has 1000 samples. This example uses these two datasets to evaluate and test the performance of the proposed method.

[0064] The dataset is collected from news websites and fact-checking websites. It consists of 19 categories of English content and does not contain digitally tampered or forged images.

[0065] Training set / validation set: In the training set, images with captions are collected from news websites. Each captioned image includes the following: an image, one or more associated captions, the source address (URL), a list of entities mentioned in the caption, a modified caption (where each entity is replaced by the corresponding ontology word), and the locations of 10 bounding boxes extracted by the pre-trained Mask-RCNN. The training data does not contain images with out-of-context captions. All image captions are not-out-of-context and have no context label. The training data includes approximately 200,000 images and 450,000 matching text captions. In addition, 20% of them are divided into validation sets.

[0066] Test set: In the test set, the captioned images come from news websites and fact-checking websites. Similar to the training set, each captioned image in the test set contains the following: an image, a caption, the address (URL) of the source, a list of entities, a modified caption, and a bounding box. Each image in the test set corresponds to two captions. One of the two captions is always out of context, and the other caption may be out of context or out of context. In addition, each captioned image in the test set has a context annotation to indicate whether the caption of the image contains out-of-context content. The test set COSMOS_test includes 1700 images and 3400 text captions. The test set GCDC_test includes 1000 images and 2000 text captions.

[0067] Judging false information out of context is a binary classification task, which generally has the following measurement indicators:

[0068] Accuracy: The proportion of correctly predicted samples to the total samples;

[0069] Precision: The proportion of samples predicted to be out of context that actually belong to the category;

[0070] Recall: The proportion of samples that are actually out of context and that are correctly predicted by the model;

[0071] F1-Score: The harmonic average of precision and recall, suitable for scenarios with imbalanced categories.

[0072] ;

[0073] ;

[0074] ;

[0075] ;

[0076] True Positives (TP): the number of samples correctly identified as out of context;

[0077] True Negatives (TN): The number of samples that are correctly identified as not out of context;

[0078] False Positives (FP): The number of samples that are not out of context but are mistakenly identified as out of context;

[0079] False Negatives (FN): The number of samples that are mistakenly identified as not out of context.

[0080] Step 2: Multimodal natural language reasoning: multimodal language reasoning model;

[0081] The original Natural Language Inference (NLI) was proposed to judge the semantic relationship between two sentences. Two texts are input: Premise: a declarative sentence that describes known background information or facts. Hypothesis: another declarative sentence that needs to judge the logical relationship between it and the premise. Output a category label to indicate the logical relationship between the hypothesis and the premise, which usually includes the following three types: Entailment: the hypothesis can be derived from the premise (that is, the hypothesis is logically true); Contradiction: the hypothesis contradicts the premise (that is, the hypothesis is logically false); Neutral: there is no obvious logical relationship between the premise and the hypothesis. In terms of the current multi-tasks, the original NLI can be specified as text entailment.

[0082] The subsequent development of Visual Entailment and Multimodal Natural Language Inference is an extension of Natural Language Inference (NLI), which mainly combines visual data (such as pictures) and text information to determine the semantic relationship between pictures and text. Visual Entailment refers to reasoning between images and text to determine whether text can be derived from the semantic content of the image, with the premise being images and the assumption being text. Multimodal Natural Language Inference integrates data from multiple modalities (usually images and text) to perform natural language reasoning tasks, with the premise being images + text and the assumption being text.

[0083] Previously, only the relationship between two titles or images and titles was considered to determine the relationship of the entire sample, out of context (OOC) / not out of context (NOOC). This embodiment proposes an MNLI recognition component, which uses a multimodal language reasoning model (OFA model) to input (Image, Caption1, Caption2) triples, where Image is an image and Caption1 and Caption2 are two titles, and outputs the entailment / contradiction / neutral categories.

[0084] For the multimodal language reasoning model, an image, a text premise, and a text hypothesis are input into the model's encoder, the decoder generates the corresponding label, and the model evaluates the semantic relevance between the given image and text. In other words, the model learns to answer "yes", "no", or "maybe" to respond to questions such as "Can image and text 1 deduce text 2?".

[0085] Different from the traditional single prediction result determination, this embodiment uses a bidirectional reasoning process to exchange the position of the title for two predictions, and each output is a vector containing 3 elements (c, e, n), where c is contradiction, e is implication, and n is neutral.

[0086] ;

[0087] ;

[0088] ;

[0089] in, is the result output by the multimodal language inference model based on the triple (image, title 1, title 2), is the result output by the multimodal language inference model based on the triple (image, title 2, title 1), To apply the first dimension classification label dimension , For a function that returns the maximum value, find the index of the category with the highest score. If is 0, that is, the triple reasoning result is contradictory, and the output label is OOC; if is 1, that is, the triple inference result is implication, and the output label is NOOC; if is 2, that is, the triple reasoning result is neutral, the output label is neutral, and it enters the visual semantic reasoning model for further judgment.

[0090] For the input and output of the multimodal language reasoning model, the OFA model directly uses the ResNet module (residual learning module) to convert image information Convolution is the image patch feature of P hidden sizes, is the image of the input sample; the ResNet module is a deep convolutional neural network architecture that solves the gradient vanishing and gradient exploding problems in deep neural network training by introducing residual blocks, and can train very deep networks. In the OFA model, the ResNet module is used to perform convolution operations on the input image. The convolution operation is specifically as follows: the image of the input sample is divided and converted into multiple small blocks, and each small block is convolved to obtain a feature vector, which is the image block feature. Among them, P represents the number of image block features obtained after processing. By dividing the image into multiple small blocks for convolution, the processing complexity and the amount of calculation will be reduced, and more detailed local features can be captured.

[0091] For the processing of language information, byte pair encoding is applied to the text sequence of a given title, which is converted into a subword sequence and then embedded into the image block features. In order to process data of different modalities without a task-specific output structure, the data of various modalities are represented in a unified space, the text, images, and objects are discretized, and they are represented by tokens in a unified vocabulary. An image with a resolution of 256×256 can be represented as a code sequence of length 16×16, and each discrete code is strongly associated with the corresponding image block. For each object, its label and bounding box are extracted. The continuous coordinates of the bounding box (upper left corner and lower right corner) are uniformly discretized into integers as position tokens ,in is the coordinate value of the upper left corner of the image, is the coordinate value of the lower right corner of the image. For object labels, they are essentially words, so they can be represented by byte-pair encoding tokens. The final model uses a unified vocabulary for all language and visual tokens (including subwords, image codes, and position tokens). This approach unifies data representations of different modalities into the same space, thereby supporting joint modeling of multimodal tasks.

[0092] The multimodal language reasoning model (OFA model) selects Transformer as the backbone network and adopts the encoder-decoder network as the unified architecture for all tasks. This embodiment uses it to implement multimodal natural language reasoning. The encoder consists of a self-attention mechanism 1 (self-attention) and a feedforward network 1 (FFN). The decoder consists of a self-attention mechanism 2 (self-attention), a feedforward network 2 (FFN) and a cross-attention mechanism (cross-attention), which is used to establish a connection between the decoder and the encoder output representation. In order to stabilize training and accelerate convergence, the OFA model adds the following improvements to the architecture: add head scaling to the self-attention mechanism 1, add a normalization layer after the self-attention mechanism 1, and add a normalization layer to the first layer of the feedforward network 1.

[0093] In addition, in constructing the training set, the acquired images and titles need to be preprocessed, and the preprocessed images and titles are used as training sets. Specifically, the input images and title texts need to be converted into a format acceptable to the model, including image processing, text encoding, and the input structure required to generate the information detection model. Specifically, the information detection model reads, Base64 encodes, and extracts features (such as scaling and cropping) on ​​the image to generate a tensor representation of the image; the text is cleaned, truncated, and encoded to convert it into a token sequence. Subsequently, the encoder input sequence, the decoder input sequence, and the target sequence are generated according to the configuration, and a sample dictionary containing image features, text features, decoder prompts, and reference labels is constructed, while supporting batching and device migration (such as GPU or FP16). These processes ensure that the input data matches the structure and task requirements of the model. This embodiment uses the OFA model, so the fixed label is out of context, and the information detection model is expected to perform reasoning output.

[0094] Step 3: Visual semantic reasoning model;

[0095] The triples with neutral category index are input into the pre-trained visual semantic reasoning model, and a top-down attention mechanism is used on the image to generate a group of encoded features. The graph convolutional neural network is used to infer the encoded feature group, and global semantic reasoning is performed on the obtained relationship enhanced features to obtain image representation features. The image representation features are aligned with the two titles and the relationship scores between the image and the two titles are output.

[0096] Since all titles in the training set are real information without being separated from the context, during the fine-tuning process of the pre-trained visual semantic reasoning model, the positive samples of the image-text matching task are images and titles from the same set, and titles from different sets are randomly selected to form negative samples. The pre-trained visual semantic reasoning model is fine-tuned based on positive and negative samples. The text generation task is to convert the final image representation features into text sequences and match them with real text.

[0097] (A) The pre-training process of the visual semantic reasoning model is as follows (a1) to (a4):

[0098] (a1) Acquire triplets multiple times to build a pre-training set.

[0099] (a2) Generate a set of encoded features for each image based on a top-down attention mechanism, where each encoded feature in the set of encoded features represents a region of the image;

[0100] Using bottom-up attention, each image can eventually be represented as a set of features Indicates the image object, or the first The encoding of the salient regions, It can be the number of objects or the number of regions, and the meanings of the two are consistent. The model used is the target detection algorithm (Faster RCNN) with ResNet-101 as the backbone, including the region generation network (RPN) and the full convolution network (FCN). The model is trained to predict instance classes and attribute classes, rather than object classes, so it can help learn feature representations with rich semantic meanings. Specifically, instance classes include objects and salient content that is difficult to identify. For example, attributes like "furry" and things like "building", "grass" and "sky". The IoU threshold is set to 0.7, the location threshold is set to 0.3, and the top 36 ROIs are selected by sorting according to the location score. Features are extracted from each selected region, and after average pooling and full connection layers, a D-dimensional (2048) vector is obtained.

[0101] Among them, IoU (Intersection over Union) is a common indicator for evaluating the degree of overlap between the predicted results and the true label (Ground Truth) in target detection and segmentation tasks. Its value is between 0 and 1, and the larger the value, the closer the predicted result is to the true value. ROIs (Regions of Interest) are common terms in computer vision and image processing, referring to areas in an image that require special attention. These areas usually contain key information required for target detection, segmentation, recognition or analysis tasks.

[0102] (a3) Regional relations and global semantic reasoning

[0103] Based on the obtained features , calculate the affinity between any two coding features in the coding feature set , , , They are respectively The encoding of the salient regions, for The area indicates that for The area indicates that and are weights, and Through back propagation learning, the characteristics of the image regions with false information in the pre-training set are learned based on affinity.

[0104] The graph structure is constructed with encoded features as nodes and affinity as edges. The points and edges of a region represent the degree of correlation between its own features and other regions. Based on graph convolutional network reasoning, each node is added to the corresponding edge to obtain an enhanced node. Global reasoning is performed on the enhanced node to filter out irrelevant information and obtain the final predicted representation of the image. , a collection of enhanced nodes , For the image Enhanced representation of salient regions.

[0105] In filtering out irrelevant information, the regional feature sequence is input into the recurrent gate unit. During this reasoning process, the description of the entire scene will gradually grow and update in the memory unit (hidden state). The update gate will decide the extent to which its storage unit is updated based on the current state and the description of the entire scene in the last step.

[0106] (a4) Learning alignment through joint matching and generation

[0107] In order to extract a caption representation with the same dimension as the global features of the image, a GRU encoder is used. Before the GRU encoder, this embodiment applies named entity recognition (NER) to classify entities in the caption into corresponding types, thereby reducing the abstract relationship between the image and the caption in cheap fake information. For the objective function, this embodiment uses a hinge-based triple ranking loss , and select the negative sample with the closest margin to each pre-training sample. is the marginal parameter, , It is a similarity function in the common vector space, selecting the hardest negative samples in each mini-batch to reduce computation, rather than the entire pre-training set.

[0108] ;

[0109] in, is the triple ranking loss, is the marginal parameter, yes and The similarity function of yes and The similarity function of yes and The similarity function of for , and are positive sample pairs, and In small batches and The hardest negative sample in .

[0110] At the same time, during the training process, a title with real matching ability should be generated based on the visual representation, and the title generation loss as follows:

[0111] ;

[0112] in, is a trainable parameter, is the length of the output word sequence, For the given previous word , some contextual information And the model parameters Under the condition of The probability of is the index of the word in the word sequence.

[0113] The final pre-training loss function used for:

[0114] .

[0115] (B) Processing of triples with neutral category index based on the pre-trained visual semantic reasoning model;

[0116] The visual semantic reasoning model has learned the matching of images and title texts very well during the pre-training stage, and has taken into account that the title may involve multiple objects in the image. The pre-trained visual semantic reasoning model fine-tuned on the training set of this embodiment can use the global information of the image and the title for cross-modal image-text retrieval, and together with the pre-trained language model (SBERT-WK), it can judge triples with neutral category indexes.

[0117] Input the (Image, Caption1, Caption2) triple into the pre-trained visual semantic reasoning model, which will perform global semantic reasoning on the image, select discriminative information and gradually generate a representation of the entire scene. Specifically, 1. For the image part, first use a top-down attention model to generate image regions and their features; 2. The pre-trained visual semantic reasoning model establishes connections between these regions, and uses the graph convolutional neural network GCN for reasoning to generate features with semantic relationship information, i.e., relationship-enhanced features; 3. Perform global semantic reasoning on the relationship-enhanced features, select discriminative information, filter out unimportant information, generate the final image representation, and then input the features into the multimodal alignment module to align with the text. The pre-trained visual semantic reasoning model outputs two scores, representing the relationship between the image and the two captions respectively.

[0118] This example uses a pre-trained language model that is trained on the Sentence Text Similarity (STS) task. The pre-trained language model takes two captions (Caption1, Caption2) as input and outputs a similarity score in the range [0, 1] , the similarity score Indicates the semantic similarity between two titles (higher scores indicate the same context).

[0119] In this embodiment, the threshold Set to 0.25, Set to 0.5, samples that meet the following conditions are marked as out of context: For the triple, it can be considered that the image is relatively relevant to both titles, but the semantic similarity between the two titles is not high, so the sample is marked as out of context, and the rest are judged as not out of context, specifically:

[0120] ;

[0121] is the similarity between the image and title 1, is the similarity between the image and title 2, is the similarity score between two titles.

[0122] It should be noted that the pre-trained language model does not participate in the training process of the information detection model and is directly used in the execution process of the information detection model.

[0123] Step 4: Construct a total loss function to train the multimodal language reasoning model and the pre-trained visual semantic reasoning model.

[0124] Through steps one to four, the target triplet is detected, considering the image and title, title and title, optimizing the understanding of the global context, and using multi-step discrimination to better detect false information out of context.

[0125] In addition, the trained information detection model is tested based on the test set, where each sample in the test set includes an image and two titles. During the testing and execution of the information detection model, for each triple, the two titles need to be checked for forgery claims first, which includes two judgment rules:

[0126] Judgment rule 1: By checking the opposition relationship between the two titles of "whether they are true or not", determine whether the two titles are self-contradictory or contradictory;

[0127] Check whether the "true" and "not true" assumptions of Title 1 lead to a contradiction in Title 2. First, add the true and false tags to the given Title 1, that is, "was true / genuine / real / authentic" and "was nottrue / genuine / real / authentic". The two new titles generated are called positive probes and negative probes, respectively. Calculate the positive probe The semantic similarity between the two titles and , is the semantic similarity, For Title 1; Calculate negative probes The semantic similarity between the two titles, and , For title 2.

[0128] If the two original titles have opposite meanings (i.e., they are taken out of context), we expect these scores to move in different directions between positive and negative probes (i.e., when one increases, the other decreases); otherwise, we expect them to move in the same direction. Title 1 and Title 2 are contradictory, so the rule should be met: If Less than and At The difference is greater than the preset threshold of 0.01, or Less than and Difference More than the preset threshold of 0.01. For example, when title 1 is assumed to be true, its similarity with title 2 is higher, while when title 1 is assumed to be false, its similarity with title 2 is lower, which meets the above rules and the sample is marked as out of context.

[0129] Judgment rule 2: Check whether the two titles have similarities with keywords related to forgery, and determine whether the two titles are false information out of context;

[0130] In order to identify misleading words to improve detection accuracy, check whether Title 1 and Title 2 have significant similarity with forgery-related keywords, and judge the falsity of the text based on this. Generate a test title "fake, hoax, fabrication, supposedly, falsification, propaganda, deflection, deception, contradicted, defamation, lie, misleading, deceive, fraud, concocted, bluffing, made up, double meaning, alternative facts, trick, half-truth, untruth, falsehoods, inaccurate, disinformation, misconception" with some common offensive words, and calculate the semantic similarity score of the original two titles relative to the test title. If one of the scores is significantly smaller than the other, specifically, the falsity score of one title is lower than 0.1, while the falsity score of the other title is higher than 0.15, indicating that the title with the higher score is highly likely to be fabricated false information, then the sample is marked as out of context.

[0131] For two judgment rules, the semantic similarity is calculated as follows:

[0132] ;

[0133] in, for and The semantic similarity between For the sentence embedding of the model, convert the title sentence into an embedding vector, is to calculate the cosine similarity, are the objects for which similarity is to be calculated.

[0134] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for detecting false information out of context based on global information enhancement, comprising: Get a triplet of an image and two titles, swap the positions of the two titles and form two sets of input features with the image respectively, input the two sets of input features into the trained multimodal language reasoning model respectively, obtain the fused features based on the bidirectional reasoning process, and take the category index with the highest score in the fused features; Input the triples with neutral category index into the trained visual semantic reasoning model, judge the triples with implied category index as true information that has not been separated from the context, and judge the triples with contradictory category index as false information that has been separated from the context; Perform global semantic reasoning on the triples based on the visual semantic reasoning model and output the relationship scores between the image and the two titles. and ; Calculate the similarity between two titles based on the pre-trained language model ,Will and Both are greater than the preset image and text threshold and Less than the preset text threshold The triples of are judged as false information out of context; The multimodal language reasoning model, visual semantic reasoning model, and pre-trained language model constitute the information detection model. The training process of the information detection model is as follows: Obtain images and captions to construct a training set, where each sample in the training set includes an image and one or more captions associated with the image, and all captions in the training set are true information without being separated from the context; Input the triples with neutral category index into the pre-trained visual semantic reasoning model, use the top-down attention mechanism on the image to generate the encoding feature group; use the graph convolutional neural network to reason on the encoding feature group, and perform global semantic reasoning on the obtained relationship enhancement features to obtain the image representation features, align the image representation features with the title and output the relationship scores between the image and different titles; Construct a total loss function to train the multimodal language reasoning model and the pre-trained visual semantic reasoning model; The trained information detection model is tested based on the test set, where each sample in the test set includes an image and two titles. During the information detection model testing and execution process, the two titles are checked for forged claims as follows: Judgment rule 1: By checking the opposition relationship between the two titles of "whether they are true or not", determine whether the two titles are self-contradictory or contradictory; Judgment rule two: check whether the two titles have similarities with keywords related to forgery, and determine whether the two titles are false information out of context.

2. The method for detecting false information out of context based on global information enhancement according to claim 1, characterized in that: The pre-training process of the visual semantic reasoning model is as follows: Get triplets multiple times to build a pre-training set; Based on the top-down attention mechanism, each image generates a set of encoded features, and each encoded feature in the encoded feature group represents an area of ​​the image; Calculate the affinity between any two encoding features in the encoding feature group, build a graph structure with encoding features as nodes and affinity as edges, add each node to the corresponding edge based on graph convolutional network reasoning to obtain enhanced nodes, and combine all enhanced nodes to obtain the final predicted representation of the image; Input the title into the GRU encoder, output the text features in the same feature space as the image features, and then align the text features with the image features; Construct a pre-training loss function to pre-train the visual semantic reasoning model.

3. The method for detecting false information out of context based on global information enhancement according to claim 2, characterized in that: Pre-training loss function The details are as follows: ； ； ； in, is the triple ranking loss, Generate loss for title, is the marginal parameter, yes and The similarity function of yes and The similarity function of yes and The similarity function of for , and are positive sample pairs, and In small batches and The hardest negative sample in To enhance the collection of nodes, is a trainable parameter, is the length of the output word sequence, For the given previous word , some contextual information And the model parameters Under the condition of The probability of is the index of the word in the word sequence.

4. The method for detecting false information out of context based on global information enhancement according to claim 1, characterized in that: Judgment rule 1 is as follows: The two titles are Title 1 and Title 2; Add title 1 as a true marker and a false marker respectively, and generate two new titles: positive probe and negative probe ; Count positive probes The semantic similarity between the two titles and , is the semantic similarity, For Title 1; Counting negative probes The semantic similarity between the two titles, and , For Title 2; like Less than and At The difference is greater than the preset threshold, or Less than and Difference If the value is greater than a preset threshold, the corresponding triplet will be marked as false information out of context.

5. The method for detecting false information out of context based on global information enhancement according to claim 1, characterized in that: The details of judgment rule 2 are as follows: Set additional offensive words and generate test titles; The semantic similarity between the two titles and the test title is calculated respectively. If the difference between the two similarities is higher than the preset similarity threshold, it means that the title with high similarity is forged false information, and the corresponding triplet is marked as false information out of context.

6. The method for detecting false information out of context based on global information enhancement according to claim 2, characterized in that: The multimodal language inference model includes an encoder and a decoder. The encoder consists of a self-attention mechanism 1 and a feedforward network 1. The decoder consists of a self-attention mechanism 2, a feedforward network 2, and a cross-attention mechanism. Head scaling is added to the self-attention mechanism 1, a normalization layer is added after the self-attention mechanism 1, and a normalization layer is added to the first layer of the feedforward network 1.

7. The method for detecting false information out of context based on global information enhancement according to claim 2, characterized in that: During the training process of the information detection model, the pre-trained language model does not participate in the training and is directly used in the execution process of the information detection model.

8. The method for detecting false information out of context based on global information enhancement according to claim 1, characterized in that: After all the enhanced nodes are obtained, global reasoning is performed on the enhanced nodes to filter out irrelevant information and obtain the final predicted representation of the image.

Images