Multimodal recognition method and device for graphic and text information

Abstract

The present application provides a method and device for multimodal recognition of graphic information, which relates to the field of computer technology. The method includes: obtaining image data to be recognized and text data to be recognized of an object to be recognized; inputting the object to be recognized into a recognition model, determining whether the object to be recognized contains forged content, and generating a forged content recognition result when it is determined that the object to be recognized contains forged content; the forged content recognition result includes: image forged areas in the image data to be recognized, and / or forged word positions in the text data to be recognized. The method and device for multimodal recognition of graphic information provided by the present application are used to improve the recognition accuracy of forged graphic information, and to accurately locate the forged parts of forged graphic information.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for multimodal recognition of graphic and text information. Background Art

[0002] With the development of deep learning technology, especially Generative Adversarial Networks (GAN) and Large Language Model (LLM), the use of these technologies to generate realistic fake images and text information has brought many problems and caused certain negative impacts on normal work and life.

[0003] In the related art, deep learning network models can be used to detect and identify images and texts in graphic information to identify forged graphic information. However, this detection method can only identify whether the graphic information is forged, but cannot determine the specific forged part, and requires two different models to identify forged graphic information, resulting in poor recognition effect.

[0004] Based on this, there is an urgent need for a multimodal recognition method to improve the recognition accuracy of forged image and text information, and to accurately locate the forged part of the forged image and text information. Summary of the invention

[0005] The purpose of the present application is to provide a multimodal recognition method and device for graphic information, so as to improve the recognition accuracy of forged graphic information and accurately locate the forged part of the forged graphic information.

[0006] The present application provides a multimodal recognition method for graphic information, comprising:

[0007] Obtain image data and text data of an object to be identified; input the object to be identified into a recognition model, determine whether the object to be identified contains forged content, and generate a forged content recognition result if it is determined that the object to be identified contains forged content.

[0008] Optionally, the obtaining of the image data to be identified and the text data to be identified of the object to be identified includes: obtaining the image and text information to be processed; adjusting the image pixel size of the image to be processed in the image and text information to be processed to a preset pixel size to obtain the image data to be identified; the preset image pixel size is determined based on the training samples for training the recognition model; and determining the image data to be identified and the text data to be identified contained in the image and text information to be processed as the object to be identified.

[0009] Optionally, inputting the object to be identified into the recognition model to determine whether the object to be identified contains forged content includes: evenly dividing the image data to be identified into multiple image blocks; inputting the multiple image blocks into the image encoder of the recognition model to obtain a first image embedding corresponding to the image data to be identified; wherein the first image embedding includes: a first image classification embedding, and multiple first image block embeddings; the number of the multiple first image block embeddings is the same as the number of the multiple image blocks.

[0010] Optionally, the step of inputting the object to be identified into the recognition model to determine whether the object to be identified contains forged content includes: inputting the text data to be identified into the text encoder of the recognition model, and dividing the text data to be identified into multiple word units through the word segmenter of the text encoder; obtaining multiple embedding vectors to be processed from the word unit embedding vector table of the text encoder; one word unit corresponds to one embedding vector to be processed; forward processing the multiple embedding vectors to be processed to obtain a first text embedding corresponding to the text data to be identified; wherein the first text embedding includes: a first text classification embedding, and multiple first word unit embeddings; the number of the multiple first word unit embeddings is the same as the number of the multiple embedding vectors to be processed.

[0011] Optionally, inputting the object to be identified into the recognition model to determine whether the object to be identified contains forged content includes: inputting a first image embedding and a first text embedding into a cross-modal interaction module of the recognition model to obtain a second image embedding corresponding to the first image embedding and a second text embedding corresponding to the first text embedding; wherein the first image embedding is: the result output by the image encoder of the recognition model; the first text embedding is: the result output by the text encoder of the recognition model; the second image embedding includes: a plurality of second image block embeddings having the same number as the plurality of first image block embeddings in the first image embedding, and a second image classification embedding corresponding to the first image classification embedding in the first image embedding; the second text embedding includes: a plurality of second word embeddings having the same number as the plurality of first word embeddings in the first text embedding, and a second text classification embedding corresponding to the first text classification embedding in the first text embedding.

[0012] Optionally, the cross-modal interaction module includes: a first cross-attention module, a second cross-attention module, a third cross-attention module and a fourth cross-attention module; the inputting the first image embedding and the first text embedding into the cross-modal interaction module of the recognition model to obtain the second image embedding corresponding to the first image embedding and the second text embedding corresponding to the first text embedding includes: inputting the first image embedding as a query and the first text embedding as a key and value into the first cross-attention module to obtain a first output embedding; and, inputting the first text embedding as a query and the first image embedding as a key and value into the second cross-attention module to obtain a second output embedding; inputting the first output embedding as a query and the second output embedding as a key and value into the third cross-attention module to obtain the second image embedding; and, inputting the second output embedding as a query and the first output embedding as a key and value into the fourth cross-attention module to obtain the second text embedding.

[0013] Optionally, inputting the object to be identified into the recognition model to determine whether the object to be identified contains forged content includes: embedding the second image classification into a fully connected layer of an image classification module in the recognition model to obtain two latent variable values; inputting the two latent variable values ​​into a normalized exponential function layer of the image classification module in the recognition model to obtain two classification probability values; one latent variable value corresponds to one classification probability value; determining the category corresponding to the larger value of the two classification probability values ​​as the image forgery classification result; wherein the image forgery classification result is used to indicate an image forgery area in the image data to be identified.

[0014] Optionally, inputting the object to be identified into the recognition model to determine whether the object to be identified contains forged content includes: embedding the second text classification into a fully connected layer of a text classification module in the recognition model to obtain two latent variable values, and inputting the two latent variable values ​​into a normalized exponential function layer of an image classification module in the recognition model to obtain two classification probability values; one latent variable value corresponds to one classification probability value; determining the category corresponding to the larger value of the two classification probability values ​​as a text forgery classification result; wherein the text forgery classification result is used to indicate the position of forged word elements in the text data to be identified.

[0015] Optionally, the step of inputting the object to be identified into the recognition model to determine whether the object to be identified contains forged content includes: concatenating the second image classification embedding and the second text classification embedding into a first target embedding vector, and inputting the first target embedding vector into the fully connected layer of the image-text pair classification module in the recognition model to obtain two latent variable values; inputting the two latent variable values ​​into the normalized exponential function layer of the image-text pair classification module to obtain two classification probability values; one latent variable value corresponds to one classification probability value; and determining the category corresponding to the larger value of the two classification probability values ​​as the image-text pair forged classification result.

[0016] Optionally, generating a forged content identification result includes: using multiple second image block embeddings in the second image embedding as keys and values ​​and inputting a trained query vector as a query into a cross-attention module of the recognition model to obtain a second target embedding vector; inputting the second target embedding vector into a forged area positioning module of the recognition model, predicting the relative position coordinates of the forged area in the image data to be identified, and converting the predicted relative position coordinates into absolute pixel position coordinates; wherein the absolute pixel position coordinates are used to indicate the forged area in the image data to be identified when the image data to be identified is a forged image.

[0017] Optionally, the second target embedding vector is input into the forged area positioning module of the recognition model, the relative position coordinates of the forged area in the image data to be identified are predicted, and the predicted relative position coordinates are converted into absolute pixel position coordinates, including: inputting the second target embedding vector into the fully connected layer of the forged area positioning module to obtain four hidden variable values; inputting the four hidden variable values ​​into a preset activation function of the forged area positioning module to obtain four relative coordinate values; and converting the four relative coordinate values ​​into absolute pixel coordinate values ​​based on the pixel size of the image data to be identified.

[0018] Optionally, generating a forged content identification result includes: inputting any target word-meta embedding among the multiple second word-meta embeddings into the fully connected layer of the forged word-meta location module of the recognition model to obtain two latent variable values, and inputting the two latent variable values ​​into the normalized exponential function layer to obtain two classification probability values; one latent variable value corresponds to one classification probability value; determining the category corresponding to the larger value of the two classification probability values ​​as the forged word-meta classification result of the target word-meta embedding; wherein the forged word-meta classification result is used to indicate whether the word-meta corresponding to the target word-meta embedding is a forged word-meta when the image data to be identified is a forged image.

[0019] Optionally, generating the forged content identification result includes: extracting multiple image block embedding groups from the output results of the last multiple encoding layers of the image encoder of the recognition model; the output result of one encoding layer corresponds to one image block embedding group; each image block embedding group contains the same number of image block embeddings as the multiple image blocks into which the image data to be identified is divided; splicing the image block embeddings at corresponding positions in each image block embedding group in the multiple image block embedding groups and inputting them into the linear projection module of the recognition model to obtain multiple fused third image block embeddings; generating a single-channel image of a preset pixel size based on the vector dot product operation results of the multiple third image block embeddings and two prompt word embeddings; decoding the single-channel image through an image decoder to obtain a forged area mask image with the same pixel size as the image data to be identified; wherein the forged area mask image is used to indicate the image forged area in the image data to be identified when the image data to be identified is identified as a forged image.

[0020] Optionally, the generating a single-channel image of a preset pixel size based on the vector dot product operation results of the multiple third image block embeddings and the two prompt word element embeddings includes: performing vector dot product operations on the multiple third image block embeddings and the two prompt word element embeddings in sequence to obtain a first operation result corresponding to each third image block embedding and a second operation result corresponding to each third image block embedding; the first prompt word element embedding of the two prompt word element embeddings corresponds to the first operation result, and the second prompt word element embedding of the two prompt word element embeddings corresponds to the second operation result; the first prompt word element embedding is generated based on a preset first prompt word, and the first prompt word element embedding is generated based on a preset second prompt word Based on the comparison result of the first operation result and the second operation result corresponding to each third image block embedding, determining whether the image block corresponding to each third image block embedding contains a forged image; wherein, if the first operation result corresponding to any target image block embedding among the plurality of third image block embeddings is greater than or equal to the second operation result corresponding to the target image block embedding, then determining that the target image block embedding is a forged image block, otherwise, determining that the target image block is a non-forged image block; determining the label value corresponding to each image block based on the recognition result of each image block, obtaining a plurality of label values, and rearranging the positions of the plurality of label values ​​based on the position information of the image blocks corresponding to the label values, to obtain the single-channel image.

[0021] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of any of the multimodal recognition methods for graphic and text information described above.

[0022] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the multimodal recognition method for graphic information as described in any one of the above are implemented.

[0023] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the multimodal recognition method for graphic and text information as described in any one of the above are implemented.

[0024] The multimodal recognition method and device for graphic information provided by the present application first obtains the image data to be recognized and the text data to be recognized of the object to be recognized; then, the object to be recognized is input into the recognition model to determine whether the object to be recognized contains forged content, and if it is determined that the object to be recognized contains forged content, a forged content recognition result is generated. In this way, the recognition accuracy of forged graphic information can be improved through multimodal recognition, and the forged part of the forged graphic information can be accurately located. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 It is a structural diagram of the recognition model provided by this application;

[0027] Figure 2 It is a flowchart of a multimodal recognition method for graphic information provided by this application;

[0028] Figure 3 is a schematic diagram of the structure of the cross-modal interaction module provided by this application;

[0029] Figure 4 It is a structural diagram of the counterfeit area positioning module provided by this application;

[0030] Figure 5 It is a schematic diagram of the structure of the forged word unit location module provided by the present application;

[0031] Figure 6 It is a structural diagram of a forged region mask prediction module provided by the present application;

[0032] Figure 7 It is a structural schematic diagram of the electronic device provided by this application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0035] In response to the technical problems in the related art of low accuracy in identifying forged graphic and text information and inability to locate the specific forged parts, the embodiment of the present application provides a multimodal recognition method for graphic and text information, which inputs the graphic and text information to be identified into a trained recognition model for identification. It can not only identify whether the graphic and text message is forged graphic and text information, but also accurately locate the forged parts in the image and the forged words in the text when the graphic and text information is identified as forged graphic and text information, thereby achieving better recognition effect.

[0036] like Figure 1 As shown, it is a schematic diagram of the model structure of the recognition model provided by the embodiment of the present application. It comprises an image encoder, a text encoder, a cross-modal interaction module, an image classification module, a text classification module, an image-text pair classification module, a cross-attention module, a forged region positioning module, a forged word unit positioning module and a forged region mask prediction module. In the training process of the model, the embodiment of the present application constructs the loss function of each branch and linearly weights them. The design of this joint loss function takes full account of the importance of each branch and weighs their importance. The embodiment of the present application optimizes the above-mentioned joint loss function using a stochastic gradient descent algorithm to obtain a final recognition model. This optimization process continuously adjusts the model parameters in an iterative manner so that it can better adapt to the characteristics of multimodal forged image and text information and achieve outstanding performance.

[0037] It should be noted that the embodiment of the present application is used to identify whether an image-text pair (i.e., an image and the corresponding text description constitute an image-text pair) contains forged information, and the forged information includes: 1. Forged information in the image, including the facial image of a specific person in the image being replaced with the facial image of another person; the emotional state of the facial image in the image being changed (for example: happiness is changed to anger). 2. Forged information in the text, including the text corresponding to the image being replaced with another piece of text; some words / phrases in the text corresponding to the image are replaced with other words / phrases. In particular, this modification can cause the emotional state expressed by the text to change, thereby changing the tendency of the graphic information content. Based on the above two types of forged information, a variety of forged combinations can be constructed.

[0038] In order to identify the above-mentioned forged information, the embodiment of the present application collects and annotates a training data set. The original training data set includes about 10,000 image-text pairs, each image contains at least 1 face image and at most 10 face images; if it is Chinese text, the corresponding text length is at least 5 characters and at most 50 characters. If it is English text, the corresponding text length is at least 5 words and at most 50 words. Each text in the above-mentioned training data set contains at least one predicate verb and one subject or object.

[0039] After collecting the original training data set, the embodiment of the present application uses the existing image and text forgery technology to modify the image-text pairs in the original training data set, as follows:

[0040] 1. In the above training set, about 20% of the images have no modifications; the largest face in about 40% of the images is replaced with another person's face image; the expression of the largest face in the remaining images is tampered with (for example: smiling is changed to crying, etc.).

[0041] 2. In the above training set, about 20% of the text was not modified; about 40% of the text was replaced by another text; and the words / phrases related to the emotional state in the remaining text were tampered with (for example, words / phrases with positive emotions were changed to words / phrases with negative emotions, etc.).

[0042] 3. Engineering technicians manually mark the location coordinates of the modified face area, the mask of the face area, the sequence number of the modified words / phrases in the text, etc.

[0043] The multimodal recognition method for graphic and text information provided by the embodiment of the present application is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0044] like Figure 2As shown, an embodiment of the present application provides a multimodal recognition method for graphic information, which may include the following steps 201 and 202:

[0045] Step 201: Obtain image data and text data of an object to be identified.

[0046] Exemplarily, the above-mentioned object to be identified is graphic information including an image and text. The graphic information may be an article describing an image in text, or may be news including an image and text.

[0047] Exemplarily, before the original graphic message is input into the above recognition model, the original graphic message needs to be preprocessed.

[0048] Specifically, the above step 201 may further include the following steps 201a1 to 201a3:

[0049] Step 201a1, obtaining the graphic and text information to be processed.

[0050] Step 201a2: adjusting the image pixel size of the image to be processed in the image and text information to be processed to a preset pixel size to obtain the image data to be recognized.

[0051] The preset pixel size is determined based on training samples used to train the recognition model.

[0052] Step 201a3: determine the image data to be recognized and the text data to be recognized contained in the image and text information to be processed as the objects to be recognized.

[0053] Exemplarily, in an embodiment of the present application, in the training phase of the model and the model application reasoning phase, the same preprocessing operation needs to be performed on the original image, specifically including: first, the long side of the original image is scaled to L pixels (L can be any one of 256, 384, 512. In the embodiment of the present application, L=384.). After that, the short side of the original image is scaled proportionally; finally, the short side is padded with zeros to L pixels. After the above preprocessing operation, a preprocessed image can be obtained, and the image width and height are equal, both L pixels. The preprocessed image is the input image I.

[0054] Step 202: input the object to be identified into a recognition model, determine whether the object to be identified contains forged content, and generate a forged content identification result if it is determined that the object to be identified contains forged content.

[0055] The forged content identification result includes: the image forged area in the image data to be identified, and / or the forged word position in the text data to be identified.

[0056] Exemplarily, the classification results output by the recognition model include: image forgery classification results output by the image classification module in the recognition model, text forgery classification results output by the text classification module in the recognition model, and image-text pair forgery classification results output by the image-text pair classification module in the recognition model; forged content recognition results that can be obtained based on the above classification results. The image forgery area is determined based on the results output by the forgery area mask prediction module and the forgery area positioning module of the recognition model; the forgery word unit position is determined based on the results output by the forgery word unit positioning module of the recognition model.

[0057] For example, after the above-mentioned object to be recognized is input into the recognition model, it is first necessary to use an image encoder Processing image data to be recognized and using text encoder Process the text data to be recognized.

[0058] Specifically, the processing step for the image encoder in the above step 202 may further include the following steps 202a1 and 202a2:

[0059] Step 202a1: evenly divide the image data to be recognized into a plurality of image blocks.

[0060] Step 202a2: input the multiple image blocks into the image encoder of the recognition model to obtain a first image embedding corresponding to the image data to be recognized.

[0061] The first image embedding includes: first image classification embedding, and multiple first image block embeddings; the number of the multiple first image block embeddings is the same as the number of the multiple image blocks.

[0062] It should be noted that, in order to facilitate the understanding of the technical solution of the present application, in the embodiment of the present application, a sample in the model training stage, or an object to be identified in the actual application reasoning stage of the model refers to an "image-text pair", which is defined as pair (I, T), including an image I and a corresponding text T. Image I is a three-channel color image, and text T is a description of the content corresponding to image I.

[0063] For example, the image encoder The input is 256 image patches (also called image slices). In order to generate the image encoder The required input image blocks, the embodiment of the present application can divide the image I into 256 image blocks of the same size and non-overlapping along the width and height of the image I, and the width of each image block is L / 16 pixels and the height is also L / 16 pixels.

[0064] For example, Figure 1 As shown, for the convenience of description, the total number of image blocks is defined as ,So .Will The image blocks are input to the above image encoder Extracting image embeddings (patch embedding) (i.e. the first image embedded above), include Image patch embedding (i.e., the multiple first image block embeddings mentioned above) and a classification embedding for classification (i.e. the first image classification embedding mentioned above), so Image Embedding The dimension of each embedding in is , .

[0065] It should be noted that the network structure of the image encoder in the embodiment of the present application can be the same as the network structure of ViT-L / 16, and the parameters of the image encoder can be initialized using the parameters pre-trained by ViT-L / 16 on the ImageNet-21k data set. Various embeddings in the embodiments of the present application are represented in the form of vectors.

[0066] Specifically, the processing steps for the text encoder in the above step 202 may also include the following steps 202b1 to 202b3:

[0067] Step 202b1: input the text data to be recognized into the text encoder of the recognition model, and divide the text data to be recognized into a plurality of word units through the word segmenter of the text encoder.

[0068] Step 202b2: Obtain multiple embedding vectors to be processed from the word unit embedding vector table of the text encoder.

[0069] Among them, one word unit corresponds to one embedding vector to be processed.

[0070] Step 202b3: forward process the multiple embedding vectors to be processed to obtain a first text embedding corresponding to the text data to be recognized.

[0071] The first text embedding includes: a first text classification embedding, and a plurality of first word element embeddings; the number of the plurality of first word element embeddings is the same as the number of the plurality of embedding vectors to be processed.

[0072] It should be noted that the network structure of the text encoder in the embodiment of the present application can be The network structure is the same as that of . At the beginning of training, the parameters of the text encoder can be used The parameters trained on the "General Language Understanding Evaluation Benchmark Collection" are initialized. The tokenizer and the corresponding token embedding vector table used by the text encoder can be compared with the above The word segmenter and word unit embedding vector table of the text encoder are fixed settings and are not updated during training.

[0073] For example, Figure 1 As shown, the present application embodiment uses a text encoder Extracting text word embeddings (i.e., the first text embedding mentioned above). The embodiment of the present application first uses a word segmenter to divide the text T into word units (i.e., the above-mentioned multiple word units), and defines the total number of word units as ( Dynamically changes according to the input text T), then retrieve the embeddings corresponding to these words (i.e. the above-mentioned multiple word-unit vectors to be processed) from the word-unit embedding vector table and pass them through the text encoder The forward processing process, text encoder Extract the text embedding of text T , include Token Embedding (i.e., the multiple first word embeddings mentioned above) and a classification word embedding for classification (i.e. the first text classification embedding mentioned above). .in, The dimension of each word embedding in is also , .

[0074] For example, after extracting the image embedding and text embedding Afterwards, the two sets of embeddings extracted above are simultaneously input into a cross-modal interaction module (Cross Modal Interaction Module) for processing.

[0075] Specifically, in the above step 202, the processing step for the cross-modal interaction module may further include the following step 202c:

[0076] Step 202c: input the first image embedding and the first text embedding into the cross-modal interaction module of the recognition model to obtain a second image embedding corresponding to the first image embedding and a second text embedding corresponding to the first text embedding.

[0077] Among them, the first image embedding is: the result of the image encoder output of the recognition model; the first text embedding is: the result of the text encoder output of the recognition model; the second image embedding includes: a plurality of second image block embeddings having the same number as the plurality of first image block embeddings in the first image embedding, and a second image classification embedding corresponding to the first image classification embedding in the first image embedding; the second text embedding includes: a plurality of second word embeddings having the same number as the plurality of first word embeddings in the first text embedding, and a second text classification embedding corresponding to the first text classification embedding in the first text embedding.

[0078] Exemplarily, the cross-modal interaction module in the embodiment of the present application includes: a first cross-attention module m1, a second cross-attention module m2, a third cross-attention module m3 and a fourth cross-attention module m4.

[0079] Specifically, the above step 202c may further include the following steps 202c1 and 202c2:

[0080] Step 202c1, input the first image embedding as a query and the first text embedding as a key and value into the first cross-attention module to obtain a first output embedding; and, input the first text embedding as a query and the first image embedding as a key and value into the second cross-attention module to obtain a second output embedding.

[0081] Step 202c2, input the first output embedding as a query and the second output embedding as a key and value into the third cross-attention module to obtain the second image embedding; and, input the second output embedding as a query and the first output embedding as a key and value into the fourth cross-attention module to obtain the second text embedding.

[0082] For example, based on Figure 1 ,like Figure 3 As shown, it is a schematic diagram of the structure of the cross-modal interaction module provided in the embodiment of the present application. The cross-modal interaction module uses the cross-attention mechanism to realize the information interaction between the image embedding of the image modality and the text embedding of the text modality, thereby improving the ability of the image embedding of the image modality and the text embedding of the text modality to encode semantic information.

[0083] For example, Figure 3 As shown, the queries of the cross-attention module m1 are image embeddings The keys and values ​​of the cross-attention module m1 are text embeddings. , after cross-attention calculation, the output embedding of the cross-attention module m1 is obtained, that is, the first output embedding mentioned above.

[0084] The queries of the cross-attention module m2 are text embeddings The keys and values ​​of the cross-attention module m2 are the image embeddings , after cross-attention calculation, the output embedding of the cross-attention module m2 is obtained, which is the second output embedding mentioned above.

[0085] The query of the cross-attention module m3 is the output embedding of the cross-attention module m1. The keys and values ​​of the cross-attention module m3 are the output embedding of the cross-attention module m2. The output embedding of the cross-attention module m3 is obtained after the cross-attention calculation. .

[0086] The query of the cross-attention module m4 is the output embedding of the cross-attention module m2. The keys and values ​​of the cross-attention module m4 are the output embedding of the cross-attention module m1. The output embedding of the cross-attention module m4 is obtained after the cross-attention calculation. .

[0087] Exemplarily, the cross-modal interaction module embeds the first image into and the first text embedding After processing, the second image embedding is obtained and the second text embedding .

[0088] in, ,in is the embedding corresponding to all image patches, is a categorical embedding for image classification. , in is the word embedding of the text T, i.e., the multiple second word embeddings mentioned above, is the classification embedding for text classification, i.e. the second text classification embedding mentioned above.

[0089] Exemplarily, after obtaining the second image embedding, the second image classification embedding in the second image embedding may be input into the image classification module for classification.

[0090] Specifically, in the above step 202, the processing steps for the image classification module may also include the following steps 202d1 to 202d3:

[0091] Step 202d1: embed the second image classification into the fully connected layer of the image classification module in the recognition model to obtain two latent variable values.

[0092] Step 202d2: Input the two latent variable values ​​into the normalized exponential function layer of the image classification module in the recognition model to obtain two classification probability values.

[0093] Among them, one latent variable value corresponds to one classification probability value.

[0094] Step 202d3: Determine the category corresponding to the larger value of the two classification probability values ​​as the image forgery classification result.

[0095] The image forgery classification result is used to indicate an image forgery area in the image data to be identified.

[0096] For example, Figure 1 As shown, the input of the image classification module is the second image embedding Second image classification embedding in .

[0097] Exemplarily, the training process of the image classification module in the embodiment of the present application is as follows:

[0098] First, the classification embedding After a fully connected layer (weight parameter is ), get two latent variable values; then, these two latent variable values ​​pass through a Softmax layer to output the classification probability value and ,in:

[0099] ,

[0100]

[0101] Finally, according to the classification probability value and Construct the classification loss of image I :

[0102]

[0103] in, Represents the true value of the image classification, Representing images There are no fake areas in Representing images Contains fake areas; , Represents the first The transposed vector of the weight parameters of the columns, Represents the dot product between vectors.

[0104] For example, the actual application reasoning process of the image classification module is as follows:

[0105] First, the second image classification embedding of the image data to be identified After a fully connected layer, two latent variable values ​​are obtained; then, the two latent variable values ​​pass through a normalized exponential function (for example, Softmax) layer to output a classification probability value; finally, the category corresponding to the latent variable with a high classification probability value is the forged classification result of the image, and the classification result is used to indicate whether the image data to be identified is forged.

[0106] Exemplarily, after obtaining the second text embedding, the second word classification embedding in the second text embedding may be input into a text classification module for classification.

[0107] Specifically, in the above step 202, the processing steps for the image classification module may further include the following steps 202e1 and 202e2:

[0108] Step 202e1, embedding the second text classification into the fully connected layer of the text classification module in the recognition model to obtain two latent variable values, and inputting the two latent variable values ​​into the normalized exponential function layer of the image classification module in the recognition model to obtain two classification probability values.

[0109] Among them, one latent variable value corresponds to one classification probability value.

[0110] Step 202e2: Determine the category corresponding to the larger value of the two classification probability values ​​as the text forgery classification result.

[0111] The text forgery classification result is used to indicate the forged word position in the text data to be identified.

[0112] For example, Figure 1 As shown, the input of the text classification module is the second text classification embedding which is the second text embedding .

[0113] Exemplarily, the training process of the text classification module is similar to the application reasoning process, which is specifically as follows:

[0114] First, the second text classification embedding After a fully connected layer (weight parameter is ), and obtain two latent variable values; then, these two latent variable values ​​pass through a normalized exponential function (for example, Softmax) layer to output the classification probability value and ,in:

[0115]

[0116]

[0117] Finally, according to the classification probability value and Construct the classification loss of text T :

[0118]

[0119] in, Represents the true value of the annotation for text classification, Indicates that the text T has no forged tokens, Indicates that the text T contains fake tokens. , Represents the first The transposed vector of the weight parameters of the columns, Represents the dot product between vectors.

[0120] Exemplarily, the application reasoning process of the text classification module is as follows:

[0121] First, the second text classification embedding of the text data T to be recognized After a fully connected layer (weight parameter is ), and obtain two latent variable values; then, the two latent variable values ​​are passed through a normalized exponential function (for example, Softmax) layer to output a classification probability value; finally, the category corresponding to the latent variable with a high classification probability value is the forged classification result of the text T, and the forged classification result is used to indicate whether the text data to be identified is a forged text.

[0122] Exemplarily, in the embodiment of the present application, it is also necessary to classify and identify the image and text as a whole through an image-text classification module to improve the recognition accuracy of the model.

[0123] Specifically, in the above step 202, the processing step of the image text classification module may also include the following steps 202f1 to 202f3:

[0124] Step 202f1, concatenate the second image classification embedding and the second text classification embedding into a first target embedding vector, and input the first target embedding vector into the fully connected layer of the image-text pair classification module in the recognition model to obtain two latent variable values.

[0125] Step 202f2: input the two latent variable values ​​into the normalized exponential function layer of the image-text pair classification module to obtain two classification probability values.

[0126] Among them, one latent variable value corresponds to one classification probability value.

[0127] Step 202f3: Determine the category corresponding to the larger value of the two classification probability values ​​as the image-text pair forged classification result.

[0128] For example, in the embodiment of the present application, the second image classification of the image data to be identified is embedded in and the second text classification embedding of the text data to be recognized Concatenate into a new embedding vector , which is the first target embedding vector mentioned above. Among them, , and The dimension is d, The dimension of is 2d.

[0129] For example, Figure 1 As shown in the figure, the model training process is similar to the model application reasoning process, which is as follows:

[0130] First, the first target embedding vector After a fully connected layer (weight parameter is ), and obtain two latent variable values; then, these two latent variable values ​​pass through a normalized exponential function (for example, Softmax) layer to output the classification probability value and ,in:

[0131]

[0132]

[0133] Finally, the image-text pair classification loss is constructed based on the classification probability value :

[0134]

[0135] in, Represents the true value of the image text classification, Indicates that the image-text pair contains neither forged image regions nor forged word units; Indicates that the image-text pair contains forged image regions or forged words. , Represents the first The transposed vector of the weight parameters of the columns, Represents the dot product between vectors.

[0136] For example, the application reasoning process of the image text to the classification module is as follows:

[0137] First, the first target embedding vector After a fully connected layer (weight parameter is ), and obtain two latent variable values; then, these two latent variable values ​​pass through a Softmax layer and output two classification probability values; finally, the categories corresponding to the two latent variables with high classification probability values ​​are the forged classification results of the image-text pair, and the forged classification results are used to indicate whether the image data to be identified and the text data to be identified are forged as a whole.

[0138] For example, in an embodiment of the present application, in addition to identifying whether the object to be identified is forged, it is also necessary to identify the specific forged part through a cross-attention module and a forged area positioning module when it is determined that the object to be identified is forged graphic information.

[0139] Specifically, in the above step 202, the processing steps for the cross attention module may also include the following steps 202g1 and 202g2:

[0140] Step 202g1, using multiple second image block embeddings in the second image embedding as keys and values, and inputting the trained query vector as a query into the cross-attention module of the recognition model to obtain a second target embedding vector.

[0141] Step 202g2: input the second target embedding vector into the forged area positioning module of the recognition model, predict the relative position coordinates of the forged area in the image data to be recognized, and convert the predicted relative position coordinates into absolute pixel position coordinates.

[0142] The absolute pixel position coordinates are used to indicate a forged area in the image data to be identified when the image data to be identified is identified as a forged image.

[0143] For example, Figure 1 As shown, the cross attention module The input is the cross-modal interaction module Output Embedding in and a learnable query vector , The dimension of is the same as the dimension of the image patch embedding, ensuring that dot products can be performed between them.

[0144] Exemplarily, the criss-cross attention module by For query, embed the image block is the key and value, and the embedding vector is obtained after attention calculation , which is the second target embedding vector mentioned above. Embedding vector It combines all The information embedded in each image block.

[0145]

[0146] in, represent The transposed vector of represents the dot product between vectors. In the initial stage of training, is randomly initialized, and then during training is updated.

[0147] Specifically, in the above step 202, the processing step for the forged area positioning module may also include the following steps 202h1 to 202h3:

[0148] Step 202h1: input the second target embedding vector into the fully connected layer of the forged region positioning module to obtain four hidden variable values.

[0149] Step 202h2: input the four hidden variable values ​​into the preset activation function of the forged area positioning module to obtain four relative coordinate values.

[0150] Step 202h3: based on the pixel size of the image data to be identified, convert the four relative coordinate values ​​into absolute pixel coordinate values.

[0151] For example, the embodiment of the present application uses the forged region positioning module to predict the relative position coordinates of the forged region. The relative position coordinates refer to the ratio of the absolute position coordinates of the image pixel to the image width and image height. The input of the forged region positioning module is the cross attention module. Output embedding , the output is the relative position coordinates of the forged area. In this embodiment, the forged area positioning module includes a fully connected neural network and a preset activation function, for example, a Sigmoid function.

[0152] For example, based on Figure 1 ,like Figure 4 FIG. 1 is a schematic diagram of the structure of the forged area positioning module provided in the embodiment of the present application. In the model application reasoning process, The fully connected layer of the fake region positioning module outputs four hidden variable values , the four hidden variable values ​​are respectively passed through the Sigmoid function, and the value range is The four relative coordinate values ​​of Then, according to the width and height of the input image, the relative position coordinate values ​​are converted to absolute pixel position coordinate values.

[0153] For example, during the model training process, the four relative coordinate values ​​output by the forged area positioning module are compared with the pre-annotated true coordinate values. For comparison, the true value of the coordinates is the relative coordinate of the pre-marked forged area position. This embodiment uses GIoU loss to construct the forged area coordinate regression loss function :

[0154]

[0155] in, It is the input of the forged region localization module and the output of the cross attention module; Refers to the fully connected layer of the forged region localization module; It is the true position coordinate of the forged area marked in advance, that is, the relative position coordinate.

[0156] For example, in the embodiment of the present application, in addition to identifying whether the object to be identified is forged, it is also necessary to identify the specific forged part through the forged word unit location module when it is determined that the object to be identified is forged graphic information.

[0157] Specifically, in the above step 202, the processing steps for the forged word unit location module may also include the following steps 202k1 and 202k2:

[0158] Step 202k1, input any target word-unit embedding among the multiple second word-unit embeddings into the fully connected layer of the forged word-unit locating module of the recognition model to obtain two latent variable values, and input the two latent variable values ​​into the normalized exponential function layer to obtain two classification probability values.

[0159] Among them, one latent variable value corresponds to one classification probability value.

[0160] Step 202k2: Determine the category corresponding to the larger value of the two classification probability values ​​as the forged word-unit classification result of the target word-unit embedding.

[0161] The forged word-gram classification result is used to indicate whether the word-gram corresponding to the target word-gram embedding is a forged word-gram when the image data to be identified is identified as a forged image.

[0162] For example, based on Figure 1 ,like Figure 5 The figure is a schematic diagram of the structure of the forged word unit location module provided in the embodiment of the present application. The input of the forged word unit location module is the cross-modal interaction module Output In Each word embedding It is input separately into the forged word location module and outputs the classification result (0 represents a non-forged word, 1 represents a forged word).

[0163] For example, Figure 5 As shown, the model application reasoning process is as follows:

[0164] First, word embedding After a fully connected layer, two hidden variable values ​​are obtained ( and ); Then, these two latent variable values ​​pass through a Softmax layer and output the classification probability value ( and ); Finally, the category corresponding to the latent variable with high classification probability value is word embedding The classification results.

[0165] For example, Figure 5 As shown in the figure, the model training process of the forged word location module is similar to the model application reasoning process:

[0166] First, word embedding After a fully connected layer, two hidden variable values ​​are obtained ( and ); Then, these two latent variable values ​​pass through a Softmax layer to output the classification probability value and ,in

[0167]

[0168]

[0169] Finally, the classification loss of word embedding is constructed based on the classification probability value :

[0170]

[0171] in, represents the classification truth label of the i-th word, represents a non-forged word, Indicates a fake word; , Represents the first The transposed vector of the weight parameters of the columns, Represents the dot product between vectors.

[0172] Exemplarily, in an embodiment of the present application, a forged area mask prediction module can be added as an additional supervisory signal to accurately locate the fine-grained contour information of the forged area by predicting the local mask of the forged area, thereby further improving the accuracy of the forged area positioning module in predicting the position coordinates of the forged area.

[0173] Specifically, in the above step 202, the processing steps for the forged region mask prediction module may also include the following steps 202L1 to 202L4:

[0174] Step 202L1, extract multiple image block embedding groups from the output results of the last multiple encoding layers of the image encoder of the recognition model.

[0175] The output result of one coding layer corresponds to one image block embedding group; each image block embedding group contains image block embeddings having the same number as the multiple image blocks into which the image data to be identified is divided.

[0176] Step 202L2: splice the image block embeddings at corresponding positions in each image block embedding group in the multiple image block embedding groups and input them into the linear projection module of the recognition model to obtain multiple fused third image block embeddings.

[0177] Step 202L3: Generate a single-channel image of a preset pixel size based on the vector dot product operation results of the embeddings of the multiple third image blocks and the embeddings of the two prompt word units.

[0178] Step 202L4: After decoding the single-channel image through an image decoder, a forged region mask image having the same pixel size as the image data to be identified is obtained.

[0179] The forged region mask image is used to indicate the image forged region in the image data to be identified when the image data to be identified is identified as a forged image.

[0180] Specifically, the above step 202L3 may further include the following steps 202L31 to 202L33:

[0181] Step 202L31, perform vector dot product operations on the multiple third image block embeddings and the two prompt word embeddings in sequence to obtain a first operation result corresponding to each third image block embedding and a second operation result corresponding to each third image block embedding.

[0182] Among them, the first prompt word-unit embedding of the two prompt word-unit embeddings corresponds to the first operation result, and the second prompt word-unit embedding of the two prompt word-unit embeddings corresponds to the second operation result; the first prompt word-unit embedding is generated based on a preset first prompt word, and the first prompt word-unit embedding is generated based on a preset second prompt word.

[0183] Step 202L32: Based on the comparison result of the first operation result and the second operation result corresponding to each third image block being embedded, determine whether the image block corresponding to each third image block being embedded contains a forged image.

[0184] Among them, if a first operation result corresponding to the embedding of any target image block among the multiple third image block embeddings is greater than or equal to a second operation result corresponding to the embedding of the target image block, the target image block embedding is determined to be a forged image block; otherwise, the target image block is determined to be a non-forged image block.

[0185] Step 202L33: determine the label value corresponding to each image block based on the recognition result of each image block to obtain multiple label values, and rearrange the multiple label values ​​based on the position information of the image block corresponding to the label value to obtain the single-channel image.

[0186] For example, based on Figure 1 ,like Figure 6 FIG. 1 is a schematic diagram of the structure of the forged region mask prediction module provided in the embodiment of the present application. The outputs of the last three encoding layers extract three sets of image patch embeddings (excluding classification embeddings), which are defined as (Output of the third-to-last encoding layer), (output of the penultimate encoding layer), (Output of the last encoding layer). , and Each of the embeddings contains p image blocks (i.e., the above-mentioned multiple image block embedding groups), and the dimension of each image block embedding is 1024 dimensions. , and The image blocks at corresponding positions in the image are embedded and spliced ​​together, and then the spliced ​​image blocks are embedded and projected into a 1024-dimensional vector (embedding) through a shared linear projection module (including a fully connected network). After the above processing, the image blocks at all positions are obtained by fusion The image block embedding , image patch embedding It contains p embeddings (corresponding to p image blocks, i.e., the above-mentioned multiple third image block embeddings).

[0187] For example, Figure 6 As shown, this embodiment uses a text encoder The two preset prompt words and Transformed into two word embeddings and (i.e., the two hint word embeddings mentioned above). When the text descriptions in the training data are in English, is 'A photo of an object', is 'A photo of an unnatural object'. When the text description in the training data is in Chinese, is "This is a natural image", is "This is a fake image". When the text descriptions in the training data are both in Chinese and English, in each training iteration, and Randomly choose between the two settings above.

[0188] For example, the word embeddings of the two prompt words and Embedded with image blocks Each embedding in Multiply (i.e. vector dot product operation). If (i.e., the comparison result of the first operation result and the second operation result), then the corresponding image block is marked as 0, otherwise it is marked as 1. 0 represents that the corresponding image block contains a non-forged target, and 1 represents that the corresponding image block contains a forged target. represent The transposed vector of represent The transposed vector of Represents the dot product between vectors. After the above processing, each image block corresponds to a label value (0 or 1).

[0189] Afterwards, according to the position information of the image blocks where the above p label values ​​are located, the positions are rearranged to obtain 16 A single-channel image of 16 pixels (i.e. the above preset pixel size), where the width is 16 pixels and the height is 16 pixels. The above single-channel image is passed through the image decoder (bilinear interpolation) to obtain an image of the same size as the input image (width and height are L pixels), and this image is defined as the forged area mask image . Mask the image according to the forged region And the true value of the forged region mask marked, this embodiment defines the following forged region mask prediction loss function :

[0190]

[0191] in, Represents the true value of the annotated forged area mask. The pixels in the forged area are marked as 1, and other pixels are marked as 0. Represents the L1 norm.

[0192] Exemplarily, the embodiment of the present application constructs the loss function of each branch and linearly weights them. This joint loss function fully considers the importance of each branch and weighs their importance.

[0193] Specifically, during the training phase of the model, when a batch of samples containing B training samples (each training sample is an image-text pair) is input, the overall loss function of the training is defined as:

[0194]

[0195] in, represents the image classification loss of the i-th input sample, represents the text classification loss of the i-th input sample, represents the image-text classification loss of the i-th input sample, represents the forged region coordinate regression loss of the i-th input sample, represents the word classification loss of the i-th input sample, represents the forged region mask prediction loss. Other hyperparameter settings, , , , , , .

[0196] Exemplarily, in the embodiment of the present application, the method for setting small batch training samples is as follows:

[0197] The total training cycle (epoch) is set to 20 cycles. One training cycle means that all training samples are used once without repetition. In order to prevent the order of samples in the data set from affecting the performance of the model and to improve the generalization performance of the model, all samples in the training data set are randomly rearranged at the beginning of each training cycle, that is, the order is shuffled. In each training cycle, the shuffled training data set is divided into smaller sets, and this small sample set is called a mini-batch sample. Depending on the configuration of the training server, the number of samples in each mini-batch can be set to 128, 256 or 512, etc. The model training algorithm reads a mini-batch of training samples each time and uses these samples to calculate the loss value, update the model parameters, etc.

[0198] After a training cycle (epoch) ends, the model training algorithm enters the next training cycle (epoch). When the total number of training cycles reaches the preset value, the model training ends. It is worth noting that in special cases, within a training cycle, the total number of samples in the data set cannot be divided into integer batches, and individual samples are not used (discarded directly) or are used multiple times (to complete the data, for example: adding several samples that have been used to the end of the data set). The impact of these individual cases on the training results can be ignored.

[0199] For example, in the embodiment of the present application, the recognition model can be trained by the following training method:

[0200] Training input (initial parameter setting): from the training data set Read data in; initial training cycle , total training cycle ; Stochastic gradient descent algorithm related parameters: The initial learning rate is , momentum coefficient , the weight decay coefficient is , in the 10th training cycle, the learning rate drops to , in the 15th training cycle, the learning rate drops to , in the 18th training cycle, the learning rate dropped to The number of samples in a mini-batch is .

[0201] Training output: Multimodal forged image and text information recognition network model (The parameter of the model is φ);

[0202] Loss function: Set the loss function defined in this application example ;

[0203] Step 1: Initialization

[0204] Image encoding module The parameters in are initialized using the parameters of ViT-L / 16 pre-trained on the ImageNet-21k dataset; text encoder The parameters are used Initialize the parameters trained on the "General Language Understanding Evaluation Benchmark Collection"; use the random initialization method to initialize the learnable parameters of the recognition model.

[0205] Step 2: When When , do the following:

[0206] S1, ;

[0207] S2. Data collection The order of samples in is randomly shuffled;

[0208] S3, from the data collection Select a batch of B training samples;

[0209] S4. Calculate the loss value based on the above training samples and parameter settings , and then use the stochastic gradient descent algorithm to update the model parameters:

[0210]

[0211] in, represents the image classification loss of the i-th input sample, represents the text classification loss of the i-th input sample, represents the image-text classification loss of the i-th input sample, represents the forged region coordinate regression loss of the i-th input sample, represents the word classification loss of the i-th input sample, represents the forged region mask prediction loss. Other hyperparameter settings, , , , , , ;

[0212] S5, repeat steps S3 and S4 until the data set of the samples were fully utilized;

[0213] S6. End the current training cycle.

[0214] Step 3: End the training and output the multimodal forged image and text information recognition network model

[0215] For example, in the embodiment of the present application, the specific steps in the actual application reasoning stage of the model are as follows:

[0216] 1. Get the input image to be recognized and the corresponding input text description T

[0217] Multimodal recognition model It is a multimodal recognition model trained using the embodiment of the present application. Input image The specifications are consistent with those of the input training images when training the multimodal recognition model.

[0218] If the input training image when training the multimodal recognition model is an image with a width and height of L pixels, then the input image The width and height of the input image I should also be L pixels. If the input image I does not meet the above specifications, the long side of the input image I is scaled to L pixels; then, the short side of the input image is also scaled proportionally; finally, the short side is padded with zeros to L pixels. After the above preprocessing operations, the preprocessed image can be obtained ,image The width and height are equal, both L pixels. Input into the multimodal forged image and text information recognition system.

[0219] If the input training image when training the multimodal recognition model is a three-channel color image (RGB image), then the input image It should also be a three-channel color image (RGB image) or input image Converted to a three-channel color image (RGB image).

[0220] The specifications of the input text description T are consistent with those of the input text description when training the multimodal recognition model. If the input text description when training the multimodal recognition model is in a specific language (for example, Chinese), then the input text description T should also be in a specific language (for example, Chinese); if the input text description when training the multimodal recognition model is in multiple languages, then the input text description T can also be in multiple languages.

[0221] 2. Input the image and the corresponding input text description T are simultaneously input into the multimodal recognition model , get the input image The relative position coordinates of the forged area, the forged word position, the image forged classification results, the text forged classification results, and the image-text pair forged classification results.

[0222] 3. According to the input image The relative position coordinates of the forged area and the input image The size of the input image is obtained The absolute position coordinates of the forged area in . If the input image has undergone image scaling transformation, the corresponding inverse transformation needs to be performed when calculating the absolute position coordinates.

[0223] 4. Output the recognition results (not only the forged results, but also the forged areas of the image and the locations of the forged words of the text):

[0224] Result 1: If the image-text pair forgery classification result is 1, the image forgery classification result is 0, and the text forgery classification result is 1, then output the forged word unit position.

[0225] Result 2: If the image-text pair forgery classification result is 1, the image forgery classification result is 1, and the text forgery classification result is 0, then the absolute position coordinates of the forged area are output.

[0226] Result 3: If the image-text pair forgery classification result is 1, the image forgery classification result is 1, and the text forgery classification result is 1, then the forged word unit position and the forged area absolute position coordinates are output.

[0227] Result 4: If the image-text pair forgery classification result is 0, the image forgery classification result is 0, and the text forgery classification result is 0, then the "no forgery" prompt is output.

[0228] Result 5. In other cases, the prompt "The result is unreliable" is output.

[0229] Specifically, it is shown in the following Table 1:

[0230]

[0231] Table 1

[0232] It should be noted that the embodiment of the present application proposes a new neural network framework for identifying multimodal forged image and text information, including an image encoder, a text encoder, a cross-modal interaction module, an image classification module, a text classification module, an image-text pair classification module, a cross-attention module, a forged region positioning module, a forged word unit positioning module, and a forged region mask prediction module. In particular, the embodiment of the present application adds a forged region mask prediction module as an additional supervisory signal, and accurately locates the fine-grained contour information of the forged region by predicting the local mask of the forged region, further improving the accuracy of the forged region positioning module in predicting the position coordinates of the forged region.

[0233] The embodiment of the present application constructs the loss function of each branch and linearly weights them. This joint loss function fully considers the importance of each branch and weighs their importance.

[0234] The multimodal recognition method for graphic information provided in the embodiment of the present application first obtains the image data to be recognized and the text data to be recognized of the object to be recognized; then, the object to be recognized is input into the recognition model to determine whether the object to be recognized contains forged content, and if it is determined that the object to be recognized contains forged content, a forged content recognition result is generated; wherein the recognition result includes: the image forged area in the image data to be recognized, and / or the forged word position in the text data to be recognized. In this way, the recognition accuracy of forged graphic information can be improved by multimodal recognition, and the forged part of the forged graphic information can be accurately located.

[0235] Figure 7An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the multimodal recognition method of image and text information, the method comprising: first, obtaining the image data to be recognized and the text data to be recognized of the object to be recognized; then, inputting the object to be recognized into the recognition model, determining whether the object to be recognized contains forged content, and generating a forged content recognition result when it is determined that the object to be recognized contains forged content; wherein the recognition result includes: the image forged area in the image data to be recognized, and / or the forged word position in the text data to be recognized.

[0236] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0237] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the multimodal recognition method of graphic information provided by the above methods, and the method includes: first, obtaining image data to be recognized and text data to be recognized of an object to be recognized; then, inputting the object to be recognized into a recognition model, judging whether the object to be recognized contains forged content, and generating a forged content recognition result when it is determined that the object to be recognized contains forged content; wherein the recognition result includes: the image forged area in the image data to be recognized, and / or the forged word position in the text data to be recognized.

[0238] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal recognition method for the above-mentioned graphic and text information, the method comprising: first, obtaining image data to be recognized and text data to be recognized of an object to be recognized; then, inputting the object to be recognized into a recognition model, determining whether the object to be recognized contains forged content, and generating a forged content recognition result when it is determined that the object to be recognized contains forged content; wherein the recognition result comprises: an image forged area in the image data to be recognized, and / or a forged word position in the text data to be recognized.

[0239] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0240] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimodal recognition method for graphic information, characterized in that: include: Acquire image data and text data of an object to be identified; Inputting the object to be identified into the identification model, determining whether the object to be identified contains forged content, and generating a forged content identification result if it is determined that the object to be identified contains forged content; Wherein, the recognition result includes: the image forgery area in the image data to be recognized, and / or the forged word position in the text data to be recognized; The step of inputting the object to be identified into a recognition model and determining whether the object to be identified contains forged content includes: Inputting the text data to be recognized into the text encoder of the recognition model, and dividing the text data to be recognized into a plurality of word units by the word segmenter of the text encoder; Acquire multiple embedding vectors to be processed from the word unit embedding vector table of the text encoder; one word unit corresponds to one embedding vector to be processed; Performing forward processing on the multiple embedding vectors to be processed to obtain a first text embedding corresponding to the text data to be recognized; The first text embedding includes: a first text classification embedding, and a plurality of first word element embeddings; the number of the plurality of first word element embeddings is the same as the number of the plurality of embedding vectors to be processed; The step of inputting the object to be identified into a recognition model and determining whether the object to be identified contains forged content includes: Inputting the first image embedding and the first text embedding into the cross-modal interaction module of the recognition model to obtain a second image embedding corresponding to the first image embedding and a second text embedding corresponding to the first text embedding; Among them, the first image embedding is: the result of the image encoder output of the recognition model; the first text embedding is: the result of the text encoder output of the recognition model; the second image embedding includes: a plurality of second image block embeddings having the same number as the plurality of first image block embeddings in the first image embedding, and a second image classification embedding corresponding to the first image classification embedding in the first image embedding; the second text embedding includes: a plurality of second word embeddings having the same number as the plurality of first word embeddings in the first text embedding, and a second text classification embedding corresponding to the first text classification embedding in the first text embedding.

2. The method according to claim 1, characterized in that The step of obtaining the image data to be identified and the text data to be identified of the object to be identified includes: Get the image and text information to be processed; Adjusting the image pixel size of the image to be processed in the image and text information to be processed to a preset pixel size to obtain the image data to be recognized; the preset pixel size is determined based on a training sample for training the recognition model; The image data to be recognized and the text data to be recognized contained in the image and text information to be processed are determined as the objects to be recognized.

3. The method according to claim 1, characterized in that The step of inputting the object to be identified into a recognition model and determining whether the object to be identified contains forged content includes: Evenly dividing the image data to be recognized into a plurality of image blocks; The multiple image blocks are input into the image encoder of the recognition model to obtain a first image embedding corresponding to the image data to be recognized.

4. The method according to claim 1, characterized in that The cross-modal interaction module includes: a first cross-attention module, a second cross-attention module, a third cross-attention module and a fourth cross-attention module; The step of inputting the first image embedding and the first text embedding into the cross-modal interaction module of the recognition model to obtain a second image embedding corresponding to the first image embedding and a second text embedding corresponding to the first text embedding includes: Inputting the first image embedding as a query and the first text embedding as a key and value into the first criss-cross attention module to obtain a first output embedding; and inputting the first text embedding as a query and the first image embedding as a key and value into the second criss-cross attention module to obtain a second output embedding; Input the first output embedding as a query and the second output embedding as a key and value into the third cross-attention module to obtain the second image embedding; and input the second output embedding as a query and the first output embedding as a key and value into the fourth cross-attention module to obtain the second text embedding.

5. The method according to claim 1 or 4, characterized in that: The step of inputting the object to be identified into a recognition model and determining whether the object to be identified contains forged content includes: Inputting the second image classification embedding into the fully connected layer of the image classification module in the recognition model to obtain two latent variable values; Inputting the two latent variable values ​​into the normalized exponential function layer of the image classification module to obtain two classification probability values; one latent variable value corresponds to one classification probability value; Determining the category corresponding to the larger value of the two classification probability values ​​as the image forgery classification result; The image forgery classification result is used to indicate an image forgery area in the image data to be identified.

6. The method according to claim 1 or 4, characterized in that: The step of inputting the object to be identified into a recognition model and determining whether the object to be identified contains forged content includes: The second text classification embedding is input into the fully connected layer of the text classification module in the recognition model to obtain two latent variable values, and the two latent variable values ​​are input into the normalized exponential function layer of the image classification module in the recognition model to obtain two classification probability values; one latent variable value corresponds to one classification probability value; Determine the category corresponding to the larger value of the two classification probability values ​​as the text forgery classification result; The text forgery classification result is used to indicate the forged word position in the text data to be identified.

7. The method according to claim 1 or 4, characterized in that: The step of inputting the object to be identified into a recognition model and determining whether the object to be identified contains forged content includes: Concatenate the second image classification embedding and the second text classification embedding into a first target embedding vector, and input the first target embedding vector into a fully connected layer of an image-text pair classification module in the recognition model to obtain two latent variable values; Inputting the two latent variable values ​​into the normalized exponential function layer of the image-text pair classification module to obtain two classification probability values; one latent variable value corresponds to one classification probability value; The category corresponding to the larger value of the two classification probability values ​​is determined as the image-text pair forged classification result.

8. The method according to claim 1 or 4, characterized in that: The generating of the forged content identification result includes: Using the multiple second image block embeddings in the second image embedding as keys and values ​​and the trained query vector as a query into a cross-attention module of the recognition model to obtain a second target embedding vector; Inputting the second target embedding vector into the forged region positioning module of the recognition model, predicting the relative position coordinates of the forged region in the image data to be recognized, and converting the predicted relative position coordinates into absolute pixel position coordinates; The absolute pixel position coordinates are used to indicate a forged area in the image data to be identified when the image data to be identified is identified as a forged image.

9. The method according to claim 8, characterized in that The step of inputting the second target embedding vector into the forged region positioning module of the recognition model, predicting the relative position coordinates of the forged region in the image data to be recognized, and converting the predicted relative position coordinates into absolute pixel position coordinates includes: Inputting the second target embedding vector into the fully connected layer of the forged region positioning module to obtain four hidden variable values; Inputting the four hidden variable values ​​into the preset activation function of the forged area positioning module to obtain four relative coordinate values; Based on the pixel size of the image data to be recognized, the four relative coordinate values ​​are converted into absolute pixel coordinate values.

10. The method according to claim 1 or 4, characterized in that: The generating of the forged content identification result includes: Inputting any target word-unit embedding among the plurality of second word-unit embeddings into the fully connected layer of the forged word-unit locating module of the recognition model to obtain two latent variable values, and inputting the two latent variable values ​​into the normalized exponential function layer to obtain two classification probability values; one latent variable value corresponds to one classification probability value; Determine the category corresponding to the larger value of the two classification probability values ​​as the forged word-meta classification result of the target word-meta embedding; The forged word-gram classification result is used to indicate whether the word-gram corresponding to the target word-gram embedding is a forged word-gram when the image data to be identified is identified as a forged image.

11. The method according to claim 1, characterized in that The generating of the forged content identification result includes: Extracting multiple image block embedding groups from the output results of the last multiple encoding layers of the image encoder of the recognition model; the output result of one encoding layer corresponds to one image block embedding group; each image block embedding group contains the same number of image block embeddings as the multiple image blocks into which the image data to be recognized is divided; splicing the image block embeddings at corresponding positions in each image block embedding group in the multiple image block embedding groups and inputting the spliced ​​image block embeddings into the linear projection module of the recognition model to obtain multiple fused third image block embeddings; Generate a single-channel image of a preset pixel size based on the vector dot product operation results of the embeddings of the plurality of third image blocks and the embeddings of the two prompt word units; After decoding the single-channel image through an image decoder, a forged region mask image having the same pixel size as the image data to be identified is obtained; The forged region mask image is used to indicate the image forged region in the image data to be identified when the image data to be identified is identified as a forged image.

12. The method according to claim 11, characterized in that The step of generating a single-channel image of a preset pixel size based on the vector dot product operation results of the embedding of the plurality of third image blocks and the embedding of the two prompt word units comprises: Perform vector dot product operations on the multiple third image block embeddings and the two prompt word-mem embeddings in sequence to obtain a first operation result corresponding to each third image block embedding and a second operation result corresponding to each third image block embedding; the first prompt word-mem embedding of the two prompt word-mem embeddings corresponds to the first operation result, and the second prompt word-mem embedding of the two prompt word-mem embeddings corresponds to the second operation result; the first prompt word-mem embedding is generated based on a preset first prompt word, and the first prompt word-mem embedding is generated based on a preset second prompt word; Based on the comparison result of the first operation result and the second operation result corresponding to each third image block embedding, determining whether the image block corresponding to each third image block embedding contains a forged image; wherein, if the first operation result corresponding to the embedding of any target image block among the plurality of third image block embeddings is greater than or equal to the second operation result corresponding to the embedding of the target image block, then determining that the target image block embedding is a forged image block; otherwise, determining that the target image block is a non-forged image block; Based on the recognition result of each image block, the label value corresponding to each image block is determined to obtain multiple label values, and based on the position information of the image block corresponding to the label value, the multiple label values ​​are rearranged to obtain the single-channel image.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the multimodal recognition method of graphic information described in any one of claims 1 to 12 are implemented.

14. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the multimodal recognition method for graphic and text information as claimed in any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the multimodal recognition method for graphic and text information as claimed in any one of claims 1 to 12 are implemented.

Images