Information identification method and device

By extracting multimodal features of images and text in the information recognition model and predicting forged elements, the problem of low accuracy of forged information recognition in the prior art is solved, and high accuracy recognition of forged information is achieved.

CN118711198BActive Publication Date: 2024-11-15INSPUR SUZHOU INTELLIGENT TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411207260.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-15
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

The accuracy of identification of forged information in the prior art is low, and it is impossible to effectively identify various forged information on the Internet.

Method used

By introducing feature extraction module, forged element prediction module and large language model in the information recognition model, multimodal features of images and text are extracted, forged region coordinates and word element indexes are predicted, and they are integrated into the prompt words of the large language model. , improve the accuracy of identification of forged information.

Benefits of technology

It significantly improves the accuracy of identification of forged information, can accurately identify forged information, and enhances the recognition ability of large language models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118711198B_ABST
    Figure CN118711198B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of information processing technology, and provides an information recognition method and device, the method comprising: inputting an image to be recognized and a text to be recognized in the information to be recognized into a feature extraction module of an information recognition model to obtain a target image embedding feature and a target text embedding feature; inputting the target image embedding feature and the target text embedding feature into a forged element prediction module to obtain forged region coordinates in the image to be recognized and a forged word element index in the text to be recognized; generating prompt words based on the forged region coordinates and the forged word element index; inputting a large model learning vector group, prompt words, target image embedding features and target text embedding features into a large language model to obtain a recognition result of the information to be recognized output by the large language model. The present invention integrates the forged region coordinates and the forged word element index into the prompt words of the large language model, thereby improving the recognition ability of the large language model for forged information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to an information identification method and device. Background Art

[0002] With the advancement of deep learning, the continuous development of computer vision and natural language processing technology, especially the rise of Generative Adversarial Networks (GANs) and Large Language Models (LLM), image and text forgery technology has provided technical support. Therefore, there is an increasing amount of forged information (such as news) on the Internet based on image and text forgery technology. Faced with the massive amount of information on the Internet, users can hardly tell which information is forged.

[0003] In order to identify forged information and reduce the amount of forged information on the Internet as much as possible, a recognition model based on the authenticity recognition of images and texts has been proposed in the existing related technologies to identify the authenticity of information on the Internet. However, the neural network used to build such recognition models is relatively simple, and has limited ability to express the features of images and texts. It is unable to accurately identify forged information, and thus has a low recognition accuracy rate for forged information. Summary of the invention

[0004] The present invention provides an information identification method and device, which are used to solve the problem of low accuracy in identifying forged information in the prior art.

[0005] The present invention provides an information identification method, comprising the following steps.

[0006] Inputting the image to be recognized and the text to be recognized in the information to be recognized into the feature extraction module of the information recognition model to obtain the target image embedding feature and the target text embedding feature;

[0007] Inputting the target image embedding features and the target text embedding features into a forged element prediction module of an information recognition model to obtain forged region coordinates in the image to be recognized and forged word element indexes in the text to be recognized;

[0008] Generate a prompt word based on the forged area coordinates and the forged word element index;

[0009] Inputting the large model learning vector group, the prompt word, the target image embedding feature and the target text embedding feature into the large language model in the information recognition model, and obtaining the recognition result of the information to be recognized output by the large language model;

[0010] The information recognition model is obtained by training based on the sample image, sample text, forged region coordinate labels corresponding to the sample image, and word unit true value labels corresponding to the sample text in the sample information.

[0011] According to an information recognition method provided by the present invention, the feature extraction module includes: an encoding module and a cross-modal interaction module;

[0012] The step of inputting the image to be identified and the text to be identified in the information to be identified into the feature extraction module of the information recognition model to obtain the target image embedding feature and the target text embedding feature includes:

[0013] Inputting the image to be recognized and the text to be recognized into the encoding module to obtain the source image embedding feature of the image to be recognized and the source text embedding feature of the text to be recognized;

[0014] Inputting the source image embedding feature and the source text embedding feature into the cross-modal interaction module to obtain the cross-modal image embedding feature of the image to be recognized and the cross-modal text embedding feature of the text to be recognized;

[0015] The cross-modal image embedding feature is determined as the target image embedding feature, and the cross-modal text embedding feature is determined as the target text embedding feature.

[0016] According to an information recognition method provided by the present invention, the feature extraction module includes: an encoding module, a cross-modal interaction module and a feature enhancement module;

[0017] The step of inputting the image to be identified and the text to be identified in the information to be identified into the feature extraction module of the information recognition model to obtain the target image embedding feature and the target text embedding feature includes:

[0018] Inputting the image to be recognized and the text to be recognized into the encoding module to obtain the source image embedding feature of the image to be recognized and the source text embedding feature of the text to be recognized;

[0019] Inputting the source image embedding feature and the source text embedding feature into the cross-modal interaction module to obtain the cross-modal image embedding feature of the image to be recognized and the cross-modal text embedding feature of the text to be recognized;

[0020] Inputting the learnable vector and the cross-modal image embedding feature into the feature enhancement module to obtain an enhanced image embedding feature;

[0021] The enhanced image embedding feature is determined as the target image embedding feature, and the cross-modal text embedding feature is determined as the target text embedding feature.

[0022] According to an information recognition method provided by the present invention, the feature enhancement module includes: at least one of a cross attention module and a linear projection module;

[0023] The step of inputting the learnable vector and the cross-modal image embedding feature into the feature enhancement module to obtain the enhanced image embedding feature comprises:

[0024] Inputting the first learnable vector and the cross-modal image embedding feature into the cross-attention module to obtain a first image embedding feature;

[0025] Inputting the second learnable vector and the cross-modal image embedding feature into the linear projection module to obtain a second image embedding feature;

[0026] The first image embedding feature, the second image embedding feature, or the sum of the first image embedding feature and the second image embedding feature is determined as the enhanced image embedding feature.

[0027] According to an information recognition method provided by the present invention, the cross-modal interaction module includes: a first cross-attention submodule and a second cross-attention submodule;

[0028] Inputting the source image embedding feature and the source text embedding feature into the cross-modal interaction module to obtain the cross-modal image embedding feature of the image to be recognized and the cross-modal text embedding feature of the text to be recognized, including:

[0029] Inputting the source image embedding feature as a query parameter and the source text embedding feature as a key parameter and a value parameter into the first cross-attention submodule to obtain the cross-modal image embedding feature;

[0030] The source text embedding feature is used as a query parameter and the source image embedding feature is used as a key parameter and a value parameter to input into the second cross-attention submodule to obtain the cross-modal text embedding feature.

[0031] According to an information recognition method provided by the present invention, the encoding module includes: an image encoder and a text encoder;

[0032] Inputting the image to be recognized and the text to be recognized into the encoding module to obtain the source image embedding feature of the image to be recognized and the source text embedding feature of the text to be recognized, including:

[0033] Dividing the image to be recognized into a plurality of image blocks;

[0034] Input each of the image blocks into the image encoder to obtain the image classification embedding features and each image block embedding features output by the image encoder, wherein the image block embedding features and the image classification embedding features constitute the source image embedding features;

[0035] The text to be recognized is input into the text encoder, and the text encoder divides the text to be recognized into word units and extracts features to obtain the source text embedding features, which include: each word unit embedding feature and word unit classification embedding feature.

[0036] According to an information identification method provided by the present invention, the forged element prediction module includes: a forged region position prediction module and a forged word unit prediction module;

[0037] Inputting the target image embedding features and the target text embedding features into the forged element prediction module of the information recognition model to obtain the forged region coordinates in the image to be recognized and the forged word element index in the text to be recognized, including:

[0038] Inputting the target image embedding feature into the forged area position prediction module to obtain the forged area coordinates in the image to be identified;

[0039] The target text embedding feature is input into the forged word prediction module to obtain the forged word index in the text to be recognized.

[0040] According to an information identification method provided by the present invention, a forged area location prediction module includes: a first fully connected neural network and a mapping module;

[0041] Inputting the target image embedding feature into the forged area position prediction module to obtain the forged area coordinates in the image to be identified includes:

[0042] Inputting the target image embedding feature into the first fully connected neural network to obtain at least one group of latent vectors output by the first fully connected neural network, each group of latent vectors including four latent variable values;

[0043] The latent vector is input into the mapping module, and the four latent variable values ​​in the latent vector are mapped to four relative coordinate values ​​in the numerical range of [0, 1] by the mapping module, and the four relative coordinate values ​​of each group are determined as the coordinates of the forged area.

[0044] According to an information identification method provided by the present invention, the forged word unit prediction module includes: a second fully connected neural network and a classification output module;

[0045] Inputting the target text embedding feature into the forged word prediction module to obtain the forged word index in the text to be recognized includes:

[0046] Inputting a word unit embedding feature in the target text embedding feature into the second fully connected neural network, and obtaining two latent variable values ​​corresponding to the word unit embedding feature output by the second fully connected neural network;

[0047] Inputting the two latent variable values ​​into the classification output module, obtaining the forged category probability values ​​and the non-forged category probability values ​​respectively corresponding to the two latent variable values ​​output by the classification output module;

[0048] When the forged category probability value is greater than the non-forged category probability value, the position index of the word element corresponding to the latent variable value of the forged category probability value in the text is obtained as the forged word element index.

[0049] According to an information identification method provided by the present invention, a prompt word is generated based on the forged area coordinates and the forged word element index, including:

[0050] Obtain a prompt word template, and identify keywords representing the coordinates of the forged area, keywords representing the forged word element index, and keywords representing the title of the information to be identified in the prompt word template;

[0051] The forged area coordinates, the forged word element index and the title of the information to be identified are used to replace the adapters in the corresponding keywords to generate the prompt word.

[0052] According to an information recognition method provided by the present invention, the information recognition model is trained based on the following steps:

[0053] Acquire a training data set, wherein the training data set includes a plurality of sample information, each of which includes: a sample image, a sample text, a forged region coordinate label corresponding to the sample image, and a word unit true value label corresponding to the sample text;

[0054] Based on different sample information, the initial model corresponding to the information recognition model is iterated for multiple cycles of training, and when a preset training cycle threshold is reached or the total loss function converges, the training is completed to obtain the information recognition model;

[0055] Among them, the total loss function is obtained by weighted summing up the forged region coordinate regression loss function of the forged element prediction module, the word unit classification loss function and the word unit regression loss function of the large language model. In each iterative training process, the sample information is input into the initial model to obtain the forged region prediction coordinates predicted by the forged element prediction module, the word unit classification prediction result and the prediction result of the large language model. The forged region prediction coordinates, the word unit classification prediction result, the prediction result of the large language model, the forged region coordinate label and the word unit true value label are substituted into the total loss function, and the calculation result of the total loss function is back-propagated to update the model parameters of the feature extraction module and the forged element prediction module in the initial model, and to update the large model learning vector group.

[0056] According to an information recognition method provided by the present invention, based on different sample information, an initial model corresponding to the information recognition model is iteratively trained for multiple cycles, including:

[0057] In each training cycle, randomly sorting and adjusting each sample information in the training data set, and dividing the randomly sorted and adjusted training data set into a plurality of sub-training sets;

[0058] Each sub-training set is regarded as a training batch, and iterative training is performed on the basis of the initial model obtained by the previous batch training to obtain the initial model after the current batch training.

[0059] According to an information identification method provided by the present invention, in the sample information, the forged information of the sample image includes: at least one of: forged information in which a target object in the sample image is replaced and forged information in which an emotional state of the target object in the sample image is replaced, and the forged information of the sample text includes: forged information in which a paragraph, word or phrase used to represent the emotional state in the sample text is replaced.

[0060] The present invention also provides an information identification device, comprising the following functional units.

[0061] A target embedding feature extraction unit is used to input the image to be identified and the text to be identified in the information to be identified into the feature extraction module of the information recognition model to obtain the target image embedding feature and the target text embedding feature;

[0062] a forged element prediction unit, configured to input the target image embedding features and the target text embedding features into a forged element prediction module of an information recognition model, and obtain forged region coordinates in the image to be recognized and forged word element indexes in the text to be recognized;

[0063] A prompt word generating unit, used for generating a prompt word based on the forged area coordinates and the forged word element index;

[0064] A large model execution unit, used for inputting the large model learning vector group, the prompt word, the target image embedding feature and the target text embedding feature into the large language model in the information recognition model, and obtaining the recognition result of the information to be recognized output by the large language model;

[0065] The information recognition model is obtained by training based on the sample image, sample text, forged region coordinate labels corresponding to the sample image, and word unit true value labels corresponding to the sample text in the sample information.

[0066] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described information identification methods when executing the computer program.

[0067] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the information identification method as described in any one of the above is implemented.

[0068] The information recognition method and device provided by the present invention realize multimodal feature expression of the image and the text by extracting respective features of the image to be recognized and the text to be recognized in the information to be recognized, and adopt a forged element prediction module to obtain the forged area coordinates in the image to be recognized and the forged word element index in the text to be recognized based on the multimodal feature expression, and integrate the forged area coordinates and the forged word element index into the prompt words of the large language model, thereby improving the quality of the prompt words of the large language model, thereby improving the recognition ability of the large language model for forged information, being able to accurately recognize forged information, and improving the recognition accuracy of the corresponding forged information. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0070] Figure 1 This is one of the flow charts of the information identification method provided by the present invention.

[0071] Figure 2 It is a schematic diagram of the information identification model structure in the information identification method provided by the present invention.

[0072] Figure 3 This is the second flow chart of the information identification method provided by the present invention.

[0073] Figure 4It is a schematic diagram of the structure of the cross-modal interaction module in the information recognition model provided by the present invention.

[0074] Figure 5 This is the third flow chart of the information identification method provided by the present invention.

[0075] Figure 6 It is a schematic diagram of the structure of the forged area location prediction module in the information recognition model provided by the present invention.

[0076] Figure 7 It is a schematic diagram of the structure of the forged word prediction module in the information recognition model provided by the present invention.

[0077] Figure 8 It is a structural schematic diagram of the information identification device provided by the present invention.

[0078] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] The information identification method of the embodiment of the present invention is as follows: Figure 1 As shown, the process includes the following steps S110 to S140.

[0081] Step S110: Input the image to be identified and the text to be identified in the information to be identified into the feature extraction module of the information recognition model to obtain the target image embedding feature and the target text embedding feature. Specifically, the information to be identified in the network basically appears in the form of image-text pairs, such as: a piece of news, with an image and a corresponding text description. In this step, the features of the image to be identified and the text to be identified in the image-text pair are extracted to achieve multi-modal feature expression.

[0082] Step S120: Input the target image embedding features and the target text embedding features into the forged element prediction module of the information recognition model to obtain the coordinates of the forged area in the image to be recognized and the forged word element index in the text to be recognized. Specifically, the forged element prediction module is used to predict the forged area in the image to be recognized and the forged word element index in the text to be recognized, wherein the forged area in the image to be recognized, for example, is a replaced face area or a face area with a modified expression, and the forged word element index is the position index of the forged word in the text to be recognized.

[0083] Step S130: Generate prompt words based on the forged area coordinates and the forged word element index, wherein the prompt words are prompt words of the large language model. In this embodiment, prompt words are generated based on the forged area coordinates and the forged word element index, that is, the forged area coordinates and the forged word element index predicted by the forged element prediction module are integrated into the prompt words of the large language model, thereby improving the quality of the prompt words of the large language model, thereby improving the large language model's ability to recognize forged information.

[0084] Step S140: inputting the large model learning vector group, the prompt word, the target image embedding feature and the target text embedding feature into the large language model in the information recognition model to obtain the recognition result of the information to be recognized output by the large language model.

[0085] In this embodiment, the information recognition model is obtained by training based on the sample image, sample text, forged region coordinate labels corresponding to the sample image, and word unit true value labels corresponding to the sample text in the sample information.

[0086] In the information recognition method of the present embodiment, by extracting respective features from the image to be recognized and the text to be recognized in the information to be recognized, a multimodal feature expression of the image and the text is realized, and a forged element prediction module is used to obtain the forged region coordinates in the image to be recognized and the forged word element index in the text to be recognized based on the multimodal feature expression, and the forged region coordinates and the forged word element index are integrated into the prompt words of the large language model, thereby improving the quality of the prompt words of the large language model, thereby improving the recognition ability of the large language model for forged information, being able to accurately identify forged information, and improving the recognition accuracy of the corresponding forged information.

[0087] In some embodiments, Figure 2 As shown in FIG. 1 , the feature extraction module includes: an encoding module and a cross-modal interaction module (Cross Modal Interaction Module). Accordingly, the specific process of step S110 is as follows: Figure 3 As shown, the process includes the following steps S310 to S330.

[0088] Step S310: Input the image to be recognized and the text to be recognized into the encoding module to obtain the source image embedding features of the image to be recognized and the source text embedding features of the text to be recognized. The encoding module may be an integrated encoding model that can be used for both image embedding feature extraction and text embedding feature extraction, or a combined encoding model formed by an image encoder for image embedding feature extraction and a text encoder for text embedding feature extraction, which is not specifically limited in this embodiment.

[0089] Alternatively, if Figure 2As shown, the encoding module includes: an image encoder E I and text encoder E T , the two are combined to form a coding model. The image encoder in this embodiment E I The network structure can be the same as that of ViT-L / 16. At the beginning of training, the image encoder E I The parameters of can be initialized using the parameters of ViT-L / 16 pre-trained on the ImageNet-21k dataset. Text encoder E T The network structure can be compared with BERT LARGE The network structure is the same as that of E T The parameters can use BERT LARGE Initialize the parameters trained on the General Language Understanding Evaluation Benchmark. Text Encoder E T The tokenizer and the corresponding token embedding vector table used can be compared with the BERT LARGE The word segmenter and word embedding vector table are the same. Text encoder E T The word segmenter and word unit embedding vector table are fixed settings and are not updated during the training process. Various embeddings (embedding features) in this embodiment are all represented in the form of vectors.

[0090] Based on the above image encoder E I and text encoder E T , accordingly, step S310 specifically includes the following steps.

[0091] Step 1: Divide the image to be recognized into multiple image blocks. E I The input is 256 image patches (also called image slices). In order to generate the image encoder E I The input image blocks required, in this embodiment, can be divided into 256 image blocks of the same size and not overlapping along the width and height of the image I to be identified (assuming that the pixel size of the width and height are both L), and the width of each image block is L / 16 pixels, and the height is also L / 16 pixels. For the convenience of description, the total number of image blocks is defined as ,So .

[0092] Step 2: Input each of the image blocks into the image encoder to obtain the image classification embedding features and the embedding features of each image block output by the image encoder, and the embedding features of each image block and the image classification embedding features constitute the source image embedding features.

[0093] Specifically, after obtaining p image blocks, the p image blocks are input into the above-mentioned image encoder. E I Extract source image embedding features (patch embedding) , Includes p image patch embedding features and an image classification embedding feature for classification , therefore, the source image embedding feature Among them, the source image embedding feature The dimension of each embedding feature in is , .

[0094] In this embodiment, the image to be identified is subdivided into multiple image blocks and input into the image encoder one by one for deep coding feature extraction, so as to capture the image features comprehensively and meticulously, thereby generating source image embedding features including multiple image block embedding features with rich details and generalized image classification embedding features, thereby achieving comprehensive characterization and efficient analysis of the image content, so as to more accurately identify the authenticity of the information to be identified later.

[0095] Step three: input the text to be recognized into the text encoder, divide the text to be recognized into word units and extract features through the text encoder to obtain the source text embedding features, and the source text embedding features include: each word unit embedding feature and word unit classification embedding feature.

[0096] Specifically, first, the word segmenter is used to divide the text to be recognized T into multiple word units, where the total number of word units can be defined as ,Should It changes dynamically according to the different texts T to be recognized; secondly, the embedding corresponding to each word is retrieved from the word embedding vector table and passed through the text encoder E T The forward processing process, text encoder E T Extract the source text embedding features of the text to be recognized T , include word embedding features and a meta-classification embedding feature for classification Therefore, the source text embedding feature .in, The dimension of each word embedding feature in is also , .

[0097] In this embodiment, the text to be recognized is input into a text encoder, and undergoes fine word segmentation, word embedding vector retrieval and forward processing to extract source text embedding features including multiple word embedding features and word classification embedding features. This text encoding process not only retains the semantic details of the text, but also enhances the overall representation ability of the text, so that the authenticity of the information to be recognized can be more accurately identified later.

[0098] It should be noted that: in this embodiment, the above-mentioned step 2 of image encoding and step 3 of text encoding can be performed simultaneously, and the order is not limited.

[0099] Step S320: Input the source image embedding feature and the source text embedding feature into the cross-modal interaction module E CMIM , obtain the cross-modal image embedding features of the image to be recognized and the cross-modal text embedding features of the text to be recognized. Figure 2 and 4 As shown, the cross-modal interaction module E CMIM The cross-attention mechanism can be used to realize cross-modal information interaction, which is used to achieve information interaction between image embedding features of image modality and text embedding features of text modality, and enhance the expressive ability of image embedding features of image modality and text embedding features of text modality to encode semantic information.

[0100] Optionally, in this embodiment, the cross-modal interaction module E CMIM Structure Figure 4 As shown, it includes: a first cross-attention submodule m1 and a second cross-attention submodule m2. Based on the cross-modal interaction module E CMIM The structure of step S320 specifically includes the following steps.

[0101] Step 1: Input the source image embedding feature as a query parameter and the source text embedding feature as a key parameter and a value parameter into the first cross-attention submodule to obtain the cross-modal image embedding feature. Specifically, the query of the first cross-attention submodule m1 is the source image embedding feature The keys and values ​​of the first cross-attention submodule m1 are the source text embedding features. The embedding in (i.e., the embedding features of each word unit) is calculated through cross attention to obtain the output embedding of the first cross attention submodule m1 , that is, cross-modal image embedding features .

[0102] Step 2: Input the source text embedding feature as a query parameter and the source image embedding feature as a key parameter and a value parameter into the second cross-attention submodule to obtain the cross-modal text embedding feature. Specifically, the query of the second cross-attention submodule m2 is the source text embedding feature The keys and values ​​of the second cross-attention submodule m2 are the source image embedding features. The embedding in (i.e., the embedding features of each image block) is calculated through cross attention to obtain the output embedding of the second cross attention submodule m2 , that is, cross-modal text embedding features .

[0103] Among them, cross-modal image embedding features The number of embedded features in the source image is the same as the number of embedded features in the source image. The number of embedded features in is the same, that is, ,in is the image patch embedding feature corresponding to all image patches, is the image classification embedding feature for image classification. Similarly, the cross-modal text embedding feature The number of embedded features in the source text is the same as the number of embedded features in the source text. The number of embedded features is also the same, that is, ,in is the word embedding feature in the text to be recognized T, It is a word-classification embedding feature for text classification.

[0104] Step S330: Determine the cross-modal image embedding feature as the target image embedding feature, and determine the cross-modal text embedding feature as the target text embedding feature, that is, directly determine the cross-modal image embedding feature and the cross-modal text embedding feature as the target image embedding feature and the target text embedding feature, respectively.

[0105] In this embodiment, by taking the source image embedding features and the source text embedding features as query and key parameters respectively and inputting them into the first cross-attention sub-module and the second cross-attention sub-module in the corresponding cross-modal interaction module, a two-way interaction and fusion between the respective embedding features of the image and the text is achieved, thereby obtaining cross-modal image embedding features and cross-modal text embedding features that contain both the original information and are rich in cross-modal interaction information, significantly enhancing the depth and breadth of multimodal information processing, thereby utilizing the cross-modal image embedding features and the cross-modal text embedding features as the target image embedding features and the target text embedding features respectively, thereby improving the accuracy of authenticity recognition of the information to be identified.

[0106] In some embodiments, the feature extraction module includes: an encoding module, a cross-modal interaction module and a feature enhancement module. Accordingly, the specific process of step S110 is as follows: Figure 5 As shown, the process includes the following steps S510 to S540.

[0107] Step S510: input the image to be recognized and the text to be recognized into the encoding module to obtain the source image embedding features of the image to be recognized and the source text embedding features of the text to be recognized, wherein the specific network structure and encoding process of the encoding module are shown in the previous embodiment and will not be repeated here.

[0108] Step S520: input the source image embedding features and the source text embedding features into the cross-modal interaction module to obtain the cross-modal image embedding features of the image to be recognized and the cross-modal text embedding features of the text to be recognized, wherein the specific network structure and interaction process of the cross-modal interaction module are shown in the previous embodiment and will not be repeated here.

[0109] Step S530: inputting the learnable vector and the cross-modal image embedding feature into the feature enhancement module to obtain an enhanced image embedding feature.

[0110] Step S540: determining the enhanced image embedding feature as the target image embedding feature, and determining the cross-modal text embedding feature as the target text embedding feature.

[0111] Specifically, Figure 2 As shown, compared with the previous embodiment, in this embodiment, a feature enhancement module is added. The feature enhancement module fuses the learnable vector with the cross-modal image embedding feature through the external input learnable vector, further extracts the feature, and makes the final target image embedding feature more specific and expressive. In the information recognition model training stage, the learnable vector can be randomly initialized, and the learnable vector will be continuously updated during the model training process.

[0112] In some embodiments, the feature enhancement module includes: at least one of a cross attention module and a linear projection module. Based on this, step S530 specifically includes the following steps.

[0113] Step 1: Input the first learnable vector and the cross-modal image embedding feature into the cross-attention module to obtain a first image embedding feature.

[0114] Step 2: Input the second learnable vector and the cross-modal image embedding feature into the linear projection module to obtain a second image embedding feature.

[0115] Step three: determine the first image embedding feature, the second image embedding feature, or the sum of the first image embedding feature and the second image embedding feature as the enhanced image embedding feature.

[0116] Specifically, Figure 2 As shown, the feature enhancement module includes a cross-attention module C I and linear projection module Proj I .

[0117] The first learnable vector and cross-modal image embedding features Enter the criss-cross attention module C I , the first learnable vector The dimension of is the same as the dimension of the image block embedding feature, ensuring that dot product operations can be performed between them. For the convenience of description, define ,in .Cross-Attention Module C I Take the first learnable vector is the query, Embedding in As key and value, the first image embedding feature is obtained by attention calculation according to the following formula Q I , the first image embedding feature Q I Incorporated information.

[0118] .

[0119] in, Represents the first learnable vector The transposed vector of Represents the dot product operation between vectors. In the initial stage of model training, the first learnable vector is randomly initialized, and then during training is updated.

[0120] The second learnable vector α I and cross-modal image embedding features Input linear projection module Proj I , for the convenience of description, define ,in , perform linear projection operation according to the following formula to obtain the second image embedding feature V I , the second image embedding feature V I Also integrated information.

[0121] .

[0122] Among them, in the initial stage of model training, the second learnable vector α I is randomly initialized, and then during training α I is updated.

[0123] Finally, the first image is embedded into the feature Q I Embedding features with the second image V I The sum of the enhanced image embedding features is determined as In this embodiment, the cross attention module C I Output Q I and linear projection module Proj I Output V I Both incorporate cross-modal image embedding features Moreover, due to the different learnable vectors, the cross-modal image embedding features are analyzed from different angles (multi-angles). Fusion is performed to finally obtain the enhanced image embedding features of multi-angle information fusion , embed the enhanced image into features As the target image embedding feature, the expressive ability of the target image embedding feature is further improved.

[0124] In some embodiments, Figure 2 As shown, the forged element prediction module includes: a forged region position prediction module and a forged word unit prediction module. Based on this, step S120 specifically includes the following steps.

[0125] The target image embedding feature is input into the forged area position prediction module to obtain the forged area coordinates in the image to be identified.

[0126] The target text embedding feature is input into the forged word prediction module to obtain the forged word index in the text to be recognized.

[0127] In this embodiment, the forged area position prediction module and the forged word prediction module are both modules built based on neural networks, and their inputs are the target image embedding features and the target text embedding features respectively, and the target image embedding features and the target text embedding features each mean the feature information of the other party, so that the forged area position prediction module can accurately predict the forged area position in the image to be identified, and the forged word prediction module can accurately predict the forged word in the text to be identified and its forged word index.

[0128] In some embodiments, Figure 6 As shown, the forged area position prediction module includes: a first fully connected neural network and a mapping module. Accordingly, the target image embedding feature is input into the forged area position prediction module to obtain the forged area coordinates in the image to be identified, including the following steps.

[0129] The target image embedding features are input into the first fully connected neural network to obtain at least one set of latent vectors output by the first fully connected neural network, each set of latent vectors includes four latent variable values. The first fully connected neural network can be a single-layer or multi-layer fully connected neural network, and its output is related to the number of forged areas marked in the sample image in the sample information. For example, if only one forged area is marked in the sample image, then the first fully connected neural network only outputs one set of latent vectors, and the four latent variable values ​​of the latent vectors correspond to the coordinates of the two diagonal corners of the forged area (detection box). Figure 6 In the example, the target image embedding features are Input the first fully connected neural network, and the last layer of the first fully connected neural network outputs four hidden variable values .

[0130] The latent vector is input into the mapping module, and the four latent variable values ​​in the latent vector are mapped to four relative coordinate values ​​in the numerical range of [0, 1] by the mapping module, and the four relative coordinate values ​​of each group are determined as the coordinates of the forged area.

[0131] Among them, the mapping module can be implemented using the Sigmoid function. In the model application reasoning process, Sigmoid converts the four hidden variable values Mapped to four relative coordinate values ​​in the range [0,1] The relative position coordinates refer to the absolute position coordinates of the image pixels relative to the width of the image to be recognized. W and high H For example, assuming the absolute coordinates are , the relative coordinates are ,So , , , .

[0132] During the model training process, the four relative coordinate values ​​output by the forged area position prediction module are compared with the forged area coordinate labels pre-annotated in the sample image. For comparison, fake area coordinate labels is the relative coordinate of the pre-marked forged area position. In this embodiment, the GIoU loss function is used to construct the forged area coordinate regression loss function as follows.

[0133] (1).

[0134] in, FC A fully connected neural network representing the forged region location prediction module; It is the pre-marked forged area coordinate label (relative position coordinate).

[0135] In some embodiments, Figure 7 As shown, the forged word prediction module includes: a second fully connected neural network and a classification output module. Accordingly, the target text embedding feature is input into the forged word prediction module to obtain the forged word index in the text to be identified, including the following steps.

[0136] Input the word-meta embedding feature in the target text embedding feature into the second fully connected neural network, and obtain the two latent variable values ​​corresponding to the word-meta embedding feature output by the second fully connected neural network. The second fully connected neural network can be a single-layer or multi-layer fully connected neural network. Since the forged word-meta prediction module is used to determine whether the input word-meta embedding feature is forged, the last layer of the second fully connected neural network only needs to output two latent variable values. Figure 7 In the cross-modal interaction module E CMIM Output target text embedding features Each word embedding feature in It is input into the second fully connected neural network separately, and the second fully connected neural network outputs two hidden variable values ​​( and ).

[0137] The two latent variable values ​​are input into the classification output module to obtain the forged category probability value and the non-forged category probability value corresponding to the two latent variable values ​​output by the classification output module. The classification output module can be a Softmax layer, and the Softmax layer outputs the classification probability value, that is, the non-forged category probability value and the forged class probability values .

[0138] When the probability value of the forged category is greater than the probability value of the non-forged category, the position index of the word element in the text corresponding to the latent variable value of the forged category probability is obtained as the forged word element index, that is, the category corresponding to the latent variable value with a high classification probability value is the word element embedding feature The classification result is Greater than hour, The hidden variable value of The corresponding word is a fake word, and the corresponding word embedding feature , where i is the pseudo-word index of the word. It should be noted that: and They are all floating point numbers, and the probability of them being equal is very small and can be ignored.

[0139] Specifically, in the model application reasoning process, first, the word embedding feature After the second fully connected neural network, two hidden variable values ​​are obtained ( and ); Then, these two latent variable values ​​pass through a Softmax layer and output the classification probability value ( and ); Finally, the category corresponding to the latent variable value with high classification probability value is the word embedding The classification results.

[0140] The model training process is similar to the model application reasoning process. First, the word embedding feature After the second fully connected neural network, two hidden variable values ​​are obtained ( and ); Then, these two latent variable values ​​pass through a Softmax layer to output the classification probability value and .

[0141] .

[0142] .

[0143] Finally, the word classification loss function is constructed based on the classification probability value .

[0144] (2).

[0145] in, represents the true value label of the i-th word in the sample text, represents a non-forged word, Indicates a fake word; , Represents the second fully connected neural network The transposed vector of the weight parameters of the columns, e represents a natural constant, Represents the dot product operation between vectors.

[0146] In some embodiments, step S130 specifically includes: obtaining a prompt word template, identifying keywords representing forged area coordinates, keywords representing forged word element indexes, and keywords representing the title of information to be identified in the prompt word template; replacing the adapters in the corresponding keywords with the forged area coordinates, forged word element indexes, and the title of information to be identified to generate the prompt word. Different prompt word templates are provided according to different large language models. According to the requirements of different large language models for coordinates in the image, the forged area coordinates of relative coordinate values ​​need to be converted into absolute coordinate values.

[0147] Specifically, the prompt word template may be the following template.

[0148]

[0149] The keyword representing the coordinates of the forged area may be: image area (?), the keyword representing the forged word element index may be: note the text of?, and the keyword representing the title of the information to be identified may be: the content of the input text is "?", where? represents an adapter.

[0150] For example, in the above prompt word template, the title of the information to be identified is "Zhang San and Li Si visit the park together", (0, 0, 100, 100) is the absolute coordinate value after the relative coordinate value output by the forged area position prediction module is converted, and "index 4 and index 5" are the positions of the forged word units output by the forged word unit prediction module. In each prediction process, the prompt word needs to be updated according to the input image and text. The part is the content that needs to be updated, and the other parts remain unchanged. The final generated prompt word is as follows.

[0151]

[0152] The above prompt word is an example, and other prompt methods are also feasible, as long as the input text, the forged area coordinates output by the forged area position prediction module and the position value of the forged word unit output by the forged word unit prediction module are added to the prompt word.

[0153] In this embodiment, by replacing the adapters of the corresponding keywords in the prompt template, the prompt can be automatically, efficiently, and accurately generated.

[0154] In some embodiments, in step S140, the large language model can be, but is not limited to, Vicuna 7B. Since the large language model itself is a trained and mature model, in this embodiment, the parameters of the large language model Vicuna 7B are frozen and not updated during the training process.

[0155] The inputs of the large language model Vicuna 7B include: 1) the target image embedding features and the target text embedding features , where , ; 2) the large model learning vector group as the soft prompt; 3) the prompt. Among them, the learnable vector group includes 6 to 10 learnable query vectors.

[0156] The output of the large language model Vicuna 7B is the embedding vector group . The number of elements included in the embedding vector group dynamically changes according to the length of the input of the large language model Vicuna 7B. In the above embedding vector group , the last four embeddings correspond to the expected output answers "Yes!" or "No!" in this embodiment, and there is also a sentence end symbol. For example: in the embedding vector group output by the large language model Vicuna 7B, the last four embeddings , , , and correspond to "Yes", "of", "!", and "sentence end symbol" respectively, indicating that the information to be recognized is forged information, otherwise, it is not forged information.

[0157] It can be understood that: there is a classification module in the large language model Vicuna 7B, and this classification module includes a linear projection layer and a Softmax layer. The linear projection layer is a shared fully connected layer used to process the above four embeddings. Through the linear projection layer, each embedding is transformed into a vector with a dimension of , where represents the size of the vocabulary of the large language model Vicuna 7B (in this embodiment, ). In this way, four vectors with a dimension of can be obtained. The above four vectors with a dimension of are respectively transformed into probability value vectors through the Softmax layer. , , Then, the probability values ​​corresponding to the above four embeddings are , , , ,in, , , and They are the ordinal numbers of the four tokens corresponding to the above four embeddings in the vocabulary of the large language model Vicuna7B.

[0158] During the training process, the word unit regression loss function of the large language model Vicuna7B is .

[0159] (3).

[0160] in, It is The number of the word corresponding to the embedding in the vocabulary of the large language model Vicuna7B, It is The predicted probability value vector corresponding to the word element corresponding to the embedding, yes The element values, n represents the number of embeddings or tokens, here .

[0161] During the model inference process, according to the probability value vector output by the model , , , , find the word element corresponding to the maximum probability value in each probability value vector. These word elements are the final output results of the model reasoning, that is, "Yes!" or "No!".

[0162] In this embodiment, the forged area coordinates and the forged word element index are integrated into the prompt words of the large language model, which improves the quality of the prompt words of the large language model, thereby improving the recognition ability of the large language model for forged information and being able to accurately recognize forged information.

[0163] In some embodiments, the information recognition model is trained based on the following steps.

[0164] Step 1: Obtain a training data set, wherein the training data set includes a plurality of sample information, each of which includes: a sample image, a sample text, a forged region coordinate label corresponding to the sample image, and a word unit true value label corresponding to the sample text.

[0165] Since the information recognition model is used to identify forged information in sample information, the sample information is a picture-text pair (i.e., an image and a corresponding text description constitute a picture-text pair), and the forged information in the sample information includes: at least one of forged information that the target object in the sample image is replaced and forged information that the emotional state of the target object in the sample image is replaced, and the forged information of the sample text includes: forged information that the paragraphs, words or phrases used to represent the emotional state in the sample text are replaced. For example: the forged information in the image includes that the face image of a specific person in the image is replaced with the face image of another person; the emotional state of the face image in the image is changed (for example: happiness is changed to anger); the forged information in the text includes that the text corresponding to the image is replaced with another paragraph of text; some paragraphs, words or phrases in the text corresponding to the image are replaced with other paragraphs, words or phrases, especially such modification can cause the emotional state expressed by the text to change, thereby changing the tendency of the picture-text information content. Based on the above-mentioned two kinds of forged information, image and text, a variety of forged combinations can be constructed.

[0166] Therefore, in order to identify the above-mentioned forged information, this embodiment needs to collect a large amount of original information that has not been forged, modify the original information using existing image and text forgery technology, obtain a large amount of sample information, and mark the sample information as forged to obtain corresponding forged classification labels, thereby constructing a training data set.

[0167] Specifically, the original training data set is formed through collection and annotation. The original training data set includes about 10,000 sample image-text pairs, wherein each sample image contains at least 1 face image and at most 10 face images; if it is Chinese text, the corresponding text length is at least 5 characters and at most 50 characters. If it is English text, the corresponding text length is at least 5 words and at most 50 words. Each sample text in the above original training data set contains at least one predicate verb and one subject or object. In the above original training data set, about 20% of the sample images have no modifications; the largest face in about 40% of the sample images is replaced with another person's face image; the expression of the largest face in the remaining sample images is tampered with (for example: smile is changed to cry, etc.). Correspondingly, in the above original training data set, for the sample text corresponding to the sample image, about 20% of the sample text has no modification; about 40% of the sample text has been replaced by another text; the words / phrases related to the emotional state in the remaining sample text have been tampered with (for example, words / phrases of positive emotions have been changed to words / phrases of negative emotions, etc.). Professionals manually mark the position coordinates (relative position coordinates) of the modified (face) area in the sample image, as well as whether the modified words or phrases in the sample text have been modified as word units (0 means unmodified, 1 means modified) and the sequence number of the word unit in the sample text, so as to obtain the forged area coordinate label corresponding to the sample text and the word unit true value label corresponding to the sample text, thereby forming the training data set required for this embodiment. For the above 20% sample images, the forged area coordinate label can be set to (0, 0, 0, 0), indicating that it has not been modified.

[0168] Step 2: Based on different sample information, the initial model corresponding to the information recognition model is iterated for multiple cycles. When the preset training cycle threshold is reached or the total loss function converges, the training is completed to obtain the information recognition model. Each sample information is an image-text pair, which can be specifically defined as pair (I, T). The sample image I is a three-channel color image, and the sample text T can be a description of the content in the image I.

[0169] The total loss function is obtained by weighted summing the forged region coordinate regression loss function of the forged element prediction module, the word unit classification loss function, and the word unit regression loss function of the large language model. Specifically, the total loss function is L total .

[0170] .

[0171] Specifically, the total loss function L totalBy adding the above formulas (1), (2) and (3), the weights of the loss functions corresponding to the above formulas (1), (2) and (3) are all 1. i Indicates i Sample information, B Indicates the number of sample information in a training batch.

[0172] In each iterative training process, the sample information is input into the initial model to obtain the forged area prediction coordinates, word unit classification prediction results and prediction results of the large language model predicted by the forged element prediction module, and the forged area prediction coordinates, word unit classification prediction results, prediction results of the large language model, forged area coordinate labels and word unit true value labels are substituted into the total loss function, and back propagation is performed based on the calculation result of the total loss function to update the model parameters of the feature extraction module and the forged element prediction module in the initial model, and to update the large model learning vector group.

[0173] In some embodiments, based on different sample information, the initial model corresponding to the information recognition model is iteratively trained for multiple cycles, including the following steps.

[0174] In each training cycle, the sample information in the training data set is randomly sorted and adjusted, and the randomly sorted and adjusted training data set is divided into multiple sub-training sets. Each sub-training set is a training batch, and iterative training is performed on the basis of the initial model obtained by the previous batch training to obtain the initial model after the current batch training.

[0175] Specifically, the total training cycle (epoch) is set to 20 cycles, and one training cycle means that all training samples are used once without repetition. In order to prevent the order of samples in the data set from affecting the performance of the model and to improve the generalization performance of the model, all samples in the training data set are randomly rearranged at the beginning of each training cycle. In each training cycle, the shuffled training data set is divided into smaller sub-training sets, which are called a small batch of samples. Depending on the configuration of the training server, the number of samples in each small batch can be set to 128, 256 or 512, etc. The model training algorithm reads a small batch of training samples each time and uses these samples to calculate the loss value, update the model parameters, etc.

[0176] If all sub-training sets have completed training within a cycle, the training of this cycle ends and the next cycle of training begins. When the total number of training cycles reaches the preset training cycle threshold (such as 20) or the total loss function converges, the model training ends. It is worth noting that in special cases, within a training cycle, the total number of samples in the data set cannot be divided into integer batches, and individual samples are not used (discarded directly) or are used multiple times (in order to complete the data, for example, adding several samples that have been used to the end of the data set), the impact of these individual cases on the training results can be ignored.

[0177] Training Figure 2 The information recognition model in is taken as an example as follows.

[0178] Training input (initial parameter setting): from the training data set Read data in; initial training cycle , total training cycle ; Stochastic gradient descent algorithm related parameters: The initial learning rate is , momentum coefficient , the weight decay coefficient is , in the 10th training cycle, the learning rate drops to , in the 15th training cycle, the learning rate drops to , in the 18th training cycle, the learning rate dropped to The number of samples in a mini-batch is .

[0179] Training output: Information recognition model based on large language model (The parameter of the model is φ)

[0180] Loss function: Set the loss function defined in this invention .

[0181] The training process includes three steps: Step 1~ Step 3.

[0182] Step 1: Initialization, image encoder E I The parameters in are initialized with the parameters of ViT-L / 16 pre-trained on the ImageNet-21k dataset; text encoder E T The parameters of BERT LARGE The parameters trained on the "General Language Understanding Evaluation Benchmark Collection" are initialized; the learnable parameters of the recognition model are initialized using a random initialization method; the parameters of the large language model Vicuna7B are frozen and are not updated during training.

[0183] Step 2: When In this case, perform the following operations, where: E is the preset training cycle threshold.

[0184] a) .

[0185] b) Collect data The order of samples in is randomly adjusted.

[0186] c) From the data set Pick a batch B ( B ≥1) training samples.

[0187] d) Calculate the loss function value based on the above training samples and parameter settings , and then the stochastic gradient descent algorithm is used to update the initial model parameters.

[0188] .

[0189] in, Represents the word regression loss function of the i-th input sample information, represents the forged region coordinate regression loss function, represents the word classification loss function, Representative Sample images of sample information, Representative Sample text of sample information, Represents the total number of training samples in one batch training iteration.

[0190] e) Repeat steps c) and d) until the data set All samples were used.

[0191] f) End the current iterative training cycle.

[0192] Step 3: Jump to Step 2 until , end the training and output the information recognition model based on the large language model Of course, during the training process, if the total loss function converges, the training can be terminated to obtain an information recognition model based on a large language model. .

[0193] The information identification device provided by the present invention is described below. The information identification device described below and the information identification method described above can be referenced to each other.

[0194] The information identification device of the embodiment of the present invention is as follows: Figure 8 As shown, it includes the following functional units.

[0195] The target embedding feature extraction unit 810 is used to input the image to be recognized and the text to be recognized in the information to be recognized into the feature extraction module of the information recognition model to obtain the target image embedding feature and the target text embedding feature.

[0196] The forged element prediction unit 820 is used to input the target image embedding features and the target text embedding features into the forged element prediction module of the information recognition model to obtain the forged area coordinates in the image to be recognized and the forged word element index in the text to be recognized.

[0197] The prompt word generating unit 830 is configured to generate a prompt word based on the forged area coordinates and the forged word element index.

[0198] The large model execution unit 840 is used to input the large model learning vector group, the prompt word, the target image embedding feature and the target text embedding feature into the large language model in the information recognition model to obtain the recognition result of the information to be recognized output by the large language model.

[0199] The information recognition model is obtained by training based on the sample image, sample text, forged region coordinate labels corresponding to the sample image, and word unit true value labels corresponding to the sample text in the sample information.

[0200] In the information recognition device of the embodiment of the present invention, by extracting respective features of the image to be recognized and the text to be recognized in the information to be recognized, a multimodal feature expression of the image and the text is realized, and a forged element prediction module is used to obtain the forged area coordinates in the image to be recognized and the forged word element index in the text to be recognized based on the multimodal feature expression, and the forged area coordinates and the forged word element index are integrated into the prompt words of the large language model, thereby improving the quality of the prompt words of the large language model, thereby improving the recognition ability of the large language model for forged information, being able to accurately identify forged information, and improving the recognition accuracy of the corresponding forged information.

[0201] In some embodiments, the feature extraction module includes: an encoding module and a cross-modal interaction module.

[0202] The target embedding feature extraction unit 810 is specifically used to input the image to be recognized and the text to be recognized into the encoding module to obtain the source image embedding feature of the image to be recognized and the source text embedding feature of the text to be recognized; input the source image embedding feature and the source text embedding feature into the cross-modal interaction module to obtain the cross-modal image embedding feature of the image to be recognized and the cross-modal text embedding feature of the text to be recognized; determine the cross-modal image embedding feature as the target image embedding feature, and determine the cross-modal text embedding feature as the target text embedding feature.

[0203] In some embodiments, the feature extraction module includes: an encoding module, a cross-modal interaction module and a feature enhancement module.

[0204] The target embedding feature extraction unit 810 is specifically used to input the image to be recognized and the text to be recognized into the encoding module to obtain the source image embedding feature of the image to be recognized and the source text embedding feature of the text to be recognized; input the source image embedding feature and the source text embedding feature into the cross-modal interaction module to obtain the cross-modal image embedding feature of the image to be recognized and the cross-modal text embedding feature of the text to be recognized; input the learnable vector and the cross-modal image embedding feature into the feature enhancement module to obtain the enhanced image embedding feature; determine the enhanced image embedding feature as the target image embedding feature, and determine the cross-modal text embedding feature as the target text embedding feature.

[0205] In some embodiments, the feature enhancement module includes: at least one of a cross-attention module and a linear projection module.

[0206] The target embedding feature extraction unit 810 is specifically used to input the first learnable vector and the cross-modal image embedding feature into the cross-attention module to obtain a first image embedding feature; input the second learnable vector and the cross-modal image embedding feature into the linear projection module to obtain a second image embedding feature; determine the first image embedding feature, the second image embedding feature, or the sum of the first image embedding feature and the second image embedding feature as the enhanced image embedding feature.

[0207] In some embodiments, the cross-modal interaction module includes: a first cross-attention sub-module and a second cross-attention sub-module.

[0208] The target embedding feature extraction unit 810 is specifically used to include: inputting the source image embedding feature as a query parameter and the source text embedding feature as a key parameter and a value parameter into the first cross-attention sub-module to obtain the cross-modal image embedding feature; inputting the source text embedding feature as a query parameter and the source image embedding feature as a key parameter and a value parameter into the second cross-attention sub-module to obtain the cross-modal text embedding feature.

[0209] In some embodiments, the encoding module includes: an image encoder and a text encoder.

[0210] The target embedding feature extraction unit 810 is specifically used to divide the image to be recognized into multiple image blocks; input each of the image blocks into the image encoder to obtain the image classification embedding features and the embedding features of each image block output by the image encoder, and the embedding features of each image block and the image classification embedding features constitute the source image embedding features; input the text to be recognized into the text encoder, divide the text to be recognized into words and extract features through the text encoder to obtain the source text embedding features, and the source text embedding features include: each word-unit embedding feature and word-unit classification embedding feature.

[0211] In some embodiments, the forged element prediction module includes: a forged region position prediction module and a forged word unit prediction module.

[0212] The forged element prediction unit 820 is specifically used to input the target image embedding feature into the forged area position prediction module to obtain the forged area coordinates in the image to be identified; and input the target text embedding feature into the forged word prediction module to obtain the forged word index in the text to be identified.

[0213] In some embodiments, the forged area location prediction module includes: a first fully connected neural network and a mapping module.

[0214] The forged element prediction unit 820 is specifically used to input the target image embedding feature into the first fully connected neural network to obtain at least one group of latent vectors output by the first fully connected neural network, each group of the latent vectors includes four latent variable values; input the latent vectors into the mapping module, and map the four latent variable values ​​in the latent vectors to four relative coordinate values ​​in the numerical range of [0,1] through the mapping module, and determine the four relative coordinate values ​​of each group as the coordinates of the forged area.

[0215] In some embodiments, the pseudo-word prediction module includes: a second fully connected neural network and a classification output module.

[0216] The forged element prediction unit 820 is specifically used to input the word-meta embedding feature in the target text embedding feature into the second fully connected neural network, and obtain two latent variable values ​​corresponding to the word-meta embedding feature output by the second fully connected neural network; input the two latent variable values ​​into the classification output module, and obtain the forged category probability value and the non-forged category probability value respectively corresponding to the two latent variable values ​​output by the classification output module; when the forged category probability value is greater than the non-forged category probability value, the position index of the word-meta corresponding to the latent variable value of the forged category probability value in the text is obtained as the forged word-meta index.

[0217] In some embodiments, the prompt word generation unit 830 is specifically used to obtain a prompt word template, identify keywords representing forged area coordinates, keywords representing forged word element indexes, and keywords representing titles of information to be identified in the prompt word template; and replace the adapters in the corresponding keywords with the forged area coordinates, forged word element indexes, and titles of information to be identified to generate the prompt word.

[0218] In some embodiments, the information identification device further includes: an information identification model training unit, configured to train the information identification model according to the following steps.

[0219] A training data set is obtained, wherein the training data set includes a plurality of sample information, each of which includes: a sample image, a sample text, a forged region coordinate label corresponding to the sample image, and a word unit true value label corresponding to the sample text.

[0220] Based on different sample information, the initial model corresponding to the information recognition model is iteratively trained for multiple cycles. When a preset training cycle threshold is reached or the total loss function converges, the training is completed to obtain the information recognition model.

[0221] Among them, the total loss function is obtained by weighted summing up the forged region coordinate regression loss function of the forged element prediction module, the word unit classification loss function and the word unit regression loss function of the large language model. In each iterative training process, the sample information is input into the initial model to obtain the forged region prediction coordinates predicted by the forged element prediction module, the word unit classification prediction result and the prediction result of the large language model. The forged region prediction coordinates, the word unit classification prediction result, the prediction result of the large language model, the forged region coordinate label and the word unit true value label are substituted into the total loss function, and the calculation result of the total loss function is back-propagated to update the model parameters of the feature extraction module and the forged element prediction module in the initial model, and to update the large model learning vector group.

[0222] In some embodiments, the information recognition model training unit is specifically used to randomly sort and adjust the sample information in the training data set in each training cycle, and divide the randomly sorted and adjusted training data set into multiple sub-training sets; take each sub-training set as a training batch, and perform iterative training based on the initial model obtained from the previous batch training to obtain the initial model after the current batch training.

[0223] In some embodiments, in the sample information, the forged information of the sample image includes: at least one of: forged information in which a target object in the sample image is replaced and forged information in which an emotional state of the target object in the sample image is replaced, and the forged information of the sample text includes: forged information in which a paragraph, word or phrase used to represent the emotional state in the sample text is replaced.

[0224] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the information identification method, which includes the following steps.

[0225] The image to be recognized and the text to be recognized in the information to be recognized are input into the feature extraction module of the information recognition model to obtain the target image embedding feature and the target text embedding feature.

[0226] The target image embedding features and the target text embedding features are input into a forged element prediction module of an information recognition model to obtain forged region coordinates in the image to be recognized and forged word element indexes in the text to be recognized.

[0227] A prompt word is generated based on the forged area coordinates and the forged word element index.

[0228] The large model learning vector group, the prompt word, the target image embedding feature and the target text embedding feature are input into the large language model in the information recognition model to obtain the recognition result of the information to be recognized output by the large language model.

[0229] The information recognition model is obtained by training based on the sample image, sample text, forged region coordinate labels corresponding to the sample image, and word unit true value labels corresponding to the sample text in the sample information.

[0230] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0231] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the information identification method provided by the above methods, which includes the following steps.

[0232] The image to be recognized and the text to be recognized in the information to be recognized are input into the feature extraction module of the information recognition model to obtain the target image embedding feature and the target text embedding feature.

[0233] The target image embedding features and the target text embedding features are input into a forged element prediction module of an information recognition model to obtain forged region coordinates in the image to be recognized and forged word element indexes in the text to be recognized.

[0234] A prompt word is generated based on the forged area coordinates and the forged word element index.

[0235] The large model learning vector group, the prompt word, the target image embedding feature and the target text embedding feature are input into the large language model in the information recognition model to obtain the recognition result of the information to be recognized output by the large language model.

[0236] The information recognition model is obtained by training based on the sample image, sample text, forged region coordinate labels corresponding to the sample image, and word unit true value labels corresponding to the sample text in the sample information.

[0237] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the information identification method provided by the above methods is implemented, and the method includes the following steps.

[0238] The image to be recognized and the text to be recognized in the information to be recognized are input into the feature extraction module of the information recognition model to obtain the target image embedding feature and the target text embedding feature.

[0239] The target image embedding features and the target text embedding features are input into a forged element prediction module of an information recognition model to obtain forged region coordinates in the image to be recognized and forged word element indexes in the text to be recognized.

[0240] A prompt word is generated based on the forged area coordinates and the forged word element index.

[0241] The large model learning vector group, the prompt word, the target image embedding feature and the target text embedding feature are input into the large language model in the information recognition model to obtain the recognition result of the information to be recognized output by the large language model.

[0242] The information recognition model is obtained by training based on the sample image, sample text, forged region coordinate labels corresponding to the sample image, and word unit true value labels corresponding to the sample text in the sample information.

[0243] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0244] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0245] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An information identification method, characterized in that: include: Inputting the image to be recognized and the text to be recognized in the information to be recognized into the feature extraction module of the information recognition model to obtain the target image embedding feature and the target text embedding feature; Inputting the target image embedding features and the target text embedding features into a forged element prediction module of an information recognition model to obtain forged region coordinates in the image to be recognized and forged word element indexes in the text to be recognized; Generate a prompt word based on the forged area coordinates and the forged word element index; Inputting the large model learning vector group, the prompt word, the target image embedding feature and the target text embedding feature into the large language model in the information recognition model, and obtaining the recognition result of the information to be recognized output by the large language model; The information recognition model is obtained by training based on the sample image, sample text, forged region coordinate labels corresponding to the sample image, and word element true value labels corresponding to the sample text in the sample information; The feature extraction module includes: an encoding module and a cross-modal interaction module; The step of inputting the image to be identified and the text to be identified in the information to be identified into the feature extraction module of the information recognition model to obtain the target image embedding feature and the target text embedding feature includes: Inputting the image to be recognized and the text to be recognized into the encoding module to obtain the source image embedding feature of the image to be recognized and the source text embedding feature of the text to be recognized; Inputting the source image embedding feature and the source text embedding feature into the cross-modal interaction module to obtain the cross-modal image embedding feature of the image to be recognized and the cross-modal text embedding feature of the text to be recognized; The cross-modal image embedding feature is determined as the target image embedding feature, and the cross-modal text embedding feature is determined as the target text embedding feature.

2. The information identification method according to claim 1, characterized in that: The feature extraction module also includes: a feature enhancement module; The step of inputting the image to be identified and the text to be identified in the information to be identified into the feature extraction module of the information recognition model to obtain the target image embedding feature and the target text embedding feature includes: Inputting the image to be recognized and the text to be recognized into the encoding module to obtain the source image embedding feature of the image to be recognized and the source text embedding feature of the text to be recognized; Inputting the source image embedding feature and the source text embedding feature into the cross-modal interaction module to obtain the cross-modal image embedding feature of the image to be recognized and the cross-modal text embedding feature of the text to be recognized; Inputting the learnable vector and the cross-modal image embedding feature into the feature enhancement module to obtain an enhanced image embedding feature; The enhanced image embedding feature is determined as the target image embedding feature, and the cross-modal text embedding feature is determined as the target text embedding feature.

3. The information identification method according to claim 2, characterized in that: The feature enhancement module includes: at least one of a cross attention module and a linear projection module; The step of inputting the learnable vector and the cross-modal image embedding feature into the feature enhancement module to obtain the enhanced image embedding feature comprises: Inputting the first learnable vector and the cross-modal image embedding feature into the cross-attention module to obtain a first image embedding feature; Inputting the second learnable vector and the cross-modal image embedding feature into the linear projection module to obtain a second image embedding feature; The first image embedding feature, the second image embedding feature, or the sum of the first image embedding feature and the second image embedding feature is determined as the enhanced image embedding feature.

4. The information identification method according to claim 1, characterized in that: The cross-modal interaction module includes: a first cross-attention submodule and a second cross-attention submodule; Inputting the source image embedding feature and the source text embedding feature into the cross-modal interaction module to obtain the cross-modal image embedding feature of the image to be recognized and the cross-modal text embedding feature of the text to be recognized, including: Inputting the source image embedding feature as a query parameter and the source text embedding feature as a key parameter and a value parameter into the first cross-attention submodule to obtain the cross-modal image embedding feature; The source text embedding feature is used as a query parameter and the source image embedding feature is used as a key parameter and a value parameter to input into the second cross-attention submodule to obtain the cross-modal text embedding feature.

5. The information identification method according to claim 1, characterized in that: The encoding module includes: an image encoder and a text encoder; Inputting the image to be recognized and the text to be recognized into the encoding module to obtain the source image embedding feature of the image to be recognized and the source text embedding feature of the text to be recognized, including: Dividing the image to be recognized into a plurality of image blocks; Input each of the image blocks into the image encoder to obtain the image classification embedding features and each image block embedding features output by the image encoder, wherein the image block embedding features and the image classification embedding features constitute the source image embedding features; The text to be recognized is input into the text encoder, and the text encoder divides the text to be recognized into word units and extracts features to obtain the source text embedding features, which include: each word unit embedding feature and word unit classification embedding feature.

6. The information identification method according to claim 1, characterized in that: The forged element prediction module includes: a forged region position prediction module and a forged word unit prediction module; Inputting the target image embedding features and the target text embedding features into the forged element prediction module of the information recognition model to obtain the forged region coordinates in the image to be recognized and the forged word element index in the text to be recognized, including: Inputting the target image embedding features into the forged area position prediction module to obtain the forged area coordinates in the image to be identified; The target text embedding feature is input into the forged word prediction module to obtain the forged word index in the text to be recognized.

7. The information identification method according to claim 6, characterized in that: The forged area location prediction module includes: a first fully connected neural network and a mapping module; Inputting the target image embedding feature into the forged area position prediction module to obtain the forged area coordinates in the image to be identified includes: Inputting the target image embedding feature into the first fully connected neural network to obtain at least one group of latent vectors output by the first fully connected neural network, each group of latent vectors including four latent variable values; The latent vector is input into the mapping module, and the four latent variable values ​​in the latent vector are mapped to four relative coordinate values ​​in the numerical range of [0, 1] by the mapping module, and the four relative coordinate values ​​of each group are determined as the coordinates of the forged area.

8. The information identification method according to claim 6, characterized in that: The forged word prediction module includes: a second fully connected neural network and a classification output module; Inputting the target text embedding feature into the forged word prediction module to obtain the forged word index in the text to be recognized includes: Inputting a word unit embedding feature in the target text embedding feature into the second fully connected neural network, and obtaining two latent variable values ​​corresponding to the word unit embedding feature output by the second fully connected neural network; Inputting the two latent variable values ​​into the classification output module, obtaining the forged category probability values ​​and the non-forged category probability values ​​respectively corresponding to the two latent variable values ​​output by the classification output module; When the forged category probability value is greater than the non-forged category probability value, the position index of the word element corresponding to the latent variable value of the forged category probability value in the text is obtained as the forged word element index.

9. The information identification method according to claim 1, characterized in that: Generating a prompt word based on the forged area coordinates and the forged word element index includes: Obtain a prompt word template, and identify keywords representing the coordinates of the forged area, keywords representing the forged word element index, and keywords representing the title of the information to be identified in the prompt word template; The forged area coordinates, the forged word element index and the title of the information to be identified are used to replace the adapters in the corresponding keywords to generate the prompt word.

10. The information identification method according to any one of claims 1 to 9, characterized in that: The information recognition model is trained based on the following steps: Acquire a training data set, wherein the training data set includes a plurality of sample information, each of which includes: a sample image, a sample text, a forged region coordinate label corresponding to the sample image, and a word unit true value label corresponding to the sample text; Based on different sample information, the initial model corresponding to the information recognition model is iterated for multiple cycles of training, and when a preset training cycle threshold is reached or the total loss function converges, the training is completed to obtain the information recognition model; Among them, the total loss function is obtained by weighted summing up the forged region coordinate regression loss function of the forged element prediction module, the word unit classification loss function and the word unit regression loss function of the large language model. In each iterative training process, the sample information is input into the initial model to obtain the forged region prediction coordinates predicted by the forged element prediction module, the word unit classification prediction result and the prediction result of the large language model. The forged region prediction coordinates, the word unit classification prediction result, the prediction result of the large language model, the forged region coordinate label and the word unit true value label are substituted into the total loss function, and the calculation result of the total loss function is back-propagated to update the model parameters of the feature extraction module and the forged element prediction module in the initial model, and to update the large model learning vector group.

11. The information identification method according to claim 10, characterized in that: Based on different sample information, the initial model corresponding to the information recognition model is iteratively trained for multiple cycles, including: In each training cycle, randomly sorting and adjusting each sample information in the training data set, and dividing the randomly sorted and adjusted training data set into a plurality of sub-training sets; Each sub-training set is regarded as a training batch, and iterative training is performed on the basis of the initial model obtained by the previous batch training to obtain the initial model after the current batch training.

12. The information identification method according to claim 10, characterized in that: In the sample information, the forged information of the sample image includes: at least one of: forged information in which the target object in the sample image is replaced and forged information in which the emotional state of the target object in the sample image is replaced, and the forged information of the sample text includes: forged information in which a paragraph, word or phrase used to represent the emotional state in the sample text is replaced.

13. An information recognition device, characterized in that: include: A target embedding feature extraction unit is used to input the image to be identified and the text to be identified in the information to be identified into the feature extraction module of the information recognition model to obtain the target image embedding feature and the target text embedding feature; a forged element prediction unit, configured to input the target image embedding features and the target text embedding features into a forged element prediction module of an information recognition model, and obtain forged region coordinates in the image to be recognized and forged word element indexes in the text to be recognized; A prompt word generating unit, used for generating a prompt word based on the forged area coordinates and the forged word element index; A large model execution unit, used for inputting the large model learning vector group, the prompt word, the target image embedding feature and the target text embedding feature into the large language model in the information recognition model, and obtaining the recognition result of the information to be recognized output by the large language model; The information recognition model is obtained by training based on the sample image, sample text, forged region coordinate labels corresponding to the sample image, and word element true value labels corresponding to the sample text in the sample information; The feature extraction module includes: an encoding module and a cross-modal interaction module; The step of inputting the image to be identified and the text to be identified in the information to be identified into the feature extraction module of the information recognition model to obtain the target image embedding feature and the target text embedding feature includes: Inputting the image to be recognized and the text to be recognized into the encoding module to obtain the source image embedding feature of the image to be recognized and the source text embedding feature of the text to be recognized; Inputting the source image embedding feature and the source text embedding feature into the cross-modal interaction module to obtain the cross-modal image embedding feature of the image to be recognized and the cross-modal text embedding feature of the text to be recognized; The cross-modal image embedding feature is determined as the target image embedding feature, and the cross-modal text embedding feature is determined as the target text embedding feature.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the information identification method according to any one of claims 1 to 12 is implemented.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the information identification method according to any one of claims 1 to 12 is implemented.

Citation Information

Patent Citations

  • Document image detection method, system and equipment

    CN116363674A

  • Training method of image sample generation model and image sample generation method and device

    CN117422783A

  • Social media false news detection method based on multi-modal fusion

    CN118211141A

  • Bimodal-fused forged information detection method based on prompt learning

    CN118364421A

  • Artificial intelligence counterfeit detection

    US20190236614A1