Information identification method, device, electronic device and storage medium
Through multimodal and multipath feature extraction and aggregation methods, the problem of difficulty in comprehensively capturing forgery traces in the prior art is solved, the accuracy and comprehensiveness of information forgery identification are improved, and the negative impact of forgery information on users is reduced.
Patent Information
- Application Number
- CN202411206316.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-30
AI Technical Summary
The prior art is difficult to fully capture the forgery traces of information to be identified, resulting in insufficient accuracy of the information forgery identification results.
Using multimodal and multipath feature extraction and feature aggregation methods, the features of the images and text to be identified are extracted and fused at multiple levels and multi-angle features through encoding modules, cross-modal interaction modules and multipath projection modules to form Image aggregation features and text aggregation features are finally forged and classified through the classification module.
It improves the accuracy and comprehensiveness of information forgery identification, and reduces the negative impact of forgery information on users' normal work and life.
Smart Images

Figure CN118711196B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to an information identification method, device, electronic equipment and storage medium. Background Art
[0002] With the development of deep learning technology, especially the rise of Generative Adversarial Networks (GANs) and Large Language Models (LLM), image and text forgery has emerged. However, forged image and text information will have a certain negative impact on the normal work and life of users. Therefore, how to efficiently and conveniently identify the authenticity of information is one of the technical problems that need to be solved urgently.
[0003] Usually, in practical applications, the authenticity of information is identified by performing independent analysis of the image pixel level on the information to be identified using a deep learning network model, or by performing independent analysis of the text content of the information to be identified to identify the authenticity of the information.
[0004] However, since there are many factors that may cause the forgery of the information to be identified, if only independent analysis of the image pixel level or independent analysis of the text content of the information to be identified is performed to identify the authenticity of the information, it will be difficult to fully capture the forgery traces of the information to be identified, and thus it is difficult to ensure the accuracy of the information forgery identification results. Summary of the invention
[0005] The present invention provides an information identification method, device, electronic device and storage medium, which are used to solve the defect in the prior art that it is difficult to fully capture the forgery traces of the information to be identified, thereby making it difficult to ensure the accuracy of the information forgery identification results, and achieve comprehensive capture of the forgery traces of the information to be identified from multiple modes and multiple perspectives, so as to improve the accuracy of information forgery identification.
[0006] The present invention provides an information identification method, comprising:
[0007] Inputting the image to be recognized and the text to be recognized in the information to be recognized into the feature extraction module of the recognition model to obtain the multi-path target image embedding features of the image to be recognized and the multi-path target text embedding features of the text to be recognized;
[0008] Inputting the target image embedding features of the multi-path and the target text embedding features of the multi-path into the multi-path aggregation module of the recognition model to obtain the image aggregation features of the image to be recognized and the text aggregation features of the text to be recognized;
[0009] Inputting the image aggregation features and the text aggregation features into the classification module of the recognition model to obtain a forgery classification result of the information to be identified;
[0010] The recognition model is obtained by iterative training based on each sample information and the forged classification labels of each sample information.
[0011] According to an information recognition method provided by the present invention, the feature extraction module includes an encoding module, a cross-modal interaction module and a multi-path projection module;
[0012] The step of inputting the image to be recognized and the text to be recognized in the information to be recognized into a feature extraction module of a recognition model to obtain a multi-path target image embedding feature of the image to be recognized and a multi-path target text embedding feature of the text to be recognized, comprises:
[0013] Inputting the image to be recognized and the text to be recognized into the encoding module to obtain a first image embedding feature of the image to be recognized and a first text embedding feature of the text to be recognized;
[0014] Inputting the first image embedding feature and the first text embedding feature into the cross-modal interaction module to obtain a second image embedding feature of the image to be recognized and a second text embedding feature of the text to be recognized;
[0015] The second image embedding feature and the second text embedding feature are input into the multi-path projection module to obtain the multi-path target image embedding feature and the multi-path target text embedding feature.
[0016] According to an information recognition method provided by the present invention, the multi-path projection module includes multiple groups of first cross attention units and multiple groups of first linear projection units;
[0017] The target image embedding features of the multipath are determined based on the following steps:
[0018] Input the first learnable embedding feature corresponding to each group of the first cross attention units as a query parameter and the second image embedding feature as a key parameter and a value parameter to each group of the first cross attention units, and obtain the target image embedding feature of the path corresponding to each group of the first cross attention units;
[0019] The first learnable vector and the second image embedding feature corresponding to each group of the first linear projection units are input into each group of the first linear projection units as input information to obtain the target image embedding feature of the path corresponding to each group of the first linear projection units.
[0020] According to an information identification method provided by the present invention, the image aggregation feature is obtained by a multi-path aggregation module based on the following steps:
[0021] According to the first learnable embedding features corresponding to each group of the first cross attention units and the features of each dimension in the second image embedding features, obtaining the first coefficients of the features of each dimension under the paths corresponding to the first cross attention units in each group;
[0022] According to each dimensional vector in the first learnable vector corresponding to each group of the first linear projection units, obtaining a second coefficient of each dimensional feature under a path corresponding to each first linear projection unit;
[0023] The aggregation coefficient of the feature in each dimension is obtained by adding the sum of the first coefficients of the feature in each dimension under the paths corresponding to the multiple groups of the first cross attention units and the sum of the second coefficients of the feature in each dimension under the paths corresponding to the multiple groups of the first linear projection units;
[0024] The aggregation coefficient of each feature is weightedly added to the features of each dimension to obtain the image aggregation feature.
[0025] According to an information recognition method provided by the present invention, the multi-path projection module includes multiple groups of second cross attention units and multiple groups of second linear projection units;
[0026] The target text embedding features of the multi-path are determined based on the following steps:
[0027] Input the second learnable embedding feature corresponding to each group of the second cross attention units as a query parameter and the second text embedding feature as a key parameter and a value parameter to each group of the second cross attention units to obtain the target text embedding feature of the path corresponding to each group of the second cross attention units;
[0028] The second learnable vector and the second text embedding feature corresponding to each group of the second linear projection units are input into each group of the second linear projection units as input information to obtain the target text embedding feature of the path corresponding to each group of the second linear projection units.
[0029] According to an information identification method provided by the present invention, the encoding module includes an image encoder;
[0030] The first image embedding feature is determined based on the following steps:
[0031] Dividing the image to be recognized into a plurality of image blocks;
[0032] Inputting each of the image blocks into the image encoder, and extracting encoding features of each of the image blocks through the image encoder to obtain the first image embedding features;
[0033] The first image embedding features include multiple image block embedding features and image classification embedding features.
[0034] According to an information identification method provided by the present invention, the encoding module includes a text encoder;
[0035] The first text embedding feature is determined based on the following steps:
[0036] Input the text to be recognized into the text encoder, divide the text to be recognized into multiple word units through the text encoder, obtain the embedding vector to be processed corresponding to each word unit in the word unit embedding vector table, perform forward processing on the embedding vectors to be processed corresponding to the multiple word units, and obtain the first text embedding feature;
[0037] The first text embedding features include multiple word embedding features and text classification embedding features.
[0038] According to an information recognition method provided by the present invention, the cross-modal interaction module includes a first cross-attention unit and a second cross-attention unit;
[0039] The step of inputting the first image embedding feature and the first text embedding feature into the cross-modal interaction module to obtain a second image embedding feature of the image to be recognized and a second text embedding feature of the text to be recognized includes:
[0040] Input the first image embedding feature as a query parameter and the first text embedding feature as a key parameter and a value parameter into the first cross-attention unit to obtain the second image embedding feature;
[0041] The first text embedding feature is used as a query parameter and the first image embedding feature is used as a key parameter and a value parameter to input into the second cross attention unit to obtain the second text embedding feature.
[0042] According to an information identification method provided by the present invention, the classification module includes a fully connected layer and a classification output layer;
[0043] The step of inputting the image aggregation features and the text aggregation features into the classification module of the recognition model to obtain the forgery classification result of the information to be identified includes:
[0044] Inputting the image aggregation features and the text aggregation features into the fully connected layer to obtain the hidden variable values of the information to be identified under each preset forgery category;
[0045] Inputting the latent variable value into the classification output layer to obtain the probability value of the information to be identified under each of the preset forgery categories;
[0046] The forgery classification result is obtained according to the probability value.
[0047] According to an information recognition method provided by the present invention, the recognition model is trained based on the following steps:
[0048] Obtain information about each sample;
[0049] Marking the forged information in each of the sample information to obtain a forged classification label for each of the sample information;
[0050] Constructing a training data set according to each of the sample information and the forged classification labels;
[0051] Iteratively training the initial neural network model according to the training data set to obtain the recognition model;
[0052] The forged information includes multiple items of forged information in which a target object in a sample image is replaced, forged information in which an emotional state of a target object in a sample image is replaced, forged information in which text content of a sample text is replaced, and forged information in which a paragraph, word or phrase used to represent an emotional state in a sample text is replaced.
[0053] According to an information recognition method provided by the present invention, the initial neural network model is iteratively trained according to the training data set to obtain the recognition model, including:
[0054] For the current iteration training, randomly adjusting the arrangement order of each sample information in the training data set;
[0055] Divide the adjusted training data set to obtain training data subsets corresponding to multiple batch trainings;
[0056] For each batch of training, the following steps are performed:
[0057] Based on the training data subset corresponding to the current batch training, the initial neural network model after the previous batch training is iteratively trained to obtain the initial neural network model after the current batch training;
[0058] The recognition model is constructed according to the initial neural network model after each batch of training.
[0059] According to an information identification method provided by the present invention, the information to be identified is obtained based on the following steps:
[0060] Get original information;
[0061] Preprocessing the original image in the original information to obtain the information to be identified;
[0062] The preprocessing includes scaling the image pixel size of the original image to a preset pixel size.
[0063] The present invention also provides an information identification device, comprising:
[0064] A first processing unit is used to input the image to be recognized and the text to be recognized in the information to be recognized into a feature extraction module of a recognition model to obtain a multi-path target image embedding feature of the image to be recognized and a multi-path target text embedding feature of the text to be recognized;
[0065] A second processing unit is used to input the target image embedding features of the multi-path and the target text embedding features of the multi-path into the multi-path aggregation module of the recognition model to obtain the image aggregation features of the image to be recognized and the text aggregation features of the text to be recognized;
[0066] A classification unit, used for inputting the image aggregation feature and the text aggregation feature into the classification module of the recognition model to obtain a forgery classification result of the information to be identified;
[0067] The recognition model is obtained by iterative training based on each sample information and the forged classification labels of each sample information.
[0068] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the information identification methods described above is implemented.
[0069] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the information identification method described in any one of the above is implemented.
[0070] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the information identification method described above is implemented.
[0071] The information recognition method, device, electronic device and storage medium provided by the present invention perform multi-modal and multi-path feature extraction on the image to be recognized and the text to be recognized in the information to be recognized by using a feature extraction module, so as to capture the key features of the image and text associated with the forged information at different levels and angles, and form multi-path target image embedding features and target text embedding features; then, a multi-path aggregation module is used to fuse the features of the multi-path target image embedding features and the target text embedding features, so as to make full use of the inherent connection and complementarity between the image and the text, and form more comprehensive and integrated image aggregation features and text aggregation features; finally, a classification module is used to capture multi-modal and multi-perspective forgery traces based on the image aggregation features and the text aggregation features, so as to obtain the information forgery classification results more comprehensively and accurately. BRIEF DESCRIPTION OF THE DRAWINGS
[0072] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0073] Figure 1 It is a flow chart of the information identification method provided by the present invention.
[0074] Figure 2 It is a structural schematic diagram of the recognition model provided by the present invention.
[0075] Figure 3 It is a structural schematic diagram of the cross-modal interaction module provided by the present invention.
[0076] Figure 4 It is a structural schematic diagram of the multi-path projection unit provided by the present invention.
[0077] Figure 5 It is a schematic diagram of the process of linear projection polymerization provided by the present invention.
[0078] Figure 6 It is a structural schematic diagram of the information identification device provided by the present invention.
[0079] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0080] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0081] With the development of deep learning technology, especially the rise of GANs, image and text forgery has emerged. Among them, GANs is a model composed of a generator and a discriminator. Through learning and optimization in the adversarial process, the generator can generate realistic false image and text information; in addition, the progress of deep learning, the continuous development of computer vision and natural language processing technology, the rise of multimodal learning, and the widespread application of open source frameworks and pre-trained models have also provided strong support for the development of image and text forgery technology.
[0082] However, while researchers continue to improve GANs and other generative artificial intelligence models and enhance their ability to generate realistic images, videos, and texts, forged text and images based on these models have a certain negative impact on users' normal work and life. Therefore, in order to ensure the legitimate use of information and reduce potential risks, some researchers are also actively exploring methods to detect and prevent forged images and texts, so as to reduce the negative impact of forged images and texts on users' normal work and life. In this regard, how to efficiently and conveniently identify the authenticity of information is one of the technical problems that need to be solved urgently.
[0083] Usually, in practical applications, the authenticity of information is identified by performing independent analysis of the image pixel level on the information to be identified using a deep learning network model, or by performing independent analysis of the text content of the information to be identified to identify the authenticity of the information.
[0084] However, since there are many factors that may cause the forgery of the information to be identified, if only an independent analysis of the information to be identified from a single perspective at the image pixel level or an independent analysis of the text content from a single perspective is performed on the information to be identified to identify its authenticity, it will be difficult to fully capture the traces of forgery of the information to be identified, and thus it will be difficult to ensure the accuracy of the information forgery identification results.
[0085] In response to the above problems, the present embodiment provides an information identification method, which can perform multi-path feature extraction and feature fusion on multimodal data, so as to capture the forgery traces of the information to be identified from multiple modes and multiple perspectives, thereby improving the comprehensiveness of the forgery traces of the information to be identified, improving the performance of information forgery identification, and thereby reducing the negative impact caused by forged information.
[0086] The information identification method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.
[0087] Figure 1 is a flow chart of the information identification method provided by the present invention; Figure 1 As shown, the method includes: step 110, step 120 and step 130.
[0088] Step 110, inputting the image to be recognized and the text to be recognized in the information to be recognized into the feature extraction module of the recognition model to obtain the multi-path target image embedding features of the image to be recognized and the multi-path target text embedding features of the text to be recognized.
[0089] Step 120, inputting the target image embedding features of the multi-path and the target text embedding features of the multi-path into the multi-path aggregation module of the recognition model to obtain the image aggregation features of the image to be recognized and the text aggregation features of the text to be recognized;
[0090] Step 130: input the image aggregation features and the text aggregation features into the classification module of the recognition model to obtain the forgery classification result of the information to be identified.
[0091] The recognition model is obtained by iterative training based on each sample information and the forged classification labels of each sample information.
[0092] It is understandable that in the related technology, the authenticity of the information is usually identified by using a deep learning network model to perform independent analysis of the image pixel level on the information to be identified, or by performing independent analysis of the text content of the information to be identified to identify the authenticity of the information. However, since there are many factors that cause the forgery of the information to be identified, if the authenticity of the information is identified in this way, it is difficult to fully capture the traces of forgery in the information, and thus it is difficult to ensure the accuracy of the information forgery identification results.
[0093] To address this problem, in this embodiment, a recognition model is established that can perform multi-modal and multi-path feature extraction and feature aggregation to comprehensively capture the traces of forgery of the information to be identified. By using this recognition model to classify the forgery of the information to be identified, the accuracy of information forgery identification can be effectively improved.
[0094] Optionally, before executing step 110, a recognition model may be pre-trained. Specifically, an initialized neural network model may be constructed, and the initialized neural network model at least includes an initialized feature extraction module, an initialized multipath aggregation module, and an initialized classification module, wherein the initialized feature extraction module may be a model prepared for multi-modal and multi-path feature extraction after parameter initialization, or a pre-trained model with multi-modal and multi-path feature extraction functions; the initialized multipath aggregation module may be a model prepared for multi-path feature aggregation after parameter initialization, or a pre-trained model with multi-path feature aggregation functions; the initialized classification module may be a model prepared for counterfeit classification after parameter initialization, or a pre-trained model with counterfeit classification functions, and this embodiment does not specifically limit this.
[0095] Here, the initialized feature extraction module can be constructed based on one or more initialization modules for feature extraction, such as based on an initialized multi-path projection module, or based on an initialized encoding module, an initialized cross-modal interaction module and an initialized multi-path projection module. This embodiment does not specifically limit this.
[0096] In addition, a large amount of sample information needs to be collected, as well as forged classification labels for the sample information.
[0097] Subsequently, the sample information can be used as the input of the initialized neural network model, and the initialized feature extraction module in the initialized neural network model performs multi-modal and multi-path feature extraction on the sample image and sample text of the sample information to obtain multi-path sample image embedding features and multi-path sample text embedding features, and then the initialized multi-path aggregation module performs multi-path feature aggregation on the multi-path sample image embedding features and multi-path sample text embedding features to obtain sample image aggregation features and sample text aggregation features, and then the initialized classification module performs forgery trace detection based on the sample image aggregation features and sample text aggregation features to obtain the forged classification results of the sample information. After obtaining the forged classification results of the sample information, the loss function can be calculated based on the difference between the forged classification results of the sample information and the forged classification labels of the sample information, and the parameters of the initialized neural network model as a whole can be iterated based on the loss function to obtain the recognition model.
[0098] Optionally, after obtaining the recognition model, the model can be used to identify information forgery in the model application reasoning stage. The specific implementation steps include:
[0099] First, obtain the image to be identified and the text to be identified in the information to be identified; here, the information to be identified is the information required for forgery classification and identification, and the information is graphic information containing images and texts. The graphic information can be an article that describes a certain image in words, or it can be news information, blog articles, etc. containing images and texts. This embodiment does not specifically limit this.
[0100] The specifications of the image to be identified in the information to be identified here are consistent with the specifications of the sample image input when training the recognition model.
[0101] For example, if the input sample image when training the recognition model is an image with a width and height of L pixels, then the image to be recognized when applying the application is The width and height should also be L pixels. If the original image in the original information originally obtained meets the above specifications, the original image is directly input into the recognition model as the image to be recognized; if the original image does not meet the above specifications, the image to be recognized needs to be formed by preprocessing the original image.
[0102] Exemplarily, the information to be identified is obtained based on the following steps:
[0103] Get original information;
[0104] Preprocessing the original image in the original information to obtain the information to be identified;
[0105] The preprocessing includes scaling the image pixel size of the original image to a preset pixel size.
[0106] The preset pixel size here can be set according to actual needs, such as being set to L pixels, where L can be any one of 256, 384, and 512, and L=384 can be preferably selected.
[0107] Optionally, in the model application reasoning stage, the original image in the original information needs to be preprocessed, specifically including: first, scaling the long side of the original image to L pixels; then, scaling the short side of the original image proportionally; finally, padding the short side to L pixels. After the above preprocessing operation, a preprocessed image can be obtained, thereby obtaining the information to be recognized. Among them, the width and height of the preprocessed image are equal, both L pixels. The preprocessed image is the image to be recognized I input to the recognition module.
[0108] During the training phase of the model, the generation of sample information can also be implemented by referring to the above-mentioned steps of obtaining the information to be identified, which will not be repeated here.
[0109] For example, if the input sample image when training the recognition model is a three-channel color image (RGB image), then the image to be recognized input to the recognition model is It should also be a three-channel color image (RGB image), or the original image is converted into a three-channel color image (RGB image) and then input into the recognition model as the image to be recognized.
[0110] Similarly, the specifications of the text to be recognized T input to the recognition module must be consistent with the specifications of the sample text input when training the recognition model. If the sample text input when training the recognition model is in a specific language (for example, Chinese), then the text to be recognized T input to the recognition model should also be in a specific language (for example, Chinese); if the sample text input when training the recognition model is in multiple languages, then the text to be recognized T input to the recognition model can also be in multiple languages.
[0111] Subsequently, the image to be identified and the text to be identified in the information to be identified are input into the feature extraction module, so that the feature extraction module performs multi-modal and multi-path feature extraction on the image to be identified and the text to be identified, thereby obtaining the multi-path target image embedding features of the image to be identified and the multi-path target text embedding features of the text to be identified, thereby achieving the capture of forgery traces of the image or text from multiple modalities and multiple perspectives, so as to perform forgery classification based on this later, effectively improving the accuracy of forgery classification.
[0112] Subsequently, after obtaining the multi-path target image embedding features and the multi-path target text embedding features through the feature extraction module, the multi-path target image embedding features and the multi-path target text embedding features can be aggregated respectively through the multi-path aggregation module in the recognition model to obtain image aggregation features and text aggregation features.
[0113] The aggregation here can be achieved through cascade aggregation or linear projection aggregation, which can be determined specifically according to the computing resources of the computer device deployed by the recognition model. Thus, by flexibly selecting cascade aggregation or linear projection aggregation, the aggregation strategy is dynamically adjusted according to the computing resources of the computer device, thereby achieving the optimal balance between processing efficiency and resource utilization of the recognition model, and improving the adaptability and flexibility of the model. For example, when the computing resources of the computer device are limited, multi-path feature aggregation can be achieved through linear projection aggregation; when the computing resources of the computer device are sufficient, linear projection aggregation or cascade aggregation can be randomly selected to achieve multi-path feature aggregation.
[0114] Here, cascade aggregation is to obtain the output result of the previous module of the multi-path aggregation module, and then further perform aggregation calculation based on this output result. For example, if the previous module of the multi-path aggregation module is a multi-path projection module, the output of the cross-attention branch of the multi-path projection module can be embedded and summed, and the output of the linear projection branch of the multi-path projection module can be embedded and summed, and the above embedded sums can be added to realize aggregation calculation.
[0115] Here, linear projection aggregation is based on the calculation process of the previous module of the multi-path aggregation module. After linearly projecting the output result of the previous module, further aggregation calculation is performed. For example, the previous module of the multi-path aggregation module is a multi-path projection module. At this time, the weight coefficient of the cross-attention branch is calculated and stored; the stored weight coefficient of the cross-attention branch and the weight coefficient of the linear projection branch are summed to obtain the weight coefficient of the input embedding corresponding to the multi-path projection module, and the weight coefficient of the input embedding is weightedly added to the input embedding to realize aggregation calculation.
[0116] Subsequently, after obtaining the image aggregation features and the text aggregation features through the multi-path aggregation module, the image aggregation features and the text aggregation features can be spliced and input into the classification module, so that the classification module can identify the forged information of the image to be identified or the text to be identified based on the image aggregation features and the text aggregation features to obtain the forged classification result of the information to be identified.
[0117] The forged classification result here is used to identify whether there is forged information in the image-text pair of the information to be identified; the forged information includes but is not limited to at least one forged information category in the image to be identified and / or at least one forged information category in the text to be identified. Among them, the forged information category in the image to be identified includes but is not limited to forged information in which the target object in the image is replaced, such as the face image of a specific person in the image is replaced with the face image of another person, and the emotional state of the target object in the image is replaced, such as the emotional state of the face image in the image is changed from a happy state to an angry state; the forged information category in the text to be identified includes but is not limited to forged information in which the text content of the text is replaced, such as the text content corresponding to the image is replaced with another paragraph of text content, and the paragraphs, words or phrases used to represent the emotional state in the text are replaced, such as some words / phrases in the text corresponding to the image are replaced with other words / phrases, especially such modification can cause the emotional state expressed by the text to change, thereby changing the tendency of the information content.
[0118] The method provided in this embodiment uses a feature extraction module to perform multi-modal and multi-path feature extraction on the image to be identified and the text to be identified in the information to be identified, so as to capture the key features of the image and text associated with the forged information at different levels and angles, and form multi-path target image embedding features and target text embedding features; then, a multi-path aggregation module is used to fuse the features of the multi-path target image embedding features and the target text embedding features, so as to make full use of the inherent connection and complementarity between the image and the text, and form more comprehensive and integrated image aggregation features and text aggregation features; finally, a classification module is used to capture multi-modal and multi-perspective forgery traces based on the image aggregation features and the text aggregation features, so as to obtain the information forgery classification results more comprehensively and accurately.
[0119] Figure 2 A schematic diagram of the structure of the recognition model provided by the present invention; Figure 2 As shown, the feature extraction module includes an encoding module, a cross-modal interaction module and a multi-path projection module;
[0120] Accordingly, step 110 includes step 111 , step 112 and step 113 .
[0121] Step 111: input the image to be recognized and the text to be recognized into the encoding module to obtain a first image embedding feature of the image to be recognized and a first text embedding feature of the text to be recognized.
[0122] Among them, the encoding module can be formed by an integrated encoding model that can be used for both image embedding feature extraction and text embedding feature extraction, or it can be jointly constructed by an image encoder for image embedding feature extraction and a text encoder for text embedding feature extraction. This embodiment does not specifically limit this.
[0123] For example, when the encoding module is constructed as an integrated encoding model, the image to be recognized and the text to be recognized can be respectively input into the encoding module, so that the encoding module can encode and output the first image embedding feature of the image to be recognized and the first text embedding feature of the text to be recognized respectively; for another example, when the encoding module is jointly constructed as an image encoder and a text encoder, the image to be recognized can be input into the image encoder, so that the image encoder can encode and output the first image embedding feature of the image to be recognized, and at the same time, the text to be recognized can be input into the text encoder, so that the text encoder can encode and output the first text embedding feature of the text to be recognized.
[0124] like Figure 2 As shown, the encoding module includes an image encoder; the step of acquiring the first image embedding feature in step 111 can be further implemented based on the following steps:
[0125] Dividing the image to be recognized into a plurality of image blocks;
[0126] Inputting each of the image blocks into the image encoder, and extracting encoding features of each of the image blocks through the image encoder to obtain the first image embedding features;
[0127] The first image embedding features include multiple image block embedding features and image classification embedding features.
[0128] It should be noted that the image encoder in the embodiment of the present application The network structure of can be the same as that of ViT-L / 16; at the beginning of training, the parameters of the image encoder can be initialized using the parameters of ViT-L / 16 pre-trained on the ImageNet-21k dataset. Various embeddings (or embedded features) in the embodiments of the present application are all represented in the form of vectors.
[0129] Optionally, since the image encoder The input is 256 image patches (also called image slices). In order to generate the image encoder The required input image blocks, the embodiment of the present application can divide the image to be identified I into 256 image blocks of the same size and without overlap along the width and height of the image to be identified I, and the width of each image block is L / 16 pixels and the height is also L / 16 pixels.
[0130] For the convenience of description, the total number of image blocks is defined as ,So .
[0131] Then, after obtaining After the image blocks, The image blocks are input to the above image encoder Extracting image embeddings (patch embedding) (i.e. the first image embedding feature mentioned above), include Image patch embedding (i.e., the above-mentioned multiple image patch embedding features) and a classification embedding for classification (i.e., the above image classification embedding features), so . Among them, the image is embedded The dimension of each embedding in is , .
[0132] The method provided in this embodiment, by carefully dividing the image to be identified into multiple image blocks, and inputting them one by one into the image encoder for deep coding feature extraction, can capture the image features comprehensively and carefully, thereby generating a first image embedding feature including multiple image block embedding features with rich details and generalized image classification embedding features, thereby achieving a comprehensive representation and efficient analysis of the image content, so as to more accurately realize the identification of true and false information in the subsequent process.
[0133] like Figure 2 As shown, the encoding module further includes a text encoder; the step of obtaining the first text embedding feature in step 111 can be further implemented based on the following steps:
[0134] Input the text to be recognized into the text encoder, divide the text to be recognized into multiple word units through the text encoder, obtain the embedding vector to be processed corresponding to each word unit in the word unit embedding vector table, perform forward processing on the embedding vectors to be processed corresponding to the multiple word units, and obtain the first text embedding feature;
[0135] The first text embedding features include multiple word embedding features and text classification embedding features.
[0136] It should be noted that the text encoder in the embodiment of the present application The network structure can be The network structure is the same as that of . At the beginning of training, the parameters of the text encoder can be used The parameters trained on the general language understanding evaluation benchmark are initialized. The tokenizer and the corresponding token embedding vector table used by the text encoder can be compared with the above The word segmenter and word unit embedding vector table of the text encoder are fixed settings and are not updated during training.
[0137] Optionally, a text encoder can be used Extract text embeddings of the text to be recognized (i.e., the first text embedding feature). The specific implementation steps include: first, using a word segmenter to divide the text to be recognized T into multiple word units, where the total number of word units can be defined as ,Should It changes dynamically according to the different texts T to be recognized; then, the embedding corresponding to each word (i.e., the embedding vector to be processed) is retrieved from the word embedding vector table and passed through the text encoder The forward processing process, text encoder Extract the text embedding of the text to be recognized T , include Token Embedding (i.e., the multiple word embedding features mentioned above) and a classification word embedding for classification (That is, the above text classification embedding feature). Therefore .in, The dimension of each word embedding in is also , .
[0138] The method provided in this embodiment, by inputting the text to be recognized into a text encoder, undergoes fine word unit segmentation, word unit embedding vector retrieval and forward processing flow to extract the first text embedding feature containing multiple word unit embedding features and text classification embedding features. This process not only retains the semantic details of the text, but also enhances the overall representation ability of the text, so that the subsequent identification of true and false information can be realized more accurately.
[0139] Step 112, inputting the first image embedding feature and the first text embedding feature into the cross-modal interaction module to obtain a second image embedding feature of the image to be recognized and a second text embedding feature of the text to be recognized;
[0140] like Figure 2 As shown, the feature extraction module also includes a cross-modal interaction module to achieve learning of the interaction features between the different modal features output by the encoding module. The cross-modal interaction module here can be formed by constructing an integrated cross-modal interaction model that can be used to extract image features related to text content and text features related to image content, or it can be jointly constructed by a first cross-modal interaction module for extracting image features related to text content and a second cross-modal interaction module for extracting text features related to image content, and this embodiment does not specifically limit this.
[0141] For example, when the cross-modal interaction module is constructed as an integrated cross-modal interaction model, the first image embedding feature and the first text embedding feature can be input into the cross-modal interaction module, so that the cross-modal interaction module can extract image features related to the text content and extract text features related to the image content to obtain the second image embedding feature of the image to be identified and the second text embedding feature of the text to be identified; or, the first image embedding feature and the first text embedding feature can be input into the first cross-modal interaction module, so that the first cross-modal interaction module can extract image features related to the text content to obtain the second image embedding feature of the image to be identified, and the first text embedding feature and the first image embedding feature can be input into the second cross-modal interaction module, so that the second cross-modal interaction module can extract text features related to the image content to obtain the second text embedding feature of the text to be identified.
[0142] Optionally, after extracting the image embedding through the encoding module and text embedding Afterwards, the two sets of embeddings extracted above are simultaneously input into the cross-modal interaction module (Cross Modal Interaction Module) for interactive feature extraction.
[0143] Figure 3 is a schematic diagram of the structure of the cross-modal interaction module provided by the present invention; Figure 2 and Figure 3 As shown, the cross-modal interaction module The cross-attention mechanism can be used to achieve cross-modal information interaction, that is, information interaction between image embedding of image modality and text embedding of text modality, thereby improving the ability of image embedding of image modality and text embedding of text modality to encode semantic information. In this embodiment, the cross-modal interaction module includes two cross-attention modules: 1) Cross-attention module (Also known as the first cross attention unit); 2) Cross attention module (Also known as the second cross attention unit).
[0144] like Figure 3 As shown, the cross attention module The queries are image embeddings Embedding in, Cross-Attention Module The keys and values are text embeddings , after cross attention calculation, we can get the cross attention module The output embedding , which is the second image embedding feature.
[0145] Cross-Attention Module The queries are text embeddings Embedding in, Cross-Attention Module The keys and values are image embeddings , after cross attention calculation, we can get the cross attention module The output embedding , which is the second text embedding feature.
[0146] Therefore, embedding text and image embedding Through the cross-modal interaction module After processing, the cross-modal interaction module Can output the corresponding image embedding and the corresponding text embedding .
[0147] Among them, the image embedding Embedding feature dimension and image embedding The embedding feature dimension of ,in is the embedding corresponding to all image patches, is a categorical embedding for image classification.
[0148] Similarly, text embedding Embedding feature dimension and text embedding The embedding feature dimension of is also consistent, that is, ,in is the word embedding of the text T to be recognized, is a categorical embedding for text classification.
[0149] The method provided in this embodiment realizes two-way interaction and fusion between image and text embedding features by inputting the first image embedding feature and the first text embedding feature as query and key-value parameters, respectively, into the first cross-attention unit and the second cross-attention unit in the corresponding cross-modal interaction module, thereby obtaining the second image embedding feature and the second text embedding feature that contain both the original information and are rich in cross-modal interaction information, which significantly enhances the depth and breadth of multimodal information processing.
[0150] Step 113: input the second image embedding feature and the second text embedding feature into the multi-path projection module to obtain the multi-path target image embedding feature and the multi-path target text embedding feature.
[0151] like Figure 2 As shown, the feature extraction module also includes a multipath projection module to realize multipath feature extraction of embedded features of different modes output by the cross-modal interaction module. The multipath projection module here can be formed by constructing an integrated multipath projection model that can be used for multipath image feature extraction and multipath text feature extraction, or it can be jointly constructed by a first multipath projection unit for multipath image feature extraction and a second multipath projection unit for multipath text feature extraction, which is not specifically limited in this embodiment.
[0152] For example, when the multi-path projection module is constructed as an integrated multi-path projection model, the second image embedding feature and the second text embedding feature can be input into the multi-path projection module, so that the multi-path projection module can perform multi-path image feature extraction and multi-path text feature extraction, thereby obtaining the multi-path target image embedding feature and the multi-path target text embedding feature. For another example, when the multi-path projection module is jointly constructed by a first multi-path projection unit and a second multi-path projection unit, the second image embedding feature can be input into the first multi-path projection unit, so that the first multi-path projection unit can perform multi-path image feature extraction, thereby obtaining the multi-path target image embedding feature, and the second text embedding feature can be input into the second multi-path projection unit, so that the second multi-path projection unit can perform multi-path text feature extraction, thereby obtaining the multi-path target text embedding feature.
[0153] In some embodiments, Figure 2 As shown, the multi-path projection module includes two branch multi-path projection units: a multi-path projection unit (or first multi-path projection unit) and multi-path projection unit (or called the second multi-path projection unit).
[0154] Among them, the multi-path projection unit It is a multi-path projection model for image embedding, with a cross-modal interaction module Output image embedding is the input; multipath projection unit It is a multi-path projection model for text embedding, with a cross-modal interaction module Output text embedding For input.
[0155] In this embodiment, the multi-path projection unit and multi-path projection unit The network structure of is the same, but the model parameters of the two are not shared. In addition, the multi-path projection unit and multi-path projection unit The network structure configuration can also be different, such as containing different numbers of cross-attention units, different numbers of linear projection units, or different numbers of both units.
[0156] The number of cross attention units and the number of linear projection units contained in each path projection unit can be adaptively set according to actual needs, ensuring that each path projection unit includes at least two cross attention units and two linear projection units; illustratively, a multi-path projection unit can be set and multi-path projection unit Both include three groups of cross attention units and two groups of linear projection units.
[0157] The target image embedding features of the multiple paths in step 113 may be further determined based on the following steps:
[0158] Input the first learnable embedding feature corresponding to each group of the first cross attention units as a query parameter and the second image embedding feature as a key parameter and a value parameter to each group of the first cross attention units, and obtain the target image embedding feature of the path corresponding to each group of the first cross attention units;
[0159] The first learnable vector and the second image embedding feature corresponding to each group of the first linear projection units are input into each group of the first linear projection units as input information to obtain the target image embedding feature of the path corresponding to each group of the first linear projection units.
[0160] The target text embedding features of the multiple paths in step 113 may be further determined based on the following steps:
[0161] Input the second learnable embedding feature corresponding to each group of the second cross attention units as a query parameter and the second text embedding feature as a key parameter and a value parameter to each group of the second cross attention units to obtain the target text embedding feature of the path corresponding to each group of the second cross attention units;
[0162] The second learnable vector and the second text embedding feature corresponding to each group of the second linear projection units are input into each group of the second linear projection units as input information to obtain the target text embedding feature of the path corresponding to each group of the second linear projection units.
[0163] To simplify the description, the following temporarily ignores the difference in input to explain the general settings of each multi-path projection unit, that is, the multi-path projection unit described below may be a multi-path projection unit. , or a multi-path projection unit , correspondingly, in the multipath projection unit When , the cross attention unit described below is the first cross attention unit, the linear projection unit is the first linear projection unit, and the input embedding is the image embedding , the learnable embedding is the first learnable embedding feature, and the learnable vector is the first learnable vector; in the multipath projection unit, When , the cross attention unit described below is the second cross attention unit, the linear projection unit is the second linear projection unit, and the input embedding is the text embedding , the learnable embedding is the second learnable embedding feature, and the learnable vector is the second learnable vector.
[0164] Figure 4It is a structural schematic diagram of the multi-path projection unit provided by the present invention.
[0165] Alternatively, if Figure 4 As shown, the multipath projection unit can be defined as , where m is the number of cross-attention units and n is the number of linear projection units. The input embedding of the multi-path projection unit is defined as ,Include embeddings, where represents classification embedding, is an optional input. In this embodiment, contains categorical embeddings .for , ,for , .
[0166] The m groups of cross attention units are , , , Cross-Attention Unit Learnable Embedding , For query, input embedding The embeddings in are keys and values. Embeddings can be learned The dimensions and The dimensions of the embeddings in are the same. At the beginning of training, is randomly initialized and continuously updated during the training process; in addition, different cross-attention unit branches correspond to different initial values of the learnable query vectors.
[0167] Cross-Attention Unit , The output calculation formula is as follows:
[0168] ;
[0169] in, Cross-Attention Unit The output corresponding path target image embedding features, represent The transposed vector of Represents the dot product between vectors.
[0170] The n groups of linear projection units are linear projection units , , , Linear projection unit , is a learnable vector , ,Include learnable parameters, each element of the learnable vector corresponds to each embedding in the input embedding F of the multi-path projection module. At the beginning of training, , It is randomly initialized and continuously updated during the training process, and the initial values of different linear projection unit branches are different.
[0171] Linear projection unit The output calculation formula is as follows:
[0172] ;
[0173] in, Cross-Attention Unit The output target image embedding features of the corresponding path.
[0174] For example, if the multipath projection unit and Both include three groups of cross attention units and two groups of linear projection units. Then, the cross attention unit The output calculation formula can be:
[0175] ;
[0176] in, represent The transposed vector of Represents the dot product between vectors.
[0177] Linear projection module The output calculation formula can be:
[0178] .
[0179] The method provided in this embodiment captures image embedding features and text embedding features from multiple perspectives by utilizing multiple groups of cross-attention units and multiple groups of linear projection units in parallel, which not only enhances the feature representation capability of images and texts, but also further enriches the diversity of the feature space by introducing learnable embedding features and vectors, thereby effectively improving the accuracy and generalization capability of forgery classification.
[0180] In some embodiments, after obtaining the multi-path target image embedding features and the multi-path target text embedding features through step 110, step 120 may be further performed to obtain image aggregation features and text aggregation features.
[0181] The multipath aggregation module may include two branches: a multipath aggregation module and multipath aggregation module ;Multi-path aggregation module Used to perform multi-path target image embedding feature aggregation to obtain image aggregation features; multi-path aggregation module Used to perform multi-path target text embedding feature aggregation to obtain text aggregation features.
[0182] To simplify the description, the following temporarily ignores the difference in input to explain the general acquisition process of each aggregation feature, that is, the output embedding of the multi-path aggregation module described below. It can be a text aggregation feature or an image aggregation feature; accordingly, in the output embedding When it is an image aggregation feature, the multi-path projection unit described below is , the multipath aggregation module is ; Embed in output When it is a text aggregation feature, the multi-path projection unit described below is , the multipath aggregation module is .
[0183] Optionally, the multipath aggregation module may perform the following steps to obtain the aggregation feature:
[0184] Since the multi-path projection unit outputs two sets of output results, they are m embeddings output by m groups of cross-attention modules , and n embeddings output by n linear projection modules . Embed m Through vector addition operations, it can be aggregated into an embedding ; embed n Through vector addition operations, it can be aggregated into an embedding According to the above and , and the final output embedding is obtained by summing ; where the output embedding The calculation formula is as follows:
[0185] .
[0186] For example, in the multipath aggregation module and It includes three groups of cross attention modules and two groups of linear projection modules respectively, then the above formula can be simplified as:
[0187] ;
[0188] In this embodiment, if Figure 2 As shown, the multi-path projection module The input is , Through the multi-path projection module and multipath aggregation module Then get the output ;Multi-path projection module The input is , Through the multi-path projection module and multipath aggregation module Then get the output .
[0189] Further, in order to simplify the aggregation calculation process of the multi-path aggregation module, the image aggregation feature in step 120 may also be obtained by the multi-path aggregation module based on the following simplified process:
[0190] According to the first learnable embedding features corresponding to each group of the first cross attention units and the features of each dimension in the second image embedding features, obtaining the first coefficients of the features of each dimension under the paths corresponding to the first cross attention units in each group;
[0191] According to each dimensional vector in the first learnable vector corresponding to each group of the first linear projection units, obtaining a second coefficient of each dimensional feature under a path corresponding to each first linear projection unit;
[0192] The aggregation coefficient of the feature in each dimension is obtained by adding the sum of the first coefficients of the feature in each dimension under the paths corresponding to the multiple groups of the first cross attention units and the sum of the second coefficients of the feature in each dimension under the paths corresponding to the multiple groups of the first linear projection units;
[0193] The aggregation coefficient of each feature is weightedly added to the features of each dimension to obtain the image aggregation feature.
[0194] Similarly, in order to simplify the aggregation calculation process of the multi-path aggregation module, the step of obtaining the text aggregation feature in step 120 can also refer to the above simplified process, which will not be repeated here.
[0195] In the following, this embodiment uses a cascade combination of a multipath projection unit and a multipath aggregation module (such as and A cascade combination of and The cascade combination of ) is used as an example to illustrate the operation simplification method in the network reasoning process. Assume that the multi-path projection unit contains m cross attention units and n linear projection units.
[0196] To compute the output embedding vector of the cascade combination , need to be calculated separately and Among them, the first part is the sum of the outputs of multiple cross-attention unit branches, the second part is the sum of the outputs of multiple linear projection unit branches.
[0197] First, the simplified calculation method of the linear projection branch is analyzed and discussed.
[0198] During neural network training, in order to calculate the output of the linear projection unit branch , the calculation process is divided into the following two steps:
[0199] The first step is to calculate indivual .because , , then calculate indivual The total number of multiplications and additions required is
[0200] The second step is to obtain indivual In the case of , the total number of multiplications and additions required is .
[0201] Therefore, the calculation Total number of multiplications and additions required for:
[0202] ;
[0203] In the neural network training process, the above calculations cannot be simplified. However, in the neural network reasoning process, the above calculation steps can be recombined into the following calculation formula:
[0204] ;
[0205] In the above formula, we first calculate the embedding The corresponding coefficients and Then calculate , the total number of multiplications and additions required for:
[0206] ;
[0207] By comparison and ,By recombining the above coefficients, the computation amount of the linear projection branch in the network reasoning process is reduced.
[0208] Next, a simplified computational method for the criss-cross attention branches is analyzed and discussed.
[0209] For the crisscross attention unit Output In this embodiment, the learnable embedding of the input of the group of cross attention units and the embedding of each input The first coefficient of each dimensional feature under the corresponding path of this group of cross attention units is calculated, that is, , and define it as ,So .
[0210] Therefore, the calculation The formula can be rewritten as:
[0211] = ;
[0212] The above formula is similar to the calculation formula of the linear projection branch, so the calculation of the cross attention branch can also be calculated using the simplified method of the linear projection module.
[0213] Based on the above analysis, the calculation of the multi-path aggregation module in the neural network application deployment stage can be further optimized into the following formula:
[0214] ;
[0215] In the above formula, is the j-th dimension vector in the learnable vector corresponding to the linear projection unit i, that is, the second coefficient of the j-th dimension feature under the path corresponding to the linear projection unit i; and the aggregation coefficient of the j-th dimension feature is defined as , accordingly, .therefore, can be viewed as a set of embedding vectors The output of a linear projection layer, where the coefficients of the linear projection layer are . Can be calculated in advance.
[0216] Figure 5 A schematic diagram of the process of linear projection polymerization provided by the present invention; Figure 5 As shown above, based on the above analysis, the calculation of cascade combination can be transformed into linear projection. For the convenience of description, define ,in ;definition ,in So, for and The cascade combination is transformed into a linear projection, ;for and The cascade combination is transformed into a linear projection, . and is the aggregation coefficient obtained by aggregation calculation.
[0217] In summary, the method provided in this embodiment, during the application deployment stage of the neural network model, merges the calculations of the multi-path projection module and the multi-path aggregation module to convert them from cascade aggregation to linear projection aggregation, thereby reducing the computational complexity, improving the operating efficiency of the model, and thereby improving the efficiency of information authenticity identification, and accelerating the reduction of the negative impact caused by forged images and texts.
[0218] In some embodiments, the classification module includes a fully connected layer and a classification output layer;
[0219] The step of inputting the image aggregation features and the text aggregation features into the classification module of the recognition model to obtain the forgery classification result of the information to be identified includes:
[0220] Inputting the image aggregation features and the text aggregation features into the fully connected layer to obtain the hidden variable values of the information to be identified under each preset forgery category;
[0221] Inputting the latent variable value into the classification output layer to obtain the probability value of the information to be identified under each of the preset forgery categories;
[0222] The forgery classification result is obtained according to the probability value.
[0223] Optionally, after obtaining the embedding of the aggregation output of the multipath aggregation module (i.e., image aggregation features) and embedding (i.e., text aggregation features) can be embedded and embed Concatenate to form a new embedding vector , and The dimension is d, The dimension of is 2d.
[0224] The steps for obtaining forged classification results are similar in the model training process and the model inference process. Specifically, the classification module performs the following steps:
[0225] First, the vector After at least one fully connected layer (weight parameter is ), obtain the hidden variable value of the information to be identified under each preset forgery category. Since the preset forgery categories in this embodiment include two types, namely, the presence of forged information and the absence of forged information, the hidden variable value under the preset forgery category is 2 hidden variable values.
[0226] Then, the two latent variable values are passed through the classification output layer to output the probability values of each preset forgery category. , ;in, is the probability value under the category of no fake information, is the probability value of the existence of false information category. , is the first The transposed vector of the weight parameters of the columns, Represents the dot product between vectors. The classification output layer here can be formed by constructing a softmax network.
[0227] Finally, based on the probability values under each preset forgery category, the forgery classification result of the information to be identified is jointly determined.
[0228] Here, the method of determining the forged classification result of the information to be identified includes: taking the category corresponding to the latent variable with the highest probability value as the forged classification result; or taking the probability value of the category without forged information as the forged classification result. Respectively with the first preset threshold and a second preset threshold For comparison, if , then the forged classification result is determined to be that there is no forged information; if , then the forged classification result is determined to be empty, which means that the authenticity of the information to be identified cannot be determined; if , then the forgery classification result is determined to be the presence of forged information. Among them, the first preset threshold Less than the second preset threshold , the specific value can be set adaptively according to actual needs, such as , .
[0229] The method provided in this embodiment forms a new embedding vector by splicing image and text aggregation features, and calculates the probability value of each forgery category through a fully connected layer processing combined with a softmax classification output layer. Finally, the forgery classification result of the information to be identified is flexibly determined based on the probability value and a preset threshold, thereby achieving efficient, accurate and flexible evaluation of the authenticity of the information.
[0230] In some embodiments, the recognition model is trained based on the following steps:
[0231] Obtain information about each sample;
[0232] Marking the forged information in each of the sample information to obtain a forged classification label for each of the sample information;
[0233] Constructing a training data set according to each of the sample information and the forged classification labels;
[0234] Iteratively training the initial neural network model according to the training data set to obtain the recognition model;
[0235] The forged information includes multiple items of forged information in which a target object in a sample image is replaced, forged information in which an emotional state of a target object in a sample image is replaced, forged information in which text content of a sample text is replaced, and forged information in which a paragraph, word or phrase used to represent an emotional state in a sample text is replaced.
[0236] Optionally, the training steps of the recognition model specifically include:
[0237] First, since the recognition module is used to identify whether an image-text pair (i.e., an image and its corresponding text description constitute an image-text pair) contains forged information, the forged information includes: 1. Forged information in the image, including the facial image of a specific person in the image being replaced with the facial image of another person; the emotional state of the facial image in the image being changed (for example, happiness is changed to anger). 2. Forged information in the text, including the text corresponding to the image being replaced with another text; some words / phrases in the text corresponding to the image being replaced with other words / phrases, especially such modification can cause the emotional state expressed by the text to change, thereby changing the tendency of the image-text information content. Based on the above two types of forged information, a variety of forged combinations can be constructed.
[0238] Therefore, in order to identify the above-mentioned forged information, the embodiment of the present application needs to collect a large amount of original information that has not been forged, modify the original information using existing image and text forgery technology, obtain a large amount of sample information, and mark the forgery of the sample information to obtain the corresponding forged classification label, thereby constructing a training data set. The specific implementation steps are as follows:
[0239] Step 1: collect and annotate to form an original training data set. The original training data set includes about 10,000 sample image-text pairs, each sample image contains at least 1 face image and at most 10 face images; if it is Chinese text, the corresponding text length is at least 5 characters and at most 50 characters. If it is English text, the corresponding text length is at least 5 words and at most 50 words. Each sample text in the above original training data set contains at least one predicate verb and one subject or object.
[0240] Step 2: In the above original training data set, about 20% of the sample images are not modified at all; the largest face in about 40% of the sample images is replaced with another person's face image; the expression of the largest face in the remaining sample images is tampered with (for example, smiling is changed to crying, etc.).
[0241] Step 3: In the above original training data set, about 20% of the sample texts are not modified; about 40% of the sample texts are replaced by another piece of text; and the words / phrases related to the emotional state in the remaining sample texts are tampered with (for example, words / phrases with positive emotions are changed to words / phrases with negative emotions, etc.).
[0242] Step 4: Engineering technicians manually mark the position coordinates of the modified face area, the mask of the face area, and the serial number of the modified word / phrase in the text to obtain the corresponding forged classification label.
[0243] Each sample in the training data set constructed above is a combination of an image-text pair, and the text is a text description of the image.
[0244] Secondly, an initialized neural network model is constructed. The structure of the initialized neural network model is similar to that of the recognition model, including an initialized image encoder, an initialized text encoder, an initialized cross-modal interaction module, an initialized multi-path projection module, an initialized multi-path aggregation module, and an initialized classification module.
[0245] Subsequently, each sample in the training data set is input into the initial neural network model, and after multi-modal and multi-path feature extraction and feature aggregation, the initial neural network model identifies and outputs the sample probability values under each preset forgery category. A sample here refers to a sample image-text sample pair, which can be specifically defined as pair(I,T). The sample image I is a three-channel color image, and the sample text T can be a description of the content in the image I.
[0246] Then, the image-text pair classification loss is constructed based on the sample probability value , the specific calculation formula is as follows:
[0247] ;
[0248] in, Represents the true value of the image-text pair classification, Indicates that there is no forged information in the image-text pair, that is, it contains neither forged image regions nor forged word units; Indicates that there is forged information in the image-text pair, that is, it contains forged image regions or forged words. , Represents the first The transposed vector of the weight parameters of the columns, Represents the dot product between vectors.
[0249] Subsequently, the initial neural network model is iteratively trained based on the classification loss to obtain the recognition model.
[0250] The method provided in this embodiment constructs a training data set that comprehensively covers various types of forgery by integrating fine marking of various forgery information, including replacement of target objects in sample images, tampering of emotional states, and modification of sample text content and its emotional expression parts, and then iteratively trains the initial neural network model, ultimately obtaining a comprehensive recognition model that can efficiently identify various complex forgery information.
[0251] In some embodiments, the iterative training of the initial neural network model according to the training data set to obtain the recognition model includes:
[0252] For the current iteration training, randomly adjusting the arrangement order of each sample information in the training data set;
[0253] Divide the adjusted training data set to obtain training data subsets corresponding to multiple batch trainings;
[0254] For each batch of training, the following steps are performed:
[0255] Based on the training data subset corresponding to the current batch training, the initial neural network model after the previous batch training is iteratively trained to obtain the initial neural network model after the current batch training;
[0256] The recognition model is constructed according to the initial neural network model after each batch of training.
[0257] Here, the total iterative training cycle (epoch) can be configured according to actual needs, such as being set to 20 cycles. One iterative training cycle means that all training samples are used once without repetition.
[0258] Optionally, in order to prevent the order of samples in the data set from affecting the performance of the model and to improve the generalization performance of the model, at the beginning of each iterative training cycle, all samples in the training data set are rearranged randomly, that is, the order is shuffled.
[0259] In each iterative training cycle, the shuffled training data set is divided into smaller sets to obtain multiple small batches of samples (i.e., subsets of training data). The number of samples in each small batch can be adaptively set according to the configuration of the training server, such as 128, 256, or 512.
[0260] It should be noted that, in special cases, within an iterative training cycle, the total number of samples in the training data set cannot be divided into integer batches, and some individual samples are not used (discarded directly) or are used multiple times (to complete the data, for example: adding several samples that have been used to the end of the data set). The impact of these individual cases on the training results can be ignored.
[0261] For the current iteration training, perform the following steps for each batch training:
[0262] First, read the small batch samples corresponding to the current batch training, input the small batch samples into the initial neural network model after the previous batch training, perform multi-modal and multi-path feature extraction and feature aggregation through the initial neural network model after the previous batch training, and then identify and output the corresponding sample probability values under each preset forgery category, and construct the classification loss of the current batch training according to the sample probability values , the specific calculation formula is as follows:
[0263] ;
[0264] Among them, the small batch samples of the current batch training contain B training samples (each training sample is an image-text pair). Represents the image-text classification loss for the i-th input sample.
[0265] Subsequently, the model parameters of the initial neural network model after the previous batch training are iteratively trained according to the classification loss of the current batch training.
[0266] After the current batch training is completed, the next batch training is iteratively performed based on the initial neural network model after the current batch training until all batch training in the current iterative training cycle is completed, which indicates that the training of the current iterative training cycle is completed.
[0267] After the current iteration training cycle is completed, the model training algorithm enters the next iteration training cycle. When the total number of training cycles reaches the total iteration training cycle, the model training ends.
[0268] At this time, the trained initial neural network model with the highest forged classification performance can be selected from the initial neural network models after training in each batch as the final recognition model.
[0269] The following describes the specific training steps of the recognition model using a specific example.
[0270] First, the input, output, and loss function of the initial neural network model are defined through the following steps:
[0271] Training input (initial parameter setting): from the training data set Read samples in; initial training cycle , total training cycle ; Stochastic gradient descent algorithm related parameters: The initial learning rate is , momentum coefficient , the weight decay coefficient is , in the 10th training cycle, the learning rate drops to , in the 15th training cycle, the learning rate drops to , in the 18th training cycle, the learning rate dropped to The number of samples in a mini-batch is ;
[0272] Training output: recognition model (The parameter of the model is φ);
[0273] Loss function: Loss function ;
[0274] Secondly, the initial neural network model is iteratively trained by performing the following steps:
[0275] Step S1, initialization: Specifically, the image encoding module The parameters in are initialized with the parameters of ViT-L / 16 pre-trained on the ImageNet-21k dataset; the text encoder The parameters are used Initialize the parameters trained on the general language understanding evaluation benchmark set; use the random initialization method to initialize the learnable parameters of the initial neural network model.
[0276] Step S2, when the total number of iterative training Maximum number of iterations , do the following:
[0277] Step S21, ;
[0278] Step S22: training data set The order of samples in is randomly shuffled;
[0279] Step S23, from the training data set Pick a small batch of B training samples ( );
[0280] Step S24, based on the above training samples and parameter settings, calculate the following loss value: , and then based on the loss value, the model parameters are updated in reverse using the stochastic gradient descent algorithm:
[0281] ;
[0282] in, represents the image-text classification loss of the i-th input sample;
[0283] Step S25, repeat steps S23 and S24 until the training data set of the samples were fully utilized;
[0284] Step S26, ending the current iterative training cycle.
[0285] Step S3, end the training, and output the recognition network model based on the initial neural network model after each batch of training .
[0286] In summary, the method provided in this embodiment, through the random arrangement and batch processing of sample information in the training data set, combined with the iterative training of the initial neural network model in each batch, finally constructs a recognition model based on multi-batch optimization results, thereby effectively improving the generalization ability and training efficiency of the model.
[0287] The information identification device provided by the present invention is described below. The information identification device described below and the information identification method described above can be referenced to each other.
[0288] Figure 6 A schematic diagram of the structure of the information identification device provided by the present invention; Figure 6 As shown, the device comprises:
[0289] The first processing unit 610 is used to input the image to be recognized and the text to be recognized in the information to be recognized into the feature extraction module of the recognition model to obtain the multi-path target image embedding features of the image to be recognized and the multi-path target text embedding features of the text to be recognized;
[0290] The second processing unit 620 is used to input the target image embedding features of the multi-path and the target text embedding features of the multi-path into the multi-path aggregation module of the recognition model to obtain the image aggregation features of the image to be recognized and the text aggregation features of the text to be recognized;
[0291] The classification unit 630 is used to input the image aggregation feature and the text aggregation feature into the classification module of the recognition model to obtain the forgery classification result of the information to be identified;
[0292] The recognition model is obtained by iterative training based on each sample information and the forged classification labels of each sample information.
[0293] The device provided in this embodiment uses a feature extraction module to perform multi-modal and multi-path feature extraction on the image to be identified and the text to be identified in the information to be identified, so as to capture the key features of the image and text associated with the forged information at different levels and angles, and form multi-path target image embedding features and target text embedding features; then, a multi-path aggregation module is used to fuse the features of the multi-path target image embedding features and the target text embedding features, so as to make full use of the inherent connection and complementarity between the image and the text, and form more comprehensive and integrated image aggregation features and text aggregation features; finally, a classification module is used to capture multi-modal and multi-perspective forgery traces based on the image aggregation features and the text aggregation features, so as to obtain the information forgery classification results more comprehensively and accurately.
[0294] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the information recognition method, which includes: inputting the image to be recognized and the text to be recognized in the information to be recognized into the feature extraction module of the recognition model to obtain the multi-path target image embedding features of the image to be recognized and the multi-path target text embedding features of the text to be recognized; inputting the multi-path target image embedding features and the multi-path target text embedding features into the multi-path aggregation module of the recognition model to obtain the image aggregation features of the image to be recognized and the text aggregation features of the text to be recognized; inputting the image aggregation features and the text aggregation features into the classification module of the recognition model to obtain the forgery classification result of the information to be recognized; wherein the recognition model is obtained by iterative training based on each sample information and the forgery classification label of each sample information.
[0295] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0296] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the information recognition method provided by the above-mentioned methods, which includes: inputting the image to be recognized and the text to be recognized in the information to be recognized into the feature extraction module of the recognition model to obtain the multi-path target image embedding features of the image to be recognized and the multi-path target text embedding features of the text to be recognized; inputting the multi-path target image embedding features and the multi-path target text embedding features into the multi-path aggregation module of the recognition model to obtain the image aggregation features of the image to be recognized and the text aggregation features of the text to be recognized; inputting the image aggregation features and the text aggregation features into the classification module of the recognition model to obtain the forged classification result of the information to be recognized; wherein, the recognition model is obtained by iterative training based on each sample information and the forged classification label of each sample information.
[0297] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the information recognition method provided by the above-mentioned methods, the method comprising: inputting the image to be recognized and the text to be recognized in the information to be recognized into a feature extraction module of a recognition model to obtain multi-path target image embedding features of the image to be recognized and multi-path target text embedding features of the text to be recognized; inputting the multi-path target image embedding features and the multi-path target text embedding features into a multi-path aggregation module of the recognition model to obtain image aggregation features of the image to be recognized and text aggregation features of the text to be recognized; inputting the image aggregation features and the text aggregation features into a classification module of the recognition model to obtain a forged classification result of the information to be recognized; wherein, the recognition model is obtained by iterative training based on each sample information and the forged classification label of each sample information.
[0298] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0299] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0300] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An information identification method, characterized in that: include: Inputting the image to be recognized and the text to be recognized in the information to be recognized into the feature extraction module of the recognition model to obtain the multi-path target image embedding features of the image to be recognized and the multi-path target text embedding features of the text to be recognized; Inputting the target image embedding features of the multi-path and the target text embedding features of the multi-path into the multi-path aggregation module of the recognition model to obtain the image aggregation features of the image to be recognized and the text aggregation features of the text to be recognized; Inputting the image aggregation features and the text aggregation features into the classification module of the recognition model to obtain a forgery classification result of the information to be identified; The recognition model is obtained by iterative training based on each sample information and the forged classification labels of each sample information; The feature extraction module includes an encoding module, a cross-modal interaction module and a multi-path projection module; The step of inputting the image to be recognized and the text to be recognized in the information to be recognized into a feature extraction module of a recognition model to obtain a multi-path target image embedding feature of the image to be recognized and a multi-path target text embedding feature of the text to be recognized, comprises: The image to be recognized and the text to be recognized are input into the encoding module, and the encoding module performs image embedding feature extraction based on the image to be recognized to obtain a first image embedding feature of the image to be recognized, and performs text embedding feature extraction based on the text to be recognized to obtain a first text embedding feature of the text to be recognized; Inputting the first image embedding feature and the first text embedding feature into the cross-modal interaction module, the cross-modal interaction module extracting image features related to text content according to the first image embedding feature and the first text embedding feature to obtain a second image embedding feature of the image to be recognized, and extracting text features related to image content according to the first image embedding feature and the first text embedding feature to obtain a second text embedding feature of the text to be recognized; The second image embedding feature and the second text embedding feature are input into the multi-path projection module, and the multi-path projection module performs multi-path image feature extraction based on the second image embedding feature to obtain the multi-path target image embedding feature, and performs multi-path text feature extraction based on the second text embedding feature to obtain the multi-path target text embedding feature.
2. The information identification method according to claim 1, characterized in that: The multi-path projection module includes a plurality of groups of first cross attention units and a plurality of groups of first linear projection units; The target image embedding features of the multipath are determined based on the following steps: Input the first learnable embedding feature corresponding to each group of the first cross attention units as a query parameter and the second image embedding feature as a key parameter and a value parameter to each group of the first cross attention units, and obtain the target image embedding feature of the path corresponding to each group of the first cross attention units; The first learnable vector and the second image embedding feature corresponding to each group of the first linear projection units are input into each group of the first linear projection units as input information to obtain the target image embedding feature of the path corresponding to each group of the first linear projection units.
3. The information identification method according to claim 2, characterized in that: The image aggregation feature is obtained by the multi-path aggregation module based on the following steps: According to the first learnable embedding features corresponding to each group of the first cross attention units and the features of each dimension in the second image embedding features, obtaining the first coefficients of the features of each dimension under the paths corresponding to the first cross attention units in each group; According to each dimensional vector in the first learnable vector corresponding to each group of the first linear projection units, obtaining a second coefficient of each dimensional feature under a path corresponding to each first linear projection unit; The aggregation coefficient of the feature in each dimension is obtained by adding the sum of the first coefficients of the feature in each dimension under the paths corresponding to the multiple groups of the first cross attention units and the sum of the second coefficients of the feature in each dimension under the paths corresponding to the multiple groups of the first linear projection units; The aggregation coefficient of each feature is weightedly added to the features of each dimension to obtain the image aggregation feature.
4. The information identification method according to claim 1, characterized in that: The multi-path projection module includes multiple groups of second cross attention units and multiple groups of second linear projection units; The target text embedding features of the multi-path are determined based on the following steps: Input the second learnable embedding feature corresponding to each group of the second cross attention units as a query parameter and the second text embedding feature as a key parameter and a value parameter to each group of the second cross attention units to obtain the target text embedding feature of the path corresponding to each group of the second cross attention units; The second learnable vector and the second text embedding feature corresponding to each group of the second linear projection units are input into each group of the second linear projection units as input information to obtain the target text embedding feature of the path corresponding to each group of the second linear projection units.
5. The information identification method according to claim 1, characterized in that: The encoding module includes an image encoder; The first image embedding feature is determined based on the following steps: Dividing the image to be recognized into a plurality of image blocks; Inputting each of the image blocks into the image encoder, and extracting encoding features of each of the image blocks through the image encoder to obtain the first image embedding features; The first image embedding features include multiple image block embedding features and image classification embedding features.
6. The information identification method according to claim 1, characterized in that: The encoding module includes a text encoder; The first text embedding feature is determined based on the following steps: Input the text to be recognized into the text encoder, divide the text to be recognized into multiple word units through the text encoder, obtain the embedding vector to be processed corresponding to each word unit in the word unit embedding vector table, perform forward processing on the embedding vectors to be processed corresponding to the multiple word units, and obtain the first text embedding feature; The first text embedding features include multiple word embedding features and text classification embedding features.
7. The information identification method according to claim 1, characterized in that: The cross-modal interaction module includes a first cross-attention unit and a second cross-attention unit; The step of inputting the first image embedding feature and the first text embedding feature into the cross-modal interaction module to obtain a second image embedding feature of the image to be recognized and a second text embedding feature of the text to be recognized includes: Input the first image embedding feature as a query parameter and the first text embedding feature as a key parameter and a value parameter into the first cross-attention unit to obtain the second image embedding feature; The first text embedding feature is used as a query parameter and the first image embedding feature is used as a key parameter and a value parameter to input into the second cross attention unit to obtain the second text embedding feature.
8. The information identification method according to any one of claims 1 to 7, characterized in that: The classification module includes a fully connected layer and a classification output layer; The step of inputting the image aggregation features and the text aggregation features into the classification module of the recognition model to obtain the forgery classification result of the information to be identified includes: Inputting the image aggregation features and the text aggregation features into the fully connected layer to obtain the hidden variable values of the information to be identified under each preset forgery category; Inputting the latent variable value into the classification output layer to obtain the probability value of the information to be identified under each of the preset forgery categories; The forgery classification result is obtained according to the probability value.
9. The information identification method according to any one of claims 1 to 7, characterized in that: The recognition model is trained based on the following steps: Obtain information about each sample; Marking the forged information in each of the sample information to obtain a forged classification label for each of the sample information; Constructing a training data set according to each of the sample information and the forged classification labels; Iteratively training the initial neural network model according to the training data set to obtain the recognition model; The forged information includes multiple items of forged information in which a target object in a sample image is replaced, forged information in which an emotional state of a target object in a sample image is replaced, forged information in which text content of a sample text is replaced, and forged information in which a paragraph, word or phrase used to represent an emotional state in a sample text is replaced.
10. The information identification method according to claim 9, characterized in that: The iterative training of the initial neural network model according to the training data set to obtain the recognition model includes: For the current iteration training, randomly adjusting the arrangement order of each sample information in the training data set; Divide the adjusted training data set to obtain training data subsets corresponding to multiple batch trainings; For each batch of training, the following steps are performed: Based on the training data subset corresponding to the current batch training, the initial neural network model after the previous batch training is iteratively trained to obtain the initial neural network model after the current batch training; The recognition model is constructed according to the initial neural network model after each batch of training.
11. The information identification method according to any one of claims 1 to 7, characterized in that: The information to be identified is obtained based on the following steps: Get original information; Preprocessing the original image in the original information to obtain the information to be identified; The preprocessing includes scaling the image pixel size of the original image to a preset pixel size.
12. An information recognition device, characterized in that: include: A first processing unit is used to input the image to be recognized and the text to be recognized in the information to be recognized into a feature extraction module of a recognition model to obtain a multi-path target image embedding feature of the image to be recognized and a multi-path target text embedding feature of the text to be recognized; A second processing unit is used to input the target image embedding features of the multi-path and the target text embedding features of the multi-path into the multi-path aggregation module of the recognition model to obtain the image aggregation features of the image to be recognized and the text aggregation features of the text to be recognized; A classification unit, used for inputting the image aggregation feature and the text aggregation feature into the classification module of the recognition model to obtain a forgery classification result of the information to be identified; The recognition model is obtained by iterative training based on each sample information and the forged classification labels of each sample information; The feature extraction module includes an encoding module, a cross-modal interaction module and a multi-path projection module; The first processing unit is specifically configured to: The image to be recognized and the text to be recognized are input into the encoding module, and the encoding module performs image embedding feature extraction based on the image to be recognized to obtain a first image embedding feature of the image to be recognized, and performs text embedding feature extraction based on the text to be recognized to obtain a first text embedding feature of the text to be recognized; Inputting the first image embedding feature and the first text embedding feature into the cross-modal interaction module, the cross-modal interaction module extracting image features related to text content according to the first image embedding feature and the first text embedding feature to obtain a second image embedding feature of the image to be recognized, and extracting text features related to image content according to the first image embedding feature and the first text embedding feature to obtain a second text embedding feature of the text to be recognized; The second image embedding feature and the second text embedding feature are input into the multi-path projection module, and the multi-path projection module performs multi-path image feature extraction based on the second image embedding feature to obtain the multi-path target image embedding feature, and performs multi-path text feature extraction based on the second text embedding feature to obtain the multi-path target text embedding feature.
13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the information identification method according to any one of claims 1 to 11 is implemented.
14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the information identification method according to any one of claims 1 to 11 is implemented.
