Multi-mode image-text depth forgery detection and positioning method based on Mangban module

Abstract

The invention discloses a detection and positioning method for multi-modal image-text deep counterfeiting based on a Mangbar module. The method comprises a multi-modal feature alignment module, a multi-modal feature fusion module and a counterfeiting area detection and positioning module. The method comprises the following steps: firstly, searching a corresponding relationship among different data modals of the same instance through a multi-modal feature alignment module, then, fully fusing features in a multi-modal feature fusion module, learning feature representations among the features, and finally, inputting the fused features into a forged region detection and positioning module, so as to detect the forged region. And specific information of detection, area positioning and forgery methods is given. The method can effectively detect the multi-mode deep counterfeit information, can give a counterfeit area and a counterfeit mode, can prevent the transmission of false information, and guarantees the reliability of an information source.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a method for detecting and locating deep forgeries of multimodal data. Background Art

[0002] Deepfakes rely on deep generation algorithms to implement high-quality tampering and forgery of data such as facial images, audio, text and video. In the context of the increasingly close connection between generative artificial intelligence and social life, the malicious use of deepfakes has caused hidden dangers to the privacy and security of society and has attracted widespread social attention.

[0003] In the early stages of the development of deep fake technology, most of the research and application directions were aimed at single-modality image or video data. In addition, early detection technologies were mainly based on convolutional neural networks. Convolutional neural networks can effectively capture image or video artifacts introduced during the generation process, learn the characteristics of forgery traces from multiple dimensions such as spatial, frequency, and time domains, and thus extract possible forgery areas in images or videos for true / false discrimination, thereby achieving the purpose of authenticity identification of facial images and videos.

[0004] With the advancement of technology, deep fakes have developed into multimodal fake technologies including video fakes, sound fakes, text fakes, and micro-expression synthesis. Unlike traditional deep fakes, multimodal deep fake technology can use the complex associations and interactions between information in various modalities to generate very realistic false information in various data modalities. Therefore, in the face of the greater difficulty in detecting false information generated by multimodal deep fakes, it is necessary to design a more robust and generalized multimodal deep fake detection method.

[0005] Existing deep fake detection methods are mainly oriented to data modalities such as images or videos. Such detection methods can effectively prevent deep fake attacks in a single data modality such as images or videos, but do not consider multimodal deep fake scenarios. In addition, these methods can only provide binary discrimination of the authenticity of the input data, and cannot provide richer detection information such as positioning and forgery methods. Therefore, although existing deep fake detection methods have developed rapidly, there are still deficiencies in the detection of multimodal data.

[0006] In this work, a method for multimodal image and text deep fake detection and localization based on the Mamba module is proposed. This method can effectively detect multimodal forgeries, locate forgery traces, and provide specific forgery methods.

[0007] Glossary:

[0008] Modality: In the field of artificial intelligence, modality is a form of data that can be understood and processed by computers. These data forms correspond to different ways in which humans perceive the world, come from different human senses, and have multiple ways of interaction.

[0009] Artifacts: Artifacts refer to images of various shapes that appear on the image even though the object being photographed does not exist. After image processing, especially in composite images, they appear as unnatural traces, areas, flaws, etc. that can be seen as artificial processing.

[0010] Modal feature alignment: Modal feature alignment refers to finding the correspondence between different data modal features from the same instance.

[0011] Modal feature fusion: Modal feature fusion refers to the fusion of features of different modal data to achieve the effect of feature complementarity and improve the robustness and generalization of the model.

[0012] Cross-attention mechanism: The cross-attention mechanism refers to a mechanism that establishes mutual correlations between two or more different modal features and calculates the importance of modal features to each other.

[0013] Mamba module: Mamba is a structured state-space sequence model. The Mamba module is a specific implementation and encapsulation of Mamba. The Mamba module can efficiently process complex long sequence data. Summary of the invention

[0014] To solve the above problems, the present invention provides a multimodal image and text deep fake detection and positioning method based on the Mamba module, which not only effectively detects multimodal deep fake information, but also can give the fake area and fake method. Therefore, the use of the present invention can prevent the spread of false information and ensure the reliability of the information source.

[0015] The purpose of the present invention is achieved through the following technical solutions:

[0016] Step 1: Read a batch of data. Assume that there are k images and k texts in the data. Use the ViT model to encode the features of the image. Encode the image as v = {v cls ,v1,...,v N}, using the BERT model to encode the features of the text, encoding the text as t = {t cls ,t1,...,t N} form;

[0017] Step 2: Perform feature alignment on the extracted image features and text features to calculate the similarity s(I,T + ), the similarity between the image and the non-corresponding text s(I,T- ) and the similarity between text and image s(T + ,I),s(I,T - ), image-to-image similarity s(I,I + ), s(I,I - ), text-to-text similarity s(T,T + ),s(T,T - ), the image-text alignment loss function is

[0018]

[0019] Step 3: Define the image-text feature fusion module Aggregator, which takes image feature v and text feature t as input and outputs the fused feature. The image feature is

[0020] v′=Aggregator(v,t)={v′ cls ,v′1,...,v′ N}

[0021] The text features are

[0022] t′=Aggregator(v,t)={t′ cls ,t′1,...,t′ N};

[0023] Step 4: Define the true and false classifier C using a multi-layer perceptron b , the input is the image feature v′ and the text feature t′, and the output is the prediction of the authenticity of the data D b , using the predicted value D b and the true value y b Calculate the true and false binary classification loss, the loss function is

[0024] L BIC =E (I,T)～P CrossEntropy(D b ,y b );

[0025] Step 5: Define the image feature aggregator, whose input is {v′1,...,v′ in the image feature v′ N} part, the output is the aggregated feature v′ agg , use a multi-layer perceptron to define the image forgery locator G v , the input is the aggregated feature v′ agg , the output is the prediction D of the specific location of the forged area in the image v , calculated as D v =G v (v′agg ), using the predicted value D v and the true value y box Calculate the image positioning loss, the loss function is

[0026] L IMG =E (I,T)～P [||Sigmoid(D v )-y box ||+I CIoU (Sigmoid(D v )-y box )];

[0027] Step 6: Define the forgery method classifier C using a multi-layer perceptron m , the input is {v′1,...,v′ in the fused image feature v′ N} part and {t′1,...,t′ in the text feature t′ N} part, the output is the prediction D of the specific tampering method of tampering data m , using the predicted value D m and the true value y m Calculate the classification loss of the tampering method, the loss function is

[0028] L MLC =E (I,T)～P CrossEntropy(D m ,y m );

[0029] Step 7: Define the text forgery locator G using a multi-layer perceptron t , the input is {t′1,...,t′ in the fused text features N}, the output is the prediction of the specific location of the text forgery area D t , using the predicted value D t and the true value y t Calculate the text localization loss, the loss function is

[0030] L TMG =E (I,T)～P CrossEntropy(D t ,y t );

[0031] Step 8: Calculate the model loss function L, which is calculated as L = λ1L ITA +λ2L BIC +λ3L IMG +λ4L MLC +λ5L TMG , where λ1, λ2, λ3, λ4, and λ5 are hyperparameters that control the proportion of each loss;

[0032] Repeat steps 1 to 8 until the specified training round is reached to obtain the final global model.

[0033] In this scheme, the image and text data are first read, and the image and text features are encoded using the ViT model and the BERT model respectively. In each iteration of training, the method first performs data alignment in the multimodal semantic feature alignment module, calculates the similarity between the image and the text, and calculates the alignment loss function based on the obtained similarity. The alignment loss function guides the model to find the correspondence between different data modalities of the same instance. Then, in the multimodal feature fusion module, the features are fully fused using the Mamba module and the cross-attention mechanism. The fusion of features allows the image and text features to learn the feature representations of each other. Finally, the fused features are input into the forged region detection and positioning module, and the predictions of detection, region positioning and forgery methods are given. The loss function is calculated by the gap between the predicted value and the true value, and the model is iteratively updated according to the loss function. The above steps are iterated several times to obtain the final global model.

[0034] Preferably, the step 2 is specifically to calculate the alignment loss of image to text and text to image by calculating the similarity s(I, T + ), the similarity d(T,I + ), the calculation formula is

[0035] s(I,T + ) = g v (I) T ·g t (T),

[0036] s(T,I + ) = g t (T) T ·g v (T).

[0037] The similarity s(I,T) between the image and the non-corresponding text, and between the text and the non-corresponding image is calculated in the same way as above. - ),s(T,I - ), use the calculated similarity to calculate the alignment loss of image to text and text to image, the calculation formula is

[0038]

[0039] Where τ is a hyperparameter that controls the alignment granularity. The image-to-image and text-to-text loss L is calculated in the same way as above. v2v and L t2t , the final image-text alignment loss is

[0040]

[0041] Preferably, the step 3 is specifically as follows: first, the Mamba module is used to filter out the important components in the original features, v m =Mamba(v),t m =Mamba(t), then v m and t m Input the cross attention module for feature fusion, the calculation formula is

[0042]

[0043] The fused image v′ m and text feature t′ m Continue to repeat the above operation multiple times to obtain the final fusion results v′ and t′.

[0044] Preferably, the step 5 is specifically to transform {v′1,...,v′ N} Part of the input feature aggregator Aggregator, the formula is

[0045]

[0046] v′ agg =Aggregator(v′ cls ,({v′1,...,v′ N}),({v′1,...,v′ N})).

[0047] The beneficial effect of the present invention is that the present invention can effectively detect multimodal deep fake information, and at the same time can provide the fake area and the fake method. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a training flow chart of the present invention;

[0049] Figure 2 This is a graph showing the change in accuracy of the true and false binary classification over the 50 training rounds as the training process progresses;

[0050] Figure 3 This is a graph showing how the accuracy of locating forged image regions changes over the training process during 50 training rounds;

[0051] Figure 4 This is a graph showing how the accuracy of localizing forged text areas changes over the training process in 50 training rounds;

[0052] Figure 5This is a graph showing how the accuracy of forgery classification changes over the training process in 50 training rounds. DETAILED DESCRIPTION

[0053] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0054] Example:

[0055] A multimodal image and text deep fake detection and positioning method based on the Mamba module, such as Figure 1 As shown, first read the image and text data, and use the ViT model and BERT model to encode the image and text features respectively. In each iteration of training, the method first inputs the image and text features into the multimodal feature alignment module, and performs data alignment on the multimodal data through a contrastive learning-based method. In the module, the similarity between the image and the text is first calculated, and then the alignment loss function is calculated based on the obtained similarity. In the multimodal feature fusion module, the features are fully fused using the Mamba module and the cross-attention mechanism, so that the image and text features learn the feature representations of each other. Finally, the fused features are input into the forged area detection and positioning module, and the predictions of detection, area positioning and forgery methods are given. The loss function is calculated by the gap between the predicted value and the true value, and the model is iteratively updated according to the loss function. The above steps are iterated several times to obtain the final global model. Specifically, the following steps are included:

[0056] Step 1: Read a batch of data. Assume that there are k images and k texts in the data. Use the ViT model to encode the features of the image. Encode the image as v = {v cls ,v1,...,v N}, using the BERT model to encode the features of the text, encoding the text as t = {t cls ,t1,...,t N} form;

[0057] Step 2: Perform feature alignment on the extracted image features and text features to calculate the similarity s(I,T + ), the similarity between the image and the non-corresponding text s(I,T - ) and the similarity between text and image s(T + ,I),s(I,T - ), image-to-image similarity s(I,I + ), s(I,I - ), text-to-text similarity s(T,T + ),s(T,T - ), the image-text alignment loss function is

[0058]

[0059] Step 3: Define the image-text feature fusion module Aggregator, which takes image feature v and text feature t as input and outputs the fused feature. The image feature is

[0060] v′=Aggregator(v,t)={v′ cls ,v′1,...,v′ N},

[0061] The text features are

[0062] t′=Aggregator(v,t)={t′ cls ,t′1,...,t′ N};

[0063] Step 4: Define the true and false classifier C using a multi-layer perceptron b , the input is the image feature v′ and the text feature t′, and the output is the prediction of the authenticity of the data D b , using the predicted value D b and the true value y b Calculate the true and false binary classification loss, the loss function is

[0064] L BIC =E (I,T)～P CrossEntropy(D b ,y b );

[0065] Step 5: Define the image feature aggregator, whose input is {v′1,...,v′ in the image feature v′ N} part, the output is the aggregated feature v′ agg , use a multi-layer perceptron to define the image forgery locator G v , the input is the aggregated feature v′ agg , the output is the prediction D of the specific location of the forged area in the image v , calculated as D v =G v (v′ agg ), using the predicted value D v and the true value y box Calculate the image positioning loss, the loss function is

[0066] L IMG =E (I,T)～P [||Sigmoid(D v )-y box ||+L CIoU (Sigmoid(D v )-y box )];

[0067] Step 6: Define the forgery method classifier C using a multi-layer perceptron m , the input is {v′1,...,v′ in the fused image feature v′ N} part and {t′1,...,t′ in the text feature t′ N} part, the output is the prediction D of the specific tampering method of tampering data m , using the predicted value D m and the true value y m Calculate the classification loss of the tampering method, the loss function is

[0068] L MLC =E (I,T)～P CrossEntropy(D m ,y m );

[0069] Step 7: Define the text forgery locator G using a multi-layer perceptron t , the input is {t′1,...,t′ in the fused text features N}, the output is the prediction of the specific location of the text forgery area D t , using the predicted value D t and the true value y t Calculate the text localization loss, the loss function is

[0070] L TMG =E (I,T)～P CrossEntropy(D t ,y t );

[0071] Step 8: Calculate the model loss function L, which is calculated as L = λ1L ITA +λ2L BIC +λ3L IMG +λ4L MLC +λ5L TMG , where λ1, λ2, λ3, λ4, and λ5 are hyperparameters that control the proportion of each loss;

[0072] Repeat steps 1 to 8 until the specified training round is reached to obtain the final global model.

[0073] The design of this solution can effectively detect multi-modal deep fake information, and can also provide the fake area and the fake method. Compared with the existing deep fake detection methods, this solution can adapt to multiple data modalities and provide richer fake information.

[0074] The above-described embodiment is only a preferred solution of the present invention and does not limit the present invention in any form. There are other variations and modifications without exceeding the technical solution described in the claims.

Claims

1. A method for multimodal image and text deep fake detection and positioning tasks based on the Mamba module, characterized in that: It has a multimodal feature alignment module, a multimodal feature fusion module and a forged area detection and positioning module, and specifically includes the following steps: Step 1: Read a batch of data. Assume that there are k images and k texts in the data. Use the ViT model to encode the features of the image. Encode the image as v = {v cls ,v1,...,v N }, using the BERT model to encode the features of the text, encoding the text as t = {t cls ,t1,...,t N } form; Step 2: Perform feature alignment on the extracted image features and text features to calculate the similarity s(I,T + ), the similarity between the image and the non-corresponding text s(I,T - ) and the similarity between text and image s(T + ,I),s(I,T - ), image-to-image similarity s(I,I + ), s(I,I - ), text-to-text similarity s(T,T + ),s(T,T - ), calculate the image text alignment loss function Step 3: Define the image-text feature fusion module Aggregator, which takes image feature v and text feature t as input and outputs the fused feature. The image feature is v′=Aggregator(v,t)={v′ cls ,v′1,...,v′ N } The text features are t′=Aggregator(v,t)={t′ cls ,t′1,...,t′ N }; Step 4: Define the true and false classifier C using a multi-layer perceptron b , the input is the image feature v′ and the text feature t′, and the output is the prediction of the authenticity of the data D b , using the predicted value D b and the true value y b Calculate the true and false binary classification loss, the loss function is L BIC =E (I,T)～P CrossEntropy(D b ,y b ); Step 5: Define the image feature aggregator, whose input is {v′1,...,v′ in the image feature v′ N } part, the output is the aggregated feature v′ agg , use a multi-layer perceptron to define the image forgery locator G v , the input is the aggregated feature v′ agg , the output is the prediction D of the specific location of the forged area in the image v , calculated as D v =G v (v′ agg ), using the predicted value D v and the true value y box Calculate the image positioning loss, the loss function is L IMG =E (I,T)～P [||Sigmoid(D v )-y box ||+L CIoU (Sigmoid(D v )-y box )]; Step 6: Define the forgery method classifier C using a multi-layer perceptron m , the input is {v′1,...,v′ in the fused image feature v′ N } part and {t′1,...,t′ in the text feature t′ N } part, the output is the prediction D of the specific tampering method of tampering data m , using the predicted value D m and the true value y m Calculate the classification loss of the tampering method, the loss function is L MIC =E (I,T)～P CrossEntropy(D m ,y m ); Step 7: Define the text forgery locator G using a multi-layer perceptron t , the input is {t′1,...,t′ in the fused text features N }, the output is the prediction of the specific location of the text forgery area D t , using the predicted value D t and the true value y t Calculate the text localization loss, the loss function is L TMG =E (I,T)～P CrossEntropy(D t ,y t ); Step 8: Calculate the model loss function L, which is calculated as L = λ1L ITA +λ2L BIC +λ3L IMG +λ4L MLC +λ5L TMG , where λ1, λ2, λ3, λ4, and λ5 are hyperparameters that control the proportion of each loss; Repeat steps 1 to 8 until the specified training round is reached to obtain the final global model.

2. A method for multimodal deep fake detection and localization tasks according to claim 1, characterized in that: Specifically, step 2 is to calculate the alignment loss of image to text and text to image by calculating the similarity s(I, T + ), the similarity s(T,I + ), the calculation formula is s(I,T + )=g v (I) T ·g t (T); s(T,I + )=g t (T) T ·g v (T), The similarity s(I,T) between the image and the non-corresponding text, and between the text and the non-corresponding image is calculated in the same way as above. - ),s(T,I - ), use the calculated similarity to calculate the alignment loss of image to text and text to image, the calculation formula is Where τ is a hyperparameter that controls the alignment granularity. The image-to-image and text-to-text loss L is calculated in the same way as above. v2v and L t2t , the final image-text alignment loss is 3. A method for multimodal deep fake detection and localization tasks according to claim 1, characterized in that: Specifically, the step 3 is to first use the Mamba module to filter out the important components in the original features, v m =Mamba(v),t m =Mamba(t), then v m and t m Input the cross attention module for feature fusion, the calculation formula is The fused image v′ m and text feature t′ m Continue to repeat the above operation multiple times to obtain the final fusion results v′ and t′.

4. A method for multimodal deep fake detection and localization tasks according to claim 1, characterized in that: The step 5 is specifically to transform {v′1,...,v′ N } Part of the input feature aggregator Aggregator, the formula is v′ agg =Aggregator(v′ cls ,({v′1,...,v′ N }),({v′1,...,v′ N }))。

Images