A multimodal image and text deepfake detection and location method based on the Mamba module

Abstract

The present invention discloses a method for detecting and locating multimodal graphic deep fakes based on a Mamba module, comprising a multimodal feature alignment module, a multimodal feature fusion module, and a forged area detection and positioning module. The method first searches for the correspondence between different data modalities of the same instance through a multimodal feature alignment module, then fully fuses the features within a multimodal feature fusion module, learns the feature representations of each other, and finally inputs the fused features into a forged area detection and positioning module, giving specific information on detection, area positioning, and forgery methods. The present invention can effectively detect multimodal deep fake information, and at the same time can give forged areas and forgery methods. The use of the present invention can prevent the spread of false information and ensure the reliability of the information source.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a method for detecting and locating deep forgeries in multimodal data. Background Art

[0002] Deepfakes rely on deep generative algorithms to create high-quality tampering and fabrication of facial images, audio, text, and video data. In an era where generative artificial intelligence is increasingly intertwined with social life, the malicious use of deepfakes has posed privacy and security risks to society and has garnered widespread public attention.

[0003] In the early stages of deepfake technology development, research and application focused primarily on single-modality image or video data. Furthermore, early detection techniques were primarily based on convolutional neural networks. Convolutional neural networks can effectively capture image or video artifacts introduced during the generation process, learning features related to forgery traces from multiple dimensions, such as spatial, frequency, and temporal domains. This allows them to extract potentially forged areas in images or videos for authenticity verification, ultimately achieving the goal of verifying the authenticity of facial images and videos.

[0004] With technological advancements, deepfakes have evolved into multimodal forgeries, encompassing video, audio, text, and micro-expression synthesis. Unlike traditional deepfakes, multimodal deepfakes leverage the complex connections and interactions between information in various modalities to generate highly realistic false information across all data modalities. Therefore, given the increased difficulty in detecting false information generated by multimodal deepfakes, it is necessary to design robust and generalizable multimodal deepfake detection methods.

[0005] Existing deepfake detection methods primarily target data modalities such as images or videos. These methods effectively protect against deepfake attacks targeting a single data modality, such as images or videos, but fail to consider multimodal deepfake scenarios. Furthermore, these methods only provide a binary judgment of the authenticity of the input data and fail to provide richer detection information, such as location and method of forgery. Therefore, despite the rapid development of existing deepfake detection methods, they still face challenges in detecting multimodal data.

[0006] In this work, a method for detecting and locating multimodal deepfakes based on the Mamba module is proposed. This method can effectively detect multimodal forgeries, locate forgery traces, and provide specific forgery methods.

[0007] Glossary:

[0008] Modality: In the field of artificial intelligence, modality is a form of data that can be understood and processed by computers. These data forms correspond to different ways humans perceive the world, come from different human senses, and have multiple modes of interaction.

[0009] Artifacts: Artifacts are images of objects that do not exist in the original image. After image processing, especially in composite images, they appear as unnatural traces, areas, and flaws that can be seen as artificial processing.

[0010] Modal feature alignment: Modal feature alignment refers to finding the correspondence between different data modal features from the same instance.

[0011] Modal feature fusion: Modal feature fusion refers to the fusion of features of different modal data to achieve the effect of feature complementarity and improve the robustness and generalization of the model.

[0012] Cross-attention mechanism: The cross-attention mechanism refers to a mechanism that establishes mutual correlation between two or more different modal features and calculates the importance of each modal feature.

[0013] Mamba module: Mamba is a structured state-space sequence model. The Mamba module is a specific implementation and encapsulation of Mamba. Using the Mamba module can efficiently process complex long sequence data. Summary of the Invention

[0014] To address these issues, the present invention provides a multimodal deepfake detection and location method for images and text based on the Mamba module. This method not only effectively detects multimodal deepfakes but also identifies the forged areas and methods. Therefore, the use of this method can prevent the spread of false information and ensure the reliability of information sources.

[0015] The purpose of the present invention is achieved through the following technical solutions:

[0016] Step 1: Read a batch of data. Assume that there are k images and k texts in the data. Use the ViT model to encode the features of the image. Encode the image as v = {v cls ,v1,...,v N}, use the BERT model to encode the features of the text, and encode the text into t = {t cls ,t1,...,t N} form;

[0017] Step 2: Perform feature alignment on the extracted image features and text features to calculate the similarity s(I, T + ), the similarity between the image and the non-corresponding text s(I,T- ) and the similarity between text and image s(T + ,I),s(I,T - ), image-to-image similarity s(I,I + ), s(I,I - ), text-to-text similarity s(T,T + ),s(T,T - ), the image-text alignment loss function is

[0018]

[0019] Step 3: Define the image-text feature fusion module Aggregator, which takes image feature v and text feature t as input and outputs the fused feature. The image feature is

[0020] v′=Aggregator(v,t)={v′ cls ,v′1,...,v′ N}

[0021] The text features are

[0022] t′=Aggregator(v,t)={t′ cls ,t′1,...,t′ N};

[0023] Step 4: Use a multi-layer perceptron to define the true and false classifier C b , the input is the image feature v′ and the text feature t′, and the output is the prediction of the authenticity of the data D b , using the predicted value D b and the true value y b Calculate the true and false binary classification loss, the loss function is

[0024] L BIC =E (I,T)～P CrossEntropy(D b ,y b );

[0025] Step 5: Define the image feature aggregator, whose input is {v′1,...,v′ N} part, the output is the aggregated feature v′ agg , define the image forgery locator G using a multi-layer perceptron v , the input is the aggregated feature v′ agg , the output is the prediction D of the specific location of the forged area in the image v , calculated as D v =G v (v′agg ), using the predicted value D v and the true value y box Calculate the image positioning loss, the loss function is

[0026] L IMG =E (I,T)～P [||Sigmoid(D v )-y box ||+I CIoU (Sigmoid(D v )-y box )];

[0027] Step 6: Define the forgery method classifier C using a multilayer perceptron m , the input is {v′1,...,v′ in the fused image feature v′ N} part and {t′1,...,t′ in text feature t′ N} part, the output is the prediction D of the specific tampering method of tampering data m , using the predicted value D m and the true value y m Calculate the classification loss of the tampering method, the loss function is

[0028] L MLC =E (I,T)～P CrossEntropy(D m ,y m );

[0029] Step 7: Define the text forgery locator G using a multi-layer perceptron t , the input is {t′1,...,t′ in the fused text features N}, the output is the prediction D of the specific location of the text forgery area t , using the predicted value D t and the true value y t Calculate the text localization loss, the loss function is

[0030] L TMG =E (I,T)～P CrossEntropy(D t ,y t );

[0031] Step 8: Calculate the model loss function L, which is calculated as L = λ1L ITA +λ2L BIC +λ3L IMG +λ4L MLC +λ5L TMG , where λ1, λ2, λ3, λ4, and λ5 are hyperparameters that control the proportion of each loss;

[0032] Repeat steps 1 to 8 until the specified number of training rounds is reached to obtain the final global model.

[0033] In this solution, the image and text data are first read, and the image and text features are encoded using the ViT model and the BERT model respectively. In each iteration of training, the method first aligns the data through the multimodal semantic feature alignment module, calculates the similarity between the image and the text, and calculates the alignment loss function based on the obtained similarity. The alignment loss function guides the model to find the correspondence between different data modalities of the same instance. Then, in the multimodal feature fusion module, the features are fully fused using the Mamba module and the cross-attention mechanism. The fusion of features allows the image and text features to learn each other's feature representations. Finally, the fused features are input into the forged region detection and positioning module, which gives predictions of detection, region positioning, and forgery methods. The loss function is calculated by the difference between the predicted value and the true value, and the model is iteratively updated based on the loss function. The above steps are iterated several times to obtain the final global model.

[0034] Preferably, the step 2 is specifically to calculate the alignment loss of image to text and text to image by calculating the similarity s(I, T + ), the similarity d(T,I + ), the calculation formula is

[0035] s(I,T + )=g v (I) T ·g t (T),

[0036] s(T,I + )=g t (T) T ·g v (T).

[0037] The same method is used to calculate the similarity s(I,T between the image and the non-corresponding text, and between the text and the non-corresponding image) - ),s(T,I - ), use the calculated similarity to calculate the image-to-text and text-to-image alignment loss, the calculation formula is

[0038]

[0039] Where τ is a hyperparameter that controls the alignment granularity. The image-to-image and text-to-text loss L is calculated in the same way as above. v2v and L t2t , the final image-text alignment loss is

[0040]

[0041] As a preferred embodiment, the step 3 is specifically as follows: first, the Mamba module is used to filter out the important components in the original features, v m =Mamba(v),t m =Mamba(t), then v m and t m Input the cross attention module for feature fusion, the calculation formula is

[0042]

[0043] The fused image v′ m and text feature t′ m Repeat the above operation several times to obtain the final fusion results v′ and t′.

[0044] As an example, the step 5 is specifically to combine {v′1,...,v′ N}Part of the input feature aggregator Aggregator, the formula is

[0045]

[0046] v′ agg =Aggregator(v′ cls ,({v′1,...,v′ N}),({v′1,...,v′ N})).

[0047] The beneficial effects of the present invention are: the present invention can effectively detect multimodal deep fake information, and at the same time can provide the fake area and the fake method. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a training flow chart of the present invention;

[0049] Figure 2 This is a graph showing the true and false classification accuracy over the 50 training rounds as the training progresses.

[0050] Figure 3 This is a graph showing how the accuracy of image forgery location changes over the training process during 50 training rounds.

[0051] Figure 4 This is a graph showing how the accuracy of localizing forged text areas changes over the training process over 50 training rounds.

[0052] Figure 5This is a graph showing how the accuracy of forgery classification changes over the training process during 50 training rounds. DETAILED DESCRIPTION

[0053] The present invention will be further described below with reference to the accompanying drawings and examples.

[0054] Example:

[0055] A multimodal image and text deep fake detection and positioning method based on the Mamba module, such as Figure 1 As shown, the image and text data are first read, and the image and text features are encoded using the ViT model and the BERT model respectively. In each iteration of training, the method first inputs the image and text features into the multimodal feature alignment module, and performs data alignment on the multimodal data through a method based on contrastive learning. In the module, the similarity between the image and the text is first calculated, and then the alignment loss function is calculated based on the obtained similarity. In the multimodal feature fusion module, the Mamba module and the cross-attention mechanism are used to fully fuse the features, so that the image and text features learn the feature representations of each other. Finally, the fused features are input into the forged area detection and positioning module, and the predictions of detection, area positioning and forgery methods are given. The loss function is calculated by the gap between the predicted value and the true value, and the model is iteratively updated according to the loss function. The above steps are iterated several times to obtain the final global model. Specifically, the following steps are included:

[0056] Step 1: Read a batch of data. Assume that there are k images and k texts in the data. Use the ViT model to encode the features of the image. Encode the image as v = {v cls ,v1,...,v N}, use the BERT model to encode the features of the text, and encode the text into t = {t cls ,t1,...,t N} form;

[0057] Step 2: Perform feature alignment on the extracted image features and text features to calculate the similarity s(I, T + ), the similarity between the image and the non-corresponding text s(I,T - ) and the similarity between text and image s(T + ,I),s(I,T - ), image-to-image similarity s(I,I + ), s(I,I - ), text-to-text similarity s(T,T + ),s(T,T - ), the image-text alignment loss function is

[0058]

[0059] Step 3: Define the image-text feature fusion module Aggregator, which takes image feature v and text feature t as input and outputs the fused feature. The image feature is

[0060] v′=Aggregator(v,t)={v′ cls ,v′1,...,v′ N},

[0061] The text features are

[0062] t′=Aggregator(v,t)={t′ cls ,t′1,...,t′ N};

[0063] Step 4: Use a multi-layer perceptron to define the true and false classifier C b , the input is the image feature v′ and the text feature t′, and the output is the prediction of the authenticity of the data D b , using the predicted value D b and the true value y b Calculate the true and false binary classification loss, the loss function is

[0064] L BIC =E (I,T)～P CrossEntropy(D b ,y b );

[0065] Step 5: Define the image feature aggregator, whose input is {v′1,...,v′ N} part, the output is the aggregated feature v′ agg , define the image forgery locator G using a multi-layer perceptron v , the input is the aggregated feature v′ agg , the output is the prediction D of the specific location of the forged area in the image v , calculated as D v =G v (v′ agg ), using the predicted value D v and the true value y box Calculate the image positioning loss, the loss function is

[0066] L IMG =E (I,T)～P [||Sigmoid(D v )-y box ||+L CIoU (Sigmoid(D v )-y box )];

[0067] Step 6: Define the forgery method classifier C using a multilayer perceptron m , the input is {v′1,...,v′ in the fused image feature v′ N} part and {t′1,...,t′ in text feature t′ N} part, the output is the prediction D of the specific tampering method of tampering data m , using the predicted value D m and the true value y m Calculate the classification loss of the tampering method, the loss function is

[0068] L MLC =E (I,T)～P CrossEntropy(D m ,y m );

[0069] Step 7: Define the text forgery locator G using a multi-layer perceptron t , the input is {t′1,...,t′ in the fused text features N}, the output is the prediction D of the specific location of the text forgery area t , using the predicted value D t and the true value y t Calculate the text localization loss, the loss function is

[0070] L TMG =E (I,T)～P CrossEntropy(D t ,y t );

[0071] Step 8: Calculate the model loss function L, which is calculated as L = λ1L ITA +λ2L BIC +λ3L IMG +λ4L MLC +λ5L TMG , where λ1, λ2, λ3, λ4, and λ5 are hyperparameters that control the proportion of each loss;

[0072] Repeat steps 1 to 8 until the specified number of training rounds is reached to obtain the final global model.

[0073] The design of this solution can effectively detect multimodal deepfake information, and can also identify the forged area and the forgery method. Compared with existing deepfake detection methods, this solution can adapt to multiple data modalities and provide richer forgery information.

[0074] The embodiment described above is only a preferred solution of the present invention and does not limit the present invention in any form. Other variations and modifications are possible without exceeding the technical solution described in the claims.

Claims

1. A method for multimodal image and text deep fake detection and localization tasks based on the Mamba module, characterized by: It has a multimodal feature alignment module, a multimodal feature fusion module, and a forged area detection and positioning module, and specifically includes the following steps: Step 1: Read a batch of data. Assume that there are k images and k texts in the data. Use the ViT model to encode the features of the image. Encode the image as v = {v cls ,v1,...,v N }, use the BERT model to encode the features of the text, and encode the text into t = {t cls ,t1,...,t N } form; Step 2: Perform feature alignment on the extracted image features and text features to calculate the similarity s(I, T + ), the similarity between the image and the non-corresponding text s(I,T - ) and the similarity between text and image s(T + ,I),s(I,T - ), image-to-image similarity s(I,I + ), s(I,I - ), text-to-text similarity s(T,T + ),s(T,T - ), calculate the image text alignment loss function Step 3: Define the image-text feature fusion module Aggregator, which takes image feature v and text feature t as input and outputs the fused feature. The image feature is v′=Aggregator(v,t)={v′ cls ,v′1,...,v′ N } The text features are t′=Aggregator(v,t)={t′ cls ,t′1,...,t′ N }; Step 4: Use a multi-layer perceptron to define the true and false classifier C b , the input is the image feature v′ and the text feature t′, and the output is the prediction of the authenticity of the data D b , using the predicted value D b and the true value y b Calculate the true and false binary classification loss, the loss function is L BIC =E (I,T)～P CrossEntropy(D b ,y b ); Step 5: Define the image feature aggregator, whose input is {v′1,...,v′ N } part, the output is the aggregated feature v′ agg , define the image forgery locator G using a multi-layer perceptron v , the input is the aggregated feature v′ agg , the output is the prediction D of the specific location of the forged area in the image v , calculated as D v =G v (v′ agg ), using the predicted value D v and the true value y box Calculate the image positioning loss, the loss function is L IMG =E (I,T)～P [||Sigmoid(D v )-y box ||+L CIoU (Sigmoid(D v )-y box )]; Step 6: Define the forgery method classifier C using a multilayer perceptron m , the input is {v′1,...,v′ in the fused image feature v′ N } part and {t′1,...,t′ in text feature t′ N } part, the output is the prediction D of the specific tampering method of tampering data m , using the predicted value D m and the true value y m Calculate the classification loss of the tampering method, the loss function is L MIC =E (I,T)～P CrossEntropy(D m ,y m ); Step 7: Define the text forgery locator G using a multi-layer perceptron t , the input is {t′1,...,t′ in the fused text features N }, the output is the prediction D of the specific location of the text forgery area t , using the predicted value D t and the true value y t Calculate the text localization loss, the loss function is L TMG =E (I,T)～P CrossEntropy(D t ,y t ); Step 8: Calculate the model loss function L, which is calculated as L = λ1L ITA +λ2L BIC +λ3L IMG +λ4L MLC +λ5L TMG , where λ1, λ2, λ3, λ4, and λ5 are hyperparameters that control the proportion of each loss; Repeat steps 1 to 8 until the specified number of training rounds is reached to obtain the final global model.

2. The method for multimodal image and text deep fake detection and location tasks based on the Mamba module according to claim 1 is characterized in that: Specifically, step 2 is to calculate the alignment loss of image to text and text to image by calculating the similarity s(I, T + ), the similarity s(T,I + ), the calculation formula is s(I,T + )=g v (I) T ·g t (T); s(T,I + )=g t (T) T ·g v (T), The same method is used to calculate the similarity s(I,T between the image and the non-corresponding text, and between the text and the non-corresponding image) - ),s(T,I - ), use the calculated similarity to calculate the image-to-text and text-to-image alignment loss, the calculation formula is Where τ is a hyperparameter that controls the alignment granularity. The image-to-image and text-to-text loss L is calculated in the same way as above. v2v and L t2t , the final image-text alignment loss is .

3. The method for multimodal image and text deep fake detection and localization task based on the Mamba module according to claim 1 is characterized in that: Specifically, the step 3 is to first use the Mamba module to filter out the important components in the original features, v m =Mamba(v),t m =Mamba(t), then v m and t m Input the cross attention module for feature fusion, the calculation formula is The fused image v′ m and text feature t′ m Repeat the above operation several times to obtain the final fusion results v′ and t′.

4. The method for multimodal image and text deep fake detection and location tasks based on the Mamba module according to claim 1 is characterized in that: The step 5 is specifically to combine {v′1,...,v′ N }Part of the input feature aggregator Aggregator, the formula is v′ agg =Aggregator(v′ cls ,({v′1,...,v′ N }),({v′1,...,v′ N }))。

Images