Multi-modal image-text tampering detection and positioning method based on feature enhancement

Abstract

The invention relates to a feature enhancement-based multi-modal image-text tampering detection and positioning method, which comprises the following steps of: obtaining a tampered image-text pair, respectively preprocessing an input image and an input text, and constructing a data set; constructing a multi-modal image-text tampering model, wherein the multi-modal image-text tampering model comprises a text encoder, an image encoder, an image frequency domain enhancement module, a text enhancement module, a multi-cross attention fusion module and a multi-task tampering detection and positioning head; the image frequency domain enhancement module is used for selecting an optimal frequency band and sending the optimal frequency band out of the module; performing feature fusion on the frequency domain feature output by the image frequency domain enhancement module and the image domain feature output by the image encoder to obtain an image enhancement feature; fusing the aligned image enhancement features and text enhancement features by using a multi-cross attention fusion module; and training a multi-modal image-text tampering model by using the data set, and detecting and positioning multi-modal image-text tampering. According to the invention, the feature extraction capability of the network is enhanced, and the tampering detection and positioning capability is improved.

Description

Technical Field

[0001] The technical solution of the present invention relates to the multimodal technical field of image and text tampering, and specifically to a multimodal image and text tampering detection and positioning method based on feature enhancement. Background Art

[0002] In recent years, social media content has expanded from single text to multimedia formats such as images and videos, which provide additional information for tampering detection. However, most existing methods still focus only on text content and ignore the potential of multimodal information. To address this challenge, multimodal news tampering detection methods based on deep neural networks have been proposed. Some studies perform detection by splicing or fusing visual features with text features. For example, Jin et al. designed a multimodal framework that combines RNN modules for text and CNN modules for images for fake news detection on Weibo and Twitter. Singhal et al. used a pre-trained BERT model to extract news text features and a pre-trained VGG19 model to extract news image features. The MVAE method proposed by Khattar et al. uses VGG to extract image features and bidirectional LSTM to extract text features. It can learn shared representations of the two modalities and use these representations for fake news detection. Qian et al. fuse text and image features learned by pre-trained models through a multimodal contextual attention network. Zhou et al. proposed a multi-granularity fusion network based on pre-trained BERT, CLIP, and Swin-T to extract and fuse multi-granularity features for final classification. Xue et al. used BERT and ResNet to extract text and image features respectively, and judged the authenticity of the post by calculating the similarity between the two. Shao et al. proposed the first large-scale dataset for the problem of fake news detection and localization, and proposed the advanced HAMMER method to handle it. HAMMER uses forged features extracted from image and text domains, and adopts a multi-branch transformer structure with two dedicated decoders for tampering detection and localization. Its shallow decoder is used to locate the tampered face, while the deep decoder handles the remaining tasks.

[0003] CN118626932A discloses a multimodal false news detection system and method based on a self-attention mechanism, the system includes a news data collection module, a data enhancement module, a multimodal feature extraction module, a self-attention fusion module and a false news detection module; the false news is generated by the data enhancement module, the problem of data set imbalance is solved, and the multimodal fusion neural network of the self-attention mechanism performs feature fusion, and the accurate judgment of the authenticity of the multimodal news is realized. Although the method uses multimodal information for fusion, the method does not involve the enhancement of the extracted features and the enhancement of multimodal fusion. CN117854073A The present invention discloses a multimodal media tampering detection and positioning system based on unified reconstruction, including: an image encoder, a text encoder, a visual-text reconstruction module; a text-visual reconstruction module, a reconstruction coordinator module, and a detection and classification module. Although the method uses the reconstruction module to perform subsequent enhancement on the extracted features, it does not involve additional frequency domain information and cannot enhance the features well. The present invention not only considers the image and text domains, but also integrates the frequency domain. In forgery detection research, the frequency domain contains rich information about face forgery. By combining the frequency domain, our method effectively captures additional forgery clues, thereby improving the overall performance.

[0004] Existing methods mainly focus on the interaction between image and text domains, and lack the processing of enhancing the extracted features and the fused features. Summary of the invention

[0005] In view of the fact that the existing structure of each modal feature in the existing methods only focuses on the interaction between the image and text domains, and lacks the processing of enhancing the extracted features and the fused features, the technical problem to be solved by the present invention is to provide a multimodal image and text tampering detection and positioning method based on feature enhancement, which enhances the image and text flow information respectively, and uses a multi-cross attention fusion module to deeply fuse the enhanced multimodal features, thereby enhancing the feature extraction capability of the network, and further improving the tampering detection and positioning capability.

[0006] The technical solution adopted by the present invention to solve the technical problem is: A multimodal image and text tampering detection and positioning method based on feature enhancement, the detection and positioning method includes the following contents: Obtain tampered image and text pairs, preprocess the input image and input text respectively, and construct a data set; Constructing a multimodal image and text tampering model: The multimodal image-text tampering model includes a text encoder, an image encoder, an image frequency domain enhancement module, a text enhancement module, a multi-cross attention fusion module and a multi-task tampering detection and localization head; The image encoder is used to encode the preprocessed input image to generate a corresponding image embedding; The image frequency domain enhancement module is used to convert the pre-processed input image into frequency domain information, then mine the subtle forged information in the frequency domain, and select the optimal frequency band to send to the module; The frequency domain features output by the image frequency domain enhancement module and the image domain features output by the image encoder are fused to obtain image enhancement features; The text encoder is used to generate corresponding text embedding for the preprocessed input text; The text enhancement module is used to further mine text embedding to obtain text enhancement features; Use contrastive learning to align image enhancement features and text enhancement features; then use a multi-cross attention fusion module to fuse the aligned image enhancement features and text enhancement features; The multi-task tampering detection and positioning head is used to detect the output results of the multi-cross attention fusion module to complete the tampering detection and positioning of the multi-modal image and text; The dataset is used to train a multimodal image and text tampering model to achieve tampering detection and location of multimodal images and texts.

[0007] Furthermore, the image frequency domain enhancement module includes block DCT transformation, multi-head attention frequency integration, and frequency selective modulation to transform the input RGB image Transform to YCbCr space, and then perform block DCT transformation: Divided into a set of 8×8 blocks, each block is processed by DCT into a spectrum of components of different frequencies, then all components of the same frequency are grouped into one channel, the spectrum is flattened and reshaped to form a new input , this is the output of block DCT transform; After the block DCT transform, it enters the multi-head attention frequency integration. The specific process is: the output of the block DCT transform The signal is downsampled, and then the downsampled result is divided into low-frequency signals through the reshape operation and high frequency signals The two parts are input into two multi-head self-attention MHSAs respectively, and the outputs of the two multi-head self-attentions are spliced ​​to restore the original shape; then the spliced ​​result is input into another multi-head self-attention MHSA to coordinate all different frequency bands to form a new signal , and then The resized image is reshaped and the result is input into a multi-head self-attention MHSA again to model the relationship between all blocks. The enhanced frequency signal is obtained after resizing and upsampling. , which is the output of multi-head attention frequency integration; The output of integrating the multi-head attention frequencies Input to frequency selective modulation, calculate The relative importance score of each input frequency channel is then used to redistribute the weight of each frequency channel, and then modulated to output the final frequency domain feature .

[0008] Furthermore, the frequency selective modulation includes a frequency selective layer and a frequency modulation layer, and the frequency selective layer includes a global average pooling, a 1×1 convolution, a normalization process, a ReLU activation function, a 1×1 convolution, and a Sigmoid function connected in sequence, and the output of the multi-head attention frequency integration is Input to the global average pooling, after a series of processing, the processing result of the Sigmoid function is combined with Perform element-wise multiplication to obtain the output of the frequency selection layer; the frequency modulation layer includes 3×3 convolution, normalization, ReLU activation function, 1×1 convolution, and normalization connected in sequence. The output of the frequency selection layer is then passed through the frequency modulation layer to obtain the final frequency domain features. .

[0009] Furthermore, the text enhancement module includes a bidirectional LSTM network and an attention layer, and uses the bidirectional LSTM network to capture the semantic features and long-distance dependencies of the text from the front-back direction, wherein represents the forward LSTM in the bidirectional LSTM network, Represents backward LSTM, which converts the output of the pre-trained image encoder and the output vector generated by the bidirectional LSTM network into three matrices of the same dimension through linear transformation, namely, the value matrix V, the key matrix K, and the query matrix Q. The three matrices are sequentially input into the corresponding attention mechanism in the attention layer to perform h-times joint information representation, and the result of the h-times joint information representation is linearly transformed to obtain the output text enhancement feature vector.

[0010] Furthermore, the multi-cross attention fusion module includes two self-attention modules and three cross-attention modules. Enhance the features of the aligned image and text enhancement features First, input them into a self-attention module respectively, and enhance the image features after self-attention processing The data is used as Q, the text enhancement features after self-attention processing The data is used as K and V to enter a cross attention module, and then the image enhancement features after self-attention processing The output result is expressed as ; On the other side, the data of text enhancement features after self-attention processing is used as Q, and the data of image enhancement features after self-attention processing is used as K and V to enter a cross attention module, and then combined with the text enhancement features after self-attention processing The output result is expressed as ; The output will be As Q, output the result As K and V enter the third cross attention module, and then with Concatenate and get the output vector .

[0011] Specifically, the present invention is a multi-modal image and text tampering detection and positioning method based on feature enhancement, and the specific steps of the method are: The first step is to prepare the data set: The samples in the data set are image-text pairs, including original image-text pairs, tampered image-text pairs and mixed processing pairs; the tampered image-text pairs include face replacement tampering, face attribute tampering, text replacement tampering and text attribute tampering; the mixed processing pair is formed by combining 1 / 3 of the tampered image and 1 / 2 of the tampered text; The second step is to build the image encoder and text encoder: Input the image-text pair data (I, T) into the image encoder and text encoder respectively to obtain image features and text features ;

[0012]

[0013] Among them, I represents image data, T represents text data, represents the image encoder, using the pre-trained VIT model; Represents the text encoder, using the pre-trained BERT model; The third step is to build an image frequency domain enhancement module: Step 3.1, convert the image to frequency: The input image Transform to YCbCr space by Divide into a set of 8×8 blocks, so as to obtain , H and W represent the height and width of the input image; where, , represents the block divided by a certain color channel, and c represents the number of channels; Then, each block is processed into a spectrum by block DCT transformation. , flattening the spectrum to form a new input , expressed as

[0014] in , Indicates all The series connection; Represents flattening and reshaping; Step 3.2, modeling frequency information: right Downsample and divide it into two parts: low frequency signal and high frequency signals , where k represents the signal dimension after downsampling; The low frequency signal and high frequency signals Input them into two multi-head self-attention MHSAs respectively, and concatenate their outputs to restore the original shape; then use another multi-head self-attention MHSA to coordinate all different frequency bands to form a new signal, which is expressed as ; Will Reshape , C is the number of channels, and then multi-head self-attention MHSA is used again to model the relationship between all blocks. Finally, upsampling is performed to obtain the enhanced frequency signal ; Step 3.3, select the most appropriate frequency characteristics: For the enhanced frequency signal Reshape the dimension to obtain feature D, and calculate the relative importance of each input frequency channel according to formula (4): : in, , is the enhanced frequency signal The result of reshaping the dimension, p is the global average pooling GAP; δ represents the ReLU function, β represents the batch normalization BN; σ is the Sigmoid function; the kernel size of Conv1 is (C / r)×C×1×1, and the kernel size of Conv2 is C ×(C / r) ×1×1, where r is a hyperparameter; C is the number of frequency channels; make For input frequency channel The weights are then used to assign scores is the weight of the cth frequency channel, is the cth frequency channel, is a learnable parameter, then the learning-based frequency selection is defined as: Where ⊙ is the element product, is the weighted frequency channel, and the output of the frequency selection layer is expressed as: ; The output of the frequency selection layer is embedded into the backbone network through the frequency modulation layer. The calculation of the frequency modulation layer is:

[0015] In the formula, Conv3 represents a 3×3 convolution; Conv4 represents a 1×1 convolution with a kernel size of C×C×1×1, and the output of the frequency modulation layer is ; Indicates frequency modulation operation; Indicates normalization processing; The fourth step is to integrate frequency domain features with image features: The second image feature And the frequency domain features obtained in the third step Connect them and calculate the cross attention weight, the calculation formula is: ;

[0016]

[0017] in, and Represent image features and frequency domain characteristics , H and W are the height and width of the corresponding feature map respectively; For the connection operation, Get the cross feature map; and Indicates that two 1×1 convolution operations with different parameters are performed on the cross feature map, and the result is and ; , They are image features and frequency domain characteristics The cross attention weights are + = 1; The image enhancement feature is obtained by the following formula :

[0018] Step 5: Build the text enhancement module: Step 5.1, Bidirectional LSTM network: The text features obtained in the second step It is sent to the bidirectional LSTM network, and the specific semantic extraction process is as follows:

[0019]

[0020]

[0021] in and g are the activation functions of sigmoid and tanh respectively, Represents the weight parameters of different positions, n is an integer from 1 to 6; is the text vector input into the bidirectional LSTM network at time t, and They represent the previous sequence information obtained by the forward LSTM and the following sequence information obtained by the backward LSTM respectively. The subscript t-1 represents the corresponding information at the previous moment, and the subscript t+1 represents the corresponding information at the next moment. Represents the output of the bidirectional LSTM network; Step 5.2, Attention Layer: The forward and backward outputs generated by the bidirectional LSTM network are concatenated and then transformed into three matrices of the same dimension through linear transformation, namely the value matrix V, the key matrix K, and the query matrix Q. The three matrices are input into the attention layer in turn and the information is jointly represented h times according to formulas (13) and (14). The result of the h-times joint representation of information is transformed into the output text enhancement feature vector through linear transformation. ;

[0022]

[0023] in is the dimension value of matrix Q, K, V, represents the calculation result of a single self-attention mechanism, i is an integer from 1 to h; Att represents the operation of the self-attention mechanism; C is the connection function of the matrix concatenation process, is the transformation matrix; Step 6: Align image and text features: The image-to-text contrast loss is calculated using formulas (15) and (16): and contrastive loss for text to image :

[0024]

[0025] Where (I, T) represents a set of matching image-text pairs, τ is the temperature hyperparameter, is a set of negative text samples that do not match I and belong to tampered image-text pairs, is the positive text sample that matches I; is a set of negative image samples that do not match T and belong to tampered image-text pairs, is a positive image sample matching T; K is the number of negative samples; is the kth negative text sample that does not match I and belongs to the tampered image-text pair; is the kth negative image sample that does not match T and belongs to the tampered image-text pair; For expectations; in (,) indicates similarity, , and There are two mapping heads, through which the two enhanced modal features are mapped to a low-dimensional space with a dimension of 256 for similarity calculation; Perform intra-modal contrastive learning in both image and text modalities and calculate text-to-text contrastive loss and image-to-image contrast loss ; All contrast losses are combined to form the final contrastive learning loss ;

[0026] Step 7: Build a multi-cross attention fusion module: Enhance the features of the aligned image and text enhancement features First, input them into a self-attention module to obtain the image enhancement features after self-attention processing. and , and then perform the following cross-attention operation:

[0027]

[0028]

[0029] in, is the output of multiple cross attention fusion modules; represents a cross-attention operation; Step 8: Obtain the tampered image and text detection and positioning results: Will Input into the multi-task tampering detection and positioning head to obtain the detection result, where the constraint loss of each task is:

[0030]

[0031]

[0032]

[0033] Formula (22) is the image localization loss, where express The features include the components of image features, represents a bounding box detector, Represents the true bounding box position, and the image localization loss is calculated by combining the commonly used L1 loss and the generalized IoU loss; Formula (23) is the text localization loss, where, express The features include the components of text features, represents a text label detector, represents the true text label; Formula (24) is the multi-classification detection loss, where represent The component that contains classification information in the feature, represents a multi-label classifier, represents the true multi-classification label value; formula (25) is the binary classification detection loss, represents a two-label classifier, Represents the true binary classification label value; represents the expected value of sampling image I and text T on distribution p.

[0034] Compared with the existing method, the beneficial effects of the present invention are: The present invention has excellent positioning accuracy. Compared with other methods, the present invention can not only accurately locate the image tampering area, but also more accurately locate the text tampering content. This is due to the image frequency domain enhancement module, text enhancement module and multi-cross attention fusion module of the present invention, which effectively solves the problem of unsatisfactory tampering positioning in certain scenarios.

[0035] The method of the present invention has an image frequency domain enhancement module that enhances frequency clues and adaptively selects appropriate channels for fine-grained prediction. The image frequency domain enhancement module first converts the input image to the frequency domain, then completes the enhancement and screening of the frequency domain, and finally uses feature fusion to align the features of the image domain and the frequency domain, and then outputs it. By combining the frequency domain information, additional forgery clues are effectively captured, thereby completing the further mining of image information. In order to enhance text features while maintaining the balance of the encoder size of the two branches of the model, the present invention proposes a text enhancement module that can further mine text information. The text enhancement module uses a bidirectional LSTM to further extract local information of the text. The present invention proposes a dual-branch multi-cross attention fusion mechanism to deeply interact and fuse image features and text features, effectively infer the correlation between images and texts, allow each embedding to capture modal context information, and achieve deep interaction with other modal information, so that information from different modalities can enhance each other, thereby improving detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0037] Figure 1 It is an overall flow chart of a multimodal image and text detection and positioning method based on feature enhancement of the present invention.

[0038] Figure 2 It is the overall flow chart of the image frequency domain enhancement module in the present invention.

[0039] Figure 3 It is a structural diagram of multi-head attention frequency integration in the present invention.

[0040] Figure 4 It is a structural schematic diagram of the frequency selective modulation in the present invention.

[0041] Figure 5 It is a structural schematic diagram of the text enhancement module in the present invention.

[0042] Figure 6 It is a structural diagram of the multi-cross attention fusion module in the present invention. DETAILED DESCRIPTION

[0043] Figure 1 As shown in the embodiment, the overall process of a multimodal image and text detection and positioning method based on feature enhancement of the present invention is as follows: Input tampered image and text pairs → Preprocess the input image and input text separately, remove images and texts with too small resolution, unify the image size → Input the preprocessed image into the image encoder to extract image features →At the same time, the preprocessed text is input into the text encoder to extract text features →Transform the preprocessed image into YCbCr space, and then transform it into frequency domain through DCT →Input the frequency domain information into the image frequency domain enhancement module to mine subtle forgery information and obtain frequency domain features →The obtained frequency domain features and image features Fusion to obtain enhanced image features → Simultaneous text features Input into the text enhancement module to obtain deep text enhancement features. → Enhance the image features With text enhancement features Through contrastive learning loss constraint alignment → input the aligned image enhancement features and text enhancement features into the multi-cross attention fusion module to obtain the fusion feature M → input the fusion feature into the detection head to obtain the detection and positioning results.

[0044] Figure 2 , 3 As shown in the embodiment 4, the image frequency domain enhancement module process in the present invention is as follows: The input RGB image Transform to YCbCr space, and then perform block DCT transformation: Divided into a set of 8×8 blocks, each block is processed by DCT into a spectrum of components of different frequencies, then all components of the same frequency are grouped into one channel, the spectrum is flattened and reshaped to form a new input , this is the output of block DCT transform; After the block DCT transform, it enters the multi-head attention frequency integration. The specific process is: the output of the block DCT transform The signal is downsampled. In the DCT frequency domain, by downsampling, unimportant frequency components are selectively discarded, which can compress the data volume. In the previous step, components with the same frequency are divided into one channel. Here, reshape can be used to directly separate high and low frequency signals. Reshape is used to divide it into low frequency signals. and high frequency signals The two parts are input into two multi-head self-attention (MHSA) respectively, and the outputs of the two multi-head self-attention are spliced ​​to restore the original shape; then the spliced ​​result is input into another multi-head self-attention MHSA to coordinate all different frequency bands to form a new signal , and then The resized image is reshaped and the result is input into a multi-head self-attention (MHSA) again to model the relationship between all blocks. The enhanced frequency signal is then obtained after deformation and upsampling (filling the frequency components discarded by the initial downsampling by interpolation). , which is the output of multi-head attention frequency integration; The output of integrating the multi-head attention frequencies Input to frequency selective modulation, calculate The relative importance score of each input frequency channel is then used to redistribute the weight of each frequency channel, using the frequency modulation layer to solve the problem of different channel dimensions, and output the final frequency domain features .

[0045] The frequency selective modulation includes a frequency selective layer and a frequency modulation layer. The frequency selective layer includes a global average pooling, a 1×1 convolution, a normalization process, a ReLU activation function, a 1×1 convolution, and a Sigmoid function connected in sequence. The output of the multi-head attention frequency integration is Input to the global average pooling, after a series of processing, the processing result of the Sigmoid function is combined with The frequency modulation layer includes 3×3 convolution, normalization, ReLU activation function, 1×1 convolution, and normalization, and the output of the frequency selection layer is then sequentially subjected to 3×3 convolution, normalization, ReLU activation function, 1×1 convolution, and normalization to obtain the final frequency domain features. .

[0046] Figure 5 As shown in the embodiment, the text enhancement module process of the present invention is as follows: The text features extracted by the pre-trained model BERT use a bidirectional LSTM network to capture the semantic features and long-distance dependencies of the text from the front-to-back direction, and more comprehensively mine the contextual semantic features of the text. represents the forward LSTM in the bidirectional LSTM network, Represents backward LSTM, concatenates the forward and backward output vectors generated by the bidirectional LSTM network, and then transforms them into three matrices of the same dimension through linear transformation, namely, value matrix (Value, referred to as V), key matrix (Key, referred to as K), and query matrix (Query, referred to as Q), and inputs the three matrices into the attention mechanism in sequence. H-time information joint representation, h is the number of nodes in the bidirectional LSTM, which is related to the length of the input text. In this embodiment, h=6; the result of the h-time information joint representation is transformed into the output text enhancement feature vector through linear transformation .

[0047] Figure 6 As shown in the example, the process of the multi-cross attention fusion module in the present invention is as follows: Enhance the features of the aligned image and text enhancement features First, input them into a self-attention module respectively, and enhance the image features after self-attention processing The data is used as Q, the text enhancement feature after self-attention processing The data is used as K and V to enter a cross attention module, and then the image enhancement features after self-attention processing are The output result is expressed as ; On the other side, the data of text enhancement features after self-attention processing is used as Q, and the data of image enhancement features after self-attention processing is used as K and V to enter a cross attention module, and then combined with the text enhancement features after self-attention processing The output result is expressed as ; The output will be As Q, output the result As K and V enter the third cross attention module, and then with Concatenate and get the output vector .

[0048] Example 1 The specific steps of the multi-modal tampering and positioning method based on feature enhancement in this embodiment are as follows: The first step is to prepare the data set: The dataset contains 230k sets of image-text pairs, including 77,426 original image-text pairs and 152,574 tampered image-text pairs. The tampered image-text pairs include 66,722 face replacement tampering, 56,411 face attribute tampering, 43,546 text replacement tampering and 18,588 text attribute tampering. Then 1 / 3 of the tampered images and 1 / 2 of the tampered texts are combined together to form 32,693 mixed processing pairs.

[0049] The input images and input texts in the image-text pair samples were preprocessed separately, and images and texts with too small resolution were removed. The image size was unified to 256*256 to obtain the final data set, which was roughly divided into training set, validation set and test set according to the ratio of 7:1:2. This ensures that samples from the same original image-text pair but with different tampering types will not appear in the training set and validation or test set at the same time.

[0050] The second step is to build the image encoder and text encoder: The image-text pair data (I, T) prepared in the first step above is input into the image encoder and text encoder respectively to obtain the image features and text features .

[0051]

[0052]

[0053] Among them, I represents image data, T represents text data, represents the image encoder, using the pre-trained VIT model; Represents the text encoder, using the pre-trained BERT model; The third step is to build an image frequency domain enhancement module: Step 3.1, convert the image to frequency: The RGB image prepared in the first step is subjected to block DCT transformation to utilize its frequency information. First, Transformed to YCbCr space, it can be expressed as By Divide into a set of 8×8 blocks, so as to obtain , H and W represent the height and width of the input image. Among them, , represents the block divided by a color channel, and c represents the number of channels. Then, each block is processed by DCT into a spectrum , each value of which corresponds to the intensity of a certain frequency band. The spectrum is flattened and reshaped to form a new input, which can be expressed as

[0054] in , Indicates all The series connection; Represents a flatten reshape.

[0055] Step 3.2, modeling frequency information: Construct an enhancement module within the frequency block obtained in step 3.1: First, Downsample and divide it into two parts: low frequency signal and high frequency signals , where k represents the size of the signal dimension after downsampling. To enhance the signals of the corresponding frequency bands, they are fed into two multi-head self-attention (MHSA) respectively, and their outputs are concatenated to restore the original shape. Then another MHSA is used to coordinate all different frequency bands to form a new signal, which is expressed as MHSA is able to capture the rich correlations between each item in the input features, making different spectra of the image fully interact.

[0056] Construct enhancement modules between frequency blocks obtained in step 3.1: Since each block of DCT changes is independent of each other, the above process only enhances a single block. In order to help the network better identify the location of the tampered target, it is necessary to establish a connection between blocks. Reshape , C is the number of channels, and then MHSA is used again to model the relationship between all blocks. Finally, the enhanced frequency signal is obtained by reshaping and upsampling. .

[0057] Step 3.3, select the most appropriate frequency characteristics: The frequency signal after enhancement from step 3.2 Select the most appropriate frequency feature: First calculate the relative importance of each input frequency channel , and then use it to assign scores to each frequency channel, where The calculation method is as follows: in, , is the frequency signal The result of reshaping the dimension so that the number of channels is in front, p is the global average pooling (GAP). δ represents the ReLU function, β represents the batch normalization (BN). σ is the Sigmoid function. The kernel size of Conv1 is (C / r)×C×1×1, and the kernel size of Conv2 is C ×(C / r) ×1×1, where r is a hyperparameter to reduce the amount of calculation. In the experiment, r is set to 8.

[0058] make For input frequency channel The weights are then used to assign scores is the weight of the cth frequency channel, is the cth frequency channel, is a learnable parameter, then the learning-based frequency selection is defined as: Where ⊙ is the element product, is the weighted frequency channel, and the output of the frequency selection layer is expressed as: ; The output of the frequency selection layer is embedded into the backbone network through the frequency modulation layer. The calculation of the frequency modulation layer is:

[0059] In the formula, Conv3 represents a 3×3 convolution. Conv4 represents a 1×1 convolution with a kernel size of C×C×1×1. The output of the frequency modulation layer is ; Indicates frequency modulation operation; represents normalization processing. The enhanced frequency signal The most suitable frequency feature selected in the frequency modulation layer This is the output of the image frequency domain enhancement module.

[0060] The fourth step is to integrate frequency domain features with image features: Adaptively fuse the image features obtained by the second And the frequency domain features obtained in the third step Connect them and calculate the cross attention weight, the calculation formula is: ;

[0061]

[0062] in, and Represent image features and frequency domain characteristics , H and W are the height and width of the corresponding feature map respectively; For the connection operation, Get the cross feature map; and Indicates that a 1×1 convolution operation is performed on the cross feature map to obtain the attention weight; and frequency domain characteristics The cross feature map of uses two 1×1 convolutional layers with different parameters, whose parameters are learned separately. and The values ​​of are different; , are the cross attention weights of RGB features and frequency features, and + = 1.

[0063] The image enhancement feature that combines the frequency domain and the image is obtained through the following weighting scheme :

[0064] Step 5: Build the text enhancement module: Step 5.1, Bidirectional LSTM network: The text features obtained in the second step (i.e. the output of the pre-trained model BERT) is sent to the bidirectional LSTM network for more in-depth learning of contextual features, thereby obtaining more comprehensive semantic information. The bidirectional LSTM network consists of a forward LSTM and a backward LSTM, which process the sequence from two directions respectively, and can obtain information from the forward and backward context of the input text, thereby more comprehensively capturing the semantic features of the text sequence. For the specific semantic extraction process, please refer to formulas (10, 11, 12)

[0065]

[0066]

[0067] in and g are the activation functions of sigmoid and tanh respectively, Represents the weight parameters of different positions, n is an integer from 1 to 6; is the text vector input into the bidirectional LSTM network at time t, and They represent the previous sequence information obtained by the forward LSTM and the following sequence information obtained by the backward LSTM respectively. The subscript t-1 represents the corresponding information at the previous moment, and the subscript t+1 represents the corresponding information at the next moment. Represents the output of the bidirectional LSTM network; Step 5.2, Attention Layer: The output vector generated by the network in step 5.1 contains a large number of positional features and semantic relationships. The self-attention mechanism MHA is introduced to perform residual connection and normalization on the output vector generated by the above steps, and the key information in the global features is captured through the self-attention mechanism, which further improves the weight of the key features, optimizes the network structure, and improves the accuracy of sequence prediction tasks. MHA concatenates the forward and backward outputs generated by the bidirectional LSTM network and then converts them into three matrices of the same dimension through linear changes, namely the value matrix (Value, V for short), the key matrix (Key, K for short), and the query matrix (Query, Q for short). The three matrices are input into the attention layer in turn, and the information is jointly represented h times according to formulas (13) and (14). The result of the h-times joint representation of information is obtained through linear changes to obtain the output text enhancement feature vector. .

[0068]

[0069]

[0070] in is the dimension value of matrix Q, K, V, represents the calculation result of a single self-attention mechanism, i is an integer of 1-h, and h=6 in this embodiment; Att represents the operation of the self-attention mechanism; C is the connection function of the matrix concatenation process, is the transformation matrix; Step 6: Align image and text features: Align image and text embeddings through cross-modal contrastive learning, making the image enhancement features obtained in the fourth and fifth steps and text enhancement feature vector The contrastive learning pushes away the feature embeddings of the tampered image-text pairs, further emphasizing the semantic inconsistency they produce. The image-to-text contrast loss is calculated using formulas (15) and (16): and contrastive loss for text to image :

[0071]

[0072] Where (I, T) represents a set of matching image-text pairs, τ is the temperature hyperparameter, is a set of negative text samples that do not match I and belong to tampered image-text pairs, is the positive text sample that matches I; is a set of negative image samples that do not match T and belong to tampered image-text pairs, is a positive image sample matching T; K is the number of negative samples; is the kth negative text sample that does not match I and belongs to the tampered image-text pair; is the kth negative image sample that does not match T and belongs to the tampered image-text pair; For expectations; in (,) indicates similarity, , and There are two mapping heads, through which the two enhanced modal features are mapped to a low-dimensional space with a dimension of 256 for similarity calculation; Further intra-modal contrastive learning is carried out in both image and text modalities, which can maintain reasonable semantic relationships in each single modality and calculate text-to-text contrastive loss. and image-to-image contrast loss ; All contrast losses are combined to form the final contrastive learning loss ;

[0073] Step 7: Build a multi-cross attention fusion module: Enhance the features of the image after the sixth step alignment and aligned text enhancement features Perform multimodal feature fusion, in which the aligned image enhancement features are and text enhancement features First, input them into a self-attention module to obtain the image enhancement features after self-attention processing. and . Use Figure 6 The multi-cross attention fusion module (MAF) shown in Figure 1 is used to guide the interaction between image features and text features. MAF allows each embedding to capture modal context information while enabling deep interaction with other modal information. The attention function is applied to the query (Q), key (K), and value (V) features, each with a hidden size of D', as follows:

[0074] In MAF, given a query (Q) from one modality (e.g., image), the key (K) and value (V) can only be obtained from another modality (e.g., text). The two modality interaction module is represented as and ,in The cross attention in uses image features as queries, while The cross attention in uses text features as queries, and finally the alternately fused features are fused in depth using cross attention in the same way, thus achieving complex inter-modal fusion. This can be expressed as

[0075]

[0076]

[0077] in, is the output of multiple cross-attention fusion modules; represents a cross-attention operation; represents the image enhancement features after self-attention processing, Represents the text enhanced features after self-attention processing.

[0078] Step 8: Obtain the tampered image and text detection and positioning results: The features obtained in step 7 Input into the dedicated task detection head to get the detection result. The constraint loss of each task is as follows:

[0079]

[0080]

[0081]

[0082] Formula (22) is the image localization loss, where express The features include the components of image features, represents a bounding box detector, Represents the true bounding box position, and the image localization loss is calculated by combining the commonly used L1 loss and the generalized IoU loss; Formula (23) is the text localization loss, where, express The features include the components of text features, represents a text label detector, represents the true text label; Formula (24) is the multi-classification detection loss, where represent The component that contains classification information in the feature, represents a multi-label classifier, represents the true multi-classification label value; formula (25) is the binary classification detection loss, represents a two-label classifier, Represents the true binary classification label value; represents the expected value of sampling image I and text T on distribution p.

[0083] The above-mentioned multimodal image and text tampering and positioning method based on feature enhancement, wherein the image encoder VIT and the text encoder BERT are algorithms well known in the technical field. The present invention comprises three modules: an image frequency domain enhancement module, a text enhancement module, and a multi-cross attention fusion module. The image frequency domain enhancement module uses frequency domain information to extract forged information, the text enhancement module uses bidirectional LSTM and self-attention mechanism to more comprehensively mine the contextual semantic features of the text, and the multi-cross attention fusion module realizes deep interaction and fusion between the enhanced features.

[0084] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A multimodal image and text tampering detection and positioning method based on feature enhancement, characterized in that: The detection and positioning method includes the following contents: Obtain tampered image and text pairs, preprocess the input image and input text respectively, and construct a data set; Constructing a multimodal image and text tampering model: The multimodal image-text tampering model includes a text encoder, an image encoder, an image frequency domain enhancement module, a text enhancement module, a multi-cross attention fusion module and a multi-task tampering detection and localization head; The image encoder is used to encode the preprocessed input image to generate a corresponding image embedding; The image frequency domain enhancement module is used to convert the pre-processed input image into frequency domain information, then mine the subtle forged information in the frequency domain, and select the optimal frequency band to send to the module; The frequency domain features output by the image frequency domain enhancement module and the image domain features output by the image encoder are fused to obtain image enhancement features; The text encoder is used to generate corresponding text embedding for the preprocessed input text; The text enhancement module is used to further mine text embedding to obtain text enhancement features; Use contrastive learning to align image enhancement features and text enhancement features; then use a multi-cross attention fusion module to fuse the aligned image enhancement features and text enhancement features; The multi-task tampering detection and positioning head is used to detect the output results of the multi-cross attention fusion module to complete the tampering detection and positioning of the multi-modal image and text; The dataset is used to train a multimodal image and text tampering model to achieve tampering detection and location of multimodal images and texts.

2. The method according to claim 1, characterized in that The image frequency domain enhancement module includes block DCT transformation, multi-head attention frequency integration, and frequency selective modulation. Transform to YCbCr space, and then perform block DCT transformation: Divided into a set of 8×8 blocks, each block is processed by DCT into a spectrum of components of different frequencies, then all components of the same frequency are grouped into one channel, the spectrum is flattened and reshaped to form a new input , which is the output of block DCT transform; After the block DCT transform, it enters the multi-head attention frequency integration. The specific process is: the output of the block DCT transform The signal is downsampled, and then the downsampled result is divided into low-frequency signals through the reshape operation and high frequency signals The two parts are input into two multi-head self-attention MHSAs respectively, and the outputs of the two multi-head self-attentions are spliced ​​to restore the original shape; then the spliced ​​result is input into another multi-head self-attention MHSA to coordinate all different frequency bands to form a new signal , and then The resized image is reshaped and the resized image is input into a multi-head self-attention MHSA again to model the relationship between all blocks. The enhanced frequency signal is obtained after resizing and upsampling. , which is the output of multi-head attention frequency integration; The output of integrating the multi-head attention frequencies Input to frequency selective modulation, calculate The relative importance score of each input frequency channel is then used to redistribute the weight of each frequency channel, and then modulated to output the final frequency domain feature .

3. The method according to claim 2, characterized in that The frequency selective modulation includes a frequency selective layer and a frequency modulation layer. The frequency selective layer includes a global average pooling, a 1×1 convolution, a normalization process, a ReLU activation function, a 1×1 convolution, and a Sigmoid function connected in sequence. The output of the multi-head attention frequency integration is Input to the global average pooling, after a series of processing, the processing result of the Sigmoid function is combined with Perform element-wise multiplication to obtain the output of the frequency selection layer; the frequency modulation layer includes 3×3 convolution, normalization, ReLU activation function, 1×1 convolution, and normalization connected in sequence. The output of the frequency selection layer is then passed through the frequency modulation layer to obtain the final frequency domain features. .

4. The method according to claim 1, characterized in that The text enhancement module includes a bidirectional LSTM network and an attention layer, and uses the bidirectional LSTM network to capture the semantic features and long-distance dependencies of the text from the front-to-back direction. represents the forward LSTM in the bidirectional LSTM network, Represents backward LSTM, which converts the output of the pre-trained image encoder and the output vector generated by the bidirectional LSTM network into three matrices of the same dimension through linear transformation, namely, the value matrix V, the key matrix K, and the query matrix Q. The three matrices are sequentially input into the corresponding attention mechanism in the attention layer to perform h-times joint information representation, and the result of the h-times joint information representation is linearly transformed to obtain the output text enhancement feature vector.

5. The method according to claim 1, characterized in that The multi-cross attention fusion module includes two self-attention modules and three cross-attention modules. Enhance the features of the aligned image and text enhancement features First, input them into a self-attention module respectively, and enhance the image features after self-attention processing The data is used as Q, the text enhancement features after self-attention processing The data is used as K and V to enter a cross attention module, and then the image enhancement features after self-attention processing The output result is expressed as ; On the other side, the data of text enhancement features after self-attention processing is used as Q, and the data of image enhancement features after self-attention processing is used as K and V to enter a cross attention module, and then combined with the text enhancement features after self-attention processing The output result is expressed as ; Will output the result As Q, output the result As K and V enter the third cross attention module, and then with Concatenate and get the output vector .

6. A multimodal image and text tampering detection and positioning method based on feature enhancement, characterized in that: The specific steps of the method are: The first step is to prepare the data set: The samples in the data set are image-text pairs, including original image-text pairs, tampered image-text pairs and mixed processing pairs; the tampered image-text pairs include face replacement tampering, face attribute tampering, text replacement tampering and text attribute tampering; the mixed processing pair is formed by combining 1 / 3 of the tampered image and 1 / 2 of the tampered text; The second step is to build the image encoder and text encoder: Input the image-text pair data (I, T) into the image encoder and text encoder respectively to obtain image features and text features ; Among them, I represents image data, T represents text data, represents the image encoder, using the pre-trained VIT model; Represents the text encoder, using the pre-trained BERT model; The third step is to build an image frequency domain enhancement module: Step 3.1, convert the image to frequency: The input image Transform to YCbCr space by Divide into a set of 8×8 blocks, so as to obtain , H and W represent the height and width of the input image; where, , represents the block divided by a certain color channel, and c represents the number of channels; Then, each block is processed into a spectrum by block DCT transformation. , flattening the spectrum to form a new input , expressed as in , Indicates all The series connection; Represents flattening and reshaping; Step 3.2, modeling frequency information: right Downsample and divide it into two parts: low frequency signal and high frequency signals , where k represents the signal dimension after downsampling; The low frequency signal and high frequency signals Input them into two multi-head self-attention MHSAs respectively, and concatenate their outputs to restore the original shape; then use another multi-head self-attention MHSA to coordinate all different frequency bands to form a new signal, which is expressed as ; Will Reshape , C is the number of channels, and then multi-head self-attention MHSA is used again to model the relationship between all blocks. Finally, upsampling is performed to obtain the enhanced frequency signal ; Step 3.3, select the most appropriate frequency characteristics: For the enhanced frequency signal Reshape the dimension to obtain feature D, and calculate the relative importance of each input frequency channel according to formula (4): : in, , is the enhanced frequency signal The result of reshaping the dimension, p is the global average pooling GAP; δ represents the ReLU function, β represents the batch normalization BN; σ is the Sigmoid function; the kernel size of Conv1 is (C / r)×C×1×1, and the kernel size of Conv2 is C ×(C / r) ×1×1, where r is a hyperparameter; C is the number of frequency channels; make For input frequency channel The weights are then used to assign scores is the weight of the cth frequency channel, is the cth frequency channel, is a learnable parameter, then the learning-based frequency selection is defined as: Where ⊙ is the element product, is the weighted frequency channel, and the output of the frequency selection layer is expressed as: ; The output of the frequency selection layer is embedded into the backbone network through the frequency modulation layer. The calculation of the frequency modulation layer is: In the formula, Conv3 represents a 3×3 convolution; Conv4 represents a 1×1 convolution with a kernel size of C×C×1×1, and the output of the frequency modulation layer is ; Indicates frequency modulation operation; Indicates normalization processing; The fourth step is to integrate frequency domain features with image features: The second image feature And the frequency domain features obtained in the third step Connect them and calculate the cross attention weight, the calculation formula is: ; in, and Represent image features and frequency domain characteristics , H and W are the height and width of the corresponding feature map respectively; For the connection operation, Get the cross feature map; and Indicates that two 1×1 convolution operations with different parameters are performed on the cross feature map, and the result is and ; , They are image features and frequency domain characteristics The cross attention weights are + = 1; The image enhancement feature is obtained by the following formula : Step 5: Build the text enhancement module: Step 5.1, Bidirectional LSTM network: The text features obtained in the second step It is sent to the bidirectional LSTM network, and the specific semantic extraction process is as follows: in and g are the activation functions of sigmoid and tanh respectively, Represents the weight parameters of different positions, n is an integer from 1 to 6; is the text vector input into the bidirectional LSTM network at time t, and They represent the previous sequence information obtained by the forward LSTM and the following sequence information obtained by the backward LSTM respectively. The subscript t-1 represents the corresponding information at the previous moment, and the subscript t+1 represents the corresponding information at the next moment. Represents the output of the bidirectional LSTM network; Step 5.2, Attention Layer: The forward and backward outputs generated by the bidirectional LSTM network are concatenated and then transformed into three matrices of the same dimension through linear transformation, namely the value matrix V, the key matrix K, and the query matrix Q. The three matrices are input into the attention layer in turn and the information is jointly represented h times according to formulas (13) and (14). The result of the h-times joint representation of information is transformed into the output text enhancement feature vector through linear transformation. ; in is the dimension value of matrix Q, K, V, represents the calculation result of a single self-attention mechanism, i is an integer from 1 to h; Att represents the operation of the self-attention mechanism; C is the connection function of the matrix concatenation process, is the transformation matrix; Step 6: Align image and text features: The image-to-text contrast loss is calculated using formulas (15) and (16): and contrastive loss for text to image : Where (I, T) represents a set of matching image-text pairs, τ is the temperature hyperparameter, is a set of negative text samples that do not match I and belong to tampered image-text pairs, is the positive text sample that matches I; is a set of negative image samples that do not match T and belong to tampered image-text pairs, is a positive image sample matching T; K is the number of negative samples; is the kth negative text sample that does not match I and belongs to the tampered image-text pair; is the kth negative image sample that does not match T and belongs to the tampered image-text pair; For expectations; in (,) indicates similarity, , and There are two mapping heads, through which the two enhanced modal features are mapped to a low-dimensional space with a dimension of 256 for similarity calculation; Perform intra-modal contrastive learning in both image and text modalities and calculate text-to-text contrastive loss and image-to-image contrast loss ; All contrast losses are combined to form the final contrastive learning loss ; Step 7: Build a multi-cross attention fusion module: Enhance the features of the aligned image and text enhancement features First, input them into a self-attention module to obtain the image enhancement features after self-attention processing. and , and then perform the following cross-attention operation: in, is the output of multiple cross attention fusion modules; represents a cross-attention operation; Step 8: Obtain the tampered image and text detection and positioning results: Will Input into the multi-task tampering detection and positioning head to obtain the detection result, where the constraint loss of each task is: Formula (22) is the image localization loss, where express The features include the components of image features, represents a bounding box detector, Represents the true bounding box position, and the image localization loss is calculated by combining the commonly used L1 loss and the generalized IoU loss; Formula (23) is the text localization loss, where, express The features include the components of text features, represents a text label detector, represents the true text label; Formula (24) is the multi-classification detection loss, where represent The component that contains classification information in the feature, represents a multi-label classifier, represents the true multi-classification label value; formula (25) is the binary classification detection loss, represents a two-label classifier, Represents the true binary classification label value; represents the expected value of sampling image I and text T on distribution p.

Images