A Weakly Supervised Image Tampering and Forgery Detection Method Based on Mask Consistency

Abstract

This invention proposes a weakly supervised image tampering and forgery detection method based on mask consistency. This method uses SRM and Bayar filters to obtain noise textures in tampered and forged images, employs a self-attention mechanism to establish global dependencies, and obtains better feature representations of tampered and forged images. Furthermore, it introduces a mask consistency mechanism to overcome the problem of insufficient effective supervision information in weakly supervised learning. The method first extracts features from the image from multiple perspectives, obtaining information from different viewpoints through SRM and Bayar filters; it then further supervises the information in different regions through global consistency learning; it independently extracts features based on the input information from different viewpoints, and predicts all pixels in the image, including the mask, from different viewpoints. Pseudo-labels are obtained through ensemble voting as pixel-level predictions; finally, image-level predictions are obtained through adaptive pooling.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image tampering and forgery detection, specifically a weakly supervised image tampering and forgery detection method based on mask consistency. Background Technology

[0002] With the maturity and application of deep learning technology, more and more people can tamper with and forge images at a relatively low cost, which has had a certain impact on people's normal lives. Existing research on tampering and forgery detection has achieved some success. Most studies rely on supervised learning methods, which require a large dataset of tampered images and corresponding precise annotations of the forged regions. These pixel-level annotations are very expensive, time-consuming, and prone to errors. With the maturity of technologies such as GANs and Diffusion, new tampering and forgery techniques have emerged, making it even more difficult to obtain pixel-level masks. Secondly, some existing studies have focused too much on identifying tampered regions, neglecting the comprehensive features of the overall image, leading to increased false positives and misclassification of many real images.

[0003] The differences between the existing technologies are as follows:

[0004] In comparison with the technology of patent CN116452857A, "A Weakly Supervised Remote Sensing Target Detection Method Based on Feature Consistency and Location Update",

[0005] 1. Patent CN116452857A discloses a weakly supervised automatic multi-class image segmentation method, belonging to the field of image segmentation technology. For training images containing only sparse doodle annotations, this invention proposes a three-branch multi-scale network. It obtains robust soft pseudo-labels by fusing predictions from different receptive fields and performs uncertainty correction on the fused soft pseudo-labels. Three different decoders are supervised to improve the performance of the segmentation network. Simultaneously, a class similarity matrix is ​​calculated for the prediction results of each decoder branch, and consistency constraints are applied to fully utilize the inter-class relationship information of the unlabeled data. In the prediction stage, a robust segmentation result is obtained through weighted fusion of multiple prediction results.

[0006] 2. This patent relates to a weakly supervised image tampering and forgery detection method based on mask consistency. This method uses SRM filters and Bayar filters to obtain noise textures in tampered and forged images, employs a self-attention mechanism to establish global dependencies, and obtains better feature representations of tampered and forged images. Furthermore, a mask consistency mechanism is introduced to overcome the problem of insufficient effective supervision information in weakly supervised learning. The three-branch network in this method performs multi-view feature extraction by fusing information from three different perspectives and frequency domains. Global consistency learning further supervises information in different regions; feature extraction is performed independently based on input information from different perspectives, and predictions are made for all pixels in the image, including the mask, from different perspectives. Pseudo-labels are obtained through ensemble voting as pixel-level predictions; finally, image-level predictions are obtained through adaptive pooling.

[0007] 3. In patent CN116452857A, a dataset of images with graffiti annotations is used during training. Graffiti labels require annotators to draw one or two curves for each target category; bounding box annotations outline the approximate location of the target object without providing detailed internal information; point annotations simply draw points on the target object, providing rough location information; image-level labels only provide image-level labels to determine whether a target object exists in an image, lacking information on the target's location and size. To fully utilize the graffiti annotation information, a partial cross-entropy loss function is designed, which calculates partial supervised loss only on the labeled pixel set.

[0008] 4. This patent uses an image dataset containing image-level labels. The lack of target location and size information in real tampered / forged images is a common problem in this field. Therefore, how to utilize the limited image-level label information for weakly supervised learning is an effective approach in tampering / forgery detection. This invention introduces three independent training models to learn from RGB, SRM, and Bayar images respectively, predicting all image regions, including the mask. An integrated voting module is used to vote pixel-by-pixel according to a fixed voting ratio. The integrated voting result is used as a pseudo-label in weakly supervised learning, and multi-source consistency loss is calculated using the prediction results from the three backbone networks from different perspectives to constrain the voting in the mask region in different flows. Finally, the pseudo-label is used to predict the image-level label, calculating the image-level supervised loss. For the lack of mask-level supervision information, prediction supervision is provided for the predictions of each source flow through mask consistency.

[0009] In comparison with the technology of patent CN116912184A, "A Weakly Supervised Deep Inpainting Image Tamper Location Method and System Based on Tamper Region Separation and Region Constraint Loss",

[0010] 1. Patent CN116912184A discloses a weakly supervised deep image restoration tamper localization method and system based on tamper region separation and region constraint loss. This invention mainly targets image restoration detection and localization technology. The key technical points of the invention include: acquiring pre-training set image data and training set image data; preprocessing the image data; inputting the preprocessed pre-training set image data into a neural network-based pre-trained detection model for pre-training to obtain trained pre-trained model parameters; using the pre-trained model parameters as initial values ​​for network parameters, and inputting the preprocessed training set image data into a neural network-based weakly supervised tamper localization model for training to obtain a trained weakly supervised tamper localization model. This model focuses more on tamper localization of restored images and provides more detailed classification and labeling of restored images.

[0011] 2. This patent relates to a weakly supervised image tampering and forgery detection method based on mask consistency. This method uses SRM filters and Bayar filters to obtain noise textures in tampered and forged images, employs a self-attention mechanism to establish global dependencies, and obtains better feature representations of tampered and forged images. Furthermore, it introduces a mask consistency mechanism to overcome the problem of insufficient effective supervision information in weakly supervised learning. The three-branch network in this method performs multi-view feature extraction by fusing information from three different perspectives and frequency domains. Global consistency learning further supervises information in different regions; feature extraction is performed independently based on input information from different perspectives, and predictions are made for all pixels in the image, including the mask, from different perspectives. Pseudo-labels are obtained through ensemble voting as pixel-level predictions; finally, image-level predictions are obtained through adaptive pooling. This method is applicable to the detection of tampered images with various tampering methods, is not limited to a single tampering method, and requires no additional image preprocessing.

[0012] 3. In patent CN116912184A, image preprocessing refers to using a steganalysis rich model to convert the real image or the deep-restoration tampered image from the RGB color space to a noise space, obtaining the corresponding noise image; then, the real image or deep-restoration tampered image and its corresponding noise image are concatenated. The concatenated image and the noise image are input into a neural network to obtain the image's depth features; a convolutional layer reduces the number of channels corresponding to the depth features to 2; features of category 1 are selected as the initial class activation map for the tampering category. Features of the tampered region and non-tampered region in the deep-restoration tampered image are extracted, and image-level label loss function and region constraint loss function are calculated.

[0013] 4. In this patent, image preprocessing refers to using SRM and Bayar models to preprocess the image separately, and then independently training the three source streams along with the RGB image. For image depth features, a region consistency module is used to supervise learning different regions' features using shallow features from the model and pseudo-labels obtained through combined voting by the three independent models. Furthermore, during training, masking is used to expand the model's receptive field, and pseudo-labels are obtained through the combined prediction results of the three source streams, thereby constraining the prediction results of different models. Based on this, a composite loss function is designed, including region consistency loss, mask-based multi-source consistency loss, and image classification loss. Summary of the Invention

[0014] To address the aforementioned technical problems, this invention proposes a weakly supervised image tampering and forgery detection method based on mask consistency. This method fully integrates multi-view information of tampered and forged images and provides additional supervision through mask consistency, enabling the prediction of tampered areas in tampered and forged images using only image labels.

[0015] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0016] A weakly supervised image tampering and forgery detection method based on mask consistency includes a pre-processing stage, a training stage, and a prediction stage, as follows:

[0017] 1) Preliminary stage:

[0018] The input tampered and forged image is uniformly divided into regions and blocks. Using a predefined fixed ratio parameter, the blocks of the tampered and forged image are randomly selected proportionally. The selected regions are then masked to obtain the masked RGB image. The SRM filter and Bayar filter are used to extract features from the masked RGB image to obtain the SRM image and Bayar image. The two images and the masked RGB image are used as the three input streams in the training phase.

[0019] 2) Training phase:

[0020] Mask consistency refers to the process of training a model by introducing three independent training models to learn from RGB, SRM, and Bayar images respectively, predicting all image regions, including the mask. An ensemble voting module votes on each pixel according to a fixed voting ratio. The ensemble voting results are used as pseudo-labels in weakly supervised learning. These pseudo-labels are then used to calculate multi-source consistency loss with the predictions from the three backbone networks from different perspectives. This constrains the voting in the mask region in different streams, ultimately causing the three independent training models to tend to be consistent in their image predictions, including the mask region. The region consistency learning module divides the image into different regions and predicts the consistency between different regions. This prediction result is then used to calculate the region consistency loss with the pseudo-labels obtained from the final ensemble voting.

[0021] 3) Prediction phase:

[0022] After the training phase is completed, the pseudo-labels obtained through the ensemble voting phase are used as the final pixel-level prediction results. The pseudo-labels are then calculated using adaptive pooling to obtain the final image-level prediction results. The classification loss is calculated together with the provided image-level pseudo-labels, and together with the region consistency loss and multi-source consistency loss, it constitutes the final overall loss.

[0023] Furthermore, step 1) the preparatory stage specifically includes the following steps:

[0024] Step 1.1: Mask the input tampered image to form a masked image. Consider an original image as input. The size is represented as H×W, where H and W represent the height and width of the image, respectively. The image is divided into a series of square image blocks of size a×a, where a is a predefined block side length, satisfying the conditions H mod a = 0 and W mod a = 0, ensuring that the image size is divisible by the block size. Let N = H / a and M = W / a, representing the number of blocks that can be divided in the vertical and horizontal directions, respectively, where N and M are both integers.

[0025] Step 1.2: For each region P in the image i,j Where i∈{0,1,…,N-1} and j∈{0,1,…,M-1}, it is expressed as follows: P i,j =I[ai:a(i+1),aj:a(j+1)], where the image block P is located. i,j Determined by index (i,j), a portion of these blocks is randomly selected for masking according to a given ratio r;

[0026] Step 1.3: Let Φ and Ψ represent the encoding of the SRM filter and the Bayar filter, respectively, and then... mask SRM encoding is performed separately for Φ(X)mask ) and Ψ(X is processed by Bayar coding mask ), the obtained Φ(X mask ), Ψ(X mask ), X mask are used as different view representations of the input image X and input into the model, and are trained independently.

[0027] Further, in said pre-stage step 1.2:

[0028] dividing the tampered and forged image into regions, selecting partial regions according to a set proportion, and performing masking processing on these regions:

[0029] X mask =RandomMask(X S , r·|N×M|)

[0030] wherein X S is a set representation of the input image X after being divided into image blocks, r, 0<r<1 is the proportion of regions selected for masking processing, |N×M| is the total number of image blocks, X mask is an image obtained by masking randomly selected image blocks according to the proportion. Further, considering that actual tampered and forged regions are mostly distributed in the middle area of the image, no masking operation is performed on the outermost image blocks during random masking.

[0031] Further, the training stage in step 2) specifically comprises the following steps:

[0032] Step 2.1: subjecting the obtained Φ(X mask ), Ψ(X mask ), X mask to feature extraction to obtain tampered and forged image features under different views;

[0033] Step 2.2: extracting and fusing image information through a multi-layer perceptron embedded with depthwise separable convolution and a self-attention fusion network, and performing long-range modeling on the image information;

[0034] Step 2.3: upsampling the features through a C1 decoder and predicting all pixel-level masks of tampered and forged images including the masked region;

[0035] Step 2.4: performing pixel-by-pixel integrated voting on the results of the tampered and forged prediction streams from three different views according to proportions to obtain a final pixel-level prediction result;

[0036] Step 2.5: using the final pixel-level prediction result as a pseudo label, supervising the prediction results of the three source streams, calculating quality parameters of the pseudo label and calculating multi-source consistency loss;

[0037] Step 2.6: Calculate the consistency volume by performing region consistency calculation on the shallow features of the model, and use pseudo-labels for supervision. Calculate the quality parameters of the pseudo-labels and the region consistency loss.

[0038] Step 2.7: Perform adaptive pooling based on the pseudo-labels to obtain the final image-level prediction result, and calculate the image-level loss with the provided image and label;

[0039] Step 2.8: Backpropagate the network based on the errors calculated in Steps 2.5, 2.6 and 2.7, and repeat the training until the network converges.

[0040] Furthermore, in step 2.2 of the training phase:

[0041] Using 1x1 convolution as the basic unit for convolution and self-attention computation, the results of 3x3 convolution are simulated through shift and combination operations. At the same time, 1x1 convolution also serves as the basic Q, K, and V units for self-attention computation.

[0042] Furthermore, in step 2.5 of the training phase:

[0043] Mask consistency for tampered and forged images refers to learning and predicting images containing masks from different viewpoints using three independent models. The prediction results from different viewpoints are then integrated through ensemble voting, and the three independent models undergo supervised learning. Ultimately, the three models will predict the region consistency of the mask. The voting results from the three different source streams are as follows:

[0044] M R,S,B =f R,S,B (X mask )

[0045] The result of the integrated voting is used as the pseudo-label P. label The calculated quality parameters will be used as adjustment parameters for the loss function:

[0046]

[0047] Where the threshold τ is a pre-set parameter, the BCE loss is calculated as follows:

[0048]

[0049] Furthermore, in step 2.6 of the training phase:

[0050] By combining masks to further distinguish different regions of the image and calculating the similarity between all regions, the final consistency volume is obtained. The image consistency volume is calculated by calculating pseudo-labels and then compared with the image consistency volume calculated by shallow features. BCE loss is calculated, and a quality adjustment parameter q is introduced. tThe loss function used to adjust the learning of regional consistency;

[0051]

[0052]

[0053] Where the threshold τ is a preset parameter, V h,w,j,k For shallow layer features, the region consistency volume The region consistency volume for pseudo-labels.

[0054] Furthermore, in step 2.7 of the training phase:

[0055] In the absence of effective supervision using pixel-level masks, a threshold ω0 for binary classification of image pixels is determined by minimizing the intra-class prediction variance. Based on this threshold ω0, the average pixel response score of high-response points is calculated to predict the image's class. The algorithm for minimizing the intra-class prediction variance yields the appropriate threshold ω0 using the following formula:

[0056]

[0057] in This represents the response value of pixel (i,j) in the pseudo-label. The obtained threshold ω0 divides the pixels of the pseudo-label into two classes, where... This represents the pixel value with a high response value. Image-level prediction. Represented as:

[0058]

[0059] Where |P h | represents the number of points in the set of high response value points, and ∑ represents the cumulative set P. h The average of all high response values ​​above the threshold ω0 is obtained. The provided labels are used when calculating the loss. As a supervised learning method, the loss is ultimately calculated:

[0060]

[0061] Furthermore, in step 2.8 of the training phase:

[0062] The total loss is a weighted sum of multi-source consistency loss, region consistency loss, and image classification loss:

[0063]

[0064] Where α and β are the weighting coefficients for different loss terms.

[0065] Compared with the prior art, the significant advantages of this invention are:

[0066] The method of this invention can perform mask-level prediction of tampered and forged regions by applying only image-level labels, thus eliminating the dependence of image tampering and forgery detection on pixel-level masks; it provides effective contextual information through mask consistency, enabling the model to obtain effective contextual information even under weak supervision, and has good generalization ability; the neural network used has a small number of parameters, and 6D pose estimation can be performed on ordinary computers. Attached Figure Description

[0067] Figure 1 This is a flowchart of the image feature acquisition process for the weakly supervised tampering and forgery detection model of the present invention;

[0068] Figure 2 This is a flowchart of the multi-source consistency process in the weakly supervised tampering and forgery detection model of the present invention;

[0069] Figure 3 This is an overall flowchart of the weakly supervised tampering and forgery detection model of the present invention. Detailed Implementation

[0070] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0071] Example: The weakly supervised image tampering and forgery detection method based on mask consistency in this example is a tampering and forgery detection method based on image feature information.

[0072] The method in this embodiment mainly includes a pre-process, a training process, and a prediction process.

[0073] (I) Pre-construction stage

[0074] The input image is masked to form the masked image. Consider an original image as input. The size is represented as H×W, where H and W represent the height and width of the image, respectively. The image is divided into a series of square image blocks of size a×a, where a is a predefined block side length, satisfying the conditions H mod a = 0 and W mod a = 0, ensuring that the image size is divisible by the block size. Let N = H / a and M = W / a, representing the number of blocks that can be divided in the vertical and horizontal directions, respectively, where N and M are integers. For each region P in the image... i,j Where i∈{0,1,…,N-1} and j∈{0,1,…,M-1}, it can be expressed as follows: P i,j =I[ai:a(i+1),aj:a(j+1)], where the image block P is located. i,jDetermined by index (i,j). A portion of these blocks is randomly selected for masking according to a given ratio r. Let Φ and Ψ represent the encoding of the SRM filter and the Bayar filter, respectively. For X mask SRM encoding is performed separately for Φ(X) mask ) and Bayar encoding processing Ψ(X mask The obtained Φ(X) mask ), Ψ(X) mask ), X mask The input model is represented by different perspectives of the input image X and trained independently.

[0075] (II) Training Phase

[0076] 1. Multi-source consistency module:

[0077] like Figure 1 As shown, the obtained Φ(Xmask), Ψ(Xmask), and Xmask are used for feature extraction to obtain tampered and forged image features from different perspectives. In the backbone network, a multilayer perceptron with embedded deep separable convolutions and a self-attention fusion network are used to extract and fuse image information, performing long-range modeling of the image information to obtain the final features. The models from three different perspectives use a C1 decoder to upsample the features and predict all pixel-level masks for tampering and forging, including the mask region. The prediction stream results of the three different perspectives on the tampered and forged image are integrated and voted pixel-by-pixel according to a fixed ratio to obtain the final pixel-level prediction result. The final pixel-level prediction result is used as a pseudo-label to supervise the prediction results of the three source streams and effectively control them using the quality parameters of the pseudo-label. The quality parameters of the pseudo-label are calculated as follows:

[0078]

[0079] The threshold τ is a pre-set parameter, and the calculated quality parameters will be used as adjustment parameters for the loss function. Calculate the multi-source consistency loss:

[0080]

[0081] 2. Regional Consistency Module:

[0082] Region consistency is calculated on the input image, and the consistency volume is obtained by performing region consistency calculation on the shallow features of the model. Supervision is performed by calculating the region consistency volume of the pseudo-labels, and the quality parameter of the pseudo-labels is calculated as a parameter to adjust the loss.

[0083]

[0084] Calculate the region consistency loss:

[0085]

[0086] Where the threshold τ is a preset parameter, V h,w,j,k For shallow layer features, the region consistency volume The region consistency volume for pseudo-labels.

[0087] 3. Image Prediction Module:

[0088] Adaptive pooling is applied to the pseudo-labels, and a suitable threshold ω0 is calculated using an algorithm that minimizes the intra-class prediction variance to perform binary classification of the image.

[0089]

[0090] in This represents the response value of pixel (i,j) in the pseudo-label. The obtained threshold ω0 divides the pixels of the pseudo-label into two classes, where... This represents the pixel value with a high response value. Image-level prediction. Represented as:

[0091]

[0092] Where |P h | represents the number of points in the set of high response value points, and ∑ represents the cumulative set P. h The average of all high response values ​​above the threshold ω0 is obtained. The provided labels are used when calculating the loss. As a supervised learning method, the loss is ultimately calculated:

[0093]

[0094] 4. Overall loss function:

[0095] The overall framework diagram of the model is as follows Figure 2 As shown. The total loss is a weighted sum of the multi-source consistency loss, region consistency loss, and image classification loss:

[0096]

[0097] Where α and β are the weighting coefficients for different loss terms.

[0098] (III) Forecasting Phase

[0099] like Figure 3 As shown, during the prediction phase, assuming all networks have been trained, inputting an RGB image will yield image-level predictions and mask-level localization of the tampered regions. The specific steps of this phase are as follows:

[0100] Step 1: Process the input image using SRM and Bayar filters to obtain the SRM-processed image and the Bayar-processed image of the target image.

[0101] Step 2: Perform independent inference on the RGB image, SRM-processed image, and Bayar-processed image using separate backbone networks.

[0102] Step 3: Integrate and vote on the pixel-level predictions obtained from the three models, and obtain the final pixel-level voting result through a fixed ratio coefficient, which will be used as the pixel-level prediction of the tampered and forged region of the image.

[0103] Step 4: Perform adaptive pooling on the obtained pixel-level predictions to determine the optimal threshold ω0 for binary classification of the image.

[0104] Step 5: Determine the pixel portion with high response value based on the optimal threshold ω0, and calculate the average response value of this portion as the image-level prediction result.

[0105] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A weakly supervised image tampering and forgery detection method based on mask consistency, comprising a pre-processing stage, a training stage, and a prediction stage, characterized in that, The stages are as follows: 1) Preliminary stage: The input tampered and forged image is uniformly divided into regions and blocks. Using a predefined fixed ratio parameter, the blocks of the tampered and forged image are randomly selected proportionally. The selected regions are then masked to obtain the masked RGB image. The SRM filter and Bayar filter are used to extract features from the masked RGB image to obtain the SRM image and Bayar image. The two images and the masked RGB image are used as the three input streams in the training phase. 2) Training phase: Mask consistency refers to the process of training a model by introducing three independent training models to learn from RGB, SRM, and Bayar images respectively, predicting all image regions, including the mask. An ensemble voting module votes on each pixel according to a fixed voting ratio. The ensemble voting results are used as pseudo-labels in weakly supervised learning. These pseudo-labels are then used to calculate multi-source consistency loss with the predictions from the three backbone networks from different perspectives. This constrains the voting in the mask region in different streams, ultimately causing the three independent training models to tend to be consistent in their image predictions, including the mask region. The region consistency learning module divides the image into different regions and predicts the consistency between different regions. This prediction result is then used to calculate the region consistency loss with the pseudo-labels obtained from the final ensemble voting. 3) Prediction phase: After the training phase is completed, the pseudo-labels obtained through the ensemble voting phase are used as the final pixel-level prediction results. The pseudo-labels are then calculated using adaptive pooling to obtain the final image-level prediction results. The classification loss is calculated together with the provided image-level pseudo-labels, and together with the region consistency loss and multi-source consistency loss, it constitutes the final overall loss.

2. The weakly supervised image tampering and forgery detection method based on mask consistency according to claim 1, characterized in that: Step 1) The preparatory stage specifically includes the following steps: Step 1.1: Mask the input tampered image to form a masked image. Consider an original image as input. The size is represented as H×W, where H and W represent the height and width of the image, respectively. The image is divided into a series of square image blocks of size a×a, where a is a predefined block side length, satisfying the conditions H mod a = 0 and W mod a = 0, ensuring that the image size is divisible by the block size. Let N = H / a and M = W / a, representing the number of blocks that can be divided in the vertical and horizontal directions, respectively, where N and M are both integers. Step 1.2: For each region P in the image i,j Where i∈{0,1,…,N-1} and j∈{0,1,…,M-1}, it is expressed as follows: P i,j =I[ai:a(i+1),aj:a(j+1)], where the image block P is located. i,j Determined by index (i,j), a portion of these blocks is randomly selected for masking according to a given ratio r; Step 1.3: Let Φ and Ψ represent the encoding of the SRM filter and the Bayar filter, respectively, and then... mask SRM encoding is performed separately for Φ(X) mask ) and Bayar encoding processing Ψ(X mask ), the obtained Φ(X mask ), Ψ(X) mask ), X mask The input model is represented by different perspectives of the input image X and trained independently.

3. The weakly supervised image tampering and forgery detection method based on mask consistency according to claim 2, characterized in that: In step 1.2 of the pre-stage: The tampered and forged image is divided into regions, and selected regions are masked according to a set ratio. X mask =RandomMask(X S ,r·|N×M|) wherein X S is a set representation after dividing the input image X into image blocks, r, 0 < r < 1 is the proportion of selected image blocks for masking processing, |N×M| is the total number of image blocks, X mask is an image obtained by masking randomly selected image blocks according to the proportion; further, considering that actual tampered and forged regions are mostly distributed in the middle area of the image, the outermost image blocks are not subjected to masking operation in the random masking process.

4. The weakly supervised image tampering and forgery detection method based on mask consistency according to claim 1, characterized in that: Step 2) The training phase specifically includes the following steps: Step 2.1: Calculate the obtained Φ(X) mask ), Ψ(X) mask ), X mask Feature extraction is performed to obtain the features of tampered and forged images from different perspectives; Step 2.2: Extract and fuse image information by embedding a multilayer perceptron with depthwise separable convolutions and a self-attention fusion network, and perform long-range modeling of the image information; Step 2.3: Upsample the features using the C1 decoder and predict all pixel-level masks for tampering and forgery, including the masked region; Step 2.4: Perform pixel-by-pixel integration voting on the three different perspectives of tampered and forged prediction streams according to their proportions to obtain the final pixel-level prediction result; Step 2.5: Use the final pixel-level prediction results as pseudo-labels to supervise the prediction results of the three source streams, calculate the quality parameters of the pseudo-labels and calculate the multi-source consistency loss. Step 2.6: Calculate the consistency volume by performing region consistency calculation on the shallow features of the model, and use pseudo-labels for supervision. Calculate the quality parameters of the pseudo-labels and the region consistency loss. Step 2.7: Perform adaptive pooling based on the pseudo-labels to obtain the final image-level prediction result, and calculate the image-level loss with the provided image and label; Step 2.8: Backpropagate the network based on the errors calculated in Steps 2.5, 2.6 and 2.7, and repeat the training until the network converges.

5. The weakly supervised image tampering and forgery detection method based on mask consistency according to claim 4, characterized in that: In step 2.2 of the training phase: Using 1x1 convolution as the basic unit for convolution and self-attention computation, the results of 3x3 convolution are simulated through shift and combination operations. At the same time, 1x1 convolution also serves as the basic Q, K, and V units for self-attention computation.

6. The weakly supervised image tampering and forgery detection method based on mask consistency according to claim 4, characterized in that: In step 2.5 of the training phase: Mask consistency for tampered and forged images refers to learning and predicting images containing masks from different viewpoints using three independent models. The prediction results from different viewpoints are then integrated through ensemble voting, and the three independent models undergo supervised learning. Ultimately, the three models will predict the region consistency of the mask. The voting results from the three different source streams are as follows: M R,S,B =f R,S,B (X mask ) The result of the integrated voting is used as the pseudo-label P. label The calculated quality parameters will be used as adjustment parameters for the loss function: Where the threshold τ is a pre-set parameter, the BCE loss is calculated as follows:

7. The weakly supervised image tampering and forgery detection method based on mask consistency according to claim 4, characterized in that: In step 2.6 of the training phase: By combining masks to further distinguish different regions of the image and calculating the similarity between all regions, the final consistency volume is obtained. The image consistency volume is calculated by calculating pseudo-labels and then compared with the image consistency volume calculated by shallow features. BCE loss is calculated, and a quality adjustment parameter q is introduced. t The loss function used to adjust the learning of regional consistency; Where the threshold τ is a preset parameter, V h,w,j,k For shallow layer features, the region consistency volume The region consistency volume for pseudo-labels.

8. The weakly supervised image tampering and forgery detection method based on mask consistency according to claim 4, characterized in that: In step 2.7 of the training phase: In the absence of effective supervision using pixel-level masks, a threshold ω0 for binary classification of image pixels is determined by minimizing the intra-class prediction variance. Based on this threshold ω0, the average pixel response score of high-response points is calculated to predict the image's class. The algorithm for minimizing the intra-class prediction variance yields the appropriate threshold ω0 using the following formula: in This represents the response value of pixel (i,j) in the pseudo-label. The obtained threshold ω0 divides the pixels of the pseudo-label into two classes, where... This represents the pixel value with a high response value. Image-level prediction. Represented as: Where |P h | represents the number of points in the set of high response value points, and ∑ represents the cumulative set P. h The provided labels are used to calculate the loss by averaging all high response values ​​above the threshold ω0. As a supervised learning method, the loss is ultimately calculated:

9. The weakly supervised image tampering and forgery detection method based on mask consistency according to claim 4, characterized in that: In step 2.8 of the training phase: The total loss is a weighted sum of multi-source consistency loss, region consistency loss, and image classification loss: Where α and β are the weighting coefficients for different loss terms.

Images