Image tampering positioning method based on multi-source unsupervised domain generalization

Abstract

The invention discloses an image tampering positioning method based on multi-source unsupervised domain generalization. The method comprises the following steps: extracting domain specific features on each domain by adopting a domain encoder with an edge feature enhancement module; then, a domain alignment strategy is adopted, distribution of domain specific features is aligned by minimizing a loss function based on manual design and based on optimizable distance measurement, and therefore a domain invariant subspace is obtained to reduce domain offset of an unknown target domain. According to the method, the dependence of the model on marked data can be reduced, and the generalization ability on unknown tampering types is improved.

Description

Technical Field

[0001] This invention belongs to the technical field of image recognition technology, specifically relating to an image tampering localization method based on multi-source unsupervised domain generalization. Background Technology

[0002] Traditional image tampering primarily targets natural images, and the types of tampering are wide-ranging, mainly including copy-paste, splicing, erasing, and restoration. These techniques are used to alter image content, aiming to change the original meaning for deceptive or misleading purposes. Current image tampering localization technologies mainly include methods based on traditional feature engineering and methods based on deep learning. Traditional algorithms are characterized by manually extracting features, building models, analyzing features, and classifying them. However, traditional feature engineering-based methods suffer from limited ability to identify different types of tampering and poor model robustness. Deep learning-based methods mainly extract features using deep neural network models and achieve end-to-end adaptive learning through a Softmax classifier, which improves the accuracy of the model to a certain extent. Most mainstream tampering localization methods are based on deep learning models for pixel-level classification to identify tampered regions. For example, they use model weights pre-trained on ImageNet to initialize the encoder for tampering feature extraction. Existing technical solutions mostly use supervised learning paradigms based on large amounts of labeled data to capture more effective tampering features to improve the performance of image tampering localization. They are usually built on a deep learning-based semantic segmentation framework, mainly composed of an encoder and a decoder. The encoder uses specific filters to extract spatial or frequency domain features of the image, and then the decoder and the classifier at the end of the network generate pixel-level prediction results. Although existing image tampering localization methods have performed well on multiple public datasets, they are still insufficient for large-scale deployment. This is mainly due to their shortcomings in generalization and robustness. First, most existing methods are data-driven supervised deep learning models, whose performance is heavily dependent on the distribution of the training dataset. When the training data differs from real images in terms of tampering type, image content, or post-processing, the model performance drops sharply. Second, the models have weak generalization ability to new tampering tools and methods not seen during training and cannot effectively identify unknown tampering types. In addition, fake images actually spread on the Internet are usually subjected to lossy compression, scaling, and other post-processing operations, which weaken the traces of tampering and cause the micro-features on which the model depends to become invalid, thus affecting the model's robustness against post-processing operations. Therefore, it is urgent to find ways to reduce the model's dependence on labeled data and improve its generalization ability to unknown tampering types. Summary of the Invention

[0003] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide an image tampering localization method based on multi-source unsupervised domain generalization. Through a multi-source unsupervised domain generalization training strategy, the model's dependence on labeled data is reduced, thereby improving its generalization ability for unknown tampering types.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides an image tampering localization method based on multi-source unsupervised domain generalization, comprising the following steps: Input a multi-source domain dataset, and use a shared domain encoder to extract features from the multi-source domain dataset to obtain specific domain features and edge features; the multi-source domain dataset contains data from multiple source domains, including labeled images and unlabeled images; By utilizing a domain alignment strategy, domain-specific features on each domain are aligned by minimizing a double alignment loss function to obtain a domain-invariant subspace. The double alignment loss function includes both manual and optimizable metrics. In the domain-invariant subspace, a multi-attention fusion module is used to fuse features of a specific domain to obtain fused features. The fused features are then processed using hollow space pyramid pooling to obtain fine-grained localization results.

[0005] As a preferred technical solution, the shared domain encoder includes a backbone network and an edge feature enhancement module. The backbone network includes several parallel convolutional layers and residual units. The edge feature enhancement module includes a squeeze-excitation block, a 1×1 convolution, and upsampling.

[0006] As a preferred technical solution, the parallel convolutional layer includes Bayar convolution and ordinary convolution. The Bayar convolution executes the Bayar initialization strategy to extract noise features of the image, and the ordinary convolution executes the Kaiming initialization strategy to extract spatial features of the image.

[0007] As a preferred technical solution, the residual unit includes a first residual block and a second residual block, which are alternately arranged. The first residual block includes two 3×3 convolutions and one 1×1 convolution. The 3×3 convolutions are connected in series and connected in parallel with the 1×1 convolution in the residual node. The second residual block includes two 3×3 convolutions. The 3×3 convolutions are connected in series and connected in parallel with the input in the residual node. The method of using a shared domain encoder to extract features from a multi-source domain dataset to obtain specific domain features and edge features involves: simultaneously extracting spatial and frequency domain features using parallel convolutional layers, fusing the spatial and frequency domain features to obtain fused features, inputting the fused features into residual units to obtain domain-specific features, and simultaneously inputting the output features of the second residual block into a batch normalization layer, which then passes through a ReLU activation function and an edge feature enhancement module to obtain edge features.

[0008] As a preferred technical solution, the squeeze-excitation block includes two sets of modules. The first set of modules includes a global average pooling, two fully connected layers, and a sigmoid function. The second set of modules includes upsampling, a global average pooling, two fully connected layers, and a sigmoid function. The outputs of the two sets of modules are summed element-wise, and the summed feature is multiplied element-wise with the upsampling result.

[0009] As a preferred technical solution, the manual measurement uses the maximum mean difference, as shown in the following formula: , , in, This represents mapping the features to a Gaussian kernel in the Hilbert space. p (·) represents a max pooling operation with a stride of 4. This represents the features extracted from the labeled image by the shared-domain encoder. This represents the features extracted from the unlabeled image by the shared-domain encoder. This indicates the number of labeled images with labeled source domains. Indicates the first i The number of unlabeled images in each unlabeled source domain. This indicates that there is a tagged source domain and the first... i The maximum mean difference among unlabeled source domains This represents a manual measure based on the largest mean difference.

[0010] As a preferred technical solution, each shared domain encoder is adapted with a gradient inversion layer and a domain classifier, wherein the domain classifier performs a global average pooling operation and has several fully connected layers.

[0011] As a preferred technical solution, the optimizable metric employs adversarial training to learn domain-invariant features. Specifically, it involves assigning domain labels to labeled and unlabeled images, classifying the images using a domain classifier, and calculating the alignment loss. The alignment loss uses a binary cross-entropy loss, as shown in the following formula: , in, L i d_oThe first image represents the difference between labeled and unlabeled images. i Individual domain alignment loss, This represents the total domain alignment loss. This represents the i-th field label. q i,j Let represent the predicted probability of the j-th unlabeled image based on the softmax output of the i-th domain classifier. Indicates the first i The number of unlabeled images in each unlabeled source domain; The gradient reversal layer acts as an identity transformation during forward propagation and flips the sign of the gradient and multiplies it by λ during backward propagation. The domain classifier is then optimized, as shown in the following equation: , in, L t The alignment loss represents the tampering localization between labeled and unlabeled images, where λ represents the balancing weights. C i Indicates the first i A domain classifier, E(·) represents a domain encoder with edge feature enhancement.

[0012] As a preferred technical solution, the multi-attention fusion module includes a channel attention module, a spatial attention module, and a pixel attention module. The three attention modules are connected in parallel. The specific domain features are connected to the three attention modules in a skip connection, respectively, to obtain a channel attention map, a spatial attention map, and a pixel attention map. The channel attention map, the spatial attention map, and the pixel attention map are then concatenated to obtain a fused feature map. The channel attention includes two 3×3 convolutions, global average pooling, and two 1×1 convolutions. The two 3×3 convolutions are concatenated with the two 1×1 convolutions after global average pooling. The second 1×1 convolution is activated by a sigmoid function to obtain a 1×1×256 attention map. The two 3×3 convolutions are concatenated in series using a PReLU activation function, and the two 1×1 convolutions are concatenated in series using a ReLU activation function. The output of the second 3×3 convolution is multiplied pixel-wise with the 1×1×256 attention map. The spatial attention module includes two 3×3 convolutions, channel max pooling, channel average pooling, and a 7×7 convolution. The two 3×3 convolutions are connected in series via the PReLU activation function, and the channel max pooling and channel average pooling are connected in parallel. The output features are fused and input into the 7×7 convolution. The 7×7 convolution is activated by the sigmoid function to obtain a 32×32×1 attention map. The output of the second 3×3 convolution is multiplied pixel by pixel with the 32×32×1 attention map. The pixel attention map includes two 3×3 convolutions and two 1×1 convolutions, which are connected in series. The two 3×3 convolutions are connected in series using the PReLU activation function, and the two 1×1 convolutions are connected in series using the ReLU activation function. The second 1×1 convolution is activated by the sigmoid function to obtain a 32×32×256 attention map. The output of the second 3×3 convolution is multiplied pixel by pixel with the 32×32×256 attention map.

[0013] As a preferred technical solution, the process of processing the fusion features using void space pyramid pooling specifically involves: Fine-grained localization of the fused feature map is performed using hollow spatial pyramid pooling, and pixel-level loss L is calculated for the labeled image LD0. t As shown in the following formula: , in, i Index representing spatial location, N t Indicates the number of elements. This represents the predicted probability of tampered regions of category c using the void space pyramid pooling method. and w c These represent the pixel labels of the tampered region and the weights corresponding to category c, respectively. Calculate edge loss for the labeled image LD0 L e As shown in the following formula: , in, i Index representing spatial location, N t Indicates the number of elements. This represents the predicted probability of tampering edge for category c using pyramid pooling in the void space. and These represent the pixel labels of the tampered edges and the weights corresponding to category c, respectively. Calculate and minimize the joint loss function L Specifically: , Among them, hyperparameters α and β It controls the balance between different tasks. Indicates marginal loss. This represents a manual measure based on the maximum mean difference. This represents the total domain alignment loss.

[0014] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) This invention exhibits good generalization performance for unknown tampering types. This invention proposes a multi-source unsupervised domain generalization network suitable for image tampering localization. Unlike existing supervised learning models based on pixel-level labeled images and traditional unsupervised domain generalization models that use a large amount of unlabeled data for pre-training, this network improves the generalization performance for unknown target domains by aligning the data distributions of one labeled source domain and three unlabeled source domains. Furthermore, this invention uses a shared domain encoder with enhanced edge features across multiple source domains as the feature representation for extracting source domain data, rather than designing a corresponding encoder for each source domain to extract features. To better align the domains, a dual-alignment loss function composed of a hand-designed and optimizable distance metric is proposed to mitigate the domain offset of multiple source domains, thereby extracting domain-invariant features to improve model generalization.

[0015] (2) The tamper location accuracy of the present invention is high. In order to accurately locate tampering, the present invention proposes an edge feature enhancement module to capture statistical inconsistencies on the boundary of the tampered region. Unlike using only feature maps of specific layers of the network as input, the present invention uses squeeze-excitation blocks to compensate for the information loss caused by convolution and pooling operations, combining shallow feature maps with deep feature maps to extract multi-scale and multi-semantic features. It not only utilizes the local edge and texture features extracted by the shallow network, but also captures the global boundary statistical jump effect caused by various tampering operations in the deep network, gradually refining the feature representation, thereby achieving fine localization of the tampering boundary.

[0016] (3) This invention utilizes a double alignment loss function to mitigate domain offset between multiple source domains, thereby obtaining domain-invariant feature representations, which are more suitable for detecting tampered images not visible in the training set. In the prior art, domain generalization is generally performed using manual distance metrics or adversarial training, without combining the two, which makes it impossible to effectively extract domain-invariant feature representations.

[0017] (4) This invention generates multiple attention weight maps by making full use of local and global information, and performs multi-domain feature fusion based on multiple attention mechanisms. Existing technologies usually perform feature fusion under one attention mechanism. This invention combines channel, spatial and pixel attention to generate more discriminative feature representations for tamper localization tasks. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of an image tampering localization method based on multi-source unsupervised domain generalization, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a domain encoder with an edge feature enhancement module according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the domain feature fusion module based on multi-attention according to an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the tampering with the location result in an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0021] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0022] Please see Figure 1 This embodiment provides an image tampering localization method based on multi-source unsupervised domain generalization, including the following steps: S1. Input a multi-source domain dataset, and use a shared domain encoder to extract features from the multi-source domain dataset to obtain specific domain features and edge features; the multi-source domain dataset has data from multiple source domains, including labeled images and unlabeled images.

[0023] Assuming the tampered image in the target domain is completely inaccessible, the multi-source unsupervised domain generalization network extracts domain-invariant feature representations from four source domains, thus achieving better generalization ability for unknown tampered images. Only one source domain contains a labeled image, denoted as LD0. The other three source domains consist of unlabeled images, denoted as UD. i (i=1, 2, 3).

[0024] To better demonstrate its generalization ability on unknown datasets, this embodiment performs cross-dataset tamper localization performance evaluation on 11 datasets. The network model of this invention is trained on four source domains: CASIA2, IMD2020, PSCC, and IFC datasets. CASIA2 is the labeled source domain LD0, while IMD2020, PSCC, and IFC datasets are three unlabeled source domains UD. i The remaining datasets serve as the target domain for testing (i.e., the test set). In the test set, the DSO dataset focuses on detecting stitched images containing people. The Coverage dataset is used to detect copy-and-move operations between multiple similar but real objects. The In-the-Wild dataset includes fake images of real scenes collected from multiple websites, such as fake news. In addition to DSO, Coverage, In-the-Wild, NIST16, and CASIA1, constructed using traditional image editing tools, this invention also evaluates performance on the AutoSplice and CocoGlide datasets, which respectively include locally forged images generated based on large-scale language-image models and diffusion models.

[0025] Since the data distribution of forged images differs across source domains, performing domain alignment by employing independent feature extractors in each source domain is challenging. Therefore, this embodiment proposes a shared domain encoder with an edge feature enhancement module for capturing LD0 and each UD. i Domain-specific tampering effects. The architecture of a shared domain encoder is as follows: Figure 2 As shown. Specifically, to capture more tampering artifacts in the spatial and frequency domains, two parallel convolutional layers are used to extract dual-domain features, where Bayar initialization and Kaiming initialization strategies are applied to the convolutional kernels of Bayar and ordinary convolutions, respectively. The following eight residual modules are used to extract hierarchical features at different scales, generating feature maps of size 32×32×256 for domain alignment and feature fusion. LD0 and UD are then used to extract hierarchical features at different scales. i The image sets contained in (i=1,2,3) are denoted as X. LD0 ={x 1 LD0 , ···, x N0 LD0}∈R N0×D X UDi ={x 1 UD0 , ···, x Ni UD0}∈R Ni×D Among them, x LD0 This represents a fake image with pixel-level labels, x UDiThese are unlabeled tampered images. N and D represent the number of images and the dimension of the image space (spatial resolution is 256×256×3 pixels), respectively.

[0026] To accurately locate tampering, this embodiment designs an edge feature enhancement module (EFE) in the shared domain encoder to capture statistical inconsistencies on the boundaries of the forged region. Three squeeze-excitation blocks (SEBs) are stacked within the EFE, where each SEB modifies the low-level feature map (f... SE ) and high-level feature maps (f 4,6,8 This is achieved by combining multiple semantic features, such as edges and textures, extracted from shallow to deep layers via EFE. For example, SEB_1 tends to combine local appearance features extracted by residual units 2 and 4. Tamper boundary transformation effects caused by complex tampering operations can be captured by SEB_3. Therefore, EFE iteratively refines the feature representation and integrates multiple semantic features to achieve fine-grained localization of tamper boundaries.

[0027] To more clearly explain the domain encoder with edge feature enhancement module, this embodiment provides the following description. For example... Figure 2 As shown, the shared domain encoder includes a backbone network and an edge feature enhancement module. The backbone network includes several parallel convolutional layers and residual units. The edge feature enhancement module includes squeeze-excitation blocks (SEB), 1×1 convolutions, and upsampling.

[0028] The parallel convolutional layer includes Bayar convolution and ordinary convolution. The Bayar convolution executes the Bayar initialization strategy to extract noise features of the image, and the ordinary convolution executes the Kaiming initialization strategy to extract spatial features of the image.

[0029] The structure of each residual unit and squeeze-excitation block is shown in the dashed box. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. In each SEB, a global average pooling (GAP) operation is followed by two fully connected layers (FC×2) and a sigmoid function. ⊗ and ⊕ represent element-wise multiplication and summation, respectively. The output feature map of each residual unit is represented as height × width × channels. Residual units 3, 5, and 7 use convolutional kernels with a stride of 2 to perform downsampling.

[0030] S2. Using a domain alignment strategy, domain-specific features on each domain are aligned by minimizing the double alignment loss function to obtain a domain-invariant subspace. The double alignment loss function includes a manual metric and an optimizable metric.

[0031] See here. Figure 1 To improve generalization performance in unknown target domains, the key idea of ​​multi-source unsupervised domain generalization networks lies in integrating LD0 with each UD.i The distributions are aligned to a common subspace in which the distributions of all source domains become similar.

[0032] This embodiment proposes a double-alignment loss function, combined with a manual metric based on maximum mean difference (MMD). L d_h And optimizable metrics based on domain adversarial training L d_o For LD0 and each UD i The images in (i=1,2,3) are used to extract features using the domain encoder described above, denoted as f. LD0 and f UDi Then L d_h The calculation is as follows: , , Where ∅(∙) and p(·) represent the Gaussian kernel that maps features to the Hilbert space and the max pooling operation with a stride of 4, respectively.

[0033] Despite L d_h This is an effective direct domain alignment method, but it depends on the heuristic choice of the kernel function and tends to align first-order statistics, i.e., mean differences, and therefore may not fully capture higher-order differences between domains. To address this issue, this invention utilizes L-type adversarial training-based... d_o To learn domain-invariant features to mitigate more complex domain offsets, such as nonlinear transformations or higher-order statistical properties. Specifically, for LD0 and each UD i A gradient inversion layer and a corresponding domain classifier C are used. i This is based on implicit execution domain alignment using adversarial training. For each C... i , let l i =0 and l i =1 are assigned to LD0 and UD respectively. i The domain labels. Then, C is implemented using global average pooling operations and three fully connected layers. i To predict l i The gradient inversion layer acts as an identity transformation during forward propagation and flips the sign of the gradient and multiplies it by λ during backward propagation. Therefore, C i Optimization can be effectively achieved in the following ways: , Among them, L t and L i d_o Representing LD0 and UD respectively iThe loss between tamper localization and domain alignment is calculated. λ controls the balance between these two tasks, and E represents the domain encoder with edge feature enhancement.

[0034] Calculate each L using binary cross-entropy loss i d_o And combine them to get L d_o As shown below: , in q i,j This represents the predicted probability of the softmax output of the i-th domain classifier for the j-th unlabeled image. i The domain classifier aims to predict l i E(·) tends to generate those that cannot be generated by C. i Distinguishing features, thus generating domain-invariant feature representations. Therefore, utilizing L d_h and L d_o By leveraging the respective advantages of each, the proposed dual alignment loss helps mitigate the complex domain shift effect in multi-source domains.

[0035] S3. In the domain-invariant subspace, the multi-attention fusion module is used to fuse features of a specific domain to obtain fused features. The fused features are then processed using hollow space pyramid pooling to obtain fine-grained localization results.

[0036] To generate more discriminative tamper location feature representations, this invention fully utilizes local and global information and proposes a domain feature fusion module based on multi-attention. This embodiment uses labeled source domain features f with a size of 32×32×256. LD0 For input, we capture channel, spatial, and pixel attention using three parallel attention blocks with skip connections. For the channel attention block (CAB), global pooling is applied to generate a 1×1×256 attention map to characterize the correlations between channels. In the spatial attention block (SAB), channel averaging and max pooling are used to highlight f using a 32×32×1 attention map. LD0 The salient region within each channel. Pixel attention blocks (PABs) generate a 32×32×256 fine-grained attention map, refining f. LD0 This allows for the capture of local details. In this way, attention features from CAB, SAB, and PAB are concatenated to generate fused features. Then, Spatial Pyramid Pooling with Hollows (ASPP) is used to combine multi-scale features to generate fine-grained localization results.

[0037] To clearly illustrate this step of multi-attention-based domain feature fusion, this embodiment describes the multi-attention-based domain feature fusion module in more detail. For example... Figure 3As shown, the multi-attention fusion module includes a channel attention module, a spatial attention module, and a pixel attention module. The three attention modules are connected in parallel. The specific domain features are connected to the three attention modules in a skip connection, respectively, to obtain the channel attention map, spatial attention map, and pixel attention map. The channel attention map, spatial attention map, and pixel attention map are then concatenated to obtain the fused feature map. The channel attention includes two 3×3 convolutions, global average pooling, and two 1×1 convolutions. The two 3×3 convolutions are concatenated with the two 1×1 convolutions after global average pooling. The second 1×1 convolution is activated by a sigmoid function to obtain a 1×1×256 attention map. The two 3×3 convolutions are concatenated in series using a PReLU activation function, and the two 1×1 convolutions are concatenated in series using a ReLU activation function. The output of the second 3×3 convolution is multiplied pixel-wise with the 1×1×256 attention map. The spatial attention module includes two 3×3 convolutions, channel max pooling, channel average pooling, and a 7×7 convolution. The two 3×3 convolutions are connected in series via the PReLU activation function, and the channel max pooling and channel average pooling are connected in parallel. The output features are fused and input into the 7×7 convolution. The 7×7 convolution is activated by the sigmoid function to obtain a 32×32×1 attention map. The output of the second 3×3 convolution is multiplied pixel by pixel with the 32×32×1 attention map. The pixel attention map includes two 3×3 convolutions and two 1×1 convolutions, which are connected in series. The two 3×3 convolutions are connected in series using the PReLU activation function, and the two 1×1 convolutions are connected in series using the ReLU activation function. The second 1×1 convolution is activated by the sigmoid function to obtain a 32×32×256 attention map. The output of the second 3×3 convolution is multiplied pixel by pixel with the 32×32×256 attention map.

[0038] Next, Spatial Pyramid Pooling with Dirt (ASPP) is used, combined with multi-scale features, to generate fine-grained localization results. To achieve pixel-level classification, L t The following calculation is performed within the labeled source domain LD0: , in i It is an index of spatial location, N t p is the number of elements. c This represents the probability predicted by ASPP for pixels in the tampered region of category c. c and w c These represent the pixel label of the tampered area and the weight corresponding to category c, respectively.

[0039] Simultaneously, the edge loss L is calculated for the labeled image LD0. e As shown in the following formula: , in, i Index representing spatial location, N t Indicates the number of elements. This represents the predicted probability of tampering edge for category c using pyramid pooling in the void space. and These represent the pixel labels of the tampered edges and the weights corresponding to category c, respectively. Finally, we minimize the joint loss function as follows: , Among them, hyperparameters α and β It controls the balance between different tasks.

[0040] To better demonstrate the beneficial effects, this embodiment utilizes PyTorch to implement a multi-source unsupervised domain generalization network model and performs training and testing on an RTX 3090 GPU. Image patches of size 256×256×3 are extracted from the CASIA2 dataset, including 20% ​​to 80% fake pixels, resulting in 22,600 labeled training samples. For the IMD2020, PSCC, and IFC datasets, unlabeled fake image patches are extracted from each dataset, with the same sample size and number as the labeled samples. Then, the multi-source unsupervised domain generalization network is trained using the Adam optimizer, with a mini-batch size of 16. The initial learning rate is set to 0.0001, and the weight decay during training is fixed at 2×10⁻⁶. −4 .

[0041] As shown in Tables 1, 2, and 3, this embodiment is compared with several mainstream tampering localization methods using F1, MCC, and AUC (Area Under the ROC Curve) metrics. The average performance across all datasets is calculated in the last column, and the ranking of each method on each dataset is indicated in parentheses. The deep learning-based methods used for performance comparison include ReLoc, SSJT, MVSS, PSCC, and IF-OSN. ReLoc proposes an image tampering localization-assisted recovery framework to recover high-quality copies from distorted tampered images and re-enhance tampering traces. SSJT proposes a JPEG compression trace extractor to capture JPEG compression effects caused by different JPEG compression chains. MVSS proposes a multi-view, multi-scale supervised network that jointly utilizes the statistical properties of tampering boundaries and noisy views of the input image for multi-view feature learning and is trained using multi-scale supervised signals. PSCC proposes a dual-branch tampering localization network that uses a top-down branch to extract local and global features and a bottom-up branch to detect whether the input image has been tampered with. IF-OSN decouples transport noise in social networks into predictable and invisible noise patterns, and models these two parts separately in the training framework, thereby improving model robustness. For a fair comparison, all the above methods were tested using their open-source pre-trained models.

[0042] As can be observed from Tables 1 to 3, this invention achieves the best average performance (last column) and ranks among the top three on every dataset except CASIA1 and Coverage. Notably, this invention significantly outperforms other methods on the DSO (PNG images) and AutoSplice (JPEG images) datasets. Compared to other datasets, the DSO and AutoSplice datasets contain large-size stitched images generated by traditional image editing tools and novel language image models, respectively. The stitching traces in such images are often difficult to capture using other methods. In contrast, this invention improves the network model's generalization ability to tampered images not visible in the training set by integrating different forensic traces from multiple source domains.

[0043] Table 1. Comparison of positioning performance of this embodiment with other methods on the F1 metric. Table 2. Comparison of positioning performance of this embodiment with other methods on the MCC index. Table 3. Comparison of localization performance of this embodiment with other methods in terms of AUC index. at last, Figure 4A heatmap showing the tamper location results for different methods across all test datasets is presented. From Figure 4 As can be seen, this embodiment can more accurately locate the tampered area. From top to bottom, each row represents the input tampered image, label image, and tamper location heatmap for different methods. In the label image, white and black correspond to tampered and real pixels, respectively.

[0044] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0045] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for locating image tampering based on multi-source unsupervised domain generalization, characterized in that, Includes the following steps: Input a multi-source domain dataset, and use a shared domain encoder to extract features from the multi-source domain dataset to obtain specific domain features and edge features; The multi-source domain dataset contains data from multiple source domains, including labeled images and unlabeled images; By utilizing a domain alignment strategy, domain-specific features on each domain are aligned by minimizing a double alignment loss function to obtain a domain-invariant subspace. The double alignment loss function includes both manual and optimizable metrics. In the domain-invariant subspace, a multi-attention fusion module is used to fuse features of a specific domain to obtain fused features. The fused features are then processed using hollow space pyramid pooling to obtain fine-grained localization results.

2. The image tampering localization method based on multi-source unsupervised domain generalization according to claim 1, characterized in that, The shared domain encoder includes a backbone network and an edge feature enhancement module. The backbone network includes several parallel convolutional layers and residual units. The edge feature enhancement module includes a squeeze-excitation block, a 1×1 convolution, and upsampling.

3. The image tampering localization method based on multi-source unsupervised domain generalization according to claim 2, characterized in that, The parallel convolutional layer includes Bayar convolution and ordinary convolution. The Bayar convolution executes the Bayar initialization strategy to extract noise features of the image, and the ordinary convolution executes the Kaiming initialization strategy to extract spatial features of the image.

4. The image tampering localization method based on multi-source unsupervised domain generalization according to claim 2, characterized in that, The residual unit includes a first residual block and a second residual block, which are alternately arranged. The first residual block includes two 3×3 convolutions and one 1×1 convolution. The 3×3 convolutions are connected in series and connected in parallel with the 1×1 convolution in the residual node. The second residual block includes two 3×3 convolutions. The 3×3 convolutions are connected in series and connected in parallel with the input in the residual node. The method of using a shared domain encoder to extract features from a multi-source domain dataset to obtain specific domain features and edge features involves: simultaneously extracting spatial and frequency domain features using parallel convolutional layers, fusing the spatial and frequency domain features to obtain fused features, inputting the fused features into residual units to obtain domain-specific features, and simultaneously inputting the output features of the second residual block into a batch normalization layer, which then passes through a ReLU activation function and an edge feature enhancement module to obtain edge features.

5. The image tampering localization method based on multi-source unsupervised domain generalization according to claim 2, characterized in that, The squeeze-excitation block includes two sets of modules. The first set of modules includes a global average pooling, two fully connected layers, and a sigmoid function. The second set of modules includes upsampling, a global average pooling, two fully connected layers, and a sigmoid function. The outputs of the two sets of modules are summed element-wise, and the summed feature is multiplied element-wise with the upsampling result.

6. The image tampering localization method based on multi-source unsupervised domain generalization according to claim 1, characterized in that, The manual measurement uses the maximum mean difference, as shown in the following formula: ， ， in, This represents mapping the features to a Gaussian kernel in the Hilbert space. p (·) represents a max pooling operation with a stride of 4. This represents the features extracted from the labeled image by the shared-domain encoder. This represents the features extracted from the unlabeled image by the shared-domain encoder. This indicates the number of labeled images with labeled source domains. Indicates the first i The number of unlabeled images in each unlabeled source domain. This indicates that there is a tagged source domain and the first... i The maximum mean difference among unlabeled source domains This represents a manual measure based on the largest mean difference.

7. The image tampering localization method based on multi-source unsupervised domain generalization according to claim 1, characterized in that, Each shared domain encoder is adapted with a gradient inversion layer and a domain classifier, which includes a global average pooling operation and several fully connected layers.

8. The image tampering localization method based on multi-source unsupervised domain generalization according to claim 7, characterized in that, The optimizable metric employs adversarial training to learn domain-invariant features. Specifically, it assigns domain labels to labeled and unlabeled images, classifies the images using a domain classifier, and calculates the alignment loss. The alignment loss uses a binary cross-entropy loss, as shown in the following formula: ， in, L i d_o The first image represents the difference between labeled and unlabeled images. i Individual domain alignment loss, This represents the total domain alignment loss. This represents the i-th field label. q i,j Let represent the predicted probability of the j-th unlabeled image based on the softmax output of the i-th domain classifier. Indicates the first i The number of unlabeled images in each unlabeled source domain; The gradient reversal layer acts as an identity transformation during forward propagation and flips the sign of the gradient and multiplies it by λ during backward propagation. The domain classifier is then optimized, as shown in the following equation: ， in, L t The alignment loss represents the tampering localization between labeled and unlabeled images, where λ represents the balancing weights. C i Indicates the first i A domain classifier, E(·) represents a domain encoder with edge feature enhancement.

9. The image tampering localization method based on multi-source unsupervised domain generalization according to claim 1, characterized in that, The multi-attention fusion module includes a channel attention module, a spatial attention module, and a pixel attention module. The three attention modules are connected in parallel. The specific domain features are connected to the three attention modules in a skip connection, respectively, to obtain a channel attention map, a spatial attention map, and a pixel attention map. The channel attention map, the spatial attention map, and the pixel attention map are then concatenated to obtain a fused feature map. The channel attention includes two 3×3 convolutions, global average pooling, and two 1×1 convolutions. The two 3×3 convolutions are concatenated with the two 1×1 convolutions after global average pooling. The second 1×1 convolution is activated by a sigmoid function to obtain a 1×1×256 attention map. The two 3×3 convolutions are concatenated in series using a PReLU activation function, and the two 1×1 convolutions are concatenated in series using a ReLU activation function. The output of the second 3×3 convolution is multiplied pixel-wise with the 1×1×256 attention map. The spatial attention module includes two 3×3 convolutions, channel max pooling, channel average pooling, and a 7×7 convolution. The two 3×3 convolutions are connected in series via the PReLU activation function, and the channel max pooling and channel average pooling are connected in parallel. The output features are fused and input into the 7×7 convolution. The 7×7 convolution is activated by the sigmoid function to obtain a 32×32×1 attention map. The output of the second 3×3 convolution is multiplied pixel by pixel with the 32×32×1 attention map. The pixel attention map includes two 3×3 convolutions and two 1×1 convolutions, which are connected in series. The two 3×3 convolutions are connected in series using the PReLU activation function, and the two 1×1 convolutions are connected in series using the ReLU activation function. The second 1×1 convolution is activated by the sigmoid function to obtain a 32×32×256 attention map. The output of the second 3×3 convolution is multiplied pixel by pixel with the 32×32×256 attention map.

10. The image tampering localization method based on multi-source unsupervised domain generalization according to claim 9, characterized in that, The process of using void space pyramid pooling to process fused features is as follows: Fine-grained localization of the fused feature map is performed using hollow spatial pyramid pooling, and pixel-level loss L is calculated for the labeled image LD0. t As shown in the following formula: ， in, i Index representing spatial location, N t Indicates the number of elements. This represents the predicted probability of tampered regions of category c using the void space pyramid pooling method. and w c These represent the pixel labels of the tampered region and the weights corresponding to category c, respectively. Calculate edge loss for the labeled image LD0 L e As shown in the following formula: ， in, i Index representing spatial location, N t Indicates the number of elements. This represents the predicted probability of tampering edge for category c using pyramid pooling in the void space. and These represent the pixel labels of the tampered edges and the weights corresponding to category c, respectively. Calculate and minimize the joint loss function L Specifically: ， Among them, hyperparameters α and β It controls the balance between different tasks. Indicates marginal loss. This represents a manual measure based on the maximum mean difference. This represents the total domain alignment loss.

Images