A system and method for detecting copy-paste tampering of an image

Abstract

The application discloses a system and method for detecting image copy-paste tampering, which comprises the following steps: calculating the similarity vector between image regions; dividing the image into three different matching modes according to the relationship between the regions and their matching regions; sorting the similarity vectors of all regions in descending order and recording the positions of their matching regions; fusing the similarity vectors of different regions according to the matching position relationship to obtain a fused similarity matrix; extracting an attention map from the fused similarity matrix and modulating a feature map; splicing the feature map and the fused similarity matrix to obtain a fused feature map; inputting the fused feature map into a tampering detector to extract a copy-paste detection mask; splicing the feature map after double-mask separation into a new feature in the channel dimension, and inputting the spliced feature into a source-target locator to obtain a source-target positioning mask; adopting a cross-entropy loss function tampering detection network; and inputting a single image into the trained tampering detection network to obtain a detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing and deep learning, and particularly relates to an image copy-paste tampering detection method based on information fusion and separation. BACKGROUND

[0002] The prevalence of multi-functional image editing software makes image falsification easier than ever. With the help of these tools, users can easily tamper with images without leaving any detectable traces. As one of the most important information carriers, images play a dominant role in news reporting, judicial evidence, and academic research. Image tampering greatly undermines the credibility of digital images, and malicious tampering distorts the facts conveyed by the original image. With the help of social networks, tampered images can spread wildly on the Internet and mislead the public.

[0003] To address people's concerns about the credibility of digital images, researchers have adopted image analysis and signal processing methods to detect tampering traces. Image forensics targets a variety of tampering behaviors, including splicing, removal, and copy-paste, among others. This paper focuses on the detection of copy-paste tampering. Copy-paste tampering is one of the most commonly used and simplest tampering operations, which mainly involves copying an area in an image and pasting it to another area in the same image to hide the objects in the target area or copy the objects in the source area. The starting area that is copied is usually referred to as the source, and the target area where the paste is pasted is referred to as the destination. For copy-paste behavior, the source and destination must come from the same image, ensuring that the destination blends into the background and is not easily detected, increasing the difficulty of detection.

[0004] Copy-paste tampering detection usually involves two tasks:

[0005] 1. Tampering detection: detecting the tampered area and the original area in the image.

[0006] 2. Source-destination localization: further locating the source area and the destination area in the tampered area.

[0007] Traditional image copy-paste tampering detection algorithms include two stages: feature extraction and feature matching. Feature extraction can be divided into regional feature extraction and key point feature extraction, and feature matching mainly uses global matching methods and key point matching methods. Regional feature extraction methods include Discrete Wavelet Transformation (DWT) and Discrete Cosine Transformation (DCT), among others. After dividing the image into regions, two-dimensional DWT coefficients are calculated and global matching is performed, which can detect copy-paste regions, but DWT features are easily affected by rotation and scaling, so this method is not robust to geometric transformations [1] . Kumar et al.[2] A copy-paste tampering detection method based on DCT is proposed. The DCT coefficients are robust to Gaussian noise and JPEG compression, but sensitive to transformations such as rotation and scaling. Keypoint features include scale-invariant features. [3] (Scale-Invariant Feature Transformation, SIFT), Fast Stable Features [4] (Speeded Up Robust Features, SURF), etc. SIFT has stable performance, as noted by Amerini et al. [5] SIFT is used as the feature in the detection algorithm, and a novel keypoint matching module is proposed, which can achieve multi-target tampering detection. Since SIFT has a high computational cost, SURF features, which are computationally faster, are obtained by simultaneously using dimensionality-reduced keypoint features to calculate the integral image on the Hessian matrix. Using SURF for copy-paste tampering detection, the results show that this method is more robust to rotation and blur transformations than SIFT. [6] .

[0008] The deep learning-based copy-paste tampering detection method can be simplified into three basic modules: 1) Feature extraction module: using a neural network to extract image features; 2) Similarity calculation module: calculating the similarity between features in different regions; 3) Copy-paste tampering detection module: using a neural network to detect copied and pasted regions from features and similarity. (Wu et al.) [7] BusterNet, the first copy-paste tampering detection network with source-target localization capability, is proposed. BusterNet contains two branches: the upper branch is used to detect the copy-paste region, and the lower branch is used to detect the target region. The two branches cooperate to achieve source-target localization. However, because the network only uses a single scale to extract image features, the localization accuracy of BusterNet is not high. In the literature [8], Chen et al. introduced dilated convolution into BusterNet to address the shortcomings of BusterNet and used a spatial attention structure to improve the source-target localization capability of the network. DOA-GAN [9] Spatial attention maps are extracted from the similarity matrix, and the similarity matrix is ​​normalized to obtain channel attention maps. These dual attention maps are then used to improve the accuracy of copy-paste tampering detection. AR-NET

[10] Introducing the residual module into copy-paste tampering detection helps the network capture contextual information.

[0009] Mainstream algorithms have achieved good results in copy-paste detection, but existing detection algorithms still have the following problems:

[0010] 1. In the actual copy-paste process, in order to ensure that the target area is fully integrated into the background, a large number of geometric transformations (rotation, ...) are performed.

[0011] Scaling, flipping, etc., are applied to the target. These transformations weaken the similarity between matching regions, and because existing algorithms are sensitive to geometric transformations, detected tampered regions may be missing.

[0012] 2. In source-target localization tasks, the source region is more difficult to detect than the target region because its content is consistent with the background. Therefore, the accuracy of mainstream algorithms in locating the source region is far lower than that in locating the target region. Summary of the Invention

[0013] To address the sensitivity of existing tamper detection technologies to geometric transformations, this invention fuses similarity information with location information. The resulting fused features are robust to geometric transformations such as rotation and scaling, thereby improving the robustness of the detection algorithm. Furthermore, to address the issue of poor accuracy in detecting the source region in source-to-target localization tasks, this invention uses masks of the tampered region's interior and edges to separate the features of the corresponding region and then concatenates them into new features along the channel dimension, thereby improving the network's accuracy in locating the source region.

[0014] The present invention is implemented using the following technical solution:

[0015] A system for detecting image copy-paste tampering, the system being based on a deep neural network; the network includes a detection branch and a localization branch; the detection branch comprises a feature extraction module, a similarity calculation module, an information fusion module, a spatial attention module, and a tampering detector; the localization branch comprises a mask generation module, a region separation module, and a source-destination locator; wherein:

[0016] The feature extraction module extracts feature maps F∈R from the image to be detected. h×w×d ;

[0017] The similarity calculation module calculates the similarity between a single region and all regions using the following formula to obtain the similarity vectors of the regions, which form a similarity matrix S.

[0018]

[0019] in: It is the identifier after F is reduced in dimension. express Modulo; S∈R hw×hw Each row represents a single region in the image, and each column stores the similarity between that region and all regions.

[0020] The information fusion module divides paired regions into three matching modes based on the pairing relationship between regions in the image and their matching regions.

[0021] The information fusion module obtains the matching positional relationship between the sorted similarity matrix and the corresponding original position matrix by descending the similarity vectors of all regions using the following formula;

[0022] V,I = SortColumn top-k (S)

[0023] Where: SartColumn(.) is the function to sort in descending column order, that is, to sort the similarity vector of each region; top-k represents the top k maximum values ​​of each column of the sorted region, and S is the similarity matrix before sorting; V∈R hw×k Where h and w are the number of regions in the vertical and horizontal directions of the image, and R represents the set of real numbers; I∈R hw Record the position of the maximum value in each row of the matrix, representing the matching region position for each region;

[0024] The information fusion module obtains the fusion similarity matrix S′ by fusing similarity vectors from different regions based on matching positional relationships;

[0025] The spatial attention module extracts the attention map modulation feature F′ and concatenates it with the fusion similarity matrix S′ in the channel dimension to obtain the first fusion feature map F. c ,Right now:

[0026] F c =cat(F′,S′)

[0027] The tampering detector processes the first fused feature map F using the following formula. c Calculations are performed to obtain the copy-paste area mask M. d ;

[0028] M d =argmax(upsample(P d ))

[0029] Among them: Copy and paste detection probability map P d P d ∈[0,1] h×w×2 The number of channels is 2;

[0030] The mask generation module generates the mask based on the fused feature map F. c Extracting similar region masks and calculating the tampered region mask M K and the tampering area edge mask M e ;

[0031] The region separation module uses the following formula to separate image features F′ and the tampered region mask M. K and the tampering area edge mask M e

[0032] The second fusion feature F″ is obtained by concatenating the data along the channel dimension.

[0033] F″=cat(F′·M e ,F′·M K )

[0034] The source / destination locator obtains the source / destination localization mask M from the second fusion feature using the following formula. l Specifically:

[0035] M l =argmax(upsample(P l ))

[0036] Where: P l It uses three cascaded convolutional layers to extract the source-destination probability map from the second fusion feature.

[0037] This invention can also be implemented using the following technical solution, including the following steps:

[0038] Feature maps are extracted from the detected image, and the similarity between the features of a single region and all regions is calculated to form a similarity vector; the similarity vectors of all regions are then combined to form a similarity matrix.

[0039] Based on the pairing relationship between regions in the image and their matching regions, paired regions are divided into three different matching modes;

[0040] Sort the similarity vectors of all regions in descending order and record the positions of their matching regions;

[0041] A fused similarity matrix is ​​obtained by fusing the similarity vectors of different regions based on the matching positional relationships;

[0042] An attention map is extracted from the fusion similarity matrix, and the attention map is used to modulate features. The modulated features and the fusion similarity matrix are concatenated to obtain a first fusion feature map. The first fusion feature map is then fed into a tampering detector to extract the copy-paste detection mask M. d ;

[0043] Similar region masks are extracted from the feature maps, and tampered region masks and tampered region edge masks are further generated. After separating the corresponding region feature maps using dual masks, they are stitched together in the channel dimension to form a second fused feature. The stitched second fused feature is then fed into the source-destination locator to obtain the source-destination localization mask M. l ;

[0044] The source-destination probability map P is extracted from the second fused feature using three cascaded convolutional layers. l :

[0045] P l=softmax((ReLu(Conv2D) 3×3 (F″)))3)

[0046] The kernel size of each layer is 3×3, the activation function is ReLU, and the activation function of the last layer is softmax.

[0047] P l Upsample to the original size and use the argmax function to obtain a three-valued source-destination localization mask:

[0048] M l =argmax(upsample(P l ));

[0049] A deep neural network for tamper detection is employed using the cross-entropy loss function.

[0050] A single image containing copied and pasted text is input into a pre-trained deep neural network for tamper detection. The copied and pasted region mask is obtained from the tamper detector, and the source-destination localization mask is obtained from the source-destination locator.

[0051] Furthermore, the information fusion module fuses similarity vectors from different regions based on matching positional relationships to obtain a fused similarity matrix S′; including the following steps:

[0052] Generate a matrix of all zeros with the same size as the similarity matrix V.

[0053] Based on the location of the matching region provided by I, the similarity vectors corresponding to all regions in V are assigned to the appropriate values. In its matching position:

[0054]

[0055] Wherein, the similarity matrix V∈R hw×k Each row represents one region, with a total of hw regions. Each column represents the similarity score of the k most similar regions to that region; m∈[1,...,hw] represents the index of each region, and I[m] represents the position of the best-matching region in V for the m-th region. When a region in the image is the best-matching region for another region, ... The corresponding row in the matrix stores the similarity vector of the matched region; otherwise, it remains filled with zeros. The zero line in the China-Africa section can be considered a suspected copy-paste area; It contains not only similarity information but also matching location information, and is called the recombination matrix of V;

[0056] Will Adding it to V yields the fused similarity matrix V1; then, the similarity information is fused with the location information once:

[0057]

[0058] Repeat the above fusion operation n times for V. After n fusions, a similarity matrix set {V1, V2, V3, ..., V} is obtained. n-1 V n In this set, the adjacent similarity matrix V j-1 V j The following conditions must be met:

[0059]

[0060]

[0061] Where m∈[0,...,hw], represents the index of each region; j∈[1,...,n], This represents the similarity matrix obtained after j-1 fusion processes. Represent its recombination matrix;

[0062] The fusion similarity matrix S′ is obtained by adding the n similarity matrices in the similarity matrix set.

[0063]

[0064] Among them, V j Let S′ ∈ R be the similarity matrix after j fusions. hw×k This represents the similarity matrix obtained by adding the similarity matrices obtained after n fusions, which fuses position and similarity information.

[0065] The process involves extracting an attention map from the fusion similarity matrix and modulating the feature map using the attention map; concatenating the feature map with the fusion similarity matrix to obtain a fusion feature map; and feeding the fusion feature map into a tampering detector to extract a copy-paste detection mask, including the following steps:

[0066] A spatial attention module is constructed using a convolutional network. This module takes S′ as input and uses sigmoid as the activation function to obtain a spatial attention map.

[0067] A = attention(S′)

[0068] After the sigmoid activation function, A∈[0,1] h×w The obtained attention map A is multiplied by the image feature F to retain important information in the image, resulting in the modulated image feature F′:

[0069] F′=F·A

[0070] The copy-paste tampering detection module consists of three cascaded 3×3 convolutional layers, each followed by a BatchNorm layer. Except for the last convolutional layer, which uses the softmax activation function, the remaining layers use the ReLU activation function.

[0071] The image features F′ and the fusion similarity matrix S′ are concatenated along the channel dimension to obtain the fusion features F. c :

[0072] F c =cat(F′,S′)

[0073] F c It contains important image information, similarity information, and location information. Using this as input to the tamper detector, we obtain the copy-paste detection probability map P. d P d ∈[0,1] h×w×2 The number of channels is 2, and the two channels represent the probability that the region is a copied-paste region or the original background region, respectively. Let P... d After upsampling to the original size and extracting the channel with the highest probability, the copy-paste region mask M is obtained. d

[0074] M d =argmax(upsample(P d ))

[0075] Where upsample represents an 8x upsampling bilinear interpolation function; argmax is the function that takes the channel with the maximum value. M d ∈{0,1} H×W .

[0076] Furthermore, similar region masks are extracted from the feature maps, and tampered region masks and tampered region edge masks are generated. After separating the corresponding region feature maps using dual masks, they are stitched together in the channel dimension to form new features. The stitched features are then fed into the source-destination locator to obtain the source-destination localization mask. This process includes the following steps:

[0077] F will be integrated c The input mask is fed into the input mask extractor to obtain the copy-paste detection probability map P∈[0,1]. H×W×2 P is fed into the argmax function to obtain the copy-paste detection mask M:

[0078] M = argmax(P)

[0079] The `argmax` function is used to obtain the channel containing the maximum value for each pixel. For a 2-channel probability map P, M∈{0,1} H×WM takes the value 0 or 1, indicating whether each pixel belongs to the background or the tampered area, respectively, where H and W are the height and width of the image, respectively.

[0080] To obtain a mask that indicates the exact location of the tampered area, M is dilated to obtain the dilated mask M. K :

[0081]

[0082] in: The dilation operation is represented by kernel K, with a window size of 3×3. The parameters of this kernel are fixed at all 1s. A Boolean operation (bool) is performed on the dilated mask to convert all positive values ​​to 1, resulting in a binary mask M. K Compared to M, M K The edges of the regions in the image are expanded, making the tampered regions they indicate more complete.

[0083] Use M K The edge mask M is obtained by subtracting M from M. e :

[0084] M e =M K -M

[0085] M e That is, a mask used to indicate the location of the edge of the tampered area;

[0086] The second fused feature F″ is formed by separating the corresponding region feature maps using a dual-mask method and then stitching them together along the channel dimension.

[0087] F″=cat(F′·M e ,F′·M K )

[0088] Where: the mask is multiplied by the image feature F′, and M is... e M is used to extract features of the edges of the tampered region. K Used to extract features within the tampered area.

[0089] The spliced ​​second fusion feature is fed into the source-destination locator to obtain the source-destination locator mask M. l Specifically:

[0090] First, three cascaded convolutional layers are used to extract the source-destination probability map P from the second fused feature. l :

[0091] P l =softmax((ReLu(Conv2D) 3×3 (F″)))3)

[0092] The convolutional kernels in each layer are 3×3, the activation function is ReLU, and the activation function for the last layer is softmax. Then, P... l Upsample to the original size and use the argmax function to obtain a three-valued source-destination mask image:

[0093] M l =argmax(upsample(P l )).

[0094] Beneficial effects

[0095] 1. This invention fuses similarity information with location information, which has the beneficial effect of improving the algorithm's robustness to geometric transformations such as rotation and scaling.

[0096] 2. This invention uses masks inside and at the edges of the tampered region to separate the features of the corresponding region and then splices them together into new features in the channel dimension. Its beneficial effect is to improve the network's positioning accuracy of the source region. Attached Figure Description

[0097] Figure 1 A flowchart of an image copy-paste tampering detection method based on information fusion and separation;

[0098] Figure 2 A schematic diagram of the overall network structure for testing;

[0099] Figure 3 This is a schematic diagram of the location and similarity information fusion module;

[0100] Figure 4 Diagrams illustrating various matching patterns;

[0101] Figure 5 The original image and the mask image of the copied and pasted area are used in the experimental results.

[0102] Figure 6 This is a schematic diagram of the mask generation and region separation module;

[0103] Figure 7 The original image and source / destination positioning mask are copied and pasted into the experimental results. Detailed Implementation

[0104] To make the objectives, technical solutions, and advantages of this invention clearer, the following description is provided in conjunction with the appendix. Figures 1-7 The embodiments of the present invention will be described in further detail below.

[0105] This invention provides a system and method for detecting image copy-paste tampering. The invention uses a deep neural network for image copy-paste tampering detection and source / destination localization. The network includes a detection branch and a localization branch. The detection branch includes a feature extraction module, a similarity calculation module, an information fusion module, a spatial attention module, and a tampering detector. The localization branch includes a mask generation and region separation module and a source / destination locator. This invention requires no image preprocessing or post-processing and can directly detect the copy-paste region and locate the source / destination region from a single image. Specifically, it includes the following steps:

[0106] Extract feature maps from the image, calculate the similarity between the features of a single region and all regions, and form a similarity vector; then combine the similarity vectors of all regions into a similarity matrix.

[0107] Based on the pairing relationship between regions in the image and their matching regions, paired regions are divided into three different matching modes;

[0108] Sort the similarity vectors of all regions in descending order and record the positions of their matching regions;

[0109] By fusing similarity vectors from different regions based on matching positional relationships, the matching pattern is enhanced to obtain a fused similarity matrix:

[0110] An attention map is extracted from the fusion similarity matrix, and the attention map is used to modulate the feature map. The feature map and the fusion similarity matrix are concatenated to obtain a fusion feature map. The fusion feature map is then fed into the tamper detector to extract the copy-paste detection mask.

[0111] Similar region masks are extracted from the feature map, and tampered region masks and tampered region edge masks are further generated. After separating the corresponding region feature maps using dual masks, they are spliced ​​into new features in the channel dimension. The spliced ​​features are then sent to the source-destination locator to obtain the source-destination positioning mask.

[0112] A tamper detection network is adopted using the cross-entropy loss function.

[0113] A single image with copy and paste is input into a trained deep neural network for tamper detection. The copy and paste region mask is obtained from the tamper detector, and the source and destination location mask is obtained from the source and destination locator.

[0114] The process of extracting feature maps from the image, calculating the similarity between features of a single region and all regions to form a similarity vector, and then constructing a similarity matrix from the similarity vectors of all regions is specifically as follows:

[0115] Input image to be detected X∈R H×W×3 This will be converted into a grayscale image G∈R H×WThe descriptor extraction module of the SuperPoint network

[11] is used to extract descriptor features F∈R from different regions of the image. h×w×d ,in, d is the length of the descriptor. To reduce the computational cost of subsequent similarity calculations, a pooling layer is added to SuperPoint, so the final extracted feature map F is 1 / 8 the size of the original image.

[0116] The similarity between descriptor features of different regions of an image is calculated to obtain a similarity matrix S. In this embodiment of the invention, cosine similarity is used as the similarity metric, and the similarity matrix is ​​calculated as follows:

[0117]

[0118] in It is the identifier after F is reduced in dimension. express Modulo. S∈R hw×hw Each row represents a single region in the image, and each column stores the similarity between that region and all other regions.

[0119] The specific steps of sorting the similarity vectors of all regions in descending order and recording the positions of their matching regions are as follows:

[0120] Sort the similarity matrix in descending order to obtain the sorted similarity matrix and the corresponding original position matrix:

[0121] V, I = SortColumn top-k (S)

[0122] SortColumn(.) is a function that sorts the similarity vectors of each region in descending order. top-k represents the top k maximum values ​​in each column of the sorted region; therefore, after sorting, V∈R hw×k I∈R hw The position of the maximum value in each row of the matrix is ​​recorded, representing the matching region position of each region. Therefore, V contains rich similarity information, and I contains rich matching position information.

[0123] The process of fusing similarity vectors from different regions based on matching positional relationships to enhance the matching pattern and obtain a fused similarity matrix is ​​as follows:

[0124] First, generate a matrix of all zeros with the same size as the similarity matrix V. Then, based on the location of the matching region provided by I, the similarity vectors corresponding to all regions in V are assigned to the appropriate values. In its matching position:

[0125]

[0126] Wherein, the similarity matrix V∈R hw×k Each row represents one region, with a total of hw regions. Each column represents the similarity score of the k most similar regions to that region. m∈[1,...,hw] represents the index of each region, and I[m] represents the position of the best-matching region in V for the m-th region. When a region in the image is the best-matching region for another region, ... The corresponding row in the matrix stores the similarity vector of the matched region; otherwise, it remains filled with zeros. Therefore, the matrix... The zero line in the China-Africa section can be considered a suspected copy-paste area. It contains not only similarity information but also matching location information, and is called the recombination matrix of V.

[0127] Then Adding this to V yields the fused similarity matrix V1. The similarity information is then fused with the location information once.

[0128]

[0129] After one fusion, the similarity of the bidirectional matching regions in the fusion matrix V1 is effectively enhanced. This enhancement of the matching regions can effectively improve the network's robustness to geometric transformations.

[0130] Repeat the above fusion operation n times for V. After n fusions, a similarity matrix set {V1, V2, V3, ..., V} is obtained. n-1 V n In this set, the adjacent similarity matrix V j-1 V j The following conditions must be met:

[0131]

[0132]

[0133] Where m∈[0,...,hw], represents the index of each region. j∈[1,...,n], This represents the similarity matrix obtained after j-1 fusion processes. This represents its recombination matrix.

[0134] To increase the stability of the similarity matrix, the n similarity matrices in the similarity matrix set are added together to obtain the fused similarity matrix S′.

[0135]

[0136] Among them, V j Let S′ ∈ R be the similarity matrix after j fusions.hw×k This represents the similarity matrix obtained by adding the similarity matrices obtained after n fusions, which fuses position and similarity information.

[0137] The process of extracting an attention map from the fusion similarity matrix and modulating the feature map using the attention map, concatenating the feature map and the fusion similarity matrix to obtain a fusion feature map, and then feeding the fusion feature map into the tampering detector to extract the copy-paste detection mask involves the following steps:

[0138] Attention map A is extracted from the fusion similarity matrix S′, and feature map F is modulated using the attention map. The feature map and the fusion similarity matrix are concatenated to obtain the fusion feature map. The fusion feature map is then fed into the tamper detector to extract the copy-paste detection mask M. d .

[0139] A spatial attention module is constructed using a convolutional network. This module takes S′ as input and uses sigmoid as the activation function to obtain a spatial attention map.

[0140] A = attention(S′)

[0141] After the sigmoid activation function, A∈[0,1] h×w The obtained attention map A is multiplied by the image feature F to retain important information in the image, resulting in the modulated image feature F′:

[0142] F′=F·A

[0143] The copy-paste tampering detection module consists of three cascaded 3×3 convolutional layers, each followed by a BatchNorm layer. Except for the last convolutional layer, which uses the softmax activation function, the remaining layers use the ReLU activation function.

[0144] The image feature F′ is concatenated with the fusion similarity matrix S′ along the channel dimension to obtain the first fusion feature F. c :

[0145] F c =cat(F′,S′)

[0146] F c It contains important image information, similarity information, and location information. Using this as input to the tamper detector, we obtain the copy-paste detection probability map P. d P d ∈[0,1] h×w×2 The number of channels is 2, and the two channels represent the probability that the region is a copied-paste region or the original background region, respectively. Let P... d After upsampling to the original size and extracting the channel with the highest probability, the copy-paste region mask M is obtained. d

[0147] M d =argmax(upsample(P d ))

[0148] Where `upsample` represents an 8x upsampling bilinear interpolation function. `argmax` is the function that takes the channel with the maximum value. M d ∈{0,1} H×W ,like Figure 5 As shown, the tamper detection mask M d The value can be 0 or 1, which represents the area as a copy-paste area or the original background area, respectively.

[0149] The process involves extracting similar region masks from the feature map, further generating tampered region masks and tampered region edge masks, separating the corresponding region feature maps using dual masks, and then stitching them together in the channel dimension to form new features. These stitched features are then fed into the source-destination locator to obtain the source-destination localization mask.

[0150] The image feature F′ is concatenated with the fusion similarity matrix S′ along the channel dimension to obtain the first fusion feature F. c The first fusion feature F c The input mask is fed into the input mask extractor to obtain the copy-paste detection probability map P∈[0,1]. H×W×2 P is fed into the argmax function to obtain the copy-paste detection mask M:

[0151] M = argmax(P)

[0152] The `argmax` function is used to obtain the channel containing the maximum value for each pixel. For a 2-channel probability map P, M∈{0,1} H×W M takes the value 0 or 1, indicating whether each pixel belongs to the background or the tampered area.

[0153] To obtain a mask that indicates the exact location of the tampered area, M is dilated to obtain the dilated mask M. K :

[0154]

[0155] in: The dilation operation is represented by kernel K, with a window size of 3×3. The parameters of this kernel are fixed at all 1s. A Boolean operation (bool) is performed on the dilated mask to convert all positive values ​​to 1, resulting in a binary mask M. K Compared to M, M K The edges of the regions in the image are expanded, making the tampered regions they indicate more complete.

[0156] Use MK The edge mask M is obtained by subtracting M from M. e :

[0157] M e =M K -M

[0158] M e This is a mask used to indicate the edge location of the tampered area. The image features at this location can be used to accurately distinguish between the source and destination regions.

[0159] Furthermore, the process of using dual masks to separate the corresponding region feature maps and then stitching them together in the channel dimension to form new features, and then sending the stitched features into the source-destination locator to obtain the source-destination localization mask, specifically involves:

[0160] Using the mask, multiply each image feature F′ by the dot product, M e M is used to extract features of the edges of the tampered region. K The modulated features are used to extract features within the tampered region. The modulated features are then concatenated along the channel dimension to obtain the second fused feature F″.

[0161] F″=cat(F′·M e , F′·M K )

[0162] In the second fusion feature F″, the features inside and at the edge of the tampered region are separated from each other. The features inside the tampered region enhance the common information between the source and the target, ensuring that the network detects the target without missing the source. The features at the edge of the tampered region provide the difference information between the source and the target, helping the network distinguish between them. The two types of information work together to effectively improve the accuracy of locating the source region.

[0163] The tamper detection network employing the cross-entropy loss function specifically refers to:

[0164] The copy-paste tampering detection module and source / destination location module are optimized using the cross-entropy function. Cross-entropy function L ce The expression is:

[0165]

[0166] Where: H×W is the number of pixels in the input image and the label; K represents the number of categories. For copy-paste tampering detection, K=2, where k takes the value 0 or 1, and the two categories represent the background region and the copied-paste region, respectively; for source-destination localization, K=3, where k takes the value 0, 1, or 2, and the three categories represent the background region, the source, and the target, respectively; Y i，k A value of 0 indicates whether the pixel belongs to category k (1 indicates belonging to the category, 0 indicates not belonging to the category); P i，k This represents the probability that the network predicts pixel i belongs to category k.

[0167] The loss of the detection branch is:

[0168] L d =L ce (Y d P d )

[0169] Where Y d It is the tampered location label in the dataset, Y d ∈{0, 1, 2} H×W×3 The label records the category of HW regions. The two channels indicate the location of the background and the tampered area, respectively. One of the two channels is 1, which means that the pixel belongs to the category indicated by the corresponding channel.

[0170] The loss of the positioning branch is:

[0171] L l =αL ce (Y d P s )+(1-α)L ce (Y l P l )

[0172] Where Y l Y is the source and destination location label of the dataset. l ∈{0, 1, 2} H×W×3 This label records the category of HW regions. Two channels indicate the location of the background and the tampered area, respectively. One channel has a value of 1, indicating that the pixel belongs to the corresponding region.

[0173] Example 1

[0174] A method for detecting image copy-paste tampering, see [link to relevant documentation]. Figure 1 The method includes the following steps:

[0175] 101: Extract feature maps from the image, calculate the similarity between the features of a single region and all regions, and form a similarity vector; combine the similarity vectors of all regions into a similarity matrix;

[0176] 102: Based on the pairing relationship between regions in the image and their matching regions, paired regions are divided into three different matching modes;

[0177] 103: Sort the similarity vectors of all regions in descending order and record the positions of their matching regions;

[0178] 104: Based on the matching position relationship, the similarity vectors of different regions are fused to enhance the matching pattern and obtain the fused similarity matrix;

[0179] 105: Extract the attention map from the fusion similarity matrix and use the attention map to modulate the feature map; concatenate the feature map and the fusion similarity matrix to obtain the fusion feature map; send the fusion feature map into the tampering detector to extract the copy-paste detection mask.

[0180] 106: Extract similar region masks from the feature map, and further generate tampered region masks and tampered region edge masks. After separating the corresponding region feature maps using dual masks, stitch them together in the channel dimension to form new features, and send the new feature map into the source-destination locator to obtain the source-destination positioning mask.

[0181] 107: Using USC-ISI as the training set, the copy-and-paste network was trained using the cross-entropy loss function;

[0182] 108: Input a single image with copy and paste into the trained deep neural network for tamper detection, obtain the copy and paste region mask from the tamper detector, and obtain the source and destination localization mask from the source and destination locator.

[0183] Example 2

[0184] The following examples and calculation formulas illustrate this point. Figures 2-7 The solution in Example 1 will be further described below:

[0185] 201: Extract feature maps F from the image, calculate the similarity between the features of a single region and all regions, and form a similarity vector; combine the similarity vectors of all regions into a similarity matrix S;

[0186] Step 201 above includes:

[0187] Input image to be detected X∈R H×W×3 This will be converted into a grayscale image G∈R H×W The descriptor extraction module of the SuperPoint network

[11] is used to extract descriptor features F∈R from different regions of the image. h×w×d ,in, d is the length of the descriptor. To reduce the computational cost of subsequent similarity calculations, a pooling layer is added to SuperPoint, so the final extracted feature map F is 1 / 8 the size of the original image.

[0188] The similarity between descriptor features of different regions of an image is calculated to obtain a similarity matrix S. In this embodiment of the invention, cosine similarity is used as the similarity metric, and the similarity matrix is ​​calculated as follows:

[0189]

[0190] in It is the identifier after F is reduced in dimension. express Modulo. S∈R hw×hw Each row represents a single region in the image, and each column stores the similarity between that region and all other regions.

[0191] 202: Based on the matching relationship, paired regions are divided into three different matching modes.

[0192] Step 202 above includes:

[0193] exist Figure 3 Three different matching patterns are shown, where A through E represent different regions. Arrows start from a region and point to its best matching region, which is the region most similar to it. The three different matching patterns are as follows:

[0194] (1)(a) In the figure, A and B are each other's matching areas, forming a bidirectional matching mode. The probability of copying and pasting the area in this mode is relatively high.

[0195] (2)(b) In the figure, the matching area of ​​D is C, but the matching area of ​​C is other areas. The C and D areas form a one-way matching pattern, and the probability of copy and paste is the second highest.

[0196] (3)(c) In the figure, region E is not a matching region of any region, and the probability of copying and pasting in this non-matching mode is the lowest.

[0197] 203: Sort the similarity vectors of all regions in descending order and record the positions of their matching regions.

[0198] Step 203 above includes:

[0199] Sort the similarity matrix in descending order to obtain the sorted similarity matrix and the corresponding original position matrix:

[0200] V, I = SortColumn top-k (S)

[0201] SortColumn(.) is a function that sorts the similarity vectors of each region in descending order. top-k represents the top k maximum values ​​in each column of the sorted region; therefore, after sorting, V∈R hw×k I∈R hw The position of the maximum value in each row of the matrix is ​​recorded, representing the matching region position of each region. Therefore, V contains rich similarity information, and I contains rich matching position information.

[0202] After obtaining V and I, send them into such a system as follows: Figure 2 The fusion unit shown performs the fusion of location information and similarity information.

[0203] 204: Based on the matching position relationship, the similarity vectors of different regions are fused to enhance various matching modes to different degrees, resulting in a fused similarity matrix.

[0204] Step 204 above includes:

[0205] like Figure 4 As shown in (a), five regions in the image are represented by A to E, with their position indices a to e. Based on these indices, similarity vectors V[a,:] to V[e,:] ∈ R are extracted from V row by row. k . Figure 4 Figures (b) and (c) show the similarity matrices after one and two fusions, respectively. The following section combines... Figure 4 This section introduces the fusion process and the differences in the effects of various matching modes after fusion.

[0206] (1) Perform a single fusion of the similarity of each region.

[0207] like Figure 4 (b) As shown in the figure, in order to perform a single fusion of the similarity of each region, it is necessary to traverse all regions in the image. If a certain region is the best matching region of another region, then the similarity vector of the other region is superimposed on that region. To achieve this process, a matrix of all zeros with the same size as V is first generated. Then, based on the location of the matching region provided by I, the similarity vectors corresponding to all regions in V are assigned to the appropriate values. In its matching position:

[0208]

[0209] Where m∈[1,...,hw], it represents the index of each region. I[m] represents the matching region position of the m-th region in V; when a region in the image is the best matching region of another region... The corresponding row in the matrix stores the similarity vector of the matched region; otherwise, it remains filled with zeros. Therefore, the matrix... The zero line in the China-Africa section can be considered a suspected copy-paste area. It contains not only similarity information but also matching location information, and is called the recombination matrix of V.

[0210] Then Adding this to V yields the fused similarity matrix V1. The similarity information is then fused with the location information once.

[0211]

[0212] like Figure 4As shown in (b), after one fusion, the similarity of regions A and B in bidirectional matching and region C in unidirectional matching in matrix V1 is improved compared to other regions. It can be seen that the similarity of bidirectional matching regions is effectively strengthened after only one fusion, and this enhancement of matching regions can effectively improve the robustness of the network to geometric transformations.

[0213] (2) Perform multiple fusions on the similarity of each region.

[0214] To further increase the similarity of the bidirectional matching regions and to increase the difference between this matching pattern and other patterns, regions A and B are distinguished from region C. The above fusion operation is repeated n times for V, resulting in a similarity matrix set {V1, V2, V3, ..., V...}. n-1 V n In this set, the adjacent similarity matrix V j-1 V j The following conditions must be met:

[0215]

[0216]

[0217] Where m∈[0,...,hw], represents the index of each region. j∈[1,...,n], This represents the similarity matrix obtained after j-1 fusion processes. This represents its recombination matrix.

[0218] observe Figure 4 (c) shows that after two fusion processes on the original similarity matrix, the bidirectional matching regions A and B are enhanced more significantly than the unidirectional matching region C.

[0219] To increase the stability of the similarity matrix, the n similarity matrices in the similarity matrix set are added together to obtain the fused similarity matrix S′.

[0220]

[0221] Among them, V j Let S′ ∈ R be the similarity matrix after j fusions. hw×k This represents the similarity matrix obtained by adding the similarity matrices after n fusions, which fuses positional and similarity information. As the number of fusions n increases, Figure 4 The similarity differences between the corresponding regions in the three matching modes gradually become more apparent, with the bidirectional matching region showing the most significant enhancement during the fusion process. When a geometric transformation is applied to the copied and pasted region, its similarity is weakened, making it difficult to distinguish from the background region. For example... Figure 4As shown, the fusion module can increase the similarity difference between the copied and pasted area and the background area, thereby eliminating the adverse effects of geometric transformation.

[0222] 205: Extract attention map A from the fusion similarity matrix, and modulate feature map F using the attention map; concatenate the feature map and the fusion similarity matrix to obtain a fusion feature map; feed the fusion feature map into the tamper detector to extract the copy-paste detection mask M. d .

[0223] Step 205 above includes:

[0224] A spatial attention module is constructed using a convolutional network. This module takes S′ as input and uses sigmoid as the activation function to obtain a spatial attention map.

[0225] A = attention(S′)

[0226] After the sigmoid activation function, A∈[0,1] h×w The obtained attention map A is multiplied by the image feature F to retain important information in the image, resulting in the modulated image feature F′:

[0227] F′=F·A

[0228] The copy-paste tampering detection module consists of three cascaded 3×3 convolutional layers, each followed by a BatchNorm layer. Except for the last convolutional layer, which uses the softmax activation function, the remaining layers use the ReLU activation function.

[0229] The image features F′ and the fusion similarity matrix S′ are concatenated along the channel dimension to obtain the fusion features F. c :

[0230] F c =cat(F′,S′)

[0231] F c It contains important image information, similarity information, and location information. Using this as input to the tamper detector, we obtain the copy-paste detection probability map P. d P d ∈[0,1] h×w×2 The number of channels is 2, and the two channels represent the probability that the region is a copied-paste region or the original background region, respectively. Let P... d After upsampling to the original size and extracting the channel with the highest probability, the copy-paste region mask M is obtained. d

[0232] M d =argmax(upsample(P d ))

[0233] Where `upsample` represents an 8x upsampling bilinear interpolation function. `argmax` is the function that takes the channel with the maximum value. M d ∈{0,1} H×W ,like Figure 5 As shown, the tamper detection mask M d The value can be 0 or 1, which represents the area as a copy-paste area or the original background area, respectively.

[0234] 206: Extract similar region masks M from the feature map, and further generate tampering region masks M. K and the tampering area edge mask M e Using a double mask to enlarge F c After the corresponding features are separated, they are spliced ​​together in the channel dimension to obtain F″, and F″ is sent to the source-destination locator to obtain the source-destination positioning mask.

[0235] Step 206 above includes:

[0236] like Figure 6 As shown, F c The input mask is fed into an extractor consisting of convolutional layers. The last layer uses a sigmoid activation function to constrain the output value to between 0 and 1. This yields the copy-paste detection probability map P, where P∈[0,1]. H×W×2 The two channels of P record the probability that each pixel belongs to the background or the tampered region, with values ​​ranging from 0 to 1. P is fed into the argmax function to obtain the copy-paste detection mask M.

[0237] M = argmax(P)

[0238] The `argmax` function is used to obtain the channel containing the maximum value for each pixel. For a 2-channel probability map P, M∈{0,1} H×W M takes the value 0 or 1, indicating whether each pixel belongs to the background or the tampered area, respectively. To obtain a mask that indicates the location of the complete tampered area, M is first dilated to obtain the dilated mask M. K :

[0239]

[0240] in: The operation represents dilation, with a dilation kernel K having a window size of 3×3. The parameters of this kernel are fixed at all 1s. A Boolean operation (bool) is performed on the dilated mask to convert all positive values ​​to 1, resulting in a binary mask M. K Compared to M, M K The edges of the region are expanded, making the tampered region it indicates more complete.

[0241] Use M K The edge mask M is obtained by subtracting M from M. e :

[0242] M e =M K -M

[0243] M e This is a mask used to indicate the edge location of the tampered area. The image features at this location can be used to accurately distinguish between the source and destination regions.

[0244] Generate mask M e With M K Then, the mask is used to multiply the image features F′ by M respectively. e M is used to extract features of the edges of the tampered region. K The modulated features are used to extract features within the tampered region. The modulated features are then concatenated along the channel dimension to obtain the second fused feature F″.

[0245] F″=cat(F′·M e , F′·M K )

[0246] In the fused feature F″, the features inside the tampered region and the features at the edge are separated from each other. The features inside the tampered region enhance the common information between the source and the target, ensuring that the network detects the target without missing the source. The features at the edge of the tampered region provide the difference information between the source and the target, helping the network distinguish between the source and the target. The two types of information work together to effectively improve the accuracy of locating the source region.

[0247] The source-destination localizer consists of three cascaded 3×3 convolutional layers, each followed by a BatchNorm layer. Except for the last convolutional layer, which uses the SOFIMAX activation function, the remaining layers use the ReLU activation function. The fused feature F″ is input to the source-destination localization module to obtain the source-destination probability map P. l ∈[0,1] h×w×3 The number of channels is 3, representing the probabilities that the region is background, source, and destination, respectively. Let P... l After upsampling to the original size and extracting the channel with the highest probability, the source-destination localization mask M is obtained. l .

[0248] M l =argmax(upsample(P l ))

[0249] Where upsample represents the 8x upsampling bilinear interpolation function, argmax is the function that takes the maximum value in the channel, and M... l ∈{0, 1, 2}H×W ,like Figure 7 As shown, source and destination positioning mask M l The value can be 0, 1, or 2, which represent the region as the background region, source region, or destination region, respectively.

[0250] 207: Using USC-ISI as the training set, the copy-paste network was trained using the cross-entropy loss function.

[0251] Step 205 includes:

[0252] 1) The USC-ISI dataset was used as the training set. The USC-ISI dataset is obtained by extracting objects from images using existing segmentation algorithms and pasting them into other locations in the original image. The training set contains 80,000 images.

[0253] 2) Optimize the copy-paste tampering detection module and source / destination location module using the cross-entropy function. Cross-entropy function L ce The expression is:

[0254]

[0255] Where: H×W is the number of pixels in the input image and the label; K represents the number of categories. For copy-paste tampering detection, K=2, where k takes the value 0 or 1, and the two categories represent the background region and the copied-paste region, respectively; for source-destination localization, K=3, where k takes the value 0, 1, or 2, and the three categories represent the background region, the source, and the target, respectively; Y i，k A value of 0 indicates whether the pixel belongs to category k (1 indicates belonging to the category, 0 indicates not belonging to the category); P i，k This represents the probability that the network predicts pixel i belongs to category k.

[0256] The loss of the detection branch is:

[0257] L d =L ce (Y d P d )

[0258] Where Y d It is the tampered location label in the dataset, Y d ∈{0, 1, 2} H×W×3 The label records the category of HW regions. The two channels indicate the location of the background and the tampered area, respectively. One of the two channels is 1, which means that the pixel belongs to the category indicated by the corresponding channel.

[0259] The loss of the positioning branch is:

[0260] L l =αL ce (Yd P s )+(1-α)L ce (Y l P l )

[0261] Where Y l Y is the source and destination location label of the dataset. l ∈{0, 1, 2} H×W×3 The label records the category of HW regions. The two channels indicate the location of the background and the tampered area, respectively. One of the two channels is 1, which means that the pixel belongs to the category indicated by the corresponding channel.

[0262] 208: Input a single image containing copy and paste into the trained copy and paste tampering detection network, and obtain the copy and paste region mask M from the network's tampering detector. d Obtain the source / destination positioning mask M from the source / destination locator. l .

[0263] Step 206 includes:

[0264] Input a single image X containing copy and paste into the trained copy and paste tampering detection network, where X∈R. H×W×3 Obtain the copy-paste region mask M from the network's copy-paste tampering detector. d M d ∈{0,1} H×W The value is 0 or 1, representing either the copy-paste area or the original background area, respectively. The source / destination positioning mask M is obtained from the source / destination locator. l M l ∈{0, 1, 2} H×W The value ranges from 0, 1, or 2. These represent the region as background, source, or target, respectively.

[0265] References

[0266] [1]Fattah SA, Ullah M, Ahmed M, et al.A scheme for copy-move forgerydetection in digital images based on 2D-DWT[C].In Proceedings of the IEEEInternational Midwest Symposium on Circuits and Systems, 2014: 801-804.

[0267] [2]Kumar S,Desai J,Mukherj ee S.Copy move forgery detection incontrast variant environment using binary DCT vectors[J].InternationalJournal of Image,Graphics and Signal Processing,2015,7(6):38.

[0268] [3]Lowe D G.Distinctive image features from scale-invariant keypoints[J].International Journal of Computer Vision,2004,60(2):91-110.

[0269] [4]Baya H,Essa A,Tuytelaarsb T,et al.Speeded-up robust features(SURF)[J].Computer Vision and Image Understanding,2008,110(3):346-359.

[0270] [5]Amerini I,Ballan,et al.:A SIFT-Based Forensic Method for Copy-MoveAttack Detection

[0271] [6]and Transformation Recovery[J].IEEE transactions on informationforensics and security,2011.

[0272] [7]Bo X,Junwen W,Guangjie L,et al.Image copy-move forgery detectionbased on SURF[C].In Proceedings of the International Conference on MultimediaInformation Networking and Security,201 0:889-892.Wu Y,Abd-Almageed W,Natarajan P.BusterNet:Detecting copy-move image forgery with source / targetlocalization[C].In Proceedings of the European Conference on Computer Vision,2018:168-184.

[0273] [8]Yang F,Li J,Lu W,et al.Copy-move forgery detection based on hybridfeatures[J].Engineering Applications of Artificial Intelligence,201 7,59:73-83.

[0274] [9]Islam A,Long C,Basharat A,et al.Doa-gan:dual-order attentivegenerative adversarial network for image copy-move forgery detection andlocalization[C].In Proceedings of the IEEE / CVF Conference on Computer Visionand Pattern Recognition,2020:4676-4685.

[0275]

[10] Zhu Y, Chen C, Yah G, et al.AR-Net: Adaptive attention and residualrefinement network for copy-move forgery detection[J].IEEE Transactions on Industrial Informatics, 2020,16(10):6714-6723.DeTone D,Malisiewicz T,Rabinovich A.Superpoint:self-supervised interest point detection and description[C].In Proceedings of the IEEE Conference on Computer Vision andPattern Recognition Workshops, 2018:224–236.

[0276] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0277] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A system for detecting image copy-paste tampering, characterized in that, The system is based on a deep neural network; the network includes a detection branch and a localization branch; the detection branch consists of a feature extraction module, a similarity calculation module, an information fusion module, a spatial attention module, and a tamper detector; the localization branch consists of a mask generation module, a region separation module, and a source-destination locator; wherein: The feature extraction module extracts feature maps from the image to be detected. ; The similarity calculation module calculates the similarity between features of a single region and all regions using the following formula to obtain the similarity vectors of the regions, forming a similarity matrix. ; ； in: yes The reduced-dimensional identifier express Take the mold; Each row represents a single region in the image, and each column stores the similarity between that region and all regions. The information fusion module divides paired regions into three matching modes based on the pairing relationship between regions in the image and their matching regions. The information fusion module obtains the matching positional relationship between the sorted similarity matrix and the corresponding original position matrix by descending the similarity vectors of all regions using the following formula; ； in: It is a column-based descending sorting function, which sorts the similarity vectors of each region. This indicates the first column of each sorted section. The maximum value, This is the similarity matrix before sorting; ,in h and w The number of regions in the vertical and horizontal directions of the image. R Represents the set of real numbers; Record the position of the maximum value in each row of the matrix, representing the matching region position for each region; The information fusion module obtains a fused similarity matrix by fusing similarity vectors from different regions based on matching positional relationships. ; The spatial attention module will extract attention map modulation features. With fusion similarity matrix The first fused feature map obtained by splicing along the channel dimension ,Right now: ； The tampering detector uses the following formula to analyze the first fused feature map. Calculations are performed to obtain the copy-paste area mask. ; ； Among them: Copy and paste detection probability graph , The number of channels is 2; The mask generation module generates the mask based on the first fusion feature map. Extracting similar region masks and calculating the tampered region mask. and tampering with the edge mask of the area ; The region separation module uses the following formula to separate image features. tampering with area mask and tampering with the edge mask of the area The second fusion feature is obtained by concatenating the data along the channel dimension. ; ； The source-destination locator calculates the source-destination locator mask based on the second fusion feature using the following formula. ； in: It uses three cascaded convolutional layers to extract the source-destination probability map from the second fusion feature.

2. The method for detecting image copy-paste tampering according to claim 1, characterized in that: The method includes the following steps: Feature maps are extracted from the detected image, and the similarity between the features of a single region and all regions is calculated to form a similarity vector; the similarity vectors of all regions are then combined to form a similarity matrix. Based on the pairing relationship between regions in the image and their matching regions, paired regions are divided into three different matching modes; Sort the similarity vectors of all regions in descending order and record the positions of their matching regions; A fused similarity matrix is ​​obtained by fusing the similarity vectors of different regions based on the matching positional relationships; Attention maps are extracted from the fused similarity matrix, and features are modulated using the attention maps; The modulation features and the fusion similarity matrix are concatenated to obtain the first fused feature map; the first fused feature map is then fed into the tamper detector to extract the copy-paste detection mask. ; Similar region masks are extracted from the feature maps, and tampered region masks and tampered region edge masks are further generated. After separating the corresponding region feature maps using dual masks, they are stitched together in the channel dimension to form a second fused feature. The stitched second fused feature is then fed into the source-destination locator to obtain the source-destination localization mask. ;in: Source-destination probability maps are extracted from the second fused feature using three cascaded convolutional layers. : ； Where: the size of the convolutional kernel in each layer is... Activation function uses The activation function for the last layer is selected as follows: ;Will Upsample to the original size and use the argmax function to obtain a three-valued source-destination localization mask: A deep neural network for tamper detection is employed using the cross-entropy loss function. A single image with copy and paste is input into a trained deep neural network for tamper detection. The copy and paste region mask is obtained from the tamper detector, and the source and destination location mask is obtained from the source and destination locator.

3. The system for detecting image copy-paste tampering according to claim 1, characterized in that: The information fusion module obtains a fused similarity matrix by fusing similarity vectors from different regions based on matching positional relationships. The steps include: Generation and similarity matrix All-zero matrices of the same size ; according to The location of the provided matching region will The similarity vectors corresponding to all regions in the dataset are assigned values ​​accordingly. In its matching position: ； Among them, the similarity matrix Each line represents one region, totaling... Each region is represented by a column indicating the region most similar to that region. Similarity between regions; This represents the sequence number of each region. express The Middle The optimal matching region location for each region; when one region in an image is the optimal matching region for another region. The corresponding row in the matrix stores the similarity vector of the matched region; otherwise, it remains filled with zeros. The zero line in the China-Africa section can be considered a suspected copy-paste area; It not only contains similarity information, but also carries matching location information, and is called Recombination matrix; Will and Add them together to obtain the fused similarity matrix. The similarity information and location information are then fused together. ； right repeat The above fusion operation is performed. The fusion process yields a set of similarity matrices. In this set, the similarity matrix of adjacent elements The following conditions must be met: ； ； in, , representing the sequence number of each region; , Indicates the process The similarity matrix obtained from the second fusion process Represent its recombination matrix; The similarity matrix set The fusion similarity matrix is ​​obtained by adding the group similarity matrices. ； in, Indicates the process The similarity matrix of the second fusion Indicates the process The similarity matrix obtained by adding the similarity matrices of the sub-fusions is the fusion of position and similarity information.

4. The image copy-paste tampering detection method based on information fusion and separation according to claim 2, characterized in that: The process involves extracting an attention map from the fusion similarity matrix and modulating the feature map using the attention map; concatenating the feature map with the fusion similarity matrix to obtain a fusion feature map; and feeding the fusion feature map into a tampering detector to extract a copy-paste detection mask, including the following steps: A spatial attention module is constructed using convolutional networks, which will... As input, use As the activation function, the spatial attention map is obtained: ； go through After the activation function, Use the resulting attention map Image features Multiplication preserves important information in the image, yielding the modulated image features. : ； The copy-paste tampering detection module consists of three cascaded 3×3 convolutional layers, each of which is connected to a BatchNorm layer. Except for the last convolutional layer, which uses the softmax activation function, the remaining layers use the ReLU activation function. Image features With fusion similarity matrix By concatenating along the channel dimension, the fused features are obtained. : ； It contains important image information, similarity information, and location information; it is used as input to the tamper detector to obtain a copy-paste detection probability map. , The number of channels is 2, and the two channels represent the probability that the region is a copied / pasted region or the original background region, respectively; After upsampling to the original size and extracting the channel with the highest probability, the copy-paste region mask is obtained. ； in: It represents an 8x upsampling bilinear interpolation function; It is the channel function that takes the maximum value; .

5. The image copy-paste tampering detection method based on information fusion and separation according to claim 2, characterized in that: The process involves extracting similar region masks from the feature map, generating tampered region masks and tampered region edge masks, separating the corresponding region feature maps using dual masks, concatenating them into new features along the channel dimension, and then feeding the concatenated features into the source-destination locator to obtain the source-destination localization mask. The steps include: Fusion The image is fed into the input mask extractor to obtain the copy-paste detection probability map. ,Will Send in The function obtains the copy-paste detection mask. : ； The function is used to obtain the channel containing the maximum value for each pixel in a given channel. For a 2-channel probability map... , , The value can be 0 or 1, indicating whether each pixel belongs to the background or the tampered area, respectively. H and W These are the height and width of the image, respectively. The mask indicating the location of the complete tampered area is... Perform an expansion operation to obtain the expanded mask. : )； in This indicates an expansion operation. The expansion kernel has a window size of 1. The parameters of the expansion kernel are fixed at all 1s; a Boolean operation (bool) is performed on the expanded mask to convert all positive values ​​to 1, resulting in a binary mask. and compared to, The edges of the regions in the text are expanded, making the tampered regions they indicate more complete; use and Obtain the edge mask by subtraction. : ； That is, a mask used to indicate the location of the edge of the tampered area; The feature maps of the corresponding regions are separated using a dual-mask method and then stitched together in the channel dimension to form a second fused feature. , ； Where: the mask is related to the image features. Dot product, Used to extract features of the edges of tampered regions. Used to extract features within the tampered area. The spliced ​​second fusion feature is fed into the source-destination locator to obtain the source-destination positioning mask. Specifically: Source-destination probability maps are extracted from the second fused feature using three cascaded convolutional layers. : ； Where: the size of the convolutional kernel in each layer is... Activation function uses The activation function for the last layer is selected as follows: Will Upsample to the original size and use the argmax function to obtain a three-valued source-destination mask image: 。

Images