Adaptive sampling mechanism and two-step autocorrelation replication moving tampering detection method

Abstract

The application discloses a self-adaptive sampling mechanism and a double-step self-correlation copy moving tamper detection method, proposes a self-adaptive sampling mechanism ASM, automatically adjusts the number and position of sampling points according to the characteristics of an input image, and thus helps a network to better detect tamper regions of different sizes. The application proposes a low-computational double-step self-correlation, extracts high-similarity regions on a high-resolution feature map through feature correlation on a low-resolution feature map, and only calculates correlation between features in the high-similarity regions, and thus greatly reduces the computational amount. The application proposes a double-branch adaptive fusion module, adopts a lightweight multi-scale hollow convolution module to adaptively fuse local context features before LD-SCC and relevant features after LD-SCC, and thus improves detection performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image security, and particularly relates to a self-adaptive sampling mechanism and a copy-move tampering detection method based on double-step autocorrelation. BACKGROUND

[0002] With the rapid decline in the cost of mobile devices, anyone can take photos at any time and any place using a mobile phone. Meanwhile, the rapid progress and popularity of image editing software make it easy to tamper with the content of digital images, and the tampering traces are almost imperceptible [1, 2]. Considering that digital images are often used as key evidence in the fields of news reporting, political events, criminal investigation, and insurance claims, the consequences of forged images can be very serious [3, 4]. Therefore, it is particularly important to accurately detect whether an image has been tampered with. There are various ways of image tampering, among which copy-move forgery (CMF) is a very common and simple tampering method that copies one or more regions (source regions) of an image and pastes them to other locations (target regions) of the same image. In order to make the tampering look more realistic, the method usually also adjusts the brightness, contrast, and performs geometric processing such as scaling and rotation on the copied regions.

[0003] In recent years, many copy-move forgery detection (CMFD) methods have been proposed, which can be mainly divided into two categories, namely, hand-crafted feature-based methods [5, 7-14] and deep learning-based methods [6, 15-19]. The former can be further divided into block-based methods [7, 9-11] and keypoint-based methods [8, 12-14]. Block-based methods usually divide the suspicious image into overlapping or non-overlapping blocks, and then extract the features of each image block using discrete cosine transform [9], principal component analysis

[10] , or Zernik moment

[11] and other methods. Then, the similarity features between the blocks are compared to detect the copy-move tampered regions. Block-based methods are relatively simple, but the amount of calculation is large, and it is difficult to deal with complex geometric transformations. Keypoint-based methods first extract the features of the key points using SIFT

[12] , SURF

[13] , or ORB

[14] and other methods, and then match and identify the tampered regions by using the similarity between the key point features. Although keypoint-based methods have stronger robustness to geometric transformations, they are difficult to detect smooth copy-move tampered regions.

[0004] In recent years, deep learning, especially convolutional neural networks, has been widely applied in various fields. Deep learning-based methods also provide effective solutions for CMFD. BusterNet proposed by Wu et al.

[15] is the first CMFD model based on convolutional neural networks that can be trained end-to-end. It consists of two parallel branches, one branch is used to detect the tampered target region, and the other branch is used to detect the high similarity copied and moved tampered region. However, the parallel structure of BusterNet requires both branches to be correctly positioned at the same time in order to correctly classify the source region and the target region. To solve this problem, Chen et al.

[16] proposed two serial subnetworks, namely CMSDNet and STRDNet, where CMSDNet is used to detect high similarity regions in the input image, and STRDNet is used to distinguish the source region and the target region from the detected high similarity regions. Zhu et al.

[17] proposed AR-Net, which combines adaptive attention mechanism and residual refinement module, and is an end-to-end CMFD network. Liu et al.

[18] proposed a two-stage CMFD framework that combines self-deep matching and keypoint matching. Weng et al. [6] proposed two independent U-shaped CMFD networks, namely UCM-Net-L and UCM-Net-S. UCM-Net-L is responsible for detecting large-area tampered regions, and UCM-Net-S is responsible for detecting small-area tampered regions. Each network contains three modules, namely feature extraction module (FEM), tampered region localization module (TRLM), and multiple cross-layer connections between FEM and TRLM. Although UCM-Net-L and UCM-Net-S have achieved excellent detection performance in detecting large-area and small-area tampered regions, both networks need to be trained separately, which increases the computational resource overhead. To reduce the computational overhead while maintaining good detection performance, Shi et al.

[19] proposed a lightweight high-precision network LHCM-Net that integrates low-cost SCC, gated feature fusion module (GFFM), and faster residual-U-block (FRSU) equipped with FasterNet block

[20] .

[0005] So far, the detection process of most deep learning-based CMFD networks is to first use a convolutional neural network to extract the features of the input image, then calculate the similarity between the features through a self-correlation calculation (SCC) module to obtain a correlation map, and finally use a convolution layer and an up-sampling layer to decode the correlation map to obtain the final detection result. However, most CMFD networks have defects in the above stages. First, in the feature extraction stage, the commonly used standard convolution layer cannot adaptively adjust the number and position of sampling points according to the area and shape of the tampered region, which increases the difficulty of the network to detect different sizes of tampered regions. In addition, the SCC in the CMFD network aims to calculate the similarity between each feature point in the feature map, which will inevitably produce an expensive computational burden when it is used for high-resolution feature maps. Finally, the existing CMFD network only uses the correlation features after the self-correlation module to obtain the tamper detection result, ignoring the local context features extracted by the backbone network, which leads to inaccurate prediction of the edges and interiors of the tampered region by the model.

[0006] In the feature extraction stage of the prior art, the existing CMFD network uses a standard convolution layer to extract features from the input image, and the sampling position of the standard convolution layer is fixed, so it is difficult to effectively extract features of tampered regions of different areas and shapes at the same time. In addition, since the self-correlation module in the CMFD network needs to calculate the similarity between two feature points on the feature map, it brings a huge computational overhead, so most of the current CMFD networks only use the self-correlation module on low-resolution feature maps, however, this inevitably leads these networks to ignore the similarity between high-resolution features, which damages the performance of the model. Finally, the existing CMFD network only uses a convolution layer and an up-sampling layer to process the correlation feature map obtained from the self-correlation module after the self-correlation module to obtain the copy-move tamper detection result. Although this method takes into account the correlation information between features, it ignores the local context information of the features themselves, which leads to inaccurate detection results of the edges and interiors of the tampered region by the model. SUMMARY

[0007] The purpose of the present application is to provide an adaptive sampling mechanism and a double-step self-correlation copy-move tamper detection method that can achieve excellent detection performance on multiple datasets and is lightweight and has low computational overhead.

[0008] The technical solution adopted by the present application is:

[0009] The adaptive sampling mechanism and the double-step self-correlation copy-move tamper detection method include the construction part of the model, the training and inference part of the model, and the specific steps are as follows:

[0010] Step 1: Construct an adaptive sampling mechanism and a two-step autocorrelation replication-movement tampering detection network model. The network model performs the following steps:

[0011] Step 1-1: Feature Extraction: The image to be detected is input into AL-Net, which uses ASM-Net as its backbone network. ASM-Net consists of four stages. The first two stages each consist of one "CBG 3-2" and two "CBG 3-1" blocks set sequentially, used to extract low-level features from the input image, thus helping AL-Net to better detect small-area tampering regions. The latter two stages each consist of one "CBG 3-2" and three ASM blocks with automatically adjustable sampling point numbers and positions, used to adaptively extract high-level features from the input image, thus helping AL-Net to better detect large-area tampering regions. The feature maps output by the four stages of ASM-Net are F1, F2, F3, and F4, respectively. CBG 3-2 represents a convolutional block consisting of a 3×3 standard convolutional layer with a stride of 2, a Batch Normalization layer, and a GELU activation function; CBG 3-1 represents a convolutional block consisting of a 3×3 standard convolutional layer with a stride of 1, a Batch Normalization layer, and a GELU activation function. The convolutional block consists of a normalization layer and a GELU activation function; the ASMblock consists of an adaptive sampling mechanism (ASM), two point-wise convolutions (PWConv), and cross-layer connections.

[0012] Furthermore, the Adaptive Sampling Mechanism (ASM) in step 1-1 performs the following steps:

[0013] Step 1-1-1, first input feature map E in The intermediate feature map E is obtained through "Convα-1". mid The shape is W ε ×H ε ×(β 2 +1); "Convα-1" indicates a standard convolutional layer with a kernel size of α and a stride of 1; and Indicates the input feature map E in Width and height; β 2 +1 represents the number of output channels for "Convα-1";

[0014] Step 1-1-2, intermediate feature map E mid Each sampling point E on mid (i,j) is split into the first channel and the remaining β. 2 Each channel has two parts, where i and j are the sampling points E, respectively. midthe horizontal and vertical coordinates of (i, j), E mid the value of the first channel in (i, j) the number of sampling points representing the (i, j) position predicted by “Convα-1”, and the predicted values of the remaining β 2 channels will be reshaped into a 2-dimensional matrix of shape β x β each value in the matrix (m, n ∈ [1, β]) represents the weight of each feature point in the β x β range centered at the coordinate (i, j) predicted by “Convα-1”.

[0015] Step 1-1-3, since the number of sampling points required by each position is different, ASM filters the top largest weight values according to the number of predicted sampling points and assigns the remaining weight values to 0 to obtain a new weight matrix The formula is shown in equation (4):

[0016]

[0017] where, represents the largest weight value, and represent the floor and ceiling functions, respectively. When ,the purpose of is to let “Convα-1” automatically learn the number of sampling points required by each position through backpropagation

[0018] Step 1-1-4, ASM performs L1-Normalization on to obtain and performs weighted summation with the input feature map E in to obtain the output feature map E out The formula is shown in equation (5):

[0019]

[0020] Further, where α and β are set to 5 in the third stage of ASM-Net and 3 in the fourth stage of ASM-Net.

[0021] ​Step 1-2, feature fusion: input the obtained feature maps F1, F2, F3 and F4 into a feature fusion module (FFM) for feature fusion to obtain an output feature map F f ;

[0022] Further, step 1-2 specifically comprises the following steps:

[0023] Step 1-2-1, input the feature map F4 into two "Conv 3-1" for further feature extraction; Conv 3-1 represents a standard convolution with a size of 3x3 and a step of 1;

[0024] Step 1-2-2, input the result obtained in step 1-2-1 into a global average pooling layer (GAP) to obtain a weight;

[0025] Step 1-2-3, element-wise multiplication of the weight obtained in step 1-2-2 and the channel concatenated F1, F2, F3 and F4 to obtain a weighted concatenated feature map, retaining the key information in the four feature maps;

[0026] Step 1-2-4, input the weighted concatenated feature map into "CBG 1-1" and two "FasterNet blocks" for feature fusion to obtain an output feature map F f .

[0027] Specifically, in order to let the AL-Net automatically filter the key information in the four feature maps according to the characteristics of the input image, the FFM first inputs F4 into two "Conv 3-1" for further feature extraction, and then inputs the obtained result into a global average pooling layer (GAP) to obtain a weight. The FFM performs element-wise multiplication of the obtained weight and the channel concatenated F1, F2, F3 and F4, retaining the key information in the four feature maps. The weighted concatenated feature map will be input into "CBG 1-1" and two "FasterNet blocks" for feature fusion to obtain the output feature map f f of the FFM, and the structure of the FFM is shown in Figure 5 .

[0028] Step 1-3, correlation feature extraction: input the feature map F fLow-cost dual-step self-correlation calculation (LD-SCC) is used to extract correlation information from the feature map.

[0029] Furthermore, steps 1-3 specifically include the following steps:

[0030] Step 1-3-1, LD-SCC will input feature map F f Downsampling yields low-resolution feature maps Then according to Correlation between pairwise features in the data, and selection of F f Highly similar regions in;

[0031] Steps 1-3-2, LD-SCC is only performed on F f Self-correlation calculation (SCC) is performed between features in highly similar regions to obtain the output correlation feature map F. d This method not only reduces the impact of redundant low-similarity information and improves the detection performance of AL-Net, but also significantly reduces computational overhead. The flowchart of LD-SCC is as follows: Figure 4 As shown.

[0032] Specifically, in the first step, LD-SCC first processes the shape as... Input feature map F f Perform k-fold average pooling downsampling (AP) to obtain a low-resolution feature map. Shape in This invention sets k to 8. Then, LD-SCC will... Further reshape to the shape of matrix And calculate The cosine similarity (CS) between any two feature points is calculated using the formula shown in formula (6):

[0033]

[0034] Where T represents the transpose operation. It represents The cosine similarity between the u-th feature point and the v-th feature point also represents F. f The similarity between the u-th k×k pixel block and the v-th k×k pixel block is calculated. Finally, LD-SCC assigns F based on the k×k partitioning method. ftransforming into a tensor of shape of shape for each k x k size pixel block in , LD-SCC extracts the t most similar k x k size pixel blocks according to the similarity information in , obtaining a tensor of shape of shape wherein t is set to 20 in the present application.

[0035] In the second step, LD-SCC calculates the similarity of each feature point in with the feature points of the t most similar pixel blocks of the pixel block where the feature point is located. In order to establish the connection between each pixel block in and the t pixel blocks in , LD-SCC first transposes , and then performs matrix multiplication between and , as shown in formula (7):

[0036]

[0037] Finally, LD-SCC restores the shape of to , and then sorts along the channel dimension and retains the top g maximum values in the channel dimension, obtaining an output feature map of shape wherein g is set to 32 in the present application.

[0038] Step 1-4, tampered area positioning: input the fusion feature map F f obtained in step 1-2 and the correlation feature map F d obtained in step 1-3 into a bi-branch adaptive fusion module (BAFM) for fusion. The first branch of the BAFM inputs F d into the MDCM for multi-scale feature extraction, and the second branch inputs F f into the point convolution for information interaction between channels. Then, the output results of the two branches are spliced in the channel dimension and jointly input into the MDCM. The MDCM comprehensively considers the correlation features in F d and the multi-level features in F f , and performs multi-scale feature extraction on both, so as to more accurately obtain F d and F f ​The generated weights retain key information of both. BAFM splits the obtained weights in the channel dimension to obtain two tensors with the same shape as the feature maps, and weights the two feature maps. Finally, BAFM splices the weighted two feature maps in the channel dimension, and inputs them into the lightweight FRSU for feature fusion to obtain the final detection result of the AL-Net.

[0039] Specifically, the structure DWDCa-b-c of the multi-scaledilated convolution module (MDCM) represents a deep dilated convolution layer with a size of a x a, a stride of b, and a dilated rate of c. The multi-scaledilated convolution module DWDC first uses three deep dilated convolution (DWDC) layers in series to extract multi-scale features on the input feature map. Compared with using three DWDCs in parallel, the series structure can obtain a larger receptive field with the same parameter amount, thereby better capturing multi-scale features and more accurately generating weights. The output three feature maps respectively perform information interaction between channels through point convolution, and pass through a batch normalization (BN) layer and a nonlinear GELU activation function to ensure the diversity of the features. Finally, the three feature maps are spliced in the channel dimension and input into a point convolution layer for fusion to obtain the final output result.

[0040] Step 2, training and inference of the model:

[0041] Step 2-1, data preprocessing: reading training samples from the training set of the USC-ISI dataset for preprocessing to obtain training data required for training; the preprocessing includes size adjustment, data enhancement, and normalization processing.

[0042] Data preprocessing is a key link in the CMFD task, which can improve the stability during model training and the performance during testing. The data preprocessing of the present application includes the following steps:

[0043] (1) Reading data: the present application reads training samples from the training set of the USC-ISI dataset. Each sample contains an original image and a corresponding label, and the label contains class information of each pixel point in the original image.

[0044] (2) Adjusting size: in order to ensure that the sizes of all training samples are the same, the present application simultaneously scales the original image and the corresponding label. The present application uniformly adjusts the length and width dimensions of all images to 320 x 320. This step ensures that the model can process input images of the same size, simplifying the training process.

[0045] (3) Data augmentation: In this phase, the application simultaneously rotates and horizontally flips the input image and the corresponding label according to a certain probability. Random rotation and random horizontal flipping can enhance the model's detection ability for different angles and symmetry of the image.

[0046] (4) Normalization: The application uses the statistical data of ImageNet to normalize the image. This dataset contains a large number of images in various scenes, and using its mean and variance for normalization can help the model learn more efficiently, thereby accelerating the convergence speed.

[0047] Step 2-2, model training: randomly divide the training data into multiple batches, use the cross-entropy loss function to guide the training process of AL-Net, select the stochastic gradient descent method (SGD) to adjust the model parameters by estimating the gradient of the loss function to optimize the performance, and complete the model training to obtain the final model after a certain number of iterations of training.

[0048] Further, the training process of step 2-2 is set to 60 rounds, each round representing a complete traversal of the entire dataset.

[0049] Specifically, first, the application randomly divides the training data into multiple batches, each containing 16 samples. This batching method can effectively utilize the computing power of the GPU, ensuring that the model can process multiple samples in parallel, thereby improving training efficiency. Next, the application uses the cross-entropy loss function to guide the training process of AL-Net. This loss function evaluates the deviation between the model's prediction and the actual label, and by minimizing this loss, AL-Net can more accurately adapt to the training data, thereby improving its performance in the CMFD task. To optimize the model's parameters, the application chooses the stochastic gradient descent method (SGD). This method adjusts the model parameters by estimating the gradient of the loss function with respect to the parameters, thereby optimizing the performance. In addition, the entire training process is set to 60 rounds, each round representing a complete traversal of the entire dataset.

[0050] Step 2-3, model inference: after pre-processing the image to be detected to ensure that the parameters used are identical to those used during model training, the final model is inputted and analyzed through forward propagation to estimate the probability of each pixel belonging to the tampered region in the form of a probability map. All pixels in the probability map whose predicted probability exceeds a certain value are marked as tampered pixels, thus completing the accurate identification of the tampered region in the image.

[0051] Further, the set value is 0.5.

[0052] Specifically, the model acquired from step 2-2 is equipped with the weight parameters learned during training, which provides support for subsequent copy-move image tamper detection tasks. First, this step performs necessary preprocessing on the image to be detected, including size adjustment and normalization, to ensure that it is fully consistent with the parameters used during training. Then, the preprocessed image is input into the trained model. The model analyzes the image by forward propagation and estimates the probability of each pixel belonging to the tampered region. The result is presented in the form of a probability map, where the value of each pixel in the map represents the likelihood of the pixel being tampered with. Finally, this step marks all pixels in the probability map output by AL-Net with a predicted probability greater than 0.5 as tampered pixels, thus completing the accurate identification of the tampered region of the image.

[0053] The application also discloses an adaptive sampling mechanism and a copy-move tamper detection network with double-step self-correlation, including an ASM-Net module, a double-branch adaptive fusion module BAFM, and a double-step self-correlation module. The ASM-Net contains four stages, the first two stages are each composed of one "CBG 3-2" and two "CBG 3-1", which are used to extract low-level features on the input image to help AL-Net better detect small tampered regions; the last two stages are each composed of one "CBG 3-2" and three ASM blocks that can automatically adjust the number and position of sampling points, which are used to adaptively extract high-level features on the input image to help AL-Net better detect large tampered regions; the feature maps output by the four stages of ASM-Net are F1, F2, F3 and F4 respectively; wherein CBG 3-2 represents a convolution block composed of a 3x3 standard convolution layer with a step size of 2, a Batch Normalization layer and a GELU activation function; CBG 3-1 represents a convolution block composed of a 3x3 standard convolution layer with a step size of 1, a Batch Normalization layer and a GELU activation function; the ASM block is composed of an adaptive sampling mechanism (ASM), two point-wise convolutions (PWConv) and cross-layer connections; the feature maps F1, F2, F3 and F4 are input into the feature fusion module (FFM) for feature fusion to obtain the output feature map F f ;

[0054] The LD-SCC downsamples the input feature map F f to obtain a low-resolution feature map F Then, according to the correlation between each pair of features in F , F fhigh-similarity regions in the first step; in the second step, LD-SCC only calculates the correlation between the features of high-similarity regions in F f high-similarity regions in the first step; in the second step, LD-SCC only calculates the correlation between the features of high-similarity regions in F d ;

[0055] The BAFM includes at least two MDCMs, the first branch of the BAFM inputs F d into the first MDCM to extract multi-scale features, and the second branch inputs the point convolution to interact information between channels; the output results of the two branches are spliced in the channel dimension and input to the second MDCM; the multi-scale dilated convolution module DWDC uses three deep dilated convolution layers DWDC in series to extract multi-scale features on the input feature map to obtain three feature maps; the three feature maps are interacted between channels by point convolution, and pass through the batch normalization (BN) layer and the nonlinear GELU activation function to ensure the diversity of the features; finally, the three feature maps are spliced in the channel dimension and fused with the input point convolution layer to obtain the final output result.

[0056] The ASM of the present application reduces the computational overhead because the coordinates of the sampling points are all integers, unlike deformable convolution which needs to obtain the sampling points at floating-point coordinates through bilinear interpolation. The ASM (adaptive sampling mechanism) is proposed, which can automatically adjust the number and position of the sampling points according to the characteristics of the input image, thereby helping the network to better detect tampered regions of different sizes. The LD-SCC (low-cost dual-step self-correlation calculation) is proposed, which first extracts high-similarity regions in the high-resolution feature map through the correlation of the features in the low-resolution feature map, and then only calculates the correlation between the features of the high-similarity regions, thereby greatly reducing the computational cost. The BAFM (bi-branch adaptive fusion module) is proposed, which uses a lightweight multi-scale dilated convolution module to adaptively fuse the local context features before LD-SCC and the correlation features after LD-SCC, thereby improving the detection performance. The AL-Net (adaptive lightweight network for CMFD) is designed by combining the three structures, which is a lightweight, low-computational-cost and high-precision CMFD network. The experimental results on four public datasets show that the AL-Net is superior to the above related CMFD networks in terms of detection accuracy, number of parameters, computational cost, etc. BRIEF DESCRIPTION OF DRAWINGS

[0057] The application will be further described in detail below in combination with the drawings and specific embodiments.

[0058] Figure 1 Fig. 1 is a schematic diagram of the ASM-Net backbone network structure of the application;

[0059] Figure 2 Fig. 2 is a schematic diagram of the ASM block structure of the application;

[0060] Figure 3 Fig. 3 is a schematic diagram of the input feature fusion module FFM structure of the application;

[0061] Figure 4 Fig. 4 is a schematic diagram of the flow of the low-computational-load two-step autocorrelation LD-SCC;

[0062] Figure 5 Fig. 5 is a schematic diagram of the two-branch adaptive fusion module BAFM structure;

[0063] Figure 6 Fig. 6 is a schematic diagram of the multi-scale dilated convolution module MDCM structure;

[0064] Figure 7 Fig. 7 is a schematic diagram of the structure of the replication moving tampering detection network with adaptive sampling mechanism and two-step autocorrelation of the application;

[0065] Figure 8 Fig. 8 is a schematic diagram of the F1 comparison of the six methods on four public data sets;

[0066] Figure 9 Fig. 9 is a schematic diagram of the parameter amount and computational amount comparison of the six methods.

[0067] Embodiment

[0068] To make the purposes, technical solutions and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described clearly and completely below in combination with the drawings in the embodiments of the application.

[0069] As shown in Fig. 1, the application discloses a replication moving tampering detection method with adaptive sampling mechanism and two-step autocorrelation, which comprises a model construction part and a model training and reasoning part, and the specific steps are as follows: Figures 1 to 9 Step 1, model construction: step 1-1, feature extraction: inputting a 3-channel image to be detected with a resolution of 224*224*3 into an AL-Net with ASM-Net as a backbone network;

[0070] Step 1-2, input feature fusion: inputting the feature extracted in step 1-1 into an input feature fusion module FFM;

[0071] As shown in Fig. 1, the application discloses a replication moving tampering detection method with adaptive sampling mechanism and two-step autocorrelation, which comprises a model construction part and a model training and reasoning part, and the specific steps are as follows: Figure 1 ​As shown, ASM-Net comprises four stages. The first two stages each consist of one "CBG 3-2" and two "CBG 3-1" blocks set sequentially, which extract low-level features from the input image to help ASM-Net better detect small-area tampering regions. The latter two stages each consist of one "CBG 3-2" and three ASM blocks with automatically adjustable sampling point numbers and positions, which adaptively extract high-level features from the input image to help ASM-Net better detect large-area tampering regions. The feature maps output by the four stages of ASM-Net are F1, F2, F3, and F4, respectively. Figure 1 The diagram shows the ASM-Net backbone network structure. CBGa-b represents a convolutional block consisting of a standard convolutional layer of size a×a with a stride of b, a batch normalization layer, and a GELU activation function. For example, CBG 3-2 represents a convolutional block consisting of a 3×3 standard convolutional layer with a stride of 2, a batch normalization layer, and a GELU activation function. The ASM block consists of an adaptive sampling mechanism (ASM), two point-wise convolutions (PWConv), and cross-layer connections. The structure of the ASM block is as follows: Figure 2 As shown, Conve-f represents a standard convolution of size e×e with stride f. For example, Conv 1-1 represents a standard convolution of size 1×1 with stride 1.

[0072] ASM first inputs feature map E in The intermediate feature map E is obtained through "Convα-1". mid , shape is “Convα-1” represents a standard convolutional layer with a kernel size of α and a stride of 1. and Indicates the input feature map E in Width and height; β 2 +1 represents the number of output channels for "Convα-1"; where α and β are set to 5 in the third stage of ASM-Net and 3 in the fourth stage of ASM-Net. Intermediate Feature Map E mid Each sampling point E on mid (i,j), It is split into two parts: the first channel and the remaining β. 2 There are 3 channels, where i and j are the sampling points E. mid The x and y coordinates of (i,j); the value of one of the channels. the number of sampling points of the (i, j) position predicted by "Conv a-1", and the rest of the values of the prediction of the i-th channel will be reshaped into a 2-dimensional matrix of shape β x β 2 Each value in the matrix (m, n ∈ [1, β]) represents the weight of each feature point in the β x β range centered at coordinate (i, j) predicted by "Conv a-1". Since the number of sampling points required by each position is different, ASM filters the top largest weight values and assigns the rest of the weight values as 0 to obtain a new weight matrix The formula is shown in equation (4):

[0073]

[0074] wherein, represents the th largest weight value in the matrix, and and represent the floor and ceiling functions, respectively. When ,the purpose of is to let "Conv a-1" automatically learn the number of sampling points required by each position through backpropagation

[0075] Finally, ASM performs L1-Normalization on to obtain and performs weighted summation with the input feature map E in to obtain the output feature map E out , the formula is shown in equation (5):

[0076]

[0077] ​​​​​Step 1-2, Feature Fusion: The feature maps F1, F2, F3 and F4 obtained in step 1-1 are input into a feature fusion module (FFM) for feature fusion. In order to let the AL-Net automatically filter the key information in the four feature maps according to the characteristics of the input image, the FFM first inputs F4 into two "Conv 3-1" for further feature extraction, and then inputs the obtained result into a global average pooling (GAP) layer to obtain weights. The FFM performs element-wise product on the obtained weights and the channel-concatenated F1, F2, F3 and F4, and retains the key information in the four feature maps. The weighted and concatenated feature map is input into "CBG 1-1" and two "FasterNet blocks" for feature fusion, and the output feature map F f of the FFM is obtained. Figure 5

[0078] Step 1-3, Correlation Feature Extraction: The feature map F f obtained in step 1-2 is input into a low-cost dual-step self-correlation (LD-SCC) calculation for low-computing to extract the correlation information in the feature map. In the first step, the LD-SCC downsamples the input feature map F f to obtain a low-resolution feature map Then, according to the correlation between each two features in , the high-similarity region in F f is filtered out; in the second step, the LD-SCC only performs self-correlation calculation (SCC) between the features in the high-similarity region of F f to obtain the output correlation feature map F d . This method not only reduces the influence of redundant low-similarity information and improves the detection performance of the AL-Net, but also significantly reduces the computing cost. The flowchart of the LD-SCC is shown in Figure 4

[0079] Specifically, in the first step, the LD-SCC first performs k times of average pooling downsampling (AP) on the input feature map F f with a shape of to obtain a low-resolution feature map with a shape of wherein ​​The present application sets k as 8. Then, the LD-SCC reshapes F into a matrix with shape and calculates the cosine similarity (CS) between each pair of feature points as shown in equation (6):

[0080]

[0081] where T represents the transpose operation, represents the cosine similarity between the u-th feature point and the v-th feature point in F f and also represents the similarity between the u-th k x k pixel block and the v-th k x k pixel block in F f Finally, the LD-SCC converts F into a tensor with shape For each k x k pixel block in F , the LD-SCC extracts the t k x k pixel blocks with the highest similarity from F , resulting in a tensor with shape where t is set to 20 in the present application.

[0082] In the second step, the LD-SCC calculates the similarity between each feature point in F and the feature points of the t pixel blocks with the highest similarity to the pixel block containing the feature point. To establish the connection between each pixel block in F and the t pixel blocks in F , the LD-SCC first transposes F , then performs matrix multiplication between F and F as shown in equation (7):

[0083]

[0084] Finally, the LD-SCC restores the shape of F to F , then sorts along the channel dimension and keeps the top g values in the channel dimension, resulting in an output feature map with shape where g is set to 32 in the present application.

[0085] Step 1-4, tampered region positioning: the fusion feature map F f ​​​​​The correlation feature map F d is input into a bi-branch adaptive fusion module (BAFM) for fusion, and the structure of the BAFM is as shown in Figure 5 .

[0086] The structure of the multi-scaledilated convolution module (MDCM) is as shown in Figure 6 . DWDCa-b-c represents a deep dilated convolution layer with a size of a x a, a stride of b, and a dilation rate of c. It first uses three deep dilated convolution layers (DWDC) in series to extract multi-scale features on the input feature map. Compared with using three DWDCs in parallel, the series structure can obtain a larger receptive field with the same number of parameters, thereby better capturing multi-scale features and more accurately generating weights. The output of the three feature maps is subjected to point convolution for information interaction between channels, and is subjected to a batch normalization (BN) layer and a nonlinear GELU activation function to ensure the diversity of the features. Finally, the three feature maps are spliced in the channel dimension, input into a point convolution layer for fusion, and an output result is obtained.

[0087] The first branch of the BAFM inputs F d into the MDCM for multi-scale feature extraction, and the second branch inputs F f into the point convolution for information interaction between channels. Then, the output results of the two branches are spliced in the channel dimension and jointly input into the MDCM. The MDCM comprehensively considers the correlation features in F d and the multi-level features in F f , and performs multi-scale feature extraction on both, thereby more accurately generating weights for F d and F f and retaining key information of both. The BAFM splits the obtained weights in the channel dimension to obtain two tensors with the same shape as the two feature maps, and weights the two feature maps. Finally, the BAFM splices the weighted two feature maps in the channel dimension and jointly inputs a lightweight FRSU for feature fusion to obtain the final detection result of the AL-Net. The overall structure diagram of the AL-Net is as shown in Figure 7 .

[0088] Step 2, training and reasoning of the model:

[0089] Step 2-1, data preprocessing: Data preprocessing is a key step in the CMFD task, which can improve the stability of model training and the performance of testing. The data preprocessing of the present application includes the following steps:

[0090] (1) Reading data: The present application reads training samples from the training set of the USC-ISI dataset. Each sample contains an original image and the corresponding label, which contains the class information of each pixel point in the original image.

[0091] (2) Adjusting size: In order to ensure that the size of all training samples is the same, the present application simultaneously scales the original image and the corresponding label. The present application adjusts the length and width size of all images to 320x320. This step ensures that the model can process input images of the same size, simplifying the training process.

[0092] (3) Data augmentation: In this stage, the present application simultaneously rotates and horizontally flips the input image and the corresponding label according to a certain probability. Random rotation and random horizontal flipping can enhance the model's detection ability for different angles and symmetry of images.

[0093] (4) Normalization: The present application uses the statistical data of ImageNet to normalize the image. This dataset contains a large number of images in various scenes, and using its mean and variance for normalization can help the model learn more efficiently, thereby accelerating the convergence speed.

[0094] Step 2-2, model training: First, the present application randomly divides the training data into multiple batches, each containing 16 samples. This batching method can effectively utilize the computing power of GPU, ensuring that the model can process multiple samples in parallel, thereby improving training efficiency. Next, the present application uses the cross-entropy loss function to guide the training process of AL-Net. This loss function evaluates the deviation between the model's prediction and the actual label. By minimizing this loss, AL-Net can more accurately adapt to the training data, thereby improving its performance in the CMFD task. In order to optimize the parameters of the model, the present application chooses the stochastic gradient descent method (SGD). This method adjusts the model parameters by estimating the gradient of the loss function with respect to the parameters, in order to optimize the performance. In addition, the entire training process is set to 60 rounds, each round representing a complete traversal of the entire dataset.

[0095] Step 2-3, inference of the model: The model obtained from step 2-2 is loaded with the weight parameters learned during training, which support subsequent copy-move image forgery detection tasks. First, this step performs necessary preprocessing on the image to be detected, including resizing and normalization, to ensure that it is fully consistent with the parameters used during training. Then, the preprocessed image is input into the trained model. The model analyzes the image by forward propagation and estimates the probability that each pixel belongs to the tampered region. The result is presented in the form of a probability map, where the value of each pixel in the map represents the likelihood of the pixel being tampered with. Finally, this step marks all pixels in the probability map output by AL-Net with a predicted probability greater than 0.5 as tampered pixels, thus completing the accurate identification of the tampered region in the image.

[0096] The effect experiment of the present application shows that the detection performance of AL-Net, BusterNet, CMSDNet, UCM-Net and LHCM-Net is compared, and the evaluation index used is F1-score.

[0097] F1-score combines the two important performance indicators of precision (Precision) and recall (Recall), where Precision is the ratio of the number of pixels that the model correctly predicts as tampered pixels to the total number of pixels that the model predicts as tampered pixels, and the calculation formula of Precision is shown in formula (1).

[0098]

[0099] where TP represents true positive, i.e., the number of pixels that the model correctly predicts as tampered pixels, and FP represents false positive, i.e., the number of pixels that the model incorrectly predicts as tampered pixels.

[0100] Recall is the ratio of the number of pixels that the model correctly predicts as tampered pixels to the total number of tampered pixels, and the calculation formula of Recall is shown in formula (2).

[0101]

[0102] where FN represents false negative, i.e., the number of pixels that the model incorrectly predicts as true pixels.

[0103] F1-score is the harmonic mean of Precision and Recall, used to evaluate the balanced performance of the model on these two indicators. The value of F1-score ranges from 0 to 1, where the closer the value is to 1, the better the balance between precision and recall achieved by the model. The calculation formula of F1-score is shown in formula (3):

[0104]

[0105] The four public data sets of USC-ISI, COVERAGE, CoMoFoD and CASIA are used in the experiment. The USC-ISI data set contains 100,000 copy-move type tampered images, of which 80,000 are training set images, 10,000 are verification set images, and 10,000 are test set images. The training set is selected to train the AL-Net, and the test set, COVERAGE, CASIA and CoMoFoD data sets are selected to test the detection performance of the AL-Net. The CASIA data set contains 1313 pairs of copy-move type tampered samples in the CASIA v2.0 data set. The CoMoFoD data set contains 200 pairs of test samples and 4800 pairs of test samples after different types and different intensity of post-processing attacks. The COVERAEG data set contains 100 copy-move tampered images and their corresponding labels. The test results are shown in Figure 8 The detection performance of the AL-Net proposed in the application is compared with several other advanced CMFD models, and it can be seen that the detection performance of the AL-Net on all data sets exceeds all current networks, except that the detection performance on the COVERAGE data set is the same as that of UCM-Net-L-DenseNet121. Specifically, UCM-Net designs two sub-networks for detecting different sizes of tampered regions, namely UCM-Net-L-DenseNet121 and UCM-Net-S-VGG16, wherein UCM-Net-L-DenseNet121 is a sub-network specially designed for detecting large-sized tampered regions, which uses a deep DenseNet121 backbone network to extract features of large-area tampered regions, so it achieves excellent detection performance on the COVERAGE data set mainly containing large-area tampered regions. However, it performs poorly on the CoMoFoD data set mainly containing small-area tampered regions. Similarly, UCM-Net-S-VGG16 is a sub-network specially designed for detecting small-sized tampered regions, which uses a shallow VGG16 backbone network to extract features of small-area tampered regions on the image, so it can achieve a high F1 on the CoMoFoD data set, but it is poor on the COVERAGE data set. Unlike UCM-Net, the AL-Net proposed in the application benefits from the adaptive sampling mechanism (ASM) that can automatically adjust the receptive field, and can simultaneously achieve excellent detection performance on data sets containing images with different sizes of tampered regions with only a single network architecture. In addition, as shown in Figure 9As shown, AL-Net has the lowest parameters and floating point operations (FLOPs) thanks to the low-cost dual-step self-correlation calculation (LD-SCC) and the bi-branch adaptive fusion module (BAFM) with low parameter quantity, which powerfully proves the high efficiency of AL-Net.

[0106] The present application adopts the above technical scheme. Since the current CMFD network adopts a standard convolutional layer for feature extraction, and the sampling position of the standard convolutional layer is fixed, it is difficult to effectively extract the features of tampered regions with different areas and shapes. In order to solve this problem, the present application proposes an adaptive sampling mechanism (ASM) which automatically learns the sampling point position and sampling point quantity required by tampered regions with different areas and shapes through a convolutional layer, thereby more accurately capturing the features of tampered regions. In addition, based on the ASM, the present application proposes an ASM block which first converges features in the spatial dimension through the ASM, and then fuses the converged features in the channel dimension using two point convolutions. Compared with variable convolution, the ASM block has smaller computational overhead. Based on the ASM block, the present application constructs a new backbone network, ASM-Net. ASM-Net extracts local context features of different ranges through layers of different depths, thereby providing AL-Net with information different from scale for detecting tampered regions, and enhancing the detection performance of AL-Net.

[0107] Since directly applying the self-correlation module to high-resolution feature maps will bring huge computational overhead, most current CMFD networks only use the self-correlation module on low-resolution feature maps, however, this inevitably leads these networks to ignore the similarity between high-resolution features, reducing the performance of the model. In order to solve this problem, the present application proposes a low-cost dual-step self-correlation (LD-SCC) which first screens high-similarity regions on high-resolution feature maps through the correlation of features on low-resolution feature maps, excludes redundant information, and then only calculates the correlation of features between high-similarity regions, which not only greatly reduces the computational overhead of AL-Net, but also maintains excellent detection performance.

[0108] The existing CMFD network only utilizes the correlation feature after the self-correlation module to obtain the tamper detection result, and ignores the local context features extracted by the backbone network, which leads to inaccurate prediction of the edges and the interior of the tamper region by the model. In order to solve this problem, the present application proposes a bi-branch adaptive fusion module (BAFM), which adaptively fuses the local features before the LD-SCC and the correlation features after the LD-SCC by using a lightweight multi-scaled dilated convolution module (MDCM), thereby improving the detection performance of the AL-Net.

[0109] The present application combines the three structures and designs a CMFD network AL-Net which is lightweight, low in calculation overhead and high in precision. The experimental results on four public data sets show that the AL-Net is superior to the above related CMFD networks in terms of detection accuracy, parameter quantity and calculation amount.

[0110] Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

Claims

1. A method for copy-move forgery detection using adaptive sampling mechanism and two-step auto-correlation, characterized in that: It includes a model construction part, a model training and inference part, and the specific steps are as follows: Step 1, construct an adaptive sampling mechanism and a double-step autocorrelation replication tamper detection network model, and the network model performs the following steps: Step 1-1, feature extraction: input the image to be detected into AL-Net with ASM-Net as the backbone network to extract low-level and high-level features for detecting different area sizes; ASM-Net contains four stages, the first two stages are composed of one "CBG 3-2" and two "CBG 3-1" arranged in turn, which are used to extract low-level features on the input image, thereby helping AL-Net better detect small-area tamper regions; The latter two stages are composed of one "CBG 3-2" and three ASM blocks with automatically adjustable sampling point number and position arranged in turn, which are used to adaptively extract high-level features on the input image, thereby helping AL-Net better detect large-area tamper regions; The feature maps output by the four stages of ASM-Net are F1, F2, F3 and F4 respectively; Wherein, CBG 3-2 represents a convolution block composed of a 3x3 size, a stride of 2 standard convolution layer, a Batch Normalization layer and a GELU activation function; CBG 3-1 represents a convolution block composed of a 3x3 size, a stride of 1 standard convolution layer, a Batch Normalization layer and a GELU activation function; ASM block is composed of an adaptive sampling mechanism ASM and two point convolution PWConv and cross-layer connection; Step 1-2, feature fusion: input the obtained feature maps F1, F2, F3 and F4 into a feature fusion module FFM to perform feature fusion to obtain an output feature map F f ; Step 1-3, correlation feature extraction: the feature map F obtained in step 1-2 is input into a low-computational double-step autocorrelation LD-SCC to extract correlation information in the feature map f The low-computational double-step autocorrelation LD-SCC is used to extract the correlation information in the feature map. Steps 1-4: Tampering with the region location: The fused feature map F obtained in Step 1-2 is... f Correlation feature map F obtained in steps 1-3 d The input is fused using the dual-branch adaptive fusion module BAFM; the first branch of BAFM will F d Input MDCM for multi-scale feature extraction, the second branch will F f Input point convolutions facilitate information exchange between channels; the outputs of the two branches are concatenated along the channel dimension and input into the MDCM; the MDCM will comprehensively consider F... d Correlation characteristics and F f The multi-level features in the model are extracted, and multi-scale features are extracted from both, thus providing F d and F f Generate weights while preserving key information from both; BAFM splits the weights along the channel dimension to obtain two tensors with the same shape as the feature maps, and weights the two feature maps; BAFM concatenates the two weighted feature maps along the channel dimension and inputs them into a lightweight FRSU for feature fusion to obtain the tamper region localization detection result of AL-Net; Step 2, model training and inference: Step 2-1, data preprocessing: read the training samples from the training set of the USC-ISI dataset for preprocessing to obtain the training data required for training; Preprocessing includes size adjustment, data enhancement and normalization processing; Step 2-2, model training: randomly divide the training data into multiple batches, and use the cross-entropy loss function to guide the training process of AL-Net; Select the stochastic gradient descent method (SGD) to adjust the model parameters by estimating the gradient of the loss function with respect to the parameters to optimize the performance; After traversing the training for a certain number of rounds, the model training is completed to obtain the final model; Step 2-3, model inference: preprocess the image to be detected so that the parameters used are exactly the same as those used during model training, then input the final model to perform forward propagation to analyze the image and estimate the probability of each pixel belonging to the tamper region in the form of a probability map, and mark all pixels in the probability map whose predicted probability exceeds a certain value as tampered pixels, thereby accurately identifying the tamper region of the image.

2. The adaptive sampling mechanism and copy-move forgery detection method of double-step auto-correlation according to claim 1, characterized in that: The adaptive sampling mechanism ASM in step 1-1 performs the following steps: Step 1-1-1, first input feature map Ein is obtained by "Conv a-1" to get intermediate feature map E mid , shape is "Conv a-1" represents a standard convolution layer with kernel size a and step size 1; and represent the width and height of input feature map E in ; β 2 +1 is the number of output channels of "Conv a-1"; Step 1-1-2, intermediate feature map E mid each sampling point E mid (i, j) is split into the first channel and the rest of the β 2 channels two parts, where i and j are the horizontal and vertical coordinates of the sampling point Emid(i, j), E mid the value of the first channel in (i, j) represents the number of sampling points at the (i, j) position predicted by "Conv α-1", and the prediction values of the rest of the β 2 channels will be reshaped into a 2-dimensional matrix of β × β shape each value (m, n ∈ [1, β]) represents the weight of each feature point in the β × β range centered at the coordinates (i, j) predicted by "Conv α-1"; Step 1-1-3, since the number of sampling points required for each position is different, the ASM assigns the predicted number of sampling points according to the number of sampling points required for each position Screening Mid-fore The largest weight value is obtained, and the remaining weight values are assigned as 0 to obtain a new weight matrix The formula is shown in formula (4): wherein denotes the largest weight value, and respectively represent the floor and ceiling functions. Step 1-1-4, ASM will L1-Normalization is performed to obtain and the input feature map E in weighted sum is performed to obtain the output feature map E out The formula is shown in equation (5):

3. The adaptive sampling mechanism and two-step auto-correlation replicated move tamper detection method of claim 2, wherein: Alpha and beta are set to 5 in the third stage of ASM-Net and 3 in the fourth stage of ASM-Net.

4. The adaptive sampling mechanism and two-step auto-correlation replicated move tamper detection method of claim 2, wherein: Step 1-2 specifically includes the following steps: Step 1-2-1, input the feature map F4 into two "Conv 3-1" for further feature extraction; Conv 3-1 represents a standard convolution with a size of 3x3 and a stride of 1; Step 1-2-2, input the result obtained in step 1-2-1 into a global average pooling layer GAP to obtain weights; Step 1-2-3, element-wise multiply the weights obtained in step 1-2-2 with the channel-spliced F1, F2, F3 and F4 to obtain a weighted spliced feature map, retaining key information in the four feature maps; Step 1-2-4, input the weighted spliced feature map into "CBG 1-1" and two "FasterNet blocks" for feature fusion to obtain an output feature map F f .

5. The adaptive sampling mechanism and two-step auto-correlation replicated move tamper detection method of claim 2, wherein: Step 1-3 specifically comprises the following steps: Step 1-3-1, LD-SCC takes input feature map F f Down-sampling to get low resolution feature map Then according to Correlation between each pair of features in the middle, filter out F f Highly similar areas in Step 1-3-2, LD-SCC only in F f The output correlation feature map F is obtained by performing a self-correlation calculation SCC between the features of the high-similarity regions d .

6. The adaptive sampling mechanism and copy-move forgery detection method of double-step auto-correlation of claim 5, wherein: Step 1-3-1 specifically comprises the following steps: Step 1 - 3 - 1 - 1, the LD-SCC first performs k times average pooling downsampling, AP, on the input feature map F f to obtain a low resolution feature map with shape ​​ Step 1-3-1-2, LD-SCC will further reshape into a matrix of shape and calculate cosine similarity CS between each pair of feature points, as shown in equation (6):​ wherein T represents a transpose operation, represents a cosine similarity between an u-th feature point and a v-th feature point, and represents F f a similarity between an u-th k x k size pixel block and a v-th k x k size pixel block. Step 1-3-1-3, LD-SCC converts F f into a tensor of shape For each k x k size pixel block in F , LD-SCC extracts the t k x k size pixel blocks with the highest similarity according to the similarity information in F , and gets a tensor of shape ​​ 7. The adaptive sampling mechanism and two-step auto-correlation replicated move tamper detection method of claim 5, wherein: Step 1-3-2 specifically comprises the following steps: Step 1-3-2-1, LD-SCC will Each feature point in the image block only calculates similarity with the feature points of the t pixel blocks with the highest similarity to the pixel block where the feature point is located. Step 1 - 3 - 2 - 2, to establish the relationship between each pixel block in and t pixel blocks in , the LD-SCC first transposes , then performs matrix multiplication between and , the formula is shown in formula (7): Step 1-3-2-3, LD-SCC reduces the shape of to Then sort along the channel dimension and keep the top g largest values in the channel dimension to get the output feature map of shape ​ 8. The adaptive sampling mechanism and two-step auto-correlation replicated move tamper detection method of claim 1, wherein: The multi-scale dilated convolution module DWDC in step 1-4 first uses three deep dilated convolution layers DWDC in series to extract multi-scale features on the input feature map to obtain three feature maps; the three feature maps are subjected to information interaction between channels through point convolution, and are subjected to batch normalization BN layer and non-linear GELU activation function; finally, the three feature maps are spliced in the channel dimension and fused with the input point convolution layer to obtain the final output result.

9. The adaptive sampling mechanism and two-step auto-correlation replicated move tamper detection method of claim 1, wherein: The data preprocessing of step 2-1 comprises the following steps: (1) reading data: reading training samples from the training set of the USC-ISI dataset, each sample containing an original image and a corresponding label, the label containing class information of each pixel point in the original image; (2) adjusting the size: scaling the original image and the corresponding label at the same time, and adjusting the length and width size of all images to 320x320; (3) data augmentation: randomly rotating and horizontally flipping the input image and the corresponding label according to a certain probability to enhance the detection ability of the model to different angles and symmetry of the image; (4) normalization: normalizing the image using the statistical data of ImageNet, and normalizing using the mean and variance of the ImageNet dataset can help the model learn more efficiently, thereby accelerating the convergence speed.

10. The adaptive sampling mechanism and two-step auto-correlation replicated move tamper detection method of claim 1, wherein: The training process of step 2-2 is set to 60 rounds, and the set value of step 2-3 is 0.5.

Images