Digital image tampering detection method based on improved HRNet

Abstract

According to the digital image tampering detection method based on the improved HRNet, high-precision detection and positioning of local region tampering are achieved in combination with a multi-scale feature extraction structure. According to the invention, through the improved attention fusion module, the recognition capability of the model on tampering-related regions from RGB streams and SRM noise streams is improved. And then local and global feature extraction is performed on each scale through a multi-scale feature extraction module based on HRNet, so that the model can have multi-scale perception capability at the same time. And for detection and positioning of the tampered area, a pyramid-like structure with a channel related module is introduced, so that a manipulation mask of the tampered area is generated from coarse to fine, and the accuracy of positioning the tampered area by the final model is improved. In addition, due to the fact that no large-scale standard data set focuses on image tampering detection, a synthetic data set is constructed to train and verify the provided model. The data set comprises a splicing image, a copying-moving image, a removing type image and an original type image, and the total number of the images is 376 and 742. Compared with other digital image tampering detection methods based on deep learning, the method provided by the invention has higher accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for detecting digital image tampering based on an improved HRNet. Background Art

[0002] With the widespread application of information technology, digital images have become very common in people's daily lives and work. As the visual basis of computers, they have become the most commonly used information carrier in various activities such as scientific research and news media. However, with the development of image processing software, the processing of digital images has become increasingly simple. Ordinary users can easily change certain attributes and specific areas of digital images, making it difficult to identify the authenticity and original source of digital images. Therefore, how to use machines to digitally, automatically and efficiently process and analyze these images, and extract and locate the tampered areas in the images has become a scientific issue that has attracted much attention.

[0003] Digital image tampering detection aims to identify tampered digital images by analyzing their features and statistical information, and using image processing and computer vision techniques. Its primary goal is to protect the authenticity and credibility of image information, prevent the spread of false information, and enhance information security and fairness in society. Furthermore, the rapid development of deep learning, which is increasingly being used in the field of image tampering, is continuously improving the performance and robustness of detection algorithms through diverse model architectures and methods. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the present invention provides a method for digital image tampering detection based on deep learning, which has stronger generalization ability, higher positioning accuracy and more accurate detection effect.

[0005] The technical solution adopted by the present invention is: a method for detecting digital image tampering based on deep learning, comprising the following steps:

[0006] Step 1: Build a large-scale image dataset containing four categories of images: splicing-type tampering, copy-move-type tampering, removal-type tampering, and original-type images, and divide it into training and test sets.

[0007] Step 2: Input the prepared dataset into the dual-branch feature extraction network, fuse the features of the two branches and send them as input to the subsequent multi-scale feature extraction network, and perform feature extraction of different scales on the input fusion features.

[0008] In step 3, the detection and localization module predicts the detection score based on the features at four scales from the feature extraction module, and then generates the operation mask, i.e., the mask of the tampered area, through a fully supervised progressive mechanism.

[0009] Step 4: Use the training set to train the proposed network model from scratch to obtain the optimal parameters, and use the test set to verify the trained network model to obtain the final detection results.

[0010] Furthermore, in step 1, a dataset for training and testing was generated based on the large-scale MS COCO dataset. This dataset includes four categories of images: spliced, copy-move, removed, and original, totaling 376,742 images. Each tampered image file includes the tampered image, the corresponding mask image, and a named folder.

[0011] Furthermore, in step 2, the input dual-branch RGB stream and SRM stream are fused using an attention feature fusion module, and after fusion, they are input into the multi-scale feature extraction structure based on HRNet.

[0012] Because commonly used summation or concatenation fusion methods have certain drawbacks, this paper uses a feature fusion module based on the attention mechanism. Inspired by SENet, this paper combines local and global features on CNN and uses the attention mechanism to fuse multi-scale information in space. The channel attention mechanism dynamically adjusts the importance of each channel, solving the problem of fusing features of different scales. The feature X′ obtained by this mechanism can be expressed as: in, g(x) adds a global maximum pooling before L(x). Assuming that the multi-scale channel attention module is M, the attention feature fusion module can be expressed as: Where Z∈R C×H×W is the characteristic of fusion, and For initial feature integration.

[0013] The multi-scale feature extraction architecture is based on the HRNet network and has been modified. The original HRNet structure is changed from a progressive structure to a four-branch parallel structure consisting of four stages. Each level has multiple branches, each processing features of different resolutions, allowing the network to retain both coarse and fine feature information at different levels.

[0014] Furthermore, in step 3, the detection and localization module is a densely connected pyramid-like structure. It first predicts a mask at the coarsest scale to locate areas that may have been tampered with, rather than directly generating it at the finest scale. This increases prediction accuracy, and the prediction references the mask predicted at the previous scale, i.e., the tampered area, allowing the model to focus more on those selected areas. This process continues until the finest manipulation mask is generated, and the output mask serves as the final prediction of the tampered area.

[0015] Denote the input image as I∈R H×W×3 , the features of the four scales output by the feature extraction module are represented as F1∈R H×W×C , F2∈R H / s×W / s×sC , and Denote the corresponding mask as M1∈R H×W , M2∈R H / s×W / s , and H, W, C are the height, width and number of channels of the image / feature respectively. They can be: M n-1 =f n-1 (τ(M n )·F n-1 ), n=2,3,4, Among them, f n represents the channel-related module at the nth scale, τ represents the operation adopted, and M4 can be directly expressed as f4 (F4).

[0016] However, this approach is disordered and random, and without explicit regional supervision at each scale, the intermediate output mask may not follow the order from coarse to fine. Therefore, full supervision is applied at all scales to guide mask prediction, and an attention mechanism is used to constrain the intermediate output mask.

[0017] Assume that the input feature is X∈R H×W×C , reshape it into To prevent the memory from being too large, r is a downscaling ratio r based on the module after the feature map is flat. In this invention, 4, 2, 2, and 1 are used. Then the Gaussian function is introduced and three functions are constructed respectively through 1×1 convolution: g, θ, and φ, which convert X′ into a new linear embedding with the same size as the input reshaped feature X′. Then the channel correlation of the two embedded features θ and φ is calculated and recorded as Finally, the spatial and channel attention mechanisms are implemented through matrix multiplication.

[0018] Note on space: Among them, Y′ s Refers to the features generated after spatial attention operation, softmax(·) is the softmax function. (i, j) represents X′ in As θ The eigenvector of the i-th row and X′ φ The more similar the two feature vectors are, the higher the correlation is. This helps the network learn feature representations to distinguish tampered areas from original areas and avoid overfitting to a specific feature during training.

[0019] Similarly, channel correlation can be expressed as: Among them, Y′ c Refers to the features generated after channel attention operation, (i, j) represents X' in Ac θ The eigenvector of the i-th row and X′ φ Since responses from different channels may be associated with the same class (e.g., manipulated or original), channel-wise correlation aggregates feature maps according to their similarity to enhance the representativeness of the tampered region.

[0020] Subsequently, the two output features are reshaped and improved by 1×1 convolution to obtain two complementary feature representations w s and w c Finally, referring to residual learning, the final output feature Z can be expressed as: Z=X+α s ·ω s (Y s )+α c ·ω c (Y c ) Among them, α s and α c are two learnable parameters, initialized to 1. The final output of the channel correlation module is a prediction mask with only one channel.

[0021] A detection head determines whether the input image has been tampered with. This is essentially a simple classifier that categorizes the input data to complete the detection. The detection head module post-processes the model output, converting the raw logarithmic probabilities into a probability distribution. It then performs a binary classification prediction based on the category with the highest probability. If the image is tampered with, the output is 'forged'; otherwise, it is 'authentic', completing the detection of the input image.

[0022] This project uses the binary cross entropy loss function (L bce ). Predicted detection score s d By groundtruth (GT) label (l d ) supervision, 0 represents the original image and 1 represents the tampered image. And perform full supervision on each predicted mask, downsample the GT mask G1 to G2, G3, and G4 according to its corresponding size. 0 represents the original pixel and 1 represents the forged pixel. Masks of different scales predicted by the progressive mechanism are considered equally important. The final loss function It can be expressed as:

[0023] Furthermore, in step 4, the proposed network model is trained from scratch using the prepared dataset to obtain optimal parameters. The present invention uses the ImageNet pre-trained weights 'hmet_w18_small_v2.pth' to initialize the backbone, and the entire model is optimized by ADAM with a batch size of 8 and an initial learning rate of 2e-4, for a total of 100 epochs. The entire model training process is performed on the NVIDA GPU platform NVIDIA TITAN Xp. Finally, using the model weights from 100 epochs, the image to be detected is fed into the network. After detection and positioning, the final result is whether the image is tampered with and the location of the tampered area.

[0024] Compared with existing technologies, the present invention has the following advantages: Based on the HRnet network, the dual-branch, multi-scale feature extraction structure with an attention mechanism improves the model's ability to identify tampered region features; the progressive coarse-to-fine detection and positioning module avoids directly predicting tampered regions at the finest scale, thereby increasing prediction accuracy. Compared with other deep learning-based image tampering detection methods, the present invention has higher detection accuracy and better generalization.

[0025] The present invention will be further described below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a schematic diagram of the network framework implementation process of the present invention.

[0027] Figure 2 The attention fusion module used in this paper.

[0028] Figure 3 Schematic diagram of the channel-related modules used for detection and positioning in the present invention.

[0029] Figure 4Schematic diagram of the visualization of Mask4, Mask3, Mask2 and Mask1 of the present invention

[0030] Figure 5 This is a diagram showing the model detection effect in the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described below with reference to the accompanying drawings.

[0032] The present invention provides a digital image tampering detection method based on an improved HRNet, comprising the following steps:

[0033] Step 1: Construct a large-scale image dataset based on MS COCO, which contains four categories of images: splicing, copy-move, removal, and original images. The dataset contains 376,742 images in total, and divides them into training and test sets in a ratio of 75:25. Each tampered image file includes an image, a mask map of the corresponding tampered area, and a named folder.

[0034] Step 2, Figure 1 It is a schematic diagram of the overall process of implementing the method of the present invention. The left side shows a dual-branch multi-scale feature extraction module. After the image is input, RGB features and SRM features are extracted through the RGB stream and SRM stream respectively. The RGB stream models the tampering traces, and the SRM stream extracts noise features to provide a basis for image processing and classification. After extracting the features of the two scales, they pass through the attention fusion module and then enter a multi-branch feature extraction module based on the HRNet network to obtain features of different resolutions. Features of different scales are output for subsequent tampering positioning and image classification.

[0035] Figure 2 Schematic diagram of the attention fusion module structure used in this invention, (a) is the multi-scale channel attention module, (b) is the attention feature fusion module. Inspired by SENet, it combines local and global features on CNN and uses the attention mechanism to fuse multi-scale information in space. Figure 2 As shown in (a), the local context is added to the global context in the attention module. At the same time, point-wise convolution (PWConv) is selected as the context aggregator of the local channel. Among them, the kernel sizes of PWConv1 and PWConv2 are and B is the BatchNorm layer, δ is the activation function, and the number of channels is restored to the same as the original input through PWConv1.

[0036] The global channel context and local channel context are g(X) and L(X) respectively. Compared with ordinary image feature fusion, image tampering detection should pay more attention to the most significant features in the feature map. Therefore, the global maximum pooling (GMP) is introduced in the attention module to make the model pay more attention to the edge and texture information in the image. g(X) adds a global maximum pooling operation before L(X). The refined feature X′∈R of MS-CAM C×H×W It can be expressed as: Where M(X)∈R C×H×W Attention weights generated for the attention module. represents the exclusive OR operation, Represents element-wise multiplication.

[0037] Suppose that given two mappings X, Y∈R C×H×W , where Y is a feature map with a larger receptive field. In the present invention, Y inputs the RGB feature stream. Assuming that the multi-scale channel attention module is M, the attention feature fusion (AFF) can be expressed as: Where Z∈R C×H×W is the characteristic of fusion, and For initial feature integration.

[0038] The multi-branch feature extraction network is based on the HRNet network, modifying the original HRNet structure from a progressive structure to a four-branch parallel structure containing four stages. Each level has multiple branches, each processing features of different resolutions, allowing the network to retain both coarse and detailed feature information at different levels. Each stage integrates feature maps of different resolutions through multi-scale information between high-resolution and low-resolution features, continuously exchanging and fusing features, effectively exchanging information and better handling scale changes. In addition, local and global features are fused at each scale, ensuring that the features output by each branch contain sufficient information to predict the manipulation mask at the corresponding scale. After extracting the manipulated features at four scales, they are input into subsequent modules for manipulation detection and localization.

[0039] Step 3, Figure 1The right part is the detection and positioning module of the present invention. The present invention simulates the thinking of solving complex problems in human daily life, abandons the method of directly generating the minimum scale mask of the traditional model, and proposes a detection and positioning module with a densely connected pyramid structure. According to the feature information of different scales output by the feature extraction module, the mask of the coarsest scale is first predicted to locate the area that may be tampered with, rather than directly generating it at the finest scale to increase the accuracy of the prediction, and the mask predicted at the previous scale, that is, the tampered area, is referenced during the prediction, so that the model pays more attention to those selected areas. This process will continue until the finest operation mask is generated, and the output mask will be used as the prediction of the final tampered area.

[0040] Denote the input image as I∈R H×W×3 , the features of the four scales output by the feature extraction module are represented as F1∈R H×W×C , F2∈R H / s×W / s×sC , and Denote the corresponding mask as M1∈R H×W , M2∈R H / s×W / s , and H, W, C are the height, width and number of channels of the image / feature respectively, and they are: M n-1 =f n-1 (τ(M n )·F n-1 ), n=2,3,4, Among them, f n represents the channel correlation module on the nth scale, and τ represents the operation. M4 can be directly expressed as f4(F4). 1-3 , f i The features on the previous scale f i-1 The upsampled mask is associated with the tampered region for feature selection. The selected features are then input into the channel correlation module to generate the operation mask, i.e., the tampered region.

[0041] Figure 3 This is a schematic diagram of the channel-related module structure of the network of the present invention. Assume that the input feature is X∈R H×W×C , reshape it into To prevent excessive memory usage, r is the downscaling ratio r after the feature map is flattened. Then, a Gaussian function is introduced and three functions are constructed through 1×1 convolution: g, θ, and φ, respectively, to transform X′ into a new linear embedding with the same size as the input reshaped feature X′. The channel correlation between the two embedded features θ and φ is then calculated and recorded as Finally, the spatial and channel attention mechanisms are implemented through matrix multiplication.

[0042] Note on space: Among them, Y′ s Refers to the features generated after spatial attention operation, softmax(·) is the soffmax function. (i, j) represents X′ in As θ The eigenvector of the i-th row and X′ φ The more similar the two feature vectors are, the higher the correlation is. This helps the network learn feature representations to distinguish tampered areas from original areas and avoid overfitting to a specific feature during training.

[0043] Similarly, channel correlation can be expressed as: Among them, Y′ c Refers to the features generated after channel attention operation, (i, j) represents X' in Ac θ The eigenvector of the i-th row and X′ φ Since responses from different channels may be associated with the same class (e.g., manipulated or original), channel-wise correlation aggregates feature maps according to their similarity to enhance the representativeness of the tampered region.

[0044] Subsequently, the two output features are reshaped and improved by 1×1 convolution to obtain two complementary feature representations w s and w c Finally, referring to residual learning, the final output feature Z can be expressed as: Z=X+α s ·ω s (Y s )+α c ·ω c (Y c ) Among them, α s and α c are two learnable parameters, initialized to 1. The final output of the channel correlation module is a prediction mask with only one channel.

[0045] Figure 4 The figure shows a visualization diagram of Mask4, Mask3, Mask2, and Mask1 of the present invention, where GT is the given tampering mask. It can be seen that the positioning performance is gradually improved from Mask4 to Mask1, with a lower false alarm rate and clearer boundaries. The predicted mask is gradually refined and is basically the same as the given mask map, which is in line with the idea of ​​progressively predicting the image tampering area from coarse to fine in the present invention.

[0046] A detection head determines whether the input image has been tampered with. This is essentially a simple classifier that categorizes the input data to complete the detection. The detection head module post-processes the model output, converting the raw logarithmic probabilities into a probability distribution. It then performs a binary classification prediction based on the category with the highest probability. If the image is tampered with, the output is 'forged'; otherwise, it is 'authentic', completing the detection of the input image.

[0047] This project uses the binary cross entropy loss function (L bce ). Predicted detection score s d By groundtruth (GT) label (l d ) supervision, where 0 represents the original image and 1 represents the tampered image. Full supervision is performed on each predicted mask, and the GT mask G1 is downsampled to G2, G3, and G4 according to its corresponding size. 0 represents the original pixel and 1 represents the forged pixel. Masks of different scales predicted by the progressive mechanism are considered equally important. The final loss function can be expressed as:

[0048] Furthermore, in step 4, the proposed network model is trained from scratch using the prepared dataset to obtain the optimal parameters. The present invention uses the ImageNet pre-trained weights 'hrnet_w18_small_y2.pth' to initialize the backbone, and the entire model is optimized by ADAM with a batch size of 8 and an initial learning rate of 2e-4, for a total of 100 epochs. The entire model training process is performed on the NVIDA GPU platform NVIDIA TITAN Xp. Finally, using the model weights of 100 epochs, the image to be detected is fed into the network. After detection and positioning, the final result is whether the image is tampered with and the location of the tampered area.

[0049] Figure 5 This is the effect diagram of the tampered image detection of the present invention. It can be seen that the false alarm rate predicted by this model is low and is consistent with the actual tampered area.

[0050] While the present invention has been disclosed above with reference to preferred embodiments, this is not intended to limit the present invention. Persons skilled in the art will readily appreciate that various modifications and variations can be made without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A digital image tampering detection method based on an improved HRNet, comprising the following steps: Step 1: Build a large-scale image dataset containing four categories of images: splicing-type tampering, copy-move-type tampering, removal-type tampering, and original class, and divide it into training and test sets; In step 2, the prepared dataset is input into the dual-branch RGB feature and SRM feature extraction network to obtain the feature map of the image. The features of the two branches are then fused and input into the subsequent multi-scale feature extraction network to perform feature extraction of different scales on the input fusion features. In step 3, the detection and localization module predicts the detection score based on the features at four scales from the feature extraction module, and then generates the operation mask, i.e., the mask of the tampered area, through a fully supervised progressive mechanism. Step 4: Use the training set to train the proposed network model from scratch to obtain the optimal parameters, and use the test set to verify the trained network model to obtain the final detection results.

2. The method according to claim 1, wherein: The specific process of step 1 is as follows: A training and testing dataset is generated based on the large-scale MS COCO dataset. This dataset includes 376,742 images in four categories: spliced, copy-move, removed, and original. Each tampered image file contains the tampered image, the corresponding mask, and a named folder. The dataset is divided into training and test sets.

3. The method according to claim 1, wherein: In step 2, the model extracts the RGB features and SRM features of the input image respectively, and then fuses them through a fusion module with an attention mechanism. Global maximum pooling (GMP) is introduced in the attention module so that the model pays more attention to the edge and texture information in the image during fusion. At the same time, the features of the other branch L(x) have the same shape as the input features, which can retain and highlight the subtle details in the low-level features. This operation enables the model to take into account both global and local information when fusing features. The fused features are input into a multi-branch feature extraction module based on the HRNet network, and then the features of different resolutions are obtained. Features of different scales are output for subsequent tampering localization and image classification. (a) Dual-branch feature fusion module The feature X′ after the attention mechanism module can be expressed as: in, g(x) adds a global maximum pooling before L(x). Assuming that the multi-scale channel attention module is M, the attention feature fusion module can be expressed as: Where Z∈R C×H×W is the characteristic of fusion, and For initial feature integration.

4. The method according to claim 1, wherein: In step 3, the detection and localization module first predicts a mask at the coarsest scale to locate areas that may have been tampered with. This avoids directly generating a mask for the tampered area at the finest scale, thereby increasing prediction accuracy. Subsequent predictions at finer scales can reference and leverage the mask estimated at the previous scale, focusing more on those selected areas. This process continues until the finest manipulation mask is generated, and the output mask serves as the final prediction of the tampered area. The details are as follows: (a) Denote the input image as I∈R H×W×3 , the features of the four scales output by the feature extraction module are represented as F1∈R H×W×C , F2∈R H / s×W / s×sC , and Denote the corresponding mask as M1∈R H×W , M2∈R H / s×W / s , and H, W, C are the height, width and number of channels of the image / feature respectively. They can be: M n-1 =f n-1 (τ(M n )·F n-1 ),n=2,3,4, Among them, f n represents the channel-related module at the nth scale, τ represents the operation adopted, and M4 can be directly expressed as f4 (F4). (b) For the channel-dependent module, assuming the input feature is X∈R H×W×C , reshape it into To prevent the memory from being too large, we introduce the Gaussian function and construct three functions: g, θ, and φ respectively through 1×1 convolution to transform X′ into a new linear embedding with the same size as the input reshaped feature X′. Then we calculate the channel correlation of the two embedded features θ and φ, which is recorded as Note on space: Among them, Y′ s Refers to the features generated after spatial attention operation, Note on the channel: Among them, Y′ c Refers to the features generated after channel attention operation, Subsequently, the two output features are reshaped and improved by 1×1 convolution to obtain two complementary feature representations w s and w c Finally, referring to residual learning, the final output feature Z can be expressed as: Z=X+α s ·oh s (Y s )+a c ·oh c (Y c ) Among them, α s and α c are two learnable parameters, initialized to 1. The final output of the channel correlation module is a prediction mask with only one channel.

5. The method according to claim 1, wherein: In step 4, the proposed network model is trained from scratch using the prepared dataset to obtain optimal parameters. The present invention uses the ImageNet pre-trained weights 'hrnet_w18_small_v2.pth' to initialize the backbone, and the entire model is optimized by ADAM with a batch size of 8 and an initial learning rate of 2e-4, for a total of 100 epochs. The entire model training process is performed on the NVIDA GPU platform NVIDIA TITAN Xp. Finally, using the model weights from 100 epochs, the image to be detected is fed into the network. After detection and positioning, the final result is the determination of whether the image is tampered and the location of the tampered area.

Images