Multi-feature fusion image forgery detection method based on high-resolution network

Abstract

The invention discloses a multi-feature fusion image forgery detection method based on a high-resolution network. The method comprises the following steps: (1) training a binary classifier by using a real image and a manipulated image, and storing a model; (2) inputting the image into a pre-trained classifier; calculating the gradient of the image; (3) inputting the generated gradient map and the original image into a feature extraction module to extract gradient features and color space information; the features of the two branches are fused through a convolutional network; (4) inputting the original image into a frequency domain feature extraction block, and extracting high-frequency features of the image; (5) further fusing the fused gradient features and color space features into frequency domain features; a high-resolution network is used as a backbone network in a feature extraction and fusion stage; and (6) the fused features are sent to a discriminator, and whether the input image is real or forged is judged. The method has higher recognition accuracy and robustness, and can be well generalized to unseen counterfeiting methods.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and information security, and specifically relates to the field of image deep forgery detection in the field of artificial intelligence computer vision, and more specifically to a multi-feature fusion image forgery detection method based on a high-resolution network. Background Art

[0002] The advent of generative adversarial networks has made it possible to generate highly realistic images that are almost indistinguishable from real photos. In recent years, diffusion models have established a new paradigm in the field of image generation. Generative models are able to create and edit high-quality images and videos, and are widely used in autonomous driving, game development, virtual try-ons, medical imaging, virtual reality, and data enhancement to improve visual effects and user experience. Advances in image generation technology have also lowered the threshold for malicious use, leading to an increase in highly realistic fake content in multimedia. Attackers may use these technologies for malicious purposes, such as making pornographic videos or fabricating political statements, stealing identities and financial fraud through face-changing, and creating a crisis of trust on social media. Therefore, detecting synthetic images has become critical.

[0003] Previous work on forged image detection has focused on content synthesized by generative adversarial networks. Due to the fundamental differences in the generation process between generative adversarial networks and diffusion models, some valid artifacts no longer exist in diffusion-generated images, making the latter more difficult to detect. Currently, there is no method that can effectively detect images generated by both models in the absence of training data diversity. Therefore, it is challenging to develop a unified and robust detection algorithm.

[0004] Chinese patent application No. 2023102917419 discloses a face image detection method and device based on multi-scale multi-level feature fusion. The patent first enlarges the face image to be detected to obtain a high-resolution image; two high-level feature maps are obtained after all convolutions of two different resolution images on two branches in the feature extraction network, and two low-level feature maps corresponding to several convolutions in the middle of the two branches are extracted at the same time, and then the different scale information of the two low-level feature maps is fully learned to obtain the fused low-level feature map; finally, the fused low-level feature map and the two high-level feature maps are hierarchically fused to ensure that the feature information extracted by the low-level features is not lost, and can be fused with the extracted high-level features, extracting and utilizing more general features in face forgery, improving the generalization ability of the model for face detection, and thus improving the accuracy of judging whether the face image is forged; finally, judging whether the face is forged based on the fused feature map.

[0005] However, the multi-feature fusion image forgery detection method based on a high-resolution network first proposed by the inventor is essentially different from the patent. First, the patent needs to change the resolution of the input image, while the present method does not change the input resolution. Second, the input of the branch network in the patent is the original image, while the input of the present method is the feature map after convolution. Before inputting the branch network, the present method performs three sets of feature extraction and stacking processing on the original image. Third, the number of branches and the fusion method are different. The patent contains 2 branches, and the feature maps obtained after all convolutions on each branch are fused; the present method contains 4 branches, each branch corresponds to feature maps of different scales, and at each stage, information is exchanged between the branches, so that the high-resolution branches can obtain the semantic information extracted by the low-resolution branches, and the low-resolution branches can also obtain the detail information extracted by the high-resolution branches. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-feature fusion image forgery detection method based on a high-resolution network in the field of image forgery detection, which can present a generalized artifact by using the gradient of a binary classifier trained with a real image and a manipulated image. Using the trained model as a conversion model, all training and test data are converted into gradient maps, and the data dependency problem is converted into a model dependency problem, which enhances the generalization ability of the detector. In addition, high-frequency features and color space are also considered. The input image is Gaussian smoothed and down-sampled to extract multi-scale high-frequency information, and the feature maps of different scales are integrated through a 1x1 convolution layer to obtain high-frequency features. These high-frequency information represent key details such as edges and textures in the image. The high-resolution network maintains high-resolution representations of these features throughout the network. At each stage of the network, features of different resolutions are fused together. This fusion helps to combine fine-grained details from the high-resolution representation with contextual information from low resolution. This multi-resolution fusion is repeated throughout the network to gradually refine the features. Finally, the RGB space is integrated to examine the color distribution and spatial relationships to capture subtle and unique clues. By integrating information from these different fields, our detection method presents a more robust representation of artifacts, and even when the diversity of training data is limited, the detector still shows significant potential in dealing with unknown forgery methods.

[0007] The technical solution of the present invention is:

[0008] A multi-feature fusion image forgery detection method based on a high-resolution network. First, a true-false image classifier is trained and a conversion model is constructed to convert the original image into a gradient map. Secondly, the generated gradient map and the original image are input into the feature extraction module to extract gradient features and color space information. The features of the two branches are fused through a convolutional network. Then the frequency branch extracts the high-frequency features of the image, and the fused gradient features and color space features are further fused with frequency domain features. Finally, the fused features are sent to the discriminator to determine the authenticity of the input image. That is, it includes the following steps:

[0009] Step 1. Use ImageNet pre-trained ResNet50 as the classifier. Use real images and manipulated images to train a binary classifier and save the model. The trained classifier parameters will no longer be updated;

[0010] Step 2: Input the image into the pre-trained classifier. Calculate the gradient of the image, draw the gradient map of the current image and save it. This step transforms the original data-dependent problem into a model-dependent problem.

[0011] Step 3: Input the generated gradient map and the original image into the feature extraction module to extract gradient features and color space information, and connect the features of the two branches along the channel dimension;

[0012] Step 4: Input the original image into the frequency domain feature extraction block to extract the high-frequency features of the image. The extracted high-frequency features are further concatenated along the channel dimension;

[0013] Step 5: Fuse the three-channel features through a high-resolution network;

[0014] Step 6: Send the fused features to the discriminator to determine the authenticity of the input image.

[0015] More specific steps are:

[0016] Step 1. Load the ResNet50 model pre-trained with ImageNet. Replace the last fully connected layer of the model with a new fully connected layer, set the output dimension to 1, adjust the model to a binary classifier, and output the model's prediction score. Using the BCEWithLogitsLoss loss function, training can be more efficient by combining the sigmoid activation function and the binary cross entropy loss function. Using the Adam optimizer, the learning rate is reduced by 20% every round, and it stops if the reduced learning rate is less than 0.000001. Save the last round result as the final performance. Then, train the classifier on the manipulated images, and the real images, forged images, and the real labels of the images are fed into the model training. Randomly crop the images during training and feed the classifier directly during testing;

[0017] Step 2: Record the image set as where y i =(0,1) is the true label of the image, X∈R 3*256*256 ,3 represents the number of channels, 256*256 represents the image size, and N is the number of images. Define the trained classification model as G. First, load the weights of the trained classifier. Then preprocess the input image and enable automatic differentiation. Then convert the image (x i ,y i ) Input classification model.

[0018] l=G[(x i ,y i )]

[0019] Where l is the output vector of the classification model G. In the forward propagation of the model, the input image is gradually downsampled through multiple convolutional layers and deeper features are extracted. At the final resolution layer, the network processes the feature map through a fully connected layer and outputs the true and false discrimination scores. Then, the gradient in the model is cleared to avoid affecting the current gradient calculation. This ensures that subsequent backpropagation only involves the calculation of the current batch. Then, the torch.autograd.grad() function is used to calculate the gradient, which calculates the current discrimination result l relative to the current input image x i gradient.

[0020]

[0021] Keep the computational graph and keep the historical information of the computational graph. torch.autograd.grad() returns a list of gradients. Since the gradient is only calculated for the input image, the output list contains only one element, which is the gradient relative to the input image. The shape of the gradient is the same as the input image, and it represents the rate of change of the model output to the input image. That is, the gradient indicates how a small change to the input image affects the discrimination result. Finally, the gradient of each image is normalized and converted to NumPy format and saved as a PNG file.

[0022] Step 3: Input the generated gradient map and the original image into the feature extraction module. In the feature extractor, the input image undergoes two convolutions, batch normalization, and activation functions, where the convolution kernel size is 3, the convolution step size is 1, and the activation function is ReLU. The two branch feature maps are concatenated along the channel dimension.

[0023] Step 4: The original input image X is Gaussian smoothed through a convolution filter to obtain a smoothed feature map. Gaussian smoothing uses a convolution kernel size of 3. The input X contains all the original information, including low-frequency components and high-frequency components. After Gaussian smoothing, the feature map mainly retains the low-frequency information, while the high-frequency information is smoothed out. Next, the smoothed feature map is downsampled at multiple scales to obtain feature maps of different scales. Different scales represent different levels of details in the feature map. By processing images at multiple scales, different levels of features can be captured, resulting in a more comprehensive understanding of the image content. Specifically, each scale corresponds to a scale factor S[i] (i=0,1,2), which controls the degree of downsampling. The smoothed feature map is downsampled at a ratio of 1 / S[i] to obtain feature maps of three different scales. These downsampled feature maps are then upsampled to the same size as the input feature map. At each scale, the difference between the original input image X and the smoothed feature map is calculated. This difference represents the high-frequency information at that scale. High-frequency information usually includes details such as edges and textures in the image, which is crucial for the target task. By performing this operation at different scales, high-frequency features at different scales can be extracted. At a smaller scale, the difference features mainly reflect details such as edges and textures; at a larger scale, the difference features mainly reflect the larger structure and contours of the image.

[0024] These different feature maps M[j] (j=1, 2, 3) are connected along the channel dimension to combine features of different scales to enhance the richness and robustness of feature representation, thereby improving the generalization of the model. The connected difference map is passed through a 1x1 convolutional layer f 1×1 Processing is performed to integrate the different features from different scales. This step maps the dimensions of the concatenated feature map to the required input dimension for further processing in the next layer.

[0025] x'=f 1×1 (x-up(down(f' 2D (X)))

[0026] up and down represent upsampling and downsampling respectively. f' 2D represents Gaussian smoothing, f 1×1 represents a 1x1 convolutional layer that integrates features of different scales, and x' represents the concatenated feature map. Then, two sets of convolution, normalization, and activation function operations are performed on the feature map x' to obtain the feature map F f Finally, the obtained high-frequency feature maps are further concatenated along the channel dimension to concatenate the three-branch features together.

[0027] Step 5. The three-channel branch features have been spliced ​​along the channel dimension, and then the three-channel features are fused. A 1x1 convolution layer is applied to the spliced ​​three-branch features, normalized and used the ReLU activation function, and some neurons are randomly discarded with a probability of 20%. Then the features are further processed by the high-resolution network. First, two convolution blocks are constructed. The convolution block is Bottleneck. Bottleneck mainly consists of a 1x1 convolution layer, a 3x3 convolution layer (depth-separable convolution) and a 1x1 convolution layer. Among them, the first 1x1 convolution layer is used to reduce the number of input channels, thereby reducing the amount of calculation, the 3x3 convolution layer performs the main feature extraction, and the last 1x1 convolution layer is used to restore the number of channels and achieve downsampling. This structure enables Bottleneck to significantly reduce the number of parameters while maintaining performance. All constructed blocks are combined to form a sequence of layers that can be called serially. The features are then input into the constructed layers. Then multiple stages of the high-resolution network and their connections are defined. The detection model uses a high-resolution network as the backbone. The core idea of ​​the high-resolution network is to maintain high-resolution feature representation throughout the network. By combining multiple branches of different resolutions and fusing them repeatedly, richer underlying information is integrated while retaining high-resolution features. This design enables the model to understand the image globally and capture fine-grained local details. Create a high-resolution module in the second, third, and fourth stages of the high-resolution network, with 4 branches, 2 blocks in each branch, 18, 36, 72, and 144 channels, respectively. The block type is Basic, and the method of feature fusion between branches is SUM. If it is not the last module, multi-scale output is generated, and multi-scale output is not generated in the last module. Update the number of input channels, set it to the number of output channels of the last module, and pass the number of output channels of the current module to the next module. Each module can process multiple branches, and different branches process feature maps of different scales. Each branch extracts features through different numbers of channels and convolution blocks, and fuses different branch features through the SUM method. Execute these stages in sequence on the features processed above, and finally return the features processed by the high-resolution network.

[0028] Step 6: The feature map first undergoes a 1x1 convolution to adjust the number of channels, and then undergoes a 3x3 convolution to extract and enhance spatial information. After two convolutions, the number of channels of the feature map is adjusted to 64, and the spatial size remains unchanged. The processed feature map undergoes another 3x3 convolution to keep the spatial dimension unchanged, followed by batch normalization and ReLU activation, and finally upsampling to enlarge the spatial size of the feature map by 2 times. The feature map is then input into the classification module for classification result prediction. In the classification module, the feature map is first subjected to two convolution operations. The first convolution layer converts the number of input channels to 32, the convolution kernel size is 3x3, the stride is 1, and the input spatial size remains unchanged. The second convolution layer changes the number of channels from 32 to 2, and is also a 3x3 convolution. Each convolution layer is followed by a ReLU activation function. Then, through adaptive average pooling, the input feature map is reduced to a size of 1x1, and then flattened into a one-dimensional vector to prepare for the subsequent fully connected layer. The flattened one-dimensional vector is batch normalized to keep the input data stable. Finally, it passes through the fully connected layer and outputs the classification result of the category. The cross entropy loss function is used to measure the final prediction result. In the forged image detection task, the input image X∈R 3*256*256 It is mapped to the detector of the binary classification task, and the detector outputs the prediction result y b The real image is labeled y b = 0, images manipulated by other methods are marked as y b =1.

[0029] The present invention has the following characteristics:

[0030] 1. This paper proposes a new unified detection framework for detecting forged images synthesized and manipulated by generative models. First, since the gradients of convolutional neural networks trained on generated images can highlight key pixels related to the target task, they provide key clues for detecting forged images. Using the trained model as a conversion model, all training and test data are converted into gradient maps, effectively mapping all data to the gradient domain. Converting the data-dependent problem to a model-dependent problem enhances the generalization ability of the detector.

[0031] 2. In addition, the present invention also takes into account high-frequency features to ensure robustness in unknown scenarios. The input image is processed by Gaussian smoothing to retain low-frequency information and remove high-frequency information. Next, the smoothed feature map is downsampled at different scales and high-frequency information is extracted, which represents key details such as edges and textures in the image. The feature maps of different scales are then connected and the features are integrated through a 1x1 convolutional layer to improve the generalization of the model. Finally, the RGB space is integrated to examine color distribution and spatial relationships to capture subtle and unique clues. By integrating information from these different fields, the present invention presents a more robust representation of artifacts, and even if the diversity of training data is limited, the detector still shows great potential when facing unknown generative models.

[0032] 3. In addition, a high-resolution network is used to maintain high-resolution representations of these features throughout the network. The high-resolution network uses multiple parallel streams to process input features of different resolutions. At each stage of the network, features of different resolutions are fused together. This fusion helps to combine fine-grained details from high-resolution representations with contextual information from low resolution. This multi-resolution fusion is repeated throughout the network to gradually refine the features. Extensive experiments and evaluations on public datasets demonstrate the effectiveness and reliability of this method in detecting synthetic and forged content. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a detection framework diagram of the present invention;

[0034] Figure 2 is a business process diagram of the present invention;

[0035] Figure 3 is a specific training flow chart of an embodiment;

[0036] Figure 4 is a flow chart of a high-frequency feature extractor of an embodiment. DETAILED DESCRIPTION

[0037] The present invention will be further described below by way of the accompanying drawings and examples.

[0038] See also Figure 1-4 , a multi-feature fusion image forgery detection method based on high-resolution network, firstly, train a true and false image classifier, and build a conversion model to convert the original image to a gradient map; secondly, input the generated gradient map and the original image into the feature extraction module to extract the gradient features and color space information. And fuse the features of the two branches through the convolutional network; then the frequency branch extracts the high-frequency features of the image, and further fuses the fused gradient features and color space features with the frequency domain features; finally, the fused features are sent to the discriminator to determine the authenticity of the input image; specifically, it includes the following steps:

[0039] Step 1. Use ImageNet pre-trained ResNet50 as the classifier. Use real images and manipulated images to train a binary classifier and save the model. The trained classifier parameters will no longer be updated;

[0040] Step 2: Input the image into the pre-trained classifier. Calculate the gradient of the image, draw the gradient map of the current image and save it. This step transforms the original data-dependent problem into a model-dependent problem.

[0041] Step 3: Input the generated gradient map and the original image into the feature extraction module to extract gradient features and color space information, and connect the features of the two branches along the channel dimension;

[0042] Step 4: Input the original image into the frequency domain feature extraction block to extract the high-frequency features of the image. The extracted high-frequency features are further concatenated along the channel dimension;

[0043] Step 5: Fuse the three-channel features through a high-resolution network;

[0044] Step 6: Send the fused features to the discriminator to determine the authenticity of the input image.

[0045] The specific steps are:

[0046] In an embodiment, step 1 includes: loading a ResNet50 model pre-trained with ImageNet. Replace the last fully connected layer of the model with a new fully connected layer, set the output dimension to 1, adjust the model to a binary classifier, and output the prediction score of the model. Using the BCEWithLogitsLoss loss function, training can be more efficient by combining the sigmoid activation function and the binary cross entropy loss function. Using the Adam optimizer, the learning rate is reduced by 20% every round, and stops if the reduced learning rate is less than 0.000001. Save the last round result as the final performance. Then, the classifier is trained on the manipulated images, and the real images, forged images, and the real labels of the images are fed into the model training. Randomly crop the images during training and feed the classifier directly during testing;

[0047] In an embodiment, step 2 includes: recording the image set as where y i =(0,1) is the true label of the image, X∈R 3*256*256 ,3 represents the number of channels, 256*256 represents the image size, and N is the number of images. Define the pre-trained classification model as G. First, load the weights of the trained classifier. Then pre-process the input image and enable automatic differentiation. Then, transform the image (x i ,yi ) Input classification model.

[0048] l=G[(x i ,y i )]

[0049] Where l is the output vector of the classification model G. In the forward propagation of the model, the input image is gradually downsampled through multiple convolutional layers and deeper features are extracted. At the final resolution layer, the network processes the feature map through a fully connected layer and outputs the true and false discrimination scores. Then, the gradient in the model is cleared to avoid affecting the current gradient calculation. This ensures that subsequent backpropagation only involves the calculation of the current batch. Then, the torch.autograd.grad() function is used to calculate the gradient, which calculates the current discrimination result l relative to the current input image x i gradient.

[0050]

[0051] Keep the computational graph and keep the historical information of the computational graph. torch.autograd.grad() returns a list of gradients. Since the gradient is only calculated for the input image, the output list contains only one element, which is the gradient relative to the input image. The shape of the gradient is the same as the input image, and it represents the rate of change of the model output to the input image. That is, the gradient indicates how a small change to the input image affects the discrimination result. Finally, the gradient of each image is normalized and converted to NumPy format and saved as a PNG file. In the embodiment, all training sets and test sets must generate gradient maps.

[0052] In an embodiment, step 3 includes: inputting the gradient map and the original image of the generated training set into the feature extraction module. In the embodiment, the experiment is divided into two groups, one of which is to verify the model validity under the condition of sufficient training data. The real data set consists of 20 categories from LSUN, each category takes 18k images, and a total of 360k images are used for training. There are 7 forged images from methods based on generative adversarial networks (CycleGAN, gougan, IMLE, ProGAN, StarGAN, StyleGAN, StyleGAN2), a total of 195k images. There are 5 forged methods based on diffusion models (ADM, DDPM, IDDPM, LDM, PNDM), each method uses 39k images for training, a total of 195k images. A total of 390k forged images. The other group is to verify the generalization of the model when training data is scarce. Each training set includes only one forgery method, each method uses 39k images for training, and 10 operation methods are selected to explore the performance of the detector. Five of them are based on generative adversarial networks, and five are based on diffusion models. All 10 methods are evaluated. In the feature extractor, the input image undergoes two convolutions, batch normalization, and activation functions, where the convolution kernel size is 3, the convolution step is 1, and the activation function is ReLU. The two branch feature maps are concatenated along the channel dimension.

[0053] In an embodiment, step 4 includes: performing Gaussian smoothing on the original input image X through a convolution filter to obtain a smoothed feature map. Gaussian smoothing uses a convolution kernel size of 3. The input X contains all the original information, including low-frequency components and high-frequency components. After Gaussian smoothing, the feature map mainly retains low-frequency information, while the high-frequency information is smoothed. Next, the smoothed feature map is downsampled at multiple scales to obtain feature maps of different scales. Different scales represent details at different levels in the feature map. By processing images at multiple scales, features at different levels can be captured, thereby having a more comprehensive understanding of the image content. Specifically, each scale corresponds to a scale factor S[i] (i=0,1,2), which controls the degree of downsampling. The smoothed feature map is downsampled at a ratio of 1 / S[i] to obtain feature maps of three different scales. Then, these downsampled feature maps are upsampled to the same size as the input feature map. At each scale, the difference between the original input image X and the smoothed feature map is calculated. This difference represents the high-frequency information at that scale. High-frequency information usually includes details such as edges and textures in the image, which is crucial for the target task. By performing this operation at different scales, high-frequency features at different scales can be extracted. At a smaller scale, the difference features mainly reflect details such as edges and textures; at a larger scale, the difference features mainly reflect the larger structure and contours of the image.

[0054] These different feature maps M[j] (j=1, 2, 3) are connected along the channel dimension to combine features of different scales to enhance the richness and robustness of feature representation, thereby improving the generalization of the model. The connected difference map is passed through a 1x1 convolutional layer f 1×1 Processing is performed to integrate the different features from different scales. This step maps the dimensions of the concatenated feature map to the required input dimension for further processing in the next layer.

[0055] x'=f 1×1 (x-up(down(f' 2D (X)))

[0056] up and down represent upsampling and downsampling respectively. f' 2D represents Gaussian smoothing, f 1×1 represents a 1x1 convolutional layer that integrates features of different scales, and x' represents the concatenated feature map. Then, two sets of convolution, normalization, and activation function operations are performed on the feature map x' to obtain the feature map F f Finally, the obtained high-frequency feature maps are further concatenated along the channel dimension to concatenate the three-branch features together.

[0057] In an embodiment, step 5 includes: the three-channel branch features have been spliced ​​along the channel dimension, and then the three-channel features are fused. A 1x1 convolution layer is applied to the spliced ​​three-branch features, normalized and using the ReLU activation function, and some neurons are randomly discarded with a probability of 20%. The features are then further processed through a high-resolution network. First, two convolution blocks are constructed. Among them, the convolution block is Bottleneck. Bottleneck is mainly composed of a 1x1 convolution layer, a 3x3 convolution layer (depth-separable convolution) and a 1x1 convolution layer. Among them, the first 1x1 convolution layer is used to reduce the number of input channels, thereby reducing the amount of calculation, the 3x3 convolution layer performs the main feature extraction, and the last 1x1 convolution layer is used to restore the number of channels and achieve downsampling. This structure enables Bottleneck to significantly reduce the number of parameters while maintaining performance. All constructed blocks are combined to form a layer sequence that can be called serially. The features are then input into the constructed layer. Then multiple stages of the high-resolution network and their connections are defined. This detection model uses a high-resolution network as the backbone. The core idea of ​​the high-resolution network is to maintain high-resolution feature representation throughout the network. By combining multiple branches of different resolutions and fusing them repeatedly, richer underlying information is integrated while retaining high-resolution features. This design enables the model to understand the image globally and capture fine-grained local details. Create a high-resolution module in the second, third, and fourth stages of the high-resolution network, with 4 branches, 2 blocks in each branch, 18, 36, 72, and 144 channels, respectively. The block type is Basic, and the method of feature fusion between branches is SUM. If it is not the last module, multi-scale output is generated, and no multi-scale output is generated in the last module. Update the number of input channels and set it to the number of output channels of the last module, and pass the number of output channels of the current module to the next module. Each module can process multiple branches, and different branches process feature maps of different scales. Each branch extracts features through different numbers of channels and convolution blocks, and fuses features of different branches through the SUM method. Execute these stages in sequence on the features processed above, and finally return the features processed by the high-resolution network.

[0058] In an embodiment, step 6 includes: the feature map is first subjected to a 1x1 convolution to adjust the number of channels, and then subjected to a 3x3 convolution to extract and enhance spatial information. After two convolutions, the number of channels of the feature map is adjusted to 64, and the spatial size remains unchanged. The processed feature map is subjected to a 3x3 convolution again, keeping the spatial dimension unchanged, followed by batch normalization and ReLU activation, and finally upsampling to enlarge the spatial size of the feature map by 2 times. The feature map is then input into the classification module for classification result prediction. In the classification module, the feature map is first subjected to two convolution operations. The first convolution layer converts the number of input channels to 32, the convolution kernel size is 3x3, the stride is 1, and the input spatial size remains unchanged. The second convolution layer changes the number of channels from 32 to 2, which is also a 3x3 convolution. Each convolution layer is followed by a ReLU activation function. Then, through adaptive average pooling, the input feature map is reduced to a size of 1x1, and then flattened into a one-dimensional vector to prepare for the subsequent fully connected layer. The flattened one-dimensional vector is batch normalized to keep the input data stable. Finally, it passes through the fully connected layer and outputs the classification result of the category. The cross entropy loss function is used to measure the final prediction result. In the forged image detection task, the input image X∈R 3*256*256 It is mapped to the detector of the binary classification task, and the detector outputs the prediction result y b The real image is labeled y b = 0, images manipulated by other methods are marked as y b =1. The trained detector is evaluated on two sets of experiments. One set is to verify the effectiveness of the model under the condition of sufficient training data. Table 1 shows the performance of the detector on these 12 forgery methods. In addition, to verify the generalization of the model, Table 2 shows the generalization performance of the embodiment on unseen forgery methods when there is sufficient training data. The other set is to verify the generalization of the model when the training data is scarce. Tables 3 and 4 show the generalization performance of the embodiment on various forgery methods. Specifically, the model is trained on 39k images forged by each method in the first column, and the generalization performance of the model is verified on the ten forgery methods on the right.

[0059] Table 1 shows the detection results of the embodiment for two major types of counterfeiting methods when there is sufficient training data;

[0060]

[0061] Table 2 shows the generalization performance of the embodiment to unseen counterfeiting methods when there is sufficient training data;

[0062]

[0063] Table 3 shows the generalization performance of the embodiment on various forgery methods. Specifically, the model is trained on the images forged by each method in the first column, and the generalization performance of the model is verified on the ten forgery methods on the right;

[0064]

[0065] Table 4 shows the generalization performance of the embodiment on various forgery methods. Specifically, the model is trained on the images forged by each method in the first column, and the generalization performance of the model is verified on the ten forgery methods on the right;

[0066]

[0067] In summary, the present invention transforms the original data dependency problem into a model dependency problem, and inputs the generated gradient map and the original image into the feature extraction module to extract gradient features and color space information. The original image is input into the frequency domain feature extraction block to extract the high-frequency features of the image; a high-resolution network is used as the backbone network in the feature extraction and fusion stages to keep the extracted features as close to the accuracy as possible throughout the entire process. The features of the three branches are fused through a convolutional network; thereby capturing a generalized artifact representation. The present invention has higher recognition accuracy and robustness, and can be well generalized to unseen forgery methods.

[0068] The above description is only a specific embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still falls within the scope of the technical solution of the present invention.

Claims

1. A multi-feature fusion image forgery detection method based on high-resolution network, characterized by: First, a true-false image classifier is trained and a conversion model is constructed to convert the original image into a gradient map. Secondly, the generated gradient map and the original image are input into the feature extraction module to extract gradient features and color space information. The features of the two branches are fused through a convolutional network. Then the frequency branch extracts the high-frequency features of the image, and the fused gradient features and color space features are further fused with frequency domain features. Finally, the fused features are sent to the discriminator to determine the authenticity of the input image. The following steps are included: Step 1: Use ImageNet pre-trained ResNet50 as the classifier; use real images and manipulated images to train a binary classifier and save the model; the trained classifier parameters will no longer be updated; Step 2: Input the image into the pre-trained classifier; calculate the gradient of the image, draw the gradient map of the current image and save it; this step transforms the original data-dependent problem into a model-dependent problem; Step 3: Input the generated gradient map and the original image into the feature extraction module to extract gradient features and color space information, and connect the features of the two branches along the channel dimension; Step 4: Input the original image into the frequency domain feature extraction block to extract the high-frequency features of the image; and continue to splice the extracted high-frequency features along the channel dimension; Step 5: Fuse the three-channel features through a high-resolution network; Step 6: Send the fused features to the discriminator to determine the authenticity of the input image.

2. The multi-feature fusion image forgery detection method based on high-resolution network as claimed in claim 1, characterized in that: Specifically, step 1 includes loading the ResNet50 model pre-trained with ImageNet, replacing the last fully connected layer of the model with a new fully connected layer, setting the output dimension to 1, adjusting the model to a binary classifier, and outputting the prediction score of the model; using the BCEWithLogitsLoss loss function by combining the sigmoid activation function and the binary cross entropy loss function; using the Adam optimizer, reducing the learning rate by 20% every round, and stopping if the reduced learning rate is less than 0.000001; The last round result is saved as the final performance; then, the classifier is trained on the manipulated images, and the real images, forged images, and the true labels of the images are fed into the model training; the images are randomly cropped during training and directly fed to the classifier during testing.

3. The multi-feature fusion image forgery detection method based on high-resolution network as claimed in claim 1, characterized in that: Specifically, step 2 is to record the image set as where y i =(0,1) is the true label of the image, X∈R 3*256*256 ,3 represents the number of channels, 256*256 represents the image size, and N is the number of images; define the pre-trained classification model as G; first, load the weights of the trained classifier; then pre-process the input image and enable automatic differentiation; then i ,y i ) Input classification model; l=G[(x i ,y i )] Where l is the output vector of the classification model G; in the forward propagation of the model, the input image is gradually downsampled through multiple convolutional layers and deeper features are extracted; at the final resolution layer, the network processes the feature map through a fully connected layer and outputs a true and false discrimination score; then the gradient in the model is cleared to ensure that the subsequent backpropagation only involves the calculation of the current batch; then, the torch.autograd.grad() function is used to calculate the gradient, which calculates the current discrimination result l relative to the current input image x i The gradient of Keep the computational graph and its history; torch.autograd.grad() returns a list of gradients; since the gradient is only calculated for the input image, the output list contains only one element, which is the gradient relative to the input image; the shape of the gradient is the same as the input image, and it represents the rate of change of the model output to the input image; that is, the gradient indicates how a small change to the input image affects the discrimination result; finally, the gradient of each image is normalized and converted to NumPy format and saved as a PNG file.

4. The multi-feature fusion image forgery detection method based on high-resolution network as claimed in claim 1, characterized in that: Specifically, step 3 includes inputting the generated gradient map and the original image into a feature extraction module; in the feature extractor, the input image undergoes two convolutions, batch normalization, and activation function operations, where the convolution kernel size is 3, the convolution step size is 1, and the activation function is ReLU; and the two branch feature maps obtained are spliced ​​together along the channel dimension.

5. The multi-feature fusion image forgery detection method based on high-resolution network as claimed in claim 1, characterized in that: Specifically, step 4 includes: performing Gaussian smoothing on the original input image X through a convolution filter to obtain a smoothed feature map; the convolution kernel size used for Gaussian smoothing is 3; the input X contains all original information, including low-frequency components and high-frequency components; after Gaussian smoothing, the feature map mainly retains low-frequency information, while the high-frequency information is smoothed out; next, downsampling the smoothed feature map at multiple scales to obtain feature maps of different scales; different scales represent different levels of details in the feature map; by processing images at multiple scales, features at different levels are captured, so as to have a more comprehensive understanding of the image content; These different feature maps M[j] (j = 1, 2, 3) are connected along the channel dimension to combine features of different scales to enhance the richness and robustness of feature representation, thereby improving the generalization of the model; the connected difference map is passed through a 1x1 convolutional layer f 1×1 Processing is performed to integrate differential features from different scales; This step maps the dimensions of the concatenated feature map to the required input dimensions for further processing in the next layer; x'=f 1×1 (x-up(down(f' 2D (X))) up and down represent upsampling and downsampling respectively; f' 2D represents Gaussian smoothing, f 1×1 represents a 1x1 convolutional layer that integrates features of different scales, and x' represents the concatenated feature map; then, two sets of convolution, normalization, and activation function operations are performed on the feature map x' to obtain the feature map F f ; Finally, the obtained high-frequency feature map is further spliced ​​along the channel dimension to splice the three-branch features together.

6. The multi-feature fusion image forgery detection method based on high-resolution network as claimed in claim 5, characterized in that: More specifically, in step 4, each scale corresponds to a scaling factor S[i] (i=0, 1, 2), and the scaling factor controls the degree of downsampling; the smoothed feature map is downsampled at a ratio of 1 / S[i] to obtain feature maps of three different scales; These downsampled feature maps are then upsampled to the same size as the input feature map; at each scale, the difference between the original input image X and the smoothed feature map is calculated; this difference represents the high-frequency information at that scale; high-frequency information usually includes details such as edges and textures in the image, which is crucial to the target task; by performing this operation at different scales, high-frequency features at different scales are extracted; at smaller scales, the difference features mainly reflect details such as edges and textures; at larger scales, the difference features mainly reflect larger structures and contours of the image.

7. The multi-feature fusion image forgery detection method based on high-resolution network as claimed in claim 1, characterized in that: Specifically, step 5 includes: the three-channel branch features are concatenated along the channel dimension, and then the three-channel features are fused; Apply a 1x1 convolution layer to the concatenated three-branch features, normalize them and use the ReLU activation function, and randomly discard some neurons with a probability of 20%; then further process the features through a high-resolution network; First, two convolution blocks are constructed, of which the convolution block is Bottleneck, which is mainly composed of a 1x1 convolution layer, a depth-separable 3x3 convolution layer and a 1x1 convolution layer; the first 1x1 convolution layer is used to reduce the number of input channels, thereby reducing the amount of calculation, the 3x3 convolution layer performs the main feature extraction, and the last 1x1 convolution layer is used to restore the number of channels and achieve downsampling; this structure enables Bottleneck to significantly reduce the number of parameters while maintaining performance; all constructed blocks are combined to form a layer sequence that can be called serially; the features are then input into the constructed layers; then multiple stages of the high-resolution network and their connections are defined; the detection model uses a high-resolution network as the backbone; the core idea of ​​the high-resolution network is to maintain high-resolution feature representation throughout the network; by combining multiple branches and repeatedly fuse them, while retaining high-resolution features, it integrates richer underlying information; this design enables the model to understand images globally and capture fine-grained local details; a high-resolution module is created in the second, third and fourth stages of the high-resolution network, with 4 branches, 2 blocks in each branch, 18, 36, 72, 144 channels, the type of the block is Basic, and the method of feature fusion between branches is SUM; if it is not the last module, a multi-scale output is generated, and no multi-scale output is generated in the last module; the number of input channels is updated and set to the number of output channels of the last module, and the number of output channels of the current module is passed to the next module; each module can process multiple branches, and different branches process feature maps of different scales; each branch extracts features through different numbers of channels and convolution blocks, and fuses features of different branches through the SUM method; The features processed above are executed in these stages in order, and finally the features processed by the high-resolution network are returned.

8. The multi-feature fusion image forgery detection method based on high-resolution network as claimed in claim 1, characterized in that: Specifically, step 6 is as follows: the feature map is first subjected to a 1x1 convolution to adjust the number of channels, and then subjected to a 3x3 convolution to extract and enhance spatial information; after two convolutions, the number of channels of the feature map is adjusted to 64, and the spatial size remains unchanged; the processed feature map is subjected to a 3x3 convolution again to keep the spatial dimension unchanged, followed by batch normalization and ReLU activation, and finally upsampling to enlarge the spatial size of the feature map by 2 times; then the feature map is input into the classification module to predict the classification result; in the classification module, the feature map is first subjected to two convolution operations, and the first convolution layer converts the number of channels of the input into The convolution kernel size is changed to 32, the stride is 1, and the input spatial size remains unchanged. The second convolution layer changes the number of channels from 32 to 2, which is also a 3x3 convolution. Each convolution layer is followed by a ReLU activation function. Then, the input feature map is reduced to 1x1 size through adaptive average pooling, and then flattened into a one-dimensional vector to prepare for the subsequent fully connected layer. The flattened one-dimensional vector is batch normalized to keep the input data stable. Finally, after the fully connected layer, the classification result of the category is output. The cross entropy loss function is used to measure the final prediction result. In the forged image detection task, the input image X∈R 3*256*256 It is mapped to the detector of the binary classification task, and the detector outputs the prediction result y b ; The real image is labeled y b = 0, images manipulated by other methods are marked as y b =1.

9. The use of a multi-feature fusion image forgery detection method based on a high-resolution network as claimed in claim 1, characterized in that: Used in the fields of digital forensics and information security.

Images