Image forgery detection method, system, electronic device, program product and medium

Abstract

The application provides a picture forgery detection method, comprising: extracting frequency domain features and spatial domain features of a to-be-detected picture and a plurality of derived pictures, wherein each derived picture is generated after a first operation is performed on the to-be-detected picture, and the to-be-detected picture is a picture of a preset size; determining intermediate feature vectors of the to-be-detected picture and the derived pictures according to the frequency domain features and the spatial domain features, respectively; inputting the intermediate features into a true-false discriminator and a model tracer in parallel, wherein the true-false discriminator is a binary classifier, and the model tracer is a multi-class classifier; determining a fake or real detection result according to the intermediate feature vectors by using the true-false discriminator, and generating a multi-class probability distribution according to the intermediate feature vectors by using the model tracer, wherein the multi-class probability distribution refers to a probability distribution of a picture forged by a forgery model; outputting information indicating whether the to-be-detected picture is real or fake based on the detection result of the true-false discriminator and the probability distribution generated by the model tracer, and outputting the probability distribution at least in the case that the picture is fake. Corresponding systems, program products, and the like are also provided.

Description

Technical Field

[0001] This application relates to image detection technology, and more specifically, to image forgery detection technology. Background Technology

[0002] Forged AI images pose numerous new challenges to banks in verifying user identities and verifying the authenticity of materials. For example, they lack interpretability for AI image judgments and cannot provide feedback on the reasons for image judgment results. They also face problems such as insufficient generalization and poor tolerance to adversarial disturbances, affecting the reliability of applications in high-risk scenarios. Summary of the Invention

[0003] According to one aspect of this application, an image detection method is provided to at least solve one of the problems mentioned herein.

[0004] The image forgery detection method according to an embodiment of this application includes extracting frequency domain features and spatial domain features of an image to be detected and multiple derived images, wherein each derived image is an image generated after performing a first operation on the image to be detected, and the image to be detected is an image of a preset size; determining intermediate feature vectors of the image to be detected and the derived images based on the frequency domain features and the spatial domain features respectively; inputting the intermediate features into a parallel real / fake discriminator and a model source tracer, wherein the real / fake discriminator is a binary classifier and the model source tracer is a multivariate classifier; determining a fake or real detection result based on the intermediate feature vectors by the real / fake discriminator, and generating a multivariate probability distribution based on the intermediate feature vectors by the model source tracer, the multivariate probability distribution corresponding to the probability distribution of a forgery model forging an image; and outputting information indicating whether the image to be detected is real or fake based on the detection result of the real / fake discriminator and the probability distribution generated by the model source tracer, and at least outputting the probability distribution in the case that the image is fake.

[0005] According to another aspect of this application, an image forgery detection system is also provided. This system may include a feature extraction module, a fusion module, a discrimination module, and an output module. The feature extraction module is used to extract frequency domain features and spatial domain features of the image to be detected and multiple derived images, wherein each derived image is an image generated after performing a first operation on the image to be detected. The fusion module is used to determine intermediate feature vectors for the image to be detected and the derived images based on the frequency domain features and the spatial domain features, respectively. The discrimination module includes a parallel real / fake discriminator and a model source tracer. The real / fake discriminator determines a detection result of forgery or authenticity based on the intermediate feature vectors, and the model source tracer determines a multivariate probability distribution based on the intermediate feature vectors. The multivariate probability distribution refers to the probability distribution of forgery models forging images; wherein the real / fake discriminator is a binary classifier, and the model source tracer is a multivariate classifier. The output module is used to output information indicating whether the image to be detected is real or forged based on the detection result of the real / fake discriminator and the probability distribution generated by the model source tracer, and, at least in the case that the image is forged, output the probability distribution.

[0006] According to another aspect of this application, an electronic device is also provided, the electronic device comprising: a memory for storing instructions; and a processor for implementing any of the methods described herein when executing the instructions.

[0007] According to another aspect of this application, a program product is also provided, the program product comprising instructions which, when executed, can implement any of the methods described herein.

[0008] According to another aspect of this application, a non-volatile storage medium is also provided, on which instructions are stored, which, when executed, can implement any of the methods described herein.

[0009] The embodiments of this application can effectively solve the problems of poor generalization and weak anti-interference in the prior art when detecting fake images, and improve detection accuracy and practicality. Attached Figure Description

[0010] This application will be more fully understood by referring to the following detailed description of specific embodiments in conjunction with the accompanying drawings, wherein the same reference numerals in the drawings refer to the same elements, wherein: Figure 1 This is a flowchart illustrating an image forgery detection method according to some embodiments of this application; Figure 2 It illustrates the image conversion process; Figure 3 yes Figure 1 A schematic diagram of a specific example of step S60; Figure 4 This is a schematic structural diagram of an image forgery detection system according to an embodiment of this application; Figure 5 This is a schematic diagram of the structure of the electronic device according to this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described implementation methods are only a part of the implementation methods of the technical solutions of this application, and not all of them.

[0012] Figure 1 This is a flowchart illustrating an image forgery detection method according to some embodiments of this application. The method can be executed by a data processing device, such as a laptop, tablet, smartphone, smart wearable device, etc., and can also be executed in the cloud.

[0013] In step S20, the frequency domain features and spatial domain features of the image to be detected and multiple derived images are extracted. Each derived image is an image generated after performing the first operation on the image to be detected. As an example, the spatial domain features of the image to be detected and the derived images are extracted through a first branch, and the frequency domain features of the image to be detected and the derived images are extracted through a second branch. The first branch and the second branch are set up in parallel, meaning that the processing of the images is not sequential but parallel. The first branch is used to extract spatial domain features, while the second branch is used to extract frequency domain features. The first branch may include an improved deep residual network to capture the spatial domain features of the image, and the second branch may obtain frequency domain features through a fast Fourier transform and a convolutional neural network (CNN). In this example, step S20 is performed on both the image to be detected and the derived images; that is, both the derived images and the image to be detected are fed into the first branch and the second branch. The first branch and the second branch are two processing paths set up in hardware such as a computer that executes the method of this application example. These two processing paths can be implemented as software modules through program instructions and configured in the hardware.

[0014] In some examples, Figure 1 The method shown also includes step S10, performing a first operation on the image to be detected to generate a derived image. It should be noted that step S10 may or may not be included in the image forgery detection method according to the embodiments of this application; that is, the image data during step S20 includes the image data to be detected and the already converted derived image data.

[0015] Here, the image to be detected is a photograph of a preset size, thereby eliminating the impact of resolution differences on subsequent processing. If the image to be detected is of the preset size, it can be used directly; otherwise, it is converted to the preset size. For example, and not restrictively, the preset size is 256×256 pixels. In the following description, the image to be detected of the preset size will be referred to as the standard image. (See reference...) Figure 2 Here's a brief example of a size conversion. (See reference...) Figure 2 The original image to be detected 10 is converted into an image of uniform size through bilinear interpolation and the pixels are normalized to the [0,1] interval, thereby obtaining a standardized image 20, wherein the image size is 256×256 pixels.

[0016] For the standardized image 20, a first operation is performed. This first operation can be adding Gaussian noise within a predetermined frequency range to the image 20, or shifting the grayscale values ​​of the RGB (red, green, blue) channels of the image to be detected, or both. The example in this application is adding Gaussian noise within a predetermined frequency range to the image 20 and then shifting the grayscale values ​​of the image with the added Gaussian noise; however, this is an example and not a limitation.

[0017] Specifically, noise is generated within a specific frequency range and superimposed on the normalized image 20 to simulate slight noise interference in a real scene, resulting in a Gaussian noise-added image. Here, the specific frequency range can be 0.1-0.3 times the pixel sampling frequency, and the generated noise intensity can be controlled between 0.01-0.03. For the Gaussian noise-added image, grayscale value shifts are applied to the RGB channels of the normalized image 20 (also called the Gaussian noise-added image) with added Gaussian noise, resulting in a color channel shift image. The shifted grayscale values ​​are within a predetermined range, from -3 to +3. For example, a +2 grayscale value shift is applied to the R channel, a -1 grayscale value shift to the G channel, and a +1 grayscale value shift to the B channel. Applying grayscale value shifts to the image breaks the fixed color pattern that a forged image might rely on, preventing it from evading detection by simply adjusting color channels. Thus, a derived image with Gaussian noise and grayscale value shifts is obtained based on the standard image 20.

[0018] According to the embodiments of this application, a first operation is performed on the standardized image 20 to obtain multiple images with added Gaussian noise and shifted color channels. For example, the first operation is performed on image 20 three times to obtain images 31, 32, and 33. It should be noted that at least one of the noise intensity and grayscale value shift degree in the process of performing the first operation on the standardized image 20 to obtain image 31 is different from that in the process of performing the second first operation on the standardized image 20 to obtain image 32; similarly, at least one of the noise intensity and grayscale value shift degree in the process of performing the first operation on the standardized image 20 to obtain image 31 is different from that in the process of performing the third first operation on the standardized image 20 to obtain image 33; similarly, at least one of the noise intensity and grayscale value shift degree in the process of performing the second operation to obtain image 32 is different from that in the process of performing the third operation to obtain image 33. In short, images 31, 32, and 33 are all derived images of the standard image 20, but their perturbation intensities are not the same. In this application, perturbation intensity includes either noise intensity or grayscale value shift degree. Compared to the standard image, each derived image simulates color changes in a real scene to enhance the model's robustness to color perturbations, while preventing forged images from evading detection by using fixed color patterns.

[0019] Return to step S20. Send standard image 20, derivative image 31, derivative image 32, and derivative image 33 into the first branch, and also into the second branch.

[0020] Specifically, in the first branch, improved ResNets are used to extract spatial domain features from standard image 20 and derived images 31, 32, and 33, respectively. The improved ResNets replace the ordinary convolutions in stages 3 and 4 of the basic ResNet50 with depthwise separable convolutions. ResNet50 is a widely used residual network model, a 50-layer convolutional neural network. In this example, the ordinary convolutions in stages 3 and 4 of ResNet50 can simultaneously process the spatial dimension of the feature map related to pixel location and the channel dimension related to feature type, resulting in a large number of parameters and computational redundancy. Depthwise separable convolution is a convolution operation optimization technique designed for lightweight neural networks. Its core idea is to separate the two processes of "spatial domain feature extraction" and "channel feature fusion" in traditional convolution, allowing for the independent processing of spatial information for each channel. By convolving features from different channels point-by-point, it significantly reduces the number of model parameters and computational cost while preserving the model's feature extraction capabilities as much as possible. The improved residual network reduces network parameters by 60%, thereby alleviating the burden on hardware settings and enhancing the model's generalization ability, making it more adaptable to unseen forged images or image perturbations. Furthermore, the improved residual network adds an SE attention module after each residual block for a compression-activation mechanism, strengthening the feature weights of anomalous regions.

[0021] In some examples, the initial convolutional layer of the improved residual network is a 7×7 convolution with a stride of 2, outputting a feature map with 64 channels and a spatial size of 128×128. As mentioned, the original size of standard image 20 and derived images 31, 32 and 33 is a preset size, which is 256×256 pixels in this example. A stride of 2 halves the spatial size. The pooling layer has a pooling kernel size of 3×3 and a stride of 2, further compressing the spatial dimension to obtain a feature map with 64 channels and a spatial size of 64×64, halving the spatial size again while preserving key features and reducing computational cost. In this example, among the four residual groups, residual group 1 contains 3 improved residual blocks with SE modules, increasing the number of channels from 64 to 256; residual group 2 contains 4 improved residual blocks, increasing the number of channels from 256 to 512; residual group 3 contains 6 improved residual blocks, increasing the number of channels from 512 to 1024; and residual group 4 contains 3 improved residual blocks, increasing the number of channels to 2046, thus obtaining a high-dimensional feature map. As the number of channels in the feature map gradually increases, the spatial size is gradually compressed to approximately 8×8 between the residual groups. Global average pooling is performed on the 280-channel feature map output from the last residual group with a spatial size of 8×8 to eliminate the spatial dimension, directly outputting a 2048-dimensional one-dimensional spatial domain feature vector, meaning each of the 2048 channels outputs one value. Through the processing of the first branch, standard image 20, derived image 31, derived image 32, and derived image 33 are transformed into 2048-dimensional one-dimensional spatial domain feature vectors.

[0022] For example, the second branch first converts the local frequency domain information of the image into a channel spectrogram that can be processed by a convolutional neural network (CNN). Finally, the CNN processes this information. Specifically, each of the input standard image 20 and derived images 31, 32, and 33 is divided into 16×16 pixel blocks. A 2D Fast Fourier Transform (FFT) is performed on each block individually. This block-by-block processing, rather than processing the entire image at once, preserves the frequency domain details of each local region and prevents small-scale anomalous frequency domain features from being masked by global information during the overall transformation. The FFT result after the FFT includes an amplitude spectrum reflecting frequency intensity and a phase spectrum reflecting frequency location. In this example, only the amplitude spectrum, which is already capable of capturing frequency domain anomalies in forged images, is used to reduce computation. The dynamic range of the amplitude spectrum is compressed using a logarithmic transformation (log(1+|F|)) and then normalized to [0,1] to eliminate numerical differences between different images, generating a 3-channel frequency domain spectrogram corresponding to the original RGB channels. More specifically, each of the standard image 20 and the derived images 31, 32 and 33 is an RGB three-channel image. In this example, a 2D Fast Fourier Transform and amplitude spectrum processing are performed on each channel of each image to obtain three single-channel spectrograms of each image (i.e., one spectrogram per channel). The three single-channel spectrograms of each image are stitched together to form the 3-channel frequency domain spectrogram of the image, which corresponds completely to the channels of the original RGB image.

[0023] The frequency domain spectrograms processed above are fed into a CNN (Convolutional Neural Network) to output frequency domain features. This CNN may include three convolutional layers, using 32, 64, and 128 3×3 convolutional kernels respectively, with a stride of 1 for each. The convolutional layers progressively increase the number of feature channels. In this example, two pooling layers with a stride of 2 are interspersed between the three convolutional layers to compress the feature map size, reduce computation, and amplify abnormal noise. A global average pooling layer is used to calculate the global mean of the feature map output by the last convolutional layer. The frequency domain spectrograms of each image, obtained after processing such as Fast Fourier Transform and amplitude spectrum analysis, are fed into the CNN, which extracts the frequency domain features from the spectrograms and outputs a 1024-dimensional one-dimensional frequency domain feature vector. Thus, after processing in the second branch, the frequency domain features of each of the standard image 20, derived images 31, 32, and 33 are output as 1024-dimensional one-dimensional frequency domain feature vectors.

[0024] In step S30, an intermediate feature vector is determined based on the spatial domain features and frequency domain features determined in step S20. The spatial domain feature vector obtained in the first branch and the frequency domain feature vector obtained in the second branch are transformed to have the same dimension. For example, the 2048-dimensional spatial domain feature vector is compressed to 1024 dimensions, so that the spatial domain and frequency domain feature vectors have the same dimension, thereby eliminating the influence of dimensional differences on subsequent processing.

[0025] Furthermore, through the mutual attention mechanism, the 1024-dimensional spatial domain feature vector and the 1024-dimensional frequency domain vector are cross-correlated to generate a 1024×1024 feature similarity matrix. Each element in this matrix corresponds to a set of correlation strengths between spatial domain features and frequency domain features. Subsequently, the similarity matrix is ​​normalized using Softmax to obtain the cross-attention weights corresponding to the spatial domain features and the frequency domain features, respectively. For example, when there is a strong correlation between high-frequency anomalies in the frequency domain features (such as periodic noise in fake images) and edge blurring in the spatial domain features (typical spatial artifacts in fake images), the weight of the corresponding correlation position will be significantly increased to 0.8-0.9, which intuitively reflects the degree of agreement between the two types of features. Finally, the two types of features are dynamically fused according to the weighted summation formula (1), and a 1024-dimensional fused feature vector is finally output as the intermediate feature vector. Formula (1) is as follows: M feature =S feature × S attwei + F feature × F attwei (1) Among them, M feature This is an intermediate feature, i.e., a fused feature; S feature Features of the spatial domain; S attwei For spatial attention weights; F feature For frequency domain characteristics; F attwei For frequency domain attention weights.

[0026] In step S40, the determined intermediate features are input into a parallel true / false discriminator and a model source tracer, wherein the true / false discriminator is a binary classifier and the model source tracer is a multi-class classifier.

[0027] In step S50, the real / fake distinguisher and the model source tracer process the intermediate feature vectors of the input, respectively. The real / fake distinguisher determines whether the image is fake or real, and the model source tracer generates a multivariate probability distribution based on the intermediate feature vectors. The multivariate probability distribution is the probability distribution of the fake image generated by the fake model. The fake model used to fake images is an application branch of generative AI technology. It simulates the feature distribution of real images through deep learning. Mainstream fake models can include the Generative Adversarial Network (GAN) series and the Diffusion Model series.

[0028] The real / fake detector receives intermediate feature vectors, which are obtained by processing standard image 20, derived image 31, derived image 32, and derived image 33. For example, it receives the 1024-dimensional intermediate feature vector output in step S40 and determines whether the image is real or fake based on its processing. As an example, the real / fake detector includes a 1024-dimensional first fully connected layer, a 512-dimensional second fully connected layer, and a 1-dimensional output layer. The first fully connected layer processes the data input to the real / fake detector, captures anomalous features, and outputs a primary feature map. The second fully connected layer takes the primary features output from the first fully connected layer as input and learns the nonlinear combination relationship between these primary features to obtain abstract features. The output layer is the decision layer for making a judgment. Based on the abstract features determined by the second fully connected layer, the output layer performs a weighted summation judgment to obtain a judgment value. Then, the sigmoid function converts this judgment value into a probability value to obtain the fake probability of the image, i.e., Pfake. For example, setting a threshold of 0.5, if Pfake is greater than 0.5, it indicates that the image is fake, and the probability of the image being real, i.e., the sum of Ptrue and Pfake, is 1. In other words, Ptrue can be obtained by subtracting Pfake from 1. The real / fake discriminator can be optimized using a binary cross-entropy loss function to make the predicted Pfake and Ptrue infinitely close to the true label of the data. The real / fake discriminator generates detection results for each image as real or fake, determined based on the intermediate feature vectors of the standard image 20, derived images 31, 32, and 33.

[0029] The model source tracer is a multivariate classifier consisting of two fully connected layers. The feature dimensions are compressed sequentially from 1024 to 512, ultimately outputting an N-dimensional vector, where N is the number of fake model types. In this example, the model source tracer includes three types (i.e., N = 3) of fake models: for example, basic generative adversarial networks (GANs), third-generation GANs (StyleGAN3), and diffusion models. After processing with the Softmax activation function, the model source tracer generates a multivariate probability distribution. For example, outputting [0.1, 0.8, 0.1] in the order of adversarial network, third-generation GAN3, and diffusion model indicates that the probability of the image being faked by the third-generation GAN3 is 80%, and the probability of it being faked by the other two models is 10% each.

[0030] In step S60, based on the detection results of the real / fake distinguisher and the probability distribution generated by the model tracer, information indicating whether the image to be detected is real or fake is output, and at least in the case that the image is fake, the probability distribution is output.

[0031] The authenticity detector determines whether the image to be detected is real or fake, and whether the derived images are real or fake, based on the intermediate feature vectors of the image to be detected and the derived images, respectively. For example, the authenticity detector processes the intermediate feature vectors of the standard image 20, derived images 31, 32, and 33 (step S50), generating a detection result for each image. Similarly, the model source tracer generates a probability distribution of forgery models for each image. Therefore, when outputting whether the image to be detected is real or fake, the authenticity detector generates a result indicating whether the image to be detected is real or fake based on the detection results of the image to be detected and the derived images derived from the image to be detected, and outputs this result. If the image to be detected is fake, a probability distribution is output; however, if the result is real, a probability distribution can also be generated, but the output of the model source tracer may show that the probability of all forgery models is very low. "Output" here may refer to sending the result indicating whether the image to be detected is real or fake, or the probability distribution, to other devices or directly to a display screen or other component of the device executing the invention, or both, depending on the application scenario.

[0032] As an example, in response to the fact that the forgery probabilities determined based on the intermediate feature vectors of both the image to be detected and the derived images satisfy the high confidence condition, the output indicates whether the image to be detected is real or forged. Specifically, satisfying the high confidence condition includes cases where the forgery probability is greater than a high confidence threshold or less than a low confidence threshold. A forgery probability greater than the high confidence threshold indicates that the image is forged. A forgery probability less than the low confidence threshold indicates that the image is real, because in this example, the sum of the forgery probability and the real probability is 1, and a forgery probability less than the low confidence threshold indicates that the real probability is greater than the real confidence threshold. For example, the high confidence threshold is, for example, 0.8, and the low confidence threshold is, for example, 0.2. Based on the processing of the intermediate feature vectors of the standard image 20, derived image 31, derived image 32, and derived image 33 in step S50, if the real / fake discriminator determines that the Pfake of each is greater than 0.8, then the image to be detected is forged, and the output indicates that the image is forged. Based on the processing of the intermediate feature vectors of the standard image 20, derived image 31, derived image 32, and derived image 33 in step S50, the real / fake discriminator determines that the Pfake value of each is less than 0.2 (corresponding to Ptrue value greater than 0.8), indicating that the image to be detected is real, and outputs information indicating that the image is real.

[0033] Furthermore, if the forgery probability is between the low confidence threshold and the high confidence threshold, then the low confidence condition is met. For example, if 0.2 ≤ Ppseudo ≤ 0.8, then Ppseudo satisfies the low confidence condition. In response to any of the following situations, a portion of the intermediate feature vectors is randomly selected, such as 20% or 50%, and input into the parallel real / fake discriminator and model source tracer for secondary judgment. The previous judgment result is then corrected based on the secondary judgment result. These situations include: the forgery probability of the image to be detected and all derived images satisfies the low confidence condition; some images satisfy the high confidence condition and some satisfy the low confidence condition; the forgery probability of the image to be detected and the derived images are inconsistent. For example, if the secondary judgment result shows that the forgery probability determined based on the intermediate feature vectors of the image to be detected and each derived image satisfies the high confidence condition, then the previous judgment result is corrected. For example, if the forgery probability is greater than the high confidence threshold, the judgment result is forgery; if the forgery probability is less than the low confidence threshold, the judgment result is real. This is then combined with the probability distribution output generated by the model source tracer. If a low-confidence condition is found in the true probability determined based on the intermediate feature vector of the image to be detected or each derived image in the secondary judgment result, then the judgment result of the true / false discriminator is corrected and output in combination with the judgment result of the model tracer.

[0034] For example, if the forgery probability Pfake of each image determined by the real / fake detector is greater than 0.8, satisfying the high-confidence condition for forgery, then the image to be detected is determined to be forged, and the information indicating forgery is directly output. Conversely, if the real / fake detector's detection result shows that the Pfake of each image is less than the low-confidence threshold of 0.2, satisfying the high-confidence condition for authenticity, i.e., high-confidence authenticity, then the image to be detected is determined to be authentic. Following this example, the detection result of the real / fake detector is output, along with the source tracing result of the model source tracing. For example, the source tracing result of the model source tracing may include information such as "this image may have been generated by StyleGAN3." In cases where the image is authentic, in some examples, source tracing results may also be output, but the source tracing probability of each forgery model will be very low (indicating non-forgery).

[0035] If the forgery probability Pfake of each image determined by the real / fake distinguisher is a low-confidence scenario (0.2 ≤ Pfake ≤ 0.8), then, for example, 20% of the intermediate feature vectors are randomly selected again and input into the classifier for a second judgment. If the second judgment result shows that each image meets the high-confidence condition, then the previous judgment result is corrected, and the probability distribution determined by the model source tracer is combined with the output, for example, true or forged (Pfake > 0.8) or true (Pfake < 0.2, i.e., Ptrue > 0.8). If the result of the second judgment still shows a low-confidence scenario, then the judgment result is corrected and output based on the second judgment result combined with the probability distribution determined by the model source tracer. For example, when the source tracer result is generated by the diffusion model and Pfake = 0.6, the final corrected output is "suspected forgery - Diffusion generation".

[0036] In other examples, if the forgery probabilities determined based on the intermediate feature vectors of the target image and the derived image differ, the forgery probability determined based on the target image differs from that determined based on the derived image. The higher number of "real" and "forged" results is used. For example, if out of four images detected, three are real and one is forged, the image is determined to be real. Based on the forgery probability determined based on the derived image, information indicating whether the target image is real or forged is output.

[0037] According to some examples of this application, the illustrated method may further include generating and outputting an interpretable heatmap. Step S60 may further include generating an interpretable heatmap based on the detection results of the real / fake detector, and overlaying the interpretable heatmap onto the image to be detected in a semi-transparent manner and outputting it. Figure 3 This is a schematic diagram of the processing flow for the interpretability heatmap section in step S60. For example... Figure 3As shown, in step S600, gradient weights are extracted from the detection results of the real / fake image discriminator using techniques such as Grad-weighted Class Activation Mapping (Grad-CAM). Specifically, a 2048×8×8 feature map output from the last convolutional layer of the real / fake image discriminator is obtained, and the gradient of this feature map with respect to the fake image Pfake is calculated. The gradients are aggregated using global average pooling to obtain the gradient weights corresponding to each feature map (e.g., a total of 2048 feature maps). The gradient weights directly reflect the contribution of the feature map to the real / fake determination decision (the higher the weight, the more significant the influence of the feature on the decision). In step S602, these feature maps are weighted and summed to merge them into an initial heatmap. For example, the 2048 feature maps are weighted and summed to merge them into one 8×8 initial heatmap, specifically by multiplying each feature map by its corresponding gradient weight and then summing the results. In step S604, the initial heatmap is processed to match it with the corresponding original image. Specifically, the initial heatmap is upsampled using bilinear interpolation, enlarged to, for example, 256×256, to match the original image. Here, the original image refers to the standard image and derived image of the corresponding image to be detected; in this example, they are the same size. Furthermore, the pixel values ​​of the heatmap are normalized and mapped to the [0, 255] range to complete numerical adaptation. In step S606, the initial heatmap is annotated to present high-contribution and low-contribution areas in different ways. For example, a red-blue dual-color coloring rule is used to render the heatmap, where red marks high-contribution areas of the forged parts in the image, and blue marks low-contribution areas. The colored heatmap is then semi-transparently overlaid on the original image for output. This allows for a direct visual indication that the image is forged and identifies which parts of the image are fake.

[0038] Figure 4This diagram illustrates the structure of an image forgery detection system according to other embodiments of this application. The system includes a feature extraction module 40, a fusion module 42, a discrimination module 44, and an output module 46. The feature extraction module 40 extracts the frequency domain features and spatial domain features of the image to be detected and multiple derived images, where each derived image is generated after performing a first operation on the image to be detected. The fusion module 42 determines intermediate feature vectors for the image to be detected and the derived images based on the frequency domain features and spatial domain features, respectively. The discrimination module 44 includes a parallel real / fake discriminator and a model tracing unit. The real / fake discriminator determines whether the image is real or fake based on the intermediate feature vectors, and the model tracing unit generates a multivariate probability distribution based on the intermediate feature vectors. This multivariate probability distribution refers to the probability distribution of a forgery model forging an image; wherein the real / fake discriminator is a binary classifier, and the model tracing unit is a multivariate classifier. The output module 46 is used to output information indicating whether the image to be detected is real or fake based on the detection result of the real / fake distinguisher and the probability distribution generated by the model tracer, and at least in the case that the image is fake, output the probability distribution.

[0039] The image to be detected and multiple derived images have been described above and will not be repeated here. Please also refer to... Figure 2 The feature extraction module 40 extracts spatial domain features and frequency domain features from the standard image 20, derived images 31, 32, and 33. The feature extraction module 40 includes a first branch and a second branch; the first branch extracts spatial domain features, and the second branch extracts frequency domain features, similar to the description above. The fusion module 42 transforms the frequency domain features and spatial domain features to the same dimension; it calculates the similarity matrix between the spatial domain features and the frequency domain features using a mutual attention mechanism, and obtains the cross-attention weights of the spatial domain features and frequency domain features by normalizing the similarity matrix; it determines the intermediate feature vector based on the spatial domain features, spatial cross-attention weights, frequency domain features, and frequency domain attention weights.

[0040] Figure 4 The illustrated image forgery detection system can perform any one of the methods described above. For example, feature extraction module 40 performs step S20, fusion module 42 performs step S30, discrimination module 44 performs steps S40 and S50, output module 46 performs step S60, and so on.

[0041] This application also provides an electronic device. Figure 5 This is a schematic diagram of the electronic device, which includes a memory 50 and a processor 52. The memory 50 is used to store instructions, and the processor 52 is used to process these instructions. In some examples, the processor 52 can implement any of the methods described herein while executing these instructions.

[0042] This application also provides a program product comprising instructions. In some cases, during the execution of these instructions, any of the methods described herein can be implemented. This application also provides a non-volatile storage medium storing instructions that, during the execution of these instructions, can implement any of the methods described herein.

[0043] According to embodiments of this application, by adding specific Gaussian noise and color shift to the input image, image degradation in real-world scenes is effectively simulated, enhancing the model's anti-interference capability. The constructed spatial-frequency domain parallel network can comprehensively capture subtle traces of AI forgery in pixel statistics and spectral residues, overcoming the limitations of single-domain analysis. The cross-modal attention fusion mechanism achieves adaptive weighting and deep interaction between features, greatly improving the information richness and discriminative power of the intermediate feature representations generated by fusion. Through a dual-task (real / fake identification and model source tracing) collaborative decision-making architecture, the dual functions of forgery identification and source tracing are realized in the same model. In a further example, interpretable heatmaps are used to make the model's decision-making process transparent and visible. Compared with the prior art, embodiments of this application improve generalization performance, enhance anti-interference capability, and provide source tracing results while outputting real / fake results. In a further example, heatmaps are also provided, improving both detection accuracy and practicality.

[0044] The technical features in the various embodiments of this application can be combined with each other to form new implementation methods without departing from the spirit of this application and without conflicting with each other. Although specific embodiments of this application have been shown and described in detail to illustrate the principles of this application, it should be understood that this application can be implemented in other ways without departing from such principles.

Claims

1. A method for detecting image forgery, characterized in that, The method includes: Extract the frequency domain features and spatial domain features of the image to be detected and multiple derived images, wherein each of the derived images is an image generated after performing a first operation on the image to be detected, and the image to be detected is an image of a preset size; The intermediate feature vectors of the image to be detected and the derived image are determined based on the frequency domain features and the spatial domain features, respectively. The intermediate features are input into a parallel true / false discriminator and a model source tracer, wherein the true / false discriminator is a binary classifier and the model source tracer is a multi-class classifier; The true / false detector determines the true or false detection result based on the intermediate feature vector, and the model tracer generates a multivariate probability distribution based on the intermediate feature vector. Based on the detection results of the real / fake distinguisher and the probability distribution generated by the model source tracer, information indicating whether the image to be detected is real or fake is output, and at least in the case that the image is fake, the probability distribution is output; The multivariate probability distribution corresponds to the probability distribution of images forged by the forgery model.

2. The method according to claim 1, characterized in that, The output indicates whether the image to be detected is real or fake, including: In response to the fact that the forgery probability determined based on the intermediate feature vectors of the image to be detected and the derived image respectively satisfies the high confidence condition, information indicating that the image to be detected is a forged image is output. In response to the fact that the true probability determined based on the intermediate feature vectors of the image to be detected and the derived image respectively satisfies the high confidence condition, information indicating that the image to be detected is a real image is output. Wherein, the forgery probability being greater than the high confidence threshold satisfies the high confidence condition, the forgery probability being less than the low confidence threshold satisfies the high confidence condition, and the sum of the true probability and the forgery probability is 1.

3. The method according to claim 2, characterized in that, The method further includes: In response to at least one of the forgery probabilities determined based on the intermediate feature vectors of the image to be detected and the derived image satisfying a low confidence condition, a portion of the intermediate feature vectors is randomly extracted and input into the parallel real / fake discriminator and the model source tracer for secondary determination. In the secondary determination result, if the true probabilities determined by the intermediate feature vectors of the image to be detected and the derived image respectively meet the high confidence condition, the previous determination result is corrected according to the secondary determination result; In the secondary judgment result, if the true probability determined by the intermediate feature vectors of the image to be detected and the derived image still satisfies at least one low confidence condition, the judgment result of the true / false distinguisher is corrected by combining the judgment result of the model source tracer; and For a given image, if the probability of forgery is between the low confidence threshold and the high confidence threshold, then the low confidence condition is met.

4. The method according to claim 1, characterized in that, The output of the detection results of the true / false detector also includes: An interpretable heatmap is generated based on the detection results of the true / false detector. The interpretable heatmap is overlaid on the image to be detected in a semi-transparent manner and then output.

5. The method according to claim 4, characterized in that, Based on the detection results of the true / false detector, an interpretable heatmap is generated, including: Obtain the feature map output by the last convolutional layer of the true / false discriminator to determine the weight gradient of each feature map, wherein the weight gradient reflects the contribution of the feature map to the judgment result; The feature maps are weighted and summed to merge them into an initial heatmap; The initial heatmap is processed to match it with the image to be detected; The initial heatmap is labeled to present high-contribution and low-contribution areas in different ways.

6. The method according to claim 1, characterized in that, Perform a first operation on the image to be detected, including at least one of the following: Add Gaussian noise within a predetermined frequency range to the image to be detected; The grayscale values ​​of the RGB three channels of the image to be tested are shifted separately; The Gaussian noise and / or grayscale value offset are different for each of the derived images.

7. The method according to claim 6, characterized in that, Extracting the frequency domain features and spatial domain features of the derived image, including: Spatial domain features of the image to be detected and the derived image are extracted using the first branch; The second branch extracts the frequency domain features of the image to be detected and the derived images.

8. The method according to claim 7, characterized in that, The spatial domain features of the image to be detected and the derived image are extracted through the first branch, including: Spatial domain features are extracted from the image to be detected and the derived image by means of an improved residual network. The improved residual network is obtained by replacing the ordinary convolutions in the third and fourth stages with depthwise separable convolutions and adding an SE attention module before the residual block on the base ResNet50.

9. The method according to claim 8, characterized in that, The second branch extracts frequency domain features from the image to be detected and the derived image, including: The image to be detected and the derived image are divided into blocks according to the set size; Perform a 2D Fast Fourier Transform on each block; Based on the amplitude spectrum of the image data obtained by the Fast Fourier Transform, a three-channel frequency domain spectrum is generated, and the generated three-channel frequency domain spectrum corresponds to the original RGB channels; The generated three-channel frequency domain spectrogram data is used as input to a convolutional neural network to generate a frequency domain feature vector with a lower dimension than the original image.

10. The method according to claim 9, characterized in that, Determining the intermediate feature vector based on the frequency domain features and the spatial domain features includes: Transform the frequency domain features and the spatial domain features to the same dimension; The similarity matrix between the spatial domain features and the frequency domain features is calculated using a mutual attention mechanism, and the cross-attention weights of the spatial domain features and the frequency domain features are obtained by normalizing the similarity matrix. The intermediate feature vector is determined and output based on spatial domain features, spatial cross-attention weights, frequency domain features, and frequency domain attention weights.

11. A system for detecting image forgery, characterized in that, The system includes: The feature extraction module is used to extract the frequency domain features and spatial domain features of the image to be detected and multiple derived images, wherein each of the derived images is an image generated after performing a first operation on the image to be detected; The fusion module is used to determine the intermediate feature vectors of the image to be detected and the derived image based on the frequency domain features and the spatial domain features, respectively. The discrimination module includes a real / fake discriminator and a model tracing device set in parallel. The real / fake discriminator determines the detection result of fake or real based on the intermediate feature vector. The model tracing device determines and generates a multivariate probability distribution based on the intermediate feature vector. The multivariate probability distribution is the probability distribution of the fake model forging the image. The real / fake discriminator is a binary classifier, and the model tracing device is a multivariate classifier. The output module is used to output information indicating whether the image to be detected is real or fake based on the detection result of the real / fake distinguisher and the probability distribution generated by the model tracer, and at least in the case that the image is fake, output the probability distribution.

12. The system according to claim 11, characterized in that, The output module is also configured to: In response to the fact that the forgery probabilities determined based on the intermediate feature vectors of the image to be detected and the derived image respectively satisfy the high confidence condition, the detection result of the real / fake distinguisher is output; wherein, if the forgery probability is greater than the high confidence threshold or less than the low confidence threshold, the high confidence condition is satisfied, and if the forgery probability is greater than the high confidence threshold, it indicates forgery, while if the forgery probability is less than the low confidence threshold, it indicates authenticity.

13. The system according to claim 12, characterized in that, The output module is also configured to: In response to the fact that at least one of the forgery probabilities determined based on the intermediate feature vectors of the image to be detected and the derived image respectively satisfies the low confidence condition, a portion of the intermediate feature vectors is randomly extracted and input into the discrimination module for secondary judgment. In the secondary determination result, if the true probability determined by the intermediate feature vector of the image to be detected and the derived image respectively meets the high confidence condition, the previous determination result is corrected according to the secondary determination result; If, in the secondary determination result, the true probability determined by the intermediate feature vectors of the image to be detected and the derived image is still at least one low confidence condition, the determination result of the true / false discriminator is corrected by combining the determination result of the model tracer. For a given image, if the probability of forgery is between the low confidence threshold and the high confidence threshold, then the low confidence condition is met.

14. The system according to claim 11, characterized in that, The system also includes: The heat map generation module is used to generate an interpretable heat map based on the detection results of the real / fake detector, and to overlay the interpretable heat map on the image to be detected in a semi-transparent manner and transmit it to the output module.

15. The system according to claim 14, characterized in that, The heat generation module is configured to generate the heat map through the following process: Obtain the feature map output by the last convolutional layer of the true / false discriminator to determine the weight gradient of each feature map, wherein the weight gradient reflects the contribution of the feature map to the judgment result; The feature maps are weighted and summed to merge them into an initial heatmap; The initial heatmap is processed to match it with the image to be detected; The initial heatmap is labeled to present high-contribution and low-contribution areas in different ways.

16. The system according to claim 11, characterized in that, Each of the multiple derived images is obtained through the following process: Add Gaussian noise within a predetermined frequency range to the image to be detected; The grayscale values ​​of the RGB three channels of the image to be tested are shifted separately; The Gaussian noise and / or grayscale value offset are different for each of the derived images.

17. The system according to claim 11, characterized in that, The feature extraction module includes a first branch for extracting spatial domain features and a second branch for extracting frequency domain features.

18. The system according to claim 17, characterized in that, The first branch includes an improved residual network, which is obtained on the base ResNet50 by replacing the ordinary convolutions in the third and fourth stages with depthwise separable convolutions and adding an SE attention module before the residual block.

19. The system according to claim 17, characterized in that, The second branch is configured to perform a fast Fourier transform on the input data and then feed it into a convolutional neural network to obtain a frequency domain feature vector.

20. The system according to claim 17, characterized in that, The fusion module is configured to determine the intermediate feature vector through the following process: Transform the frequency domain features and the spatial domain features to the same dimension; The similarity matrix between the spatial domain features and the frequency domain features is calculated using a mutual attention mechanism, and the cross-attention weights of the spatial domain features and the frequency domain features are obtained by normalizing the similarity matrix. The intermediate feature vector is determined based on spatial domain features, spatial cross-attention weights, frequency domain features, and frequency domain attention weights.

21. An electronic device, characterized in that, The electronic device includes: Memory, used to store instructions; A processor configured to implement the method according to any one of claims 1 to 10 when executing the instructions.

22. A program product, characterized in that, The program product includes instructions that, when executed, enable the implementation of the method according to any one of claims 1 to 10.

23. A non-volatile storage medium, characterized in that, Instructions are stored on the non-volatile storage medium, and when executed, the instructions are capable of implementing the method according to any one of claims 1 to 10.

Images