A method of counterfeit detection based on feature enhancement and spectral analysis

Abstract

This invention provides a forgery detection method based on feature enhancement and spectral analysis. In the RGB space, shallow texture feature enhancement preserves texture information, resulting in a texture feature enhancement map. In the RGB space, an attention map is obtained through a channel space attention mechanism module. The attention map is then multiplied element-wise with the input feature map to obtain a deep semantic feature enhancement map after attention enhancement. In the frequency domain, the image is converted from the RGB color space to the YCbCr color space through color space transformation. Different frequency domain channels are concatenated using DFT and DWT to obtain shallow frequency domain features, which are then processed by a feature extraction network to obtain a deep frequency domain feature map. The texture feature enhancement map and the deep semantic feature enhancement map are fused using bilinear pooling to obtain an RGB space feature map. Finally, the RGB space feature map and the deep frequency domain feature map are concatenated and classified using a fully connected layer. This method addresses the problem of being affected by the diversity of generative network structures and datasets.

Description

Technical Field

[0001] This invention relates to the field of deepfake detection in visual media, and in particular to a forgery detection method based on feature enhancement and spectral analysis. Background Technology

[0002] Due to limitations in generation technology, visual artifacts appear in the central region of faces within each frame in deepfakes of visual media. Furthermore, the imperfect blending of the fake area at the center of the face with the real area at the edges also leads to visual artifacts in the mixed region. These artifacts can be captured and distinguished using machine learning algorithms and deep learning models, providing a basis for forgery detection methods based on intra-frame information. Currently, forgery detection methods based on intra-frame information mainly include machine learning methods and deep learning methods.

[0003] Machine learning-based forgery detection methods typically model deepfake detection as a binary classification problem. First, a key facial feature capable of distinguishing between genuine and fake videos needs to be observed and extracted. Then, a classification model is trained by maximizing the difference of this feature between genuine and fake videos. These methods are characterized by strong interpretability, but generally have low detection accuracy and poor generalization performance.

[0004] While deep learning-based forgery detection methods offer good discrimination performance, the subtle and localized differences between real and fake images in this task, coupled with the "black box" nature of deep learning models, result in a lack of sufficient interpretability. Furthermore, these two types of solutions cannot simultaneously and fully extract texture, semantic, and frequency features useful for forgery detection. They are often heavily influenced by the generative network structure and the diversity of datasets, meaning only images and videos generated using specific methods can be identified. Therefore, their results are less than ideal when tested across different datasets.

[0005] In summary, machine learning-based methods require observing a key facial feature that can distinguish between genuine and fake videos. A classification model is trained by maximizing the difference of this feature between genuine and fake videos. These methods are characterized by strong interpretability, but their detection accuracy is generally low. Furthermore, since the methods used to forge videos are usually unknown, key features present in datasets created by a particular forging method are often not applicable to other datasets, resulting in poor generalization performance. While deep learning-based detection methods offer better discrimination performance, the "black box" nature of deep learning models hinders their interpretability. Summary of the Invention

[0006] The main objective of this invention is to overcome the aforementioned deficiencies in the prior art and propose a forgery detection method based on feature enhancement and spectral analysis. This method extracts texture features, semantic features, and frequency features that are useful for forgery detection, thereby improving the interpretability and generalization of the model. Furthermore, this method performs well on both single and mixed datasets, solving the problem of being affected by the diversity of generative network structures and datasets.

[0007] The present invention adopts the following technical solution:

[0008] A forgery detection method based on feature enhancement and spectral analysis, the method comprising:

[0009] In the RGB space, shallow texture feature enhancement is used to preserve texture information, resulting in a texture feature enhancement map.

[0010] In the RGB space, an attention map is obtained through the channel space attention mechanism module. The attention map is then multiplied element-wise with the input feature map to obtain the deep semantic feature enhancement map after attention enhancement.

[0011] In the frequency domain space, the image is converted from the RGB color space to the YCbCr color space through color space transformation. Different frequency domain channels are concatenated through two frequency domain transformations, DFT and DWT, to obtain shallow frequency domain features. Deep frequency domain feature maps are obtained through feature extraction networks.

[0012] The obtained texture feature enhancement map and the attention-enhanced deep semantic feature enhancement map are fused by bilinear pooling to obtain the RGB space feature map.

[0013] The RGB spatial feature map and the deep frequency domain feature map are concatenated along the channel dimension and then classified through a fully connected layer.

[0014] Specifically, in the RGB space, shallow texture feature enhancement is used to preserve texture information, resulting in a texture feature enhancement map, as follows:

[0015] First, global average pooling is used to downsample the feature map of a specific shallow layer to obtain pooled features D, and the difference between feature levels is defined to represent texture information:

[0016]

[0017] in, Represents shallow texture features. Represents pooling characteristics, This represents the enhanced shallow texture features;

[0018] The enhanced shallow texture features will be obtained. The final texture feature enhancement map is obtained through the dense connection module.

[0019] Specifically, an attention map is obtained in the RGB space through a channel-space attention mechanism. The attention map is then multiplied element-wise with the input feature map to obtain a deep semantic feature enhancement map after attention enhancement.

[0020] The feature map is input into the channel space attention mechanism module to obtain the corresponding attention map. Then, the attention map and the feature map are multiplied element-wise to obtain the deep semantic feature enhancement map after attention enhancement.

[0021] The channel spatial attention mechanism module includes a channel attention mechanism module and a spatial attention mechanism module;

[0022] First, the input feature map In the channel attention mechanism module, global max pooling and global flat pooling operations are performed, and then passed through a multilayer perceptron with weight sharing to obtain the corresponding channel max pooling feature maps. And channel pooling feature map ;

[0023] Then max pool the feature map Peace Bureau Pooling Feature Map Perform matrix addition, then perform a nonlinear transformation using the Sigmoid function to obtain the channel attention map;

[0024] Finally, the channel attention map and the input feature map F are multiplied element-wise to obtain the channel feature enhancement map. The channel attention mechanism can be expressed as:

[0025]

[0026] in, The weighting coefficients are used; and the resulting channel feature enhancement map is... The input graph serves as the input for the spatial attention mechanism module;

[0027] First, global max pooling (MaxPool) and global average pooling (AvgPool) are performed along the channel dimension to obtain the spatial max pooling feature map. Spatial average pooling feature map And splice them together according to the channel direction;

[0028] After another convolution operation with a kernel of 1, the number of channels is reduced to 1, and after activation by Sigmoid, a spatial attention map is obtained.

[0029] Finally, the spatial attention map and the input map of the spatial attention mechanism module are multiplied to obtain the final deep semantic feature enhancement map; the spatial attention mechanism can be expressed as:

[0030]

[0031] Where f represents the convolution operation. This is the Sigmoid activation function.

[0032] Specifically, in the frequency domain space, the image is converted from the RGB color space to the YCbCr color space through color space transformation. Different frequency domain channels are concatenated using DFT and DWT frequency domain transformations to obtain shallow frequency domain features. A feature extraction network is then used to obtain deep frequency domain feature maps. Specifically:

[0033] In the frequency domain, an image is converted from the RGB color space to the YCbCr color space using a color space transformation. The conversion formula is as follows:

[0034]

[0035]

[0036]

[0037]

[0038] in, , and This refers to the color conversion factor;

[0039] Each image is transformed using DFTDWT separately, and then fused to form the initial frequency features.

[0040] The DFT decomposes a signal into sinusoidal components of various frequencies from 0 to its maximum value based on spatial resolution. For two-dimensional data, i.e., an image of size W×H, the DFT calculation formula is as follows:

[0041]

[0042] in, For vertical spatial frequency, For horizontal spatial frequency, coordinates Pixel value at that location, Including coordinates Amplitude and phase information at the specified frequency;

[0043] An image is decomposed into four distinct sub-bands using DWT. High-pass and low-pass filters are applied to each row / column, and downsampling is performed to obtain the high-frequency and low-frequency components for each row / column. The original image is then converted into four sub-bands: HH, HL, LH, and LL. The HH sub-band retains high-frequency components in both the horizontal and vertical directions; the HL sub-band retains high-frequency components in the horizontal direction and low-frequency components in the vertical direction; and the LL sub-band retains low-frequency components in both the horizontal and vertical directions.

[0044] After extracting the shallow frequency features, the deep frequency feature map is obtained through the backbone network module.

[0045] Specifically, the obtained texture feature enhancement map and the attention-enhanced deep semantic feature enhancement map are fused using bilinear pooling to obtain the RGB spatial feature map:

[0046] set up and It is a characteristic function. It is the dimension of the output feature. It is a pooling function. It is a classifier, which has: , where the image Image location information ; here, location It contains the image's location and scale information. and Given the feature matrix; by combining the feature outputs at each position using the matrix outer product, a bilinear combination can be obtained:

[0047]

[0048] in, and The dimensions need to be consistent with Consistent; Pooling function A global image representation is obtained by performing bilinear combination at all locations. Specifically, this refers to summation pooling:

[0049] .

[0050] Specifically, the RGB spatial feature map and the deep frequency domain feature map are concatenated along the channel dimension and then classified through a fully connected layer, as follows:

[0051] The Softmax classifier maps vectors in a high-dimensional space to categories, and the final result is represented as probabilities:

[0052]

[0053] in, It is a weight vector, representing the parameters corresponding to each category in the classifier; The parameters used to calculate the probability of a unit belonging to a class are, for a training set, assumed to contain... Sample ,in Represents the vector to be predicted. represent The corresponding category label, for a specific The Softmax classifier predicts the probability of a class as follows:

[0054]

[0055] in, Let be a vector, where represent For types The probability, The sum of all elements in the matrix is ​​1; prediction When categorizing, select the value corresponding to the highest probability. As the discrimination result; parameters The value is obtained by iteratively updating and minimizing the loss function; the loss function of Softmax regression is defined as:

[0056]

[0057] in, This represents an indicator function; its value is 1 if it is true and 0 if it is false.

[0058] The loss function used is the cross-entropy loss, which is commonly used in binary classification tasks:

[0059]

[0060] in, The sample labels are set to 1 for real faces and 0 for fake faces; This represents the probability that the prediction is the true sample; correspondingly, This represents the probability that the prediction is a fake sample.

[0061] As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0062] This invention provides a forgery detection method based on feature enhancement and spectral analysis. In the RGB space, shallow texture feature enhancement preserves texture information, resulting in a texture feature enhancement map. In the RGB space, an attention map is obtained through a channel-space attention mechanism module. The attention map is then multiplied element-wise with the input feature map to obtain a deep semantic feature enhancement map after attention enhancement. In the frequency domain, the image is converted from the RGB color space to the YCbCr color space through color space transformation. Different frequency domain channels are concatenated using DFT and DWT frequency domain transformations to obtain shallow frequency domain features. These shallow features are then extracted using a feature extraction network to obtain deep semantic feature enhancement maps. The method is as follows: The obtained texture feature enhancement map and the attention-enhanced deep semantic feature enhancement map are fused using bilinear pooling to obtain the RGB spatial feature map; the RGB spatial feature map and the deep frequency domain feature map are concatenated along the channel dimension and classified through a fully connected layer. The method provided in this invention fully extracts texture, semantic, and frequency features useful for forgery detection, improving the interpretability and generalization of the model. Compared with mainstream detection methods, this method achieves advanced results on both single and mixed datasets, solving the problem of being affected by the diversity of generative network structures and datasets. Attached Figure Description

[0063] Figure 1 This is an overall method flowchart provided for embodiments of the present invention;

[0064] Figure 2 This is a framework diagram for implementing the method according to an embodiment of the present invention;

[0065] Figure 3 This is a diagram of a shallow texture feature enhancement structure provided in an embodiment of the present invention;

[0066] Figure 4 This is a deep semantic feature enhancement structure diagram provided in an embodiment of the present invention;

[0067] Figure 5 This is a structural diagram of the CBAM module provided in an embodiment of the present invention;

[0068] Figure 6 This is a structural diagram of the channel attention module provided in an embodiment of the present invention;

[0069] Figure 7 This is a structural diagram of the spatial attention module provided in an embodiment of the present invention;

[0070] Figure 8 This is a structural diagram of the initial frequency domain feature extraction spatial attention module provided in an embodiment of the present invention;

[0071] Figure 9 This is a structural diagram of the frequency domain spatial feature extraction branch spatial attention module provided in an embodiment of the present invention;

[0072] Figure 10 This is a heatmap comparison diagram of a real face and a fake face in Deepfake provided in an embodiment of the present invention;

[0073] Figure 11 This is a heatmap comparison diagram of real and fake faces provided in an embodiment of the present invention;

[0074] Figure 12 This is a heatmap comparison diagram of a real face and a fake face provided in an embodiment of the present invention.

[0075] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0076] The present invention will be further described below through specific embodiments.

[0077] The overall flowchart of the present invention is as follows: Figure 1 As shown, the method implementation framework is as follows: Figure 2 As shown, the specific implementation steps are as follows:

[0078] (1) RGB spatial feature extraction

[0079] 1) Enhanced shallow texture features

[0080] Shallow texture feature enhancement techniques are used to preserve more texture information in order to capture these artifacts, such as... Figure 3 As shown.

[0081] First, global average pooling is used to downsample the feature map of a specific shallow layer to obtain the pooled feature D.

[0082] Then, similar to the texture representation of spatial images, feature-level differences are defined to represent texture information:

[0083] (1)

[0084] in, Represents shallow texture features. Represents pooling characteristics, This represents the enhanced shallow texture features. (The result is...) The final texture feature enhancement map is then obtained through a dense connection module.

[0085] 2) Deep semantic feature enhancement

[0086] In the context of this invention, deep semantic features refer to key facial features such as eyes and mouth. From a spatial perspective, this refers to the location of key feature regions within the entire image.

[0087] The channel-space attention mechanism (CBAM) in visual attention mechanisms can learn attention maps independently in two dimensions (channel and space);

[0088] Then, the attention map and the input feature map are weighted again through element-wise multiplication to achieve adaptive feature optimization, which is then incorporated into the deep semantic feature enhancement structure, such as... Figure 4 As shown.

[0089] After the deep feature maps of the network are processed by CBAM, their corresponding attention maps are obtained. Then, the attention maps are multiplied element-wise with the deep feature maps to obtain the deep semantic feature enhancement maps after attention enhancement.

[0090] The CBAM module includes both spatial and channel attention mechanisms. The output feature maps of the convolutional layers are weighted after passing through the spatial and channel attention modules to obtain the final result. The CBAM module structure is as follows: Figure 5 As shown.

[0091] Channel attention structure such as Figure 6 As shown.

[0092] First, the input feature map The system performs global max pooling (MaxPool) and global average pooling (AvgPool) operations, and then passes them through a weight-shared multilayer perceptron (SharedMLP).

[0093] Then, the two feature maps output by the multilayer perceptron are subjected to matrix addition and nonlinear transformation by the sigmoid function to obtain the channel attention map.

[0094] Finally, the feature map is multiplied element-wise with the input feature map of this module to obtain the result. This is used as the input feature for the subsequent spatial attention module. The channel attention mechanism can be expressed as:

[0095] (2)

[0096] Spatial attention structure such as Figure 7 As shown, the output feature map of the channel attention module in the previous step is the input feature map of this module. .

[0097] First, global max pooling (MaxPool) and global average pooling (AvgPool) are performed along the channel dimension. After concatenating these two results along the channel direction, a convolution operation with a kernel of 1 is performed to reduce the number of channels to 1. After activation by Sigmoid, a spatial attention map is obtained.

[0098] Finally, the spatial attention map and the input of this module are multiplied to obtain the final generated features. Spatial attention mechanism can be expressed as:

[0099] (3)

[0100] (2) Frequency domain spatial feature extraction

[0101] The three channels in the RGB color space are interconnected. To achieve feature decoupling and better extract frequency features, we consider converting the input image to an orthogonal color space. We use the YCbCr color space, which offers better performance than RGB. The conversion formula is as follows:

[0102]

[0103]

[0104]

[0105] (4)

[0106] in, , and This is the color conversion coefficient, and its value is determined according to the standard, as shown in Table 1.

[0107] Table 1 Conversion coefficients from RGB space to YCbCr space

[0108]

[0109] After the input image is converted to the YCbCr color space, in order to fully extract frequency information from the YCbCr image, this invention performs Discrete Fourier Transform (DFT) and Discrete Wavelet Transform (DWT) on each image separately, and then fuses the features of the two to obtain the initial frequency features. This frequency domain stream fusion of two different types of spectra yields an improved frequency domain representation, which can better distinguish between real and fake visual media content.

[0110] The Distributed Fourier Transform (DFT) can decompose a signal into sinusoidal components of various frequencies from 0 to its maximum value based on spatial resolution. For two-dimensional data, i.e., an image of size W×H, the DFT calculation formula is as follows:

[0111] (5)

[0112] in, For vertical spatial frequency, For horizontal spatial frequency, coordinates Pixel value at that location, Including coordinates The amplitude and phase information at the specified frequency.

[0113] A single image is decomposed into four distinct sub-bands using DWT. High-pass and low-pass filters are applied to each row (column), and then downsampled to obtain the high-frequency and low-frequency components for each row (column). The original image is then converted into four sub-bands: High-high (HH), High-low (HL), Low-high (LH), and Low-low (LL). Each sub-band retains different characteristics: the HH region retains high-frequency components in both the horizontal and vertical directions; HL retains high-frequency components in the horizontal direction and low-frequency components in the vertical direction; LH is the opposite of HL; and finally, LL retains low-frequency components in both the horizontal and vertical directions.

[0114] Figure 8 The initial frequency domain feature extraction process is demonstrated.

[0115] First, the image is converted from the RGB color space to the YCbCr space, and then transformed to the frequency domain using DFT and DWT. Each channel is divided into a non-overlapping block of size 8×8 pixels, and the transform is applied independently to each block. The output of DFT converts one input channel into two output channels, corresponding to real coefficients and imaginary coefficients, respectively.

[0116] Similarly, DWT converts one input channel into four output channels, corresponding to low frequency (LL), high-low frequency (HL), high frequency (HH), and low-high frequency (LH). For the three input channels of YCbCr, after conversion, 18 output channels are obtained, of which 6 come from DFT and 12 come from DWT.

[0117] These frequency output channels are connected in series to form a three-dimensional cube of size H×W×C, where H is the height of the image, W is the width of the image, and C=18 is the number of channels.

[0118] In summary, the frequency domain spatial feature extraction branch is as follows: Figure 9 As shown, after extracting the shallow frequency features, the deep frequency feature map is obtained through the backbone network module.

[0119] (3) Feature fusion and classifier design

[0120] In forgery detection methods based on feature enhancement and spectral analysis, there are two feature fusion processes. One is the fusion of the texture-enhanced feature map obtained after shallow texture feature enhancement and the semantic-enhanced feature map obtained after deep semantic feature enhancement in the RGB space. The other is the fusion of the RGB feature map obtained through the RGB space feature extraction branch and the frequency domain feature map obtained through the frequency domain space feature extraction branch. The former uses a bilinear pooling feature fusion method, while the latter performs feature fusion by concatenating features along the channel dimension.

[0121] 1) Bilinear pooling

[0122] Bilinear pooling obtains two features from the same data using two feature extraction modules. and Fusion using bilinear pooling operation and The resulting feature-fused vector is then fed into a subsequent classifier for classification.

[0123] Bilinear pooling merges features from two classes originating from different branches, improving the reliability of image classification. Let... and It is a characteristic function. It is the dimension of the output feature. It is a pooling function. It is a classifier, which has: , where the image Image location information Here, location It contains the image's location and scale information. By combining the feature outputs at each location using matrix outer product, a bilinear combination can be obtained:

[0124] (6)

[0125] in, and The dimensions need to be with Consistent, The value depends on the specific category. Pooling function A global image representation is obtained by performing bilinear combination at all locations. This invention uses summation pooling in experiments:

[0126] (7)

[0127] Bilinear operations simplify gradient calculations and allow for end-to-end training of two networks using only image labels. When iterating the parameters through backpropagation, the loss function, if the matrix... and The output dimension is and Bilinear characteristics That is Dimension. Loss function. The characteristic function is set as According to the gradient chain rule, we can obtain:

[0128] (8)

[0129] 2) Softmax classifier

[0130] The fused features are classified using a Softmax classifier. The Softmax classifier maps vectors in a high-dimensional space to categories, and the final result is represented as probabilities.

[0131] (9)

[0132] in, It is a weight vector, representing the parameters corresponding to each category in the classifier. Determined through classifier training, it is a parameter used to calculate the probability of a unit belonging to a class, and then to calculate its class. For a training set, assuming it contains... Sample ,in Represents the vector to be predicted. represent The corresponding category label, for a specific The Softmax classifier predicts the probability of a class as follows:

[0133] (10)

[0134] in, Let be a vector, where represent For types The probability, The sum of all elements in the matrix is ​​1. Prediction. When categorizing, select the value corresponding to the highest probability. As the result of the judgment. Parameters The value can be obtained by iteratively updating and minimizing the loss function. The loss function of Softmax regression is defined as:

[0135] (11)

[0136] in, This represents an indicator function; its value is 1 if it is true and 0 if it is false.

[0137] The loss function used in this invention is the cross-entropy loss, which is commonly used in binary classification tasks:

[0138] (12)

[0139] in, The sample labels are set to 1 for real faces and 0 for fake faces; This represents the probability that the prediction is the true sample; correspondingly, This represents the probability that the prediction is a fake sample.

[0140] (4) Effect evaluation

[0141] The dataset used in this invention is FaceForensics++, which contains 1000 original videos, 1000 Deepfake videos, 1000 FaceSwap videos, and 1000 Face2face videos. Each video is segmented into frames, and 70 images are randomly selected independently. For each forgery type, the training set contains 68,000 images, and the test set contains 2,000 images. The input image size is 256×256. The distribution of the FaceForensics++ dataset is shown in Table 2.

[0142] Table 2. FaceForensics++ Dataset Allocation

[0143]

[0144] The essence of the method of this invention is to perform binary classification on the input data. In order to evaluate its performance, the experimental results are analyzed using the commonly used indicators in binary classification, namely accuracy and AUC

[59] .

[0145] Accuracy is the ratio of the number of correct predictions to the total amount of test data, i.e.:

[0146] (13)

[0147] Wherein, TP represents the number of correctly predicted positive samples, which in this invention is the number of real images predicted as true. TN represents the number of correctly predicted negative samples, which in this invention is the number of fake images predicted as false. FP represents the number of negative samples predicted as positive samples, which in this invention is the number of fake images predicted as true. FN represents the number of positive samples predicted as negative samples, which in this invention is the number of real images predicted as fake.

[0148] AUC is the area under the ROC curve. This area is always less than 1; a larger value indicates better classification performance. The ROC curve is plotted with TPR on the ordinate and FPR on the abscissa, calculated as follows:

[0149] (14)

[0150] (15)

[0151] The backbone networks of this invention are XceptionNet, ResNet-18, and ResNet-50, respectively, and ablation experiments are conducted on feature enhancement and spectral analysis. XceptionNet after feature enhancement and spectral analysis is denoted as XceptionNet-Fs, and correspondingly, the networks of ResNet18 and ResNet50 after feature enhancement and spectral analysis are labeled as ResNet18-Fs and ResNet50-Fs, respectively. The experimental results are shown in Table 3.

[0152] Table 3 Detection performance of different backbone networks

[0153]

[0154] As can be seen from Table 3, compared with the backbone network, the model after fusing feature enhancement and spectral analysis has a significant improvement in the accuracy of fake face detection.

[0155] XceptionNet, benefiting from its depthwise separable convolutional structure, exhibits good performance directly in forgery detection tasks. However, the dilated convolutions in XceptionNet-Fs significantly reduce the number of parameters, resulting in insufficient learning of feature map information on the same channel. Consequently, the improvement in detection accuracy after fusing feature enhancement and spectral analysis is not substantial, and it is slightly inferior to ResNet50-Fs. ResNet50-Fs, with its deeper network structure compared to ResNet18-Fs, can capture deeper forgery features in fake faces. Therefore, ResNet50-Fs achieves an ACC value of over 97% for all forgery types, making it the best among various detection methods. In particular, on the Deepfake forgery dataset, the accuracy of the method in this invention is 97.86%.

[0156] In summary, the method of this invention (ResNet50-Fs) achieved the best performance among all the methods tested.

[0157] 1) Single dataset experiment

[0158] In this set of experiments, the method of this invention is compared with three mainstream methods for deepfake detection based on intra-frame information: XceptionNet, MesoNet, and Capsule-Forensics. XceptionNet uses depthwise separable convolutions, MesoNet detects deepfakes based on mid-level semantic information, and the Capsule-Forensics method is based on "capsule networks" and uses a dynamic routing algorithm to detect deepfakes.

[0159] The proposed method and the three methods mentioned above were tested on three different types of forgery in FaceForensics++, and the detection accuracy results are shown in Table 4. It can be seen that the forgery detection method based on feature enhancement and spectral analysis of this invention achieved the best results on all three types of forgery datasets. This invention's method extracts and enhances texture and semantic features of the input image in the RGB space, fully learning and extracting key features beneficial for classification, thus verifying the effectiveness of the shallow texture feature enhancement technology and deep semantic feature enhancement technology of this invention.

[0160] Table 4 Comparison of detection accuracy of various methods

[0161]

[0162] The AUC performance of the method of this invention was compared with the three methods mentioned above, and the results are shown in Table 5. It can be seen that the method of this invention achieved relatively stable accuracy in all three types of face forgery, and achieved the best results on all three datasets. This is because the method of this invention, through RGB spatial feature analysis and frequency domain spatial feature analysis, fully extracts and enhances key features, enabling it to quickly locate features useful for classification during model training, thus making the training process more stable.

[0163] In summary, by comparing the accuracy and AUC values ​​of the four methods on three face spoofing datasets, it is demonstrated that the method of this invention has superior performance on a single dataset.

[0164] Table 5 Comparison of AUC values ​​for each method

[0165]

[0166] 2) Mixed Dataset Experiment

[0167] The method of this invention, along with the three methods mentioned above, was tested on a mixed dataset containing three types of tampering: Deepfake, FaceSwap, and Face2face. The results are shown in Table 6. It can be seen that the method of this invention achieved the best results in both ACC and AUC metrics. The method of this invention can simultaneously learn RGB spatial features and frequency spatial features; these two types of features complement each other, and feature fusion significantly improves the model's generalization performance on mixed datasets.

[0168] Table 6 shows the cross-dataset detection results of each method.

[0169]

[0170] 3) Visualization of Grad_CAM results

[0171] The Grad_CAM technique is used to visualize the trained model. Grad_CAM uses heatmaps to display regions of interest in the image, explicitly showing the criteria of the method. Forged and real images have discernible differences; Grad_CAM visualizes and explains these differences.

[0172] Compared to real images, fake face images lack detailed information. The network only focuses on certain regions of the fake face image and assigns larger activation values ​​to these regions. The model then distinguishes between real and fake images based on these differences. The network is trained using a binary classification method. Each layer's feature map contains information for classification, and the highest layer feature map, being closest to the classifier, contains more crucial information. Visualization of this layer is considered. Figure 10 , Figure 11 , Figure 12 Heatmaps of real and fake faces from three forgery methods—Deepfake, FaceSwap, and Face2face—are displayed. Each image contains two rows: the top row shows the real face and its heatmap, while the bottom row shows the forged face image and its heatmap for the corresponding forgery method. In the heatmaps, areas with high activation values ​​are represented by red and yellow, while blue represents areas with low activation values.

[0173] Observations reveal that genuine face heatmaps lack regularly occurring regions of interest, while fake face heatmaps focus on the edges or center of the face. This is because the model, failing to detect anomalies in real faces, seeks other relatively prominent areas that might facilitate detection, such as the focus of the eyes, reflections on the forehead, and lighting on the face. In contrast, fake images exhibit regularly occurring detail gaps and edge splicing. The shallow texture feature enhancement technique of this invention can focus on edge gaps, while the deep semantic feature enhancement technique can focus on detail gaps, thereby capturing key anomalous features in local and edge areas of the face. Taking Deepfake and FaceSwap as examples, because their forgery methods involve face swapping, their heatmaps show locally and edge-focused regions. The Face2face method, used for facial expression reproduction forgery, is prone to anomalies in the mouth area, thus Face2face's heatmap focuses on the mouth region.

[0174] In summary, the experimental results of this invention demonstrate that, compared with mainstream detection methods, the method of this invention achieves superior results on both single and mixed datasets.

[0175] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.

Claims

1. A method of counterfeit detection based on feature enhancement and spectral analysis, characterized in that, The method comprises: In the RGB space, texture information is reserved by shallow texture feature enhancement to obtain a texture feature enhancement map; In the RGB space, an attention map is obtained by a channel space attention mechanism module, and the attention map is multiplied element by element with the input feature map to obtain a deep semantic feature enhancement map after attention enhancement; In the frequency domain space, the picture is converted from the RGB color space to the YCbCr color space by color space conversion, different frequency domain channels are concatenated by DFT and DWT frequency domain transformation to obtain shallow frequency domain features, and a deep frequency domain feature map is obtained by a feature extraction network; The RGB space feature map and the deep frequency domain feature map in the frequency domain space are spliced in the channel dimension, and classified by a fully connected layer; The method comprises: In the frequency domain space, the picture is converted from the RGB color space to the YCbCr color space by color space conversion, and the conversion formula is as follows: After DFTDWT transformation is performed on each image, the images are fused as initial frequency features; wherein , and are color conversion coefficients; DFT decomposes signals into sinusoidal components of various frequencies from 0 to the maximum value according to the spatial resolution; for two-dimensional data, i.e. an image with a size of WxH, the calculation formula of DFT is as follows: A DWT is used to decompose an image into four different sub-band images; high-pass and low-pass filters are used on each row / column, and down-sampling is performed to obtain the high-frequency and low-frequency components of each row / column; the original image is converted into four sub-band images: HH sub-band, HL sub-band, LH sub-band and LL sub-band; the HH sub-band retains high-frequency components in the horizontal and vertical directions; the HL sub-band retains high-frequency components in the horizontal direction and low-frequency components in the vertical direction; the LH sub-band retains low-frequency components in the horizontal direction and high-frequency components in the vertical direction; and the LL sub-band retains low-frequency components in the horizontal and vertical directions; in, For vertical spatial frequency, For horizontal spatial frequency, coordinates Pixel value at that location, Including coordinates Amplitude and phase information at the specified frequency; After the shallow frequency features are extracted, a deep frequency feature map is obtained by a backbone network module. In the RGB space, texture information is reserved by shallow texture feature enhancement to obtain a texture feature enhancement map, and the method comprises:

2. The method of claim 1, wherein the method further comprises: First, a global average pooling is used to down-sample a specific shallow feature map to obtain a pooled feature D, and a difference value of the feature level is defined to represent the texture information: In the RGB space, an attention map is obtained by a channel space attention mechanism, and the attention map is multiplied element by element with the input feature map to obtain a deep semantic feature enhancement map after attention enhancement, and the method comprises: wherein, represents a shallow texture feature, represents a pooling feature, represents an enhanced shallow texture feature; the enhanced shallow texture features are obtained a final texture feature enhancement map is obtained through the dense connection module. 3.The feature enhancement and spectrum analysis based forgery detection method of claim 1, wherein, The feature map is input into a channel space attention mechanism module to obtain a corresponding attention map, and the attention map is multiplied element by element with the feature map to obtain a deep semantic feature enhancement map after attention enhancement; ​ The channel space attention mechanism module includes a channel attention mechanism module and a space attention mechanism module; The first input feature map In the channel attention mechanism module, a global maximum pooling operation and a global tie pooling operation are performed, and then a weight-shared multi-layer perceptron is respectively performed to obtain a channel maximum pooling feature map And a channel tie pooling feature map ; Then the maximum pooling feature map and the peace pool feature map Perform matrix addition, nonlinear transformation by Sigmoid function, and get channel attention map; Finally, the channel attention map and the input feature map F are multiplied element by element to obtain a channel feature enhancement map The channel attention mechanism can be expressed as: wherein, is a sigmoid activation function; and the obtained channel feature enhancement map is input into the spatial attention mechanism module; First, global max pooling (MaxPool) and global average pooling (AvgPool) are performed along the channel dimension to obtain a spatial maximum pooling feature map and a spatial average pooling feature map and are spliced in the channel direction; After a convolution operation with a convolution kernel number of 1, the channel number is reduced to 1, and the space attention map is obtained after Sigmoid activation; Finally, the space attention map and the input map of the space attention mechanism module are multiplied to obtain the finally generated deep semantic feature enhancement map; the space attention mechanism can be expressed as: Wherein, f is a convolution operation. 4.The feature enhancement and spectrum analysis based forgery detection method of claim 1, wherein, The obtained texture feature enhancement map and the deep semantic feature enhancement map after attention enhancement are subjected to bilinear pooling for feature fusion to obtain an RGB space feature map: Let and be feature functions, be the dimension of the output feature, be a pooling function, be a classifier, with: where image image location information Here, location contains the location information and scale information of the image, and are feature matrices; using matrix outer product to combine the feature outputs at each location, i.e., the bilinear combination can be obtained: wherein and the dimensionality of the must be consistent; the pooling function obtains a global image representation by performing a bilinear combination over all positions ; in particular, sum pooling: 。 5. The method of claim 4, wherein the feature enhancement and spectral analysis based counterfeit detection method is characterized by, The RGB space feature map and the deep frequency domain feature map in the frequency domain space are spliced in the channel dimension, and a full connection layer is used for classification, specifically: The Softmax classifier maps the vector in the high-dimensional space to the category, and the final result is expressed as a probability: wherein, is a weight vector, representing parameters corresponding to each class in the classifier; is a parameter used to calculate the probability of the class to which the unit to be classified belongs, for a training set, assuming that it contains samples wherein represents a vector to be predicted, represents a corresponding class label, for a certain specific , the probability formula of Softmax classifier for each class is as follows: wherein, is a vector, wherein represents is a kind of probability, all elements in sum to 1; when predicting a class, the category corresponding to the largest value of probability is taken as the discrimination result; the values of parameters are obtained by iterative updating to minimize a loss function; the loss function of Softmax regression is defined as: wherein represents an indicator function, whose value is 1 if the value is true and whose value is 0 if the value is false; The loss function uses the cross-entropy loss commonly used in binary classification tasks: wherein, is 1 for real face and 0 for fake face; represents the probability of predicting as real sample; correspondingly, represents the probability of predicting as fake sample.

Images