An efficient deepfake detection method based on spatial frequency domain multi-scale feature similarity measurement

Abstract

This invention belongs to the field of deepfake detection technology, specifically disclosing an efficient deepfake detection method based on spatial-frequency domain multi-scale feature similarity measurement. The method involves acquiring an image, extracting image information, and fusing the image information to obtain feature maps at different scales. Similarity features within the spatial domain and frequency domain are calculated based on these feature maps. The spatial and frequency domain similarity features are then fused to obtain a feature map spectrum based on spatial-frequency domain multi-scale feature similarity measurement. Finally, the method determines whether the image is a forgery. By employing a similarity measurement strategy, the accuracy of similarity calculation is significantly improved, and the overall computational burden is reduced. The use of spatial-frequency domain multi-scale feature fusion technology enables this method to detect forgery signs in both the spatial and frequency domains, enriching the content of feature extraction, achieving better performance, providing better reconstruction results, and maintaining detection efficiency.

Description

Technical Field

[0001] This invention belongs to the field of deepfake detection technology, and specifically discloses an efficient deepfake detection method based on spatial frequency domain multi-scale feature similarity measurement. Background Technology

[0002] In recent years, deepfake technology has developed rapidly, enabling attackers to manipulate facial regions in images and generate fake images. These techniques can manipulate facial expressions and identities, making it extremely difficult for even the human eye to distinguish whether an image has been manipulated. Deepfake technology is used in a variety of applications, including entertainment, filmmaking, and virtual reality. Most existing methods employ deep convolutional neural network (CNN) models, taking a cropped facial region as input and generating a binary real / fake prediction. These methods capture artifacts in facial regions at a single scale through stacked convolution operations. While stacked convolutions achieve good detection results, they excel at modeling local information but cannot consider global pixel relationships due to their limited receptive field.

[0003] With the increasing popularity of deepfake technology, there is an urgent need for an effective detection mechanism to identify these forged images that are visually indistinguishable from genuine ones. Summary of the Invention

[0004] The purpose of this invention is to provide an efficient deep forgery detection method based on multi-scale feature similarity measurement in the spatial frequency domain, so as to solve the technical problem that the existing technology cannot consider the global relationship of pixels, resulting in low recognition accuracy and efficiency.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, this invention provides an efficient deepfake detection method based on a multi-scale feature similarity metric in the spatial-frequency domain, comprising: Acquire images, extract image information, and fuse the image information to obtain feature maps at different scales; Calculate the similarity features within the spatial domain and the similarity features within the frequency domain based on feature maps of different scales; By fusing similarity features in the spatial domain and similarity features in the frequency domain, a feature map of spatial-frequency multi-scale feature similarity measurement is obtained. The image is determined to be forged based on the feature map of the spatial-frequency domain multi-scale feature similarity metric.

[0006] Preferably, the step of extracting image information and fusing the image information to obtain feature maps of different scales includes: S101: Extract semantic and spatial information of the acquired image using the EfficientNet-B4 network; S102: Fusion of image semantic information and image spatial information to obtain feature maps at different scales.

[0007] Preferably, the step of calculating similarity features within the spatial domain based on feature maps of different scales specifically includes: After slicing the feature maps at different scales, a channel attention model is used to process them to obtain the query value in the spatial domain. Feature maps of different scales are sequentially processed by depthwise separable convolution, BN layer processing, and slicing to obtain spatial key values; Calculate the similarity features within the spatial domain features based on the query value and key value of the spatial domain.

[0008] Preferably, the step of calculating similarity features within the frequency domain based on feature maps of different scales specifically includes: The feature maps at different scales are processed by average pooling to obtain a global average image patch; the global average image patch is then subjected to discrete Fourier transform, flattening and projection to obtain a global frequency label. The feature maps at different scales are processed by average pooling to obtain a global average image block; the global average image block is then subjected to discrete Fourier transform and flattening to obtain local frequency markers, and frequency embedding is introduced into the local frequency markers. The global frequency marker, local frequency marker, and frequency embedding are processed by a learnable filter to obtain the query value and key value in the frequency domain. The frequency domain key values ​​are processed by the channel attention module to obtain key features; The similarity features within the frequency domain are obtained by projection operation on the key features and the query values ​​in the frequency domain.

[0009] Preferably, the fusion of similarity features in the spatial domain and similarity features in the frequency domain to obtain a similarity measurement result spectrum specifically includes: S301: After segmenting similarity features within the spatial domain, a spatial domain value vector is obtained; after segmenting similarity features within the frequency domain, a frequency domain value vector is obtained. S302: The spatial domain value vector and the frequency domain value vector are processed through a linear embedding layer to obtain the spatial domain query value, the spatial domain key value, the frequency domain query value, and the frequency domain key value; S303: The query value in the spatial domain, the key value in the spatial domain, the query value in the frequency domain, and the key value in the frequency domain are fused to obtain a similarity measurement result map.

[0010] Preferably, S303 is implemented using the following formula: ( ), =unpatchfy( ), =unpatchfy( ), = ( ) In the formula, unpatchfy is the operation that reassembles the flattened sequence into an image based on its relative spatial position. It consists of ReLU activation layers, 1×1 convolutional layers, and BN layers; This represents the space-frequency domain joint self-attention weight matrix; This represents the Softmax normalization function, which maps the input to a probability distribution; A query vector representing frequency domain features; A query vector representing spatial domain features; Key vectors representing frequency domain features; Key vectors representing spatial domain features; superscript Indicates transpose; Indicates feature dimension; This indicates a vector concatenation operation; This represents the reconstructed features in the frequency domain after self-attention weighting; Represents a spatial domain value vector; This represents the reconstructed features in the frequency domain after self-attention weighting; Represents a frequency domain value vector; The feature map representing the spatial-frequency domain multi-scale feature similarity measure is input into a binary classifier to distinguish between real and fake faces.

[0011] Preferably, determining whether an image is forged based on the feature map of spatial-frequency domain multi-scale feature similarity measurement includes the following steps: Mapping the feature map of spatial-frequency domain multi-scale feature similarity measure to the label space in the classification task yields the unnormalized log odds. The unnormalized log odds are applied to an activation function for multi-class classification to obtain predicted labels; the activation function is shown in the following formula:

[0012] The image is determined to be forged based on the predicted label; wherein, if It is determined to be counterfeit; if It was determined to be true.

[0013] In a second aspect, the present invention provides an efficient deepfake detection system based on a multi-scale feature similarity metric in the spatial-frequency domain, comprising: The image preprocessing unit is used to acquire images, extract image information, and fuse image information to obtain feature maps at different scales; The feature extraction unit is used to calculate similarity features in the spatial domain and similarity features in the frequency domain based on feature maps of different scales. The feature fusion unit is used to fuse similarity features in the spatial domain and similarity features in the frequency domain to obtain a feature map of spatial-frequency multi-scale feature similarity measurement. The judgment unit is used to determine whether the image is forged based on the feature map of the spatial-frequency domain multi-scale feature similarity metric.

[0014] In a third aspect, the present invention provides an electronic device including a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the efficient deep forgery detection method based on spatial frequency domain multi-scale feature similarity measurement as described in any one of the preceding claims.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium storing at least one instruction, wherein the at least one instruction, when executed by a processor, implements the efficient deep forgery detection method based on spatial frequency domain multi-scale feature similarity measurement as described above.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention extracts and fuses image information to obtain feature maps at different scales, which are then used for feature extraction and processing in the spatial and frequency domains. This design can fully capture multi-level information in images, extracting local detail features and overall structural features, thereby significantly improving the model's performance and robustness. Through spatial and frequency domain feature extraction, it is ultimately used for classification tasks, effectively processing image data and extracting features to improve model performance. Calculating similarity features within spatial features based on feature maps of different scales and dividing image patches into tokens of different scales can effectively capture spatial similarity between different regions. Calculating the similarity between each image region helps the model better understand the structural and textural features in the image, improves the model's ability to perceive image content, and makes it more accurate when processing complex images. Calculating similarity features in the frequency domain based on feature maps of different scales allows for feature extraction and similarity calculation in the frequency domain. This enables the capture of details that cannot be directly perceived by features in the spatial domain, allowing for a more comprehensive understanding of the image content. This enables the model to cope with more complex and diverse image scenes, while also improving the robustness of the model in different tasks. By fusing similarity features within the spatial domain and the frequency domain, a feature map of spatial-frequency multi-scale feature similarity measurement is obtained. By fusing feature maps from different scales and branches, comprehensive capture of complex patterns and details is achieved. Through feature mapping, fusion, and weighting, these modules enhance the model's ability to handle various visual tasks, improve its performance and robustness, and strengthen the model's overall capabilities.

[0017] In summary, this invention significantly improves the accuracy of similarity calculation by designing a similarity measurement strategy, while reducing the overall computational burden. It also proposes an advanced spatial-frequency domain multi-scale feature fusion technique, which enables the model in this application to detect forgery signs in both the spatial and frequency domains, thereby enriching the content of feature extraction, achieving better performance, providing better reconstruction results, and maintaining detection efficiency. Attached Figure Description

[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is a system block diagram according to an embodiment of the present invention; Figure 3 This is a system processing flowchart of an embodiment of the present invention; wherein, (a) is the backbone network; (b) is the spatial similarity measurement module; (c) is the frequency similarity measurement module; and (d) is the spatial-frequency cross-attention module. Figure 4 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0020] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0021] See Figure 1 This application discloses an efficient deepfake detection method based on multi-scale feature similarity measurement in the spatial frequency domain, including: S1: Acquire images, extract image information, and fuse the image information to obtain feature maps at different scales; S2: Calculate the similarity features in the spatial domain and the similarity features in the frequency domain based on the feature maps at different scales; S3: Fuse similarity features in the spatial domain and similarity features in the frequency domain to obtain a feature map of spatial-frequency multi-scale feature similarity measurement; S4: Determine whether the image is forged based on the feature map of the spatial-frequency domain multi-scale feature similarity metric.

[0022] The core of this invention lies in capturing subtle manipulation traces in video data at different feature scales in both the spatial and frequency domains through a multi-scale feature similarity metric in the spatial and frequency domains. This invention calculates the similarity between each image patch. The core of the similarity metric is to represent two objects, such as images, features, or vectors, as numerical sequences and then calculate their similarity. A spatial-frequency domain cross-attention mechanism is designed to fuse the extracted multi-scale feature similarities in the spatial-frequency domain. The deepfake detection method using feature similarity metric can identify subtle forgery traces and significantly reduce computational complexity. Simultaneously, the spatial-frequency domain analysis architecture of this invention provides a multi-scale, multi-feature perspective for image processing, enhancing the capabilities of feature extraction, analysis, and image detection.

[0023] By directly treating the image as a sequential input and utilizing similarity metrics to handle the internal relationships of image features at different scales, the image is divided into a series of image patches, and each patch is converted into a vector representation as the input sequence. These vectors are then processed using Spatial Intra-Feature Dissimilarity Metric (SIDM) and Frequency Intra-Feature Dissimilarity Metric (FIDM) to capture the differences between realistic forgeries and real content across multiple scales and domains. Finally, similarity metrics are used to calculate these features, ultimately outputting a similarity feature map containing rich frequency and spatial information. Compared to existing techniques, this method achieves higher efficiency with lower model parameters for obtaining intra-spatial and intra-frequency similarity of images.

[0024] In some embodiments, the step of extracting image information and fusing the image information to obtain feature maps of different scales includes: S101: Extract semantic and spatial information of the acquired image using the EfficientNet-B4 network; S102: Fusion of image semantic information and image spatial information to obtain feature maps at different scales.

[0025] In some embodiments, calculating similarity features within the spatial domain features based on feature maps of different scales specifically includes: After slicing the feature maps at different scales, a channel attention model is used to process them to obtain the query value in the spatial domain. Feature maps of different scales are sequentially processed by depthwise separable convolution, BN layer processing, and slicing to obtain spatial key values; Calculate the similarity features within the spatial domain features based on the query value and key value of the spatial domain.

[0026] In some embodiments, calculating similarity features within the frequency domain features based on feature maps of different scales specifically includes: The feature maps at different scales are processed by average pooling to obtain a global average image patch; the global average image patch is then subjected to discrete Fourier transform, flattening and projection to obtain a global frequency label. The feature maps at different scales are processed by average pooling to obtain a global average image block; the global average image block is then subjected to discrete Fourier transform and flattening to obtain local frequency markers, and frequency embedding is introduced into the local frequency markers. The global frequency marker, local frequency marker, and frequency embedding are processed by a learnable filter to obtain the query value and key value in the frequency domain. The frequency domain key values ​​are processed by the channel attention module to obtain key features; The similarity features within the frequency domain are obtained by projection operation on the key features and the query values ​​in the frequency domain.

[0027] In some embodiments, fusing similarity features in the spatial domain and similarity features in the frequency domain to obtain a similarity measurement result map specifically includes: S301: After segmenting similarity features within the spatial domain, a spatial domain value vector is obtained; after segmenting similarity features within the frequency domain, a frequency domain value vector is obtained. S302: The spatial domain value vector and the frequency domain value vector are processed through a linear embedding layer to obtain the spatial domain query value, the spatial domain key value, the frequency domain query value, and the frequency domain key value; S303: The query value in the spatial domain, the key value in the spatial domain, the query value in the frequency domain, and the key value in the frequency domain are fused to obtain a similarity measurement result map.

[0028] In some embodiments, S303 is specifically implemented using the following formula: ( ), =unpatchfy( ), =unpatchfy( ), = ( ) In the formula, unpatchfy is the operation that reassembles the flattened sequence into an image based on its relative spatial position. It consists of ReLU activation layers, 1×1 convolutional layers, and BN layers; This represents the space-frequency domain joint self-attention weight matrix; This represents the Softmax normalization function, which maps the input to a probability distribution; A query vector representing frequency domain features; A query vector representing spatial domain features; Key vectors representing frequency domain features; Key vectors representing spatial domain features; superscript Indicates transpose; Indicates feature dimension; This indicates a vector concatenation operation; This represents the reconstructed features in the frequency domain after self-attention weighting; Represents a spatial domain value vector; This represents the reconstructed features in the frequency domain after self-attention weighting; Represents a frequency domain value vector; The feature map representing the spatial-frequency domain multi-scale feature similarity measure is input into a binary classifier to distinguish between real and fake faces.

[0029] In some embodiments, determining whether an image is forged based on the feature map of a spatial-frequency domain multi-scale feature similarity metric includes the following steps: Mapping the feature map of spatial-frequency domain multi-scale feature similarity measure to the label space in the classification task yields the unnormalized log odds. The unnormalized log odds are applied to an activation function for multi-class classification to obtain predicted labels; the activation function is shown in the following formula:

[0030] The image is determined to be forged based on the predicted label; wherein, if It is determined to be counterfeit; if It was determined to be true.

[0031] See Figure 2 This application also discloses an efficient deepfake detection system based on multi-scale feature similarity measurement in the spatial frequency domain, comprising: The image preprocessing unit is used to acquire images, extract image information, and fuse image information to obtain feature maps at different scales; The feature extraction unit is used to calculate similarity features in the spatial domain and similarity features in the frequency domain based on feature maps of different scales. The feature fusion unit is used to fuse similarity features in the spatial domain and similarity features in the frequency domain to obtain a feature map of spatial-frequency multi-scale feature similarity measurement. The judgment unit is used to determine whether the image is forged based on the feature map of the spatial-frequency domain multi-scale feature similarity metric.

[0032] In some embodiments, the feature extraction unit includes: a spatial similarity measurement module and a frequency similarity measurement module; Spatial similarity measurement module: The spatial similarity measurement module divides the input features into tokens of different sizes, and then calculates the similarity between different local regions by calculating the spatial similarity of different local regions, thereby improving the model's understanding of the structural and texture features in the image; Frequency Domain Similarity Measurement Module: The frequency domain similarity measurement module transforms images segmented into blocks of the same size into the frequency domain for processing. It extracts features and calculates similarity in the frequency domain, which can capture details that are difficult to perceive directly in the spatial domain, significantly enhancing the model's expressiveness and robustness.

[0033] This application proposes a spatial-frequency domain multi-scale, intra-feature similarity measurement architecture, which can fully mine the spatial and frequency domain information of images to improve feature extraction and analysis capabilities. The input image is segmented into multiple image patches of different sizes, and the similarity of these patches is calculated in both the spatial and frequency domains. The spatial domain intra-label similarity measurement includes two paths for calculating the query value Q and the key value K, respectively, with one path containing the input features. p After slicing, the input channel attention model generates the query value Q, and the input features are on another path. p After being fed into a depthwise separable convolutional layer, and then through a batch normalization (BN) layer, the data is sliced ​​to obtain the key value K. The spatial similarity can then be calculated based on Q and K. The frequency domain similarity measurement model first processes the input features... pAverage pooling is used to obtain a global average image patch, which is then subjected to Discrete Fourier Transform (DCT). The spectrum is flattened and projected through a linear layer to obtain a global frequency label. Additionally, the input features are sliced, then subjected to DCT, and flattened into local frequency labels, with frequency embeddings introduced to supplement frequency location information. The global and local frequency labels, along with the frequency embeddings, are fed into a learnable filter to obtain a query value Q and a key value K in the frequency domain. The key value K, after passing through a channel attention module, obtains a key feature related to the internal difference measure of the feature. This feature, along with the query value Q, is projected to obtain the intra-feature similarity in the frequency domain. By combining spatial and frequency information and using similarity measurement mechanisms at different scales and domains, the accuracy of deepfake detection can be significantly improved. Furthermore, this invention focuses on the internal differences of features rather than the values ​​of the features themselves, thus greatly reducing the number of network parameters and computational complexity. Through a series of extensive experiments, this application has fully verified the effectiveness and efficiency of the proposed method. The experimental results show that the performance of this invention in deepfake detection significantly surpasses the existing state-of-the-art detection technology, providing a more efficient and accurate method for deepfake detection, and has broad application prospects and social value.

[0034] In some embodiments, the present invention provides a lightweight detection system for detecting subtle forgery traces hidden in forged images. The image preprocessing unit is a backbone network module; the feature extraction unit includes a spatial similarity measurement module and a frequency similarity measurement module; the feature fusion unit is a spatial-frequency cross-attention module; and the judgment unit is a classifier module. Specific module descriptions are as follows: Backbone Network Module: The backbone network module is designed to extract feature maps of different scales for spatial and frequency domain feature extraction, and is ultimately used for classification tasks. It can effectively process image data, extract features, and thus improve the performance of the model. Spatial similarity measurement module: The spatial similarity measurement module divides the input features into tokens of different sizes, and then calculates the similarity between different local regions by calculating the spatial similarity of different local regions, thereby improving the model's understanding of the structural and texture features in the image; Intra-frequency similarity measurement module: The intra-frequency similarity measurement module transforms images segmented into blocks of the same size into the frequency domain for processing, extracts features in the frequency domain and calculates similarity, which can capture details that are difficult to perceive directly in the spatial domain, significantly enhancing the model's expressiveness and robustness. Spatial-Frequency Cross-Attention Module: The spatial-frequency cross-attention module integrates similarity features in the spatial and frequency domains, capturing both local details in the spatial domain and global features in the frequency domain. This effectively handles forgery and subtle changes in images, significantly enhancing the model's comprehensive capabilities in different scenarios and improving the model's overall performance and robustness.

[0035] Classifier module: The classifier module processes the input spatial-frequency domain fused similarity measure feature map and finally outputs the classification result (main output) and the fused spatial-frequency domain multi-scale feature similarity measure feature map. The system uses visualization and similarity metric weights (interpretability analysis) to classify the processed image or video features and determine whether the video is fake.

[0036]

Example

[0037] See Figure 3 (a) The backbone network adopts the EfficientNet-B4 network, extracting the outputs of its B2, B3, B5, and B6 modules as feature maps of corresponding scales. Each feature map is fed into a Feature Pyramid Network (FPN), which fuses the semantic information of high-level feature maps with the spatial information of low-level feature maps to generate feature representations with rich multi-scale information without significantly increasing computational cost. Through FPN, this application obtains four levels of features. p iLet i = 1, 2, 3, 4; their sizes are c1×(H / d1)×(W / d1), c2×(H / d2)×(W / d2), c3×(H / d3)×(W / d3), and c4×(H / d4)×(W / d4), respectively. Then, these multi-level features p are applied in different block sizes. S i The data is sent to the corresponding spatial similarity measurement module and frequency similarity measurement module. Finally, all spatial similarities x from different scales are calculated. s and frequency similarity x f The spatial frequency domain cross-attention module will be integrated into a unified representation.

[0038] Specifically, steps 2 and 3 calculate the spatial similarity metric weights and frequency similarity metric feature values ​​at different scales, respectively, then step 4 fuses them, and finally step 5 determines whether the input video is fake.

[0039] Step 2: Design of the spatial similarity measurement module.

[0040] See Figure 3 (b) The Spatial Similarity Measurement Module (SISM) contains two branches, one for generating query values ​​and the other for generating query values. Q i and key value K i One branch is to input features p i Cut into sizes s i The feature blocks are then put into the Channel Attention Module (CAM) to generate... Q Another branch is to p i The data is fed into a depthwise separable convolutional layer, then into a BatchNormal layer, and then divided into blocks as follows. K : =CAM(patchfy( )),(1) =patchfy(SepareableConv2D( (2) Where, patchfy represents the input features p i Cut into sizes s iThe feature blocks are sliced ​​and then flattened into tokens. The CAM module uses AvgPool and MaxPool to compress the spatial information of the tokens to highlight those tokens that are more important for measuring internal differences.

[0041] Ultimately, this application obtained the following features. p i similarity within spatial features And it is combined with the last similarity through a 1×1 convolutional layer. Fusion: Conv1( Softmax ))+ ), in, The key dimension is used for scaling to avoid excessively large inner product values. The Softmax operation ensures that all similarity metric weights are between [0, 1] and sum to 1. The projection consists of a ReLU activation layer, a 1×1 convolutional layer, and a BN layer, where f represents the downsampling scale factor. It is a dot product. It should be noted that... Set to empty. Since the dot product of two vectors can represent their similarity, the larger the value of the dot product, the more similar they are. By calculating inter-token similarity at multiple levels, SISM provides a measure of internal differences within the spatial domain.

[0042] Step 3: Design of frequency domain similarity measurement module.

[0043] Besides spatial modes, frequency modes can also provide supplementary information for revealing forgery clues. See also Figure 3 (c) The frequency domain similarity measurement module (FISM) first performs... p i Perform AvgPool operations to obtain a size of s i The global average plot is then transposed to the frequency domain using the Discrete Cosine Transform (DCT) operator D, and flattened and projected through a linear layer to obtain the global frequency label. f g As shown below: (D(AvgPool( p i , s i ))). In addition, FISM can also p i spliced ​​together to a size of s iThe image patch is transposed to the frequency domain and flattened into local frequency markers. As shown below: D(patchfy( ). To preserve frequency information, this application introduces frequency embedding (FE) as the position embedding of frequency embedding: =sin(j / ), =sin(j / ), Where j The position of the frequency block is indicated by k, where k represents the k-th dimension out of dm feature dimensions. Then, this application adds the frequency embedding to the local frequency marker. In the middle, the following is generated and : =CAM( ( + + ) )), = ( + ) ), in, It is a learnable filter with a configurable frequency band b, which is set to 2 empirically in this application. Finally, this application obtains p i Frequency domain similarity And it is combined with the last similarity through a 1×1 convolutional layer. Fusion: =Conv1( Softmax ))+ ))). Step 4: Design of the spatial frequency domain cross-attention module.

[0044] Inspired by cross-attention structures, this application designs a spatial frequency domain cross-attention module (SFCA), see [link to relevant documentation]. Figure 3 (d) This is a fusion module used to cross-fuse multi-scale feature similarity in the spatial and frequency domains. By extracting feature similarities at different levels and fusing them layer by layer through four-layer SISM and FISM modules, this application finally obtains spatial and frequency similarity ( and Enter the following into SFCA: =patchfy( , ), This application first performs block-based processing on the input feature map (block size is...). Then, spatial domain value vectors are generated through a flattening operation. and frequency domain value vector . Represents a value vector in the spatial domain; This represents a value vector in the frequency domain; then, this application obtains the query value and key value through a linear embedding layer, as shown below: =SQK-Emb(patchfy( )), =FQK-Emb(patchfy( )), The query values ​​and key values ​​from the spatial and frequency domains are concatenated and fused using the following self-attention mechanism: ( ), =unpatchfy( ), =unpatchfy( ), = ( ), Among them, unpatchfy is the operation that reassembles a flattened sequence into an image based on its relative spatial position. It consists of ReLU activation layers, 1×1 convolutional layers, and BN layers.

[0045] Through this fusion, the model can simultaneously capture both local details and global structure of an image, enhancing its ability to perceive complex image features. In particular, in tasks such as deepfake image detection, it can more effectively capture subtle manipulation traces in the image.

[0046] Now, this application has obtained a feature map representing a spatial-frequency domain multi-scale feature similarity measure of the input image. Finally, Input a binary classifier to distinguish between real and fake faces.

[0047] Step 5: Classifier Design First, the classifier receives the feature map of the fused spatial-frequency domain multi-scale feature similarity measure passed from step 4. In this invention, it is used for classification decision in the last layer. The role of the classifier is to map high-dimensional features to the label space in the classification task.

[0048]

[0049] Where A is the weight matrix, b is the bias term, and x is the input feature.

[0050] An activation function is applied after the fully connected layer to the output layer of a multi-class classification problem, enabling the model to learn complex patterns.

[0051] like It is determined to be counterfeit; if The result is determined to be true. During training, a loss function is used to measure the difference between the prediction and the true label:

[0052] Where y is the one-hot encoding of the real label. This represents the probability predicted by the model. The gradient, calculated using the loss function, is used to update the network weights via backpropagation if a real sample is misclassified as a fake.

[0053] The weights are updated based on the gradient using an optimization algorithm to minimize the loss function, where It is the learning rate.

[0054]

[0055] Example of the effect: To evaluate the effectiveness of the video depth forgery detection method proposed in this invention, which uses a multi-scale feature similarity metric in the spatial-frequency domain, this application compares it with existing state-of-the-art technologies such as MAT (CVPR'21), M2TR (ICMR'22), DFTM (TAI'24), and EtD (AAAI'24). Experiments were conducted on the public datasets FFDF and CelebDF-v2, and the performance of the algorithm was evaluated using two metrics: accuracy score (Acc) and area under the receiver operating characteristic curve (AUC). Furthermore, the model size and FLOPs of different algorithms were compared, and the results are shown in Table 1. With the same number of iterations, it can be seen that the model proposed in this application can achieve high-quality video depth forgery detection with lower model parameters, and it also has lower computational complexity and faster testing speed.

[0056] Table 1 Performance Comparison of Different Algorithms

[0057] like Figure 4 As shown, the present invention also provides an electronic device 100 for implementing the efficient deep forgery detection method based on multi-scale feature similarity measurement in the spatial frequency domain as described in any one of the above claims; The electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on at least one processor 102, and at least one communication bus 104.

[0058] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of an efficient deep forgery detection method based on spatial frequency domain multi-scale feature similarity measurement by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101.

[0059] The memory 101 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0060] At least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 102 may be a microprocessor or any conventional processor. Processor 102 is the control center of electronic device 100, connecting various parts of electronic device 100 via various interfaces and lines.

[0061] The memory 101 in the electronic device 100 stores multiple instructions to implement an efficient deep forgery detection method based on spatial frequency domain multi-scale feature similarity measurement, and the processor 102 can execute multiple instructions to achieve the following: Acquire images, extract image information, and fuse the image information to obtain feature maps at different scales; Calculate the similarity features within the spatial domain and the similarity features within the frequency domain based on feature maps of different scales; By fusing similarity features in the spatial domain and similarity features in the frequency domain, a feature map of spatial-frequency multi-scale feature similarity measurement is obtained. The image is determined to be forged based on the feature map of the spatial-frequency domain multi-scale feature similarity metric.

[0062] Example If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0063] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0065] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0066] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0067] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An efficient deepfake detection method based on spatial frequency domain multi-scale feature similarity measurement, characterized in that, include: Acquire images, extract image information, and fuse the image information to obtain feature maps at different scales; Calculate the similarity features within the spatial domain and the similarity features within the frequency domain based on feature maps of different scales; By fusing similarity features within the spatial domain and similarity features within the frequency domain, a feature map of spatial-frequency multi-scale feature similarity measurement is obtained; specifically including: S301: After segmenting similarity features within the spatial domain, a spatial domain value vector is obtained; after segmenting similarity features within the frequency domain, a frequency domain value vector is obtained. S302: The spatial domain value vector and the frequency domain value vector are processed through a linear embedding layer to obtain the spatial domain query value, the spatial domain key value, the frequency domain query value, and the frequency domain key value; S303: The spatial domain query value, spatial domain key value, frequency domain query value, and frequency domain key value are fused to obtain the similarity measurement result spectrum, specifically implemented using the following formula: ( ), =unpatchfy( ), =unpatchfy( ), = ( ) In the formula, unpatchfy is the operation that reassembles the flattened sequence into an image based on its relative spatial position. It consists of ReLU activation layers, 1×1 convolutional layers, and BN layers; This represents the space-frequency domain joint self-attention weight matrix; This represents the Softmax normalization function, which maps the input to a probability distribution; A query vector representing frequency domain features; A query vector representing spatial domain features; Key vectors representing frequency domain features; Key vectors representing spatial domain features; superscript Indicates transpose; Indicates feature dimension; This indicates a vector concatenation operation; This represents the reconstructed features in the frequency domain after self-attention weighting; Represents a spatial domain value vector; This represents the reconstructed features in the frequency domain after self-attention weighting; Represents a frequency domain value vector; The feature map representing the spatial-frequency domain multi-scale feature similarity measure is input into a binary classifier to distinguish between real and fake faces. The image is determined to be forged based on the feature map of the spatial-frequency domain multi-scale feature similarity metric.

2. The efficient deepfake detection method based on multi-scale feature similarity measurement in the spatial frequency domain according to claim 1, characterized in that, The process of extracting and fusing image information to obtain feature maps at different scales includes: S101: Extract semantic and spatial information of the acquired image using the EfficientNet-B4 network; S102: Fusion of image semantic information and image spatial information to obtain feature maps at different scales.

3. The efficient deepfake detection method based on multi-scale feature similarity measurement in the spatial frequency domain according to claim 1, characterized in that, The calculation of similarity features within the spatial domain based on feature maps of different scales specifically includes: After slicing the feature maps at different scales, a channel attention model is used to process them to obtain the query value in the spatial domain. Feature maps of different scales are sequentially processed by depthwise separable convolution, BN layer processing, and slicing to obtain spatial key values; Calculate the similarity features within the spatial domain features based on the query value and key value of the spatial domain.

4. The efficient deepfake detection method based on multi-scale feature similarity measurement in the spatial frequency domain according to claim 1, characterized in that, The calculation of similarity features within the frequency domain based on feature maps of different scales specifically includes: The feature maps at different scales are processed by average pooling to obtain a global average image patch; the global average image patch is then subjected to discrete Fourier transform, flattening and projection to obtain a global frequency label. The feature maps at different scales are processed by average pooling to obtain a global average image block; the global average image block is then subjected to discrete Fourier transform and flattening to obtain local frequency markers, and frequency embedding is introduced into the local frequency markers. The global frequency marker, local frequency marker, and frequency embedding are processed by a learnable filter to obtain the query value and key value in the frequency domain. The frequency domain key values ​​are processed by the channel attention module to obtain key features; The similarity features within the frequency domain are obtained by projection operation on the key features and the query values ​​in the frequency domain.

5. The efficient deepfake detection method based on multi-scale feature similarity measurement in the spatial frequency domain according to claim 1, characterized in that, The step of determining whether an image is forged based on the feature map of spatial-frequency domain multi-scale feature similarity measurement includes the following steps: Mapping the feature map of spatial-frequency domain multi-scale feature similarity measure to the label space in the classification task yields the unnormalized log odds. The unnormalized log odds are applied to an activation function for multi-class classification to obtain predicted labels; the activation function is shown in the following formula: The image is determined to be forged based on the predicted label; wherein, if It is determined to be counterfeit; if It was determined to be true.

6. A highly efficient deepfake detection system based on multi-scale feature similarity measurement in the spatial frequency domain, characterized in that, For implementing the efficient deepfake detection method based on multi-scale feature similarity measurement in the spatial frequency domain as described in any one of claims 1 to 5, the system comprises: The image preprocessing unit is used to acquire images, extract image information, and fuse image information to obtain feature maps at different scales; The feature extraction unit is used to calculate similarity features in the spatial domain and similarity features in the frequency domain based on feature maps of different scales. The feature fusion unit is used to fuse similarity features in the spatial domain and similarity features in the frequency domain to obtain a feature map of spatial-frequency multi-scale feature similarity measurement. The judgment unit is used to determine whether the image is forged based on the feature map of the spatial-frequency domain multi-scale feature similarity metric.

7. An electronic device, characterized in that, It includes a processor and a memory, the processor being used to execute a computer program stored in the memory to implement the efficient deep forgery detection method based on spatial frequency domain multi-scale feature similarity measurement as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the efficient deep forgery detection method based on spatial frequency domain multi-scale feature similarity measurement as described in any one of claims 1 to 5.

Images