Efficient depth forgery detection method based on space-frequency domain multi-scale feature similarity measurement

Abstract

The invention belongs to the technical field of depth forgery detection, and particularly discloses an efficient depth forgery detection method based on space-frequency domain multi-scale feature similarity measurement, and the method comprises the steps: obtaining an image, extracting image information, and carrying out the fusion of the image information, and obtaining feature maps of different scales; respectively calculating similarity features in the spatial domain features and similarity features in the frequency domain features according to the feature maps of different scales; fusing the similarity features in the space domain features and the frequency domain features to obtain a feature map of space-frequency domain multi-scale feature similarity measurement; and judging whether the image is forged or not. Through a similarity measurement strategy, the accuracy of similarity calculation is remarkably improved, the overall calculation burden is reduced, and a space-frequency domain multi-scale feature fusion technology is adopted, so that counterfeit signs can be detected in a space domain and a frequency domain, the content of feature extraction is enriched, better performance is obtained, and a better reconstruction result is provided; and meanwhile, the detection efficiency is kept.

Description

Technical Field

[0001] The present invention belongs to the technical field of deepfake detection, and specifically discloses an efficient deepfake detection method based on spatio-frequency domain multi-scale feature similarity measurement. Background Art

[0002] In recent years, the deepfake technology has developed rapidly, enabling attackers to manipulate the facial regions in images and generate forged images. These technologies can manipulate the facial expressions and identities of people, and it is extremely difficult for the human eye to distinguish whether the image has been manipulated. At the same time, the deepfake technology is used in a variety of applications, including entertainment, film production, and virtual reality. Most existing methods adopt deep convolutional neural network (CNN) models, input the cropped facial regions, and generate binary real / fake predictions. These methods capture the artifacts in the facial regions at a single scale by stacking convolutional operations. Although stacking convolutions has achieved good detection results, they are good at modeling local information and cannot consider the global relationship of pixels due to the limited receptive field.

[0003] With the increasing popularity of the deepfake technology, there is an urgent need for an effective detection mechanism to identify these forged images that are almost indistinguishable visually between genuine and fake. Summary of the Invention

[0004] The purpose of the present invention is to provide an efficient deepfake detection method based on spatio-frequency domain multi-scale feature similarity measurement, so as to solve the technical problems in the prior art that the global relationship of pixels cannot be considered, resulting in low recognition accuracy and efficiency.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions: In the first aspect of the present invention, an efficient deepfake detection method based on spatio-frequency domain multi-scale feature similarity measurement is provided, including: Obtain an image, extract image information and fuse the image information to obtain feature maps of different scales; Calculate the similarity features in the spatial domain features and the similarity features in the frequency domain features respectively according to the feature maps of different scales; Fuse the similarity features in the spatial domain features and the similarity features in the frequency domain features to obtain a feature map of spatio-frequency domain multi-scale feature similarity measurement; Judge whether the image is forged according to the feature map of spatio-frequency domain multi-scale feature similarity measurement.

[0006] Preferably, the extracting image information and fusing the image information to obtain feature maps of different scales includes: S101: For the obtained image, extract it by using the efficientnet-b4 network to obtain the image semantic information and the image spatial information; S102: Fuse the image semantic information and the image spatial information to obtain feature maps of different scales.

[0007] Preferably, calculating the similarity features in the spatial domain features according to the feature maps of different scales specifically includes: After slicing the feature maps of different scales, use the channel attention model to process to obtain the query values in the spatial domain; The feature maps of different scales are successively processed by depthwise separable convolution, BN layer processing and slicing division to obtain the key values in the spatial domain; Calculate the similarity features in the spatial domain features according to the query values in the spatial domain and the key values in the spatial domain.

[0008] Preferably, calculating the similarity features in the frequency domain features according to the feature maps of different scales specifically includes: Use average pooling to process the feature maps of different scales to obtain a global average image block; successively perform discrete Fourier transform, flattening and projection on the global average image block to obtain global frequency marks; Use average pooling to process the feature maps of different scales to obtain a global average image block; successively perform discrete Fourier transform and flattening on the global average image block to obtain local frequency marks, and introduce frequency embeddings in the local frequency marks; Process the global frequency marks, local frequency marks and frequency embeddings through a learnable filter to obtain the query values in the frequency domain and the key values in the frequency domain; The key values in the frequency domain are processed by a channel attention module to obtain key features; The key features and the query values in the frequency domain are subjected to a projection operation to obtain the similarity features in the frequency domain features.

[0009] Preferably, fusing the similarity features in the spatial domain features and the similarity features in the frequency domain features to obtain a similarity measurement result map specifically includes: S301: After slicing the similarity features in the spatial domain features, obtain a spatial domain value vector; after slicing the similarity features in the frequency domain features, obtain a frequency domain value vector; S302: Process the spatial domain value vector and the frequency domain value vector through a linear embedding layer to obtain the query values in the spatial domain, the key values in the spatial domain, the query values in the frequency domain and the key values in the frequency domain; S303: Fuse the query values in the spatial domain, the key values in the spatial domain, the query values in the frequency domain and the key values in the frequency domain to obtain a similarity measurement result map.

[0010] Preferably, S303 is specifically implemented by the following formula: ( ), =unpatchfy( ), =unpatchfy( ), = ( ) where unpatchfy is an operation that reorganizes a flattened sequence into an image according to the relative spatial positions, which consists of a ReLU activation layer, a 1×1 convolutional layer, and a BN layer; represents the spatio-frequency domain joint self-attention weight matrix; represents the Softmax normalization function that maps the input to a probability distribution; represents the query vector of the frequency-domain features; represents the query vector of the spatial-domain features; represents the key vector of the frequency-domain features; represents the key vector of the spatial-domain features; the superscript represents the transpose; represents the feature dimension; represents the concatenation operation of vectors; represents the reconstructed features of the frequency domain after self-attention weighting; represents the spatial-domain value vector; represents the reconstructed features of the frequency domain after self-attention weighting; represents the frequency-domain value vector; represents the feature map for spatio-frequency domain multi-scale feature similarity measurement, which is input to a binary classifier for genuine and fake face discrimination.

[0011] Preferably, judging whether the image is forged according to the feature map of spatio-frequency domain multi-scale feature similarity measurement includes the following steps: Mapping the feature map of spatio-frequency domain multi-scale feature similarity measurement to the label space in the classification task to obtain the unnormalized log-odds; Using an activation function for the unnormalized log-odds for multi-class classification to obtain the predicted label; the activation function is shown in the following formula:

[0012] Judging whether the image is forged according to the predicted label; wherein, if , it is determined to be forged; if , it is determined to be genuine.

[0013] In a second aspect of the present invention, an efficient deepfake detection system based on spatio-frequency domain multi-scale feature similarity measurement is provided, including: An image preprocessing unit for acquiring an image, extracting image information, and fusing the image information to obtain feature maps of different scales; A feature extraction unit for calculating the similarity features in the spatial domain feature and the similarity features in the frequency domain feature respectively according to the feature maps of different scales; A feature fusion unit for fusing the similarity features in the spatial domain feature and the similarity features in the frequency domain feature to obtain a feature map of the spatial-frequency multi-scale feature similarity metric; A judgment unit for judging whether the image is forged according to the feature map of the spatial-frequency multi-scale feature similarity metric.

[0014] In a third aspect of the present invention, an electronic device is provided, including a processor and a memory, where the processor is configured to execute a computer program stored in the memory to implement the efficient deep fake detection method based on the spatial-frequency domain multi-scale feature similarity metric described in any one of the above.

[0015] In a fourth aspect of the present invention, a computer-readable storage medium is provided, where the computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the efficient deep fake detection method based on the spatial-frequency domain multi-scale feature similarity metric described in any one of the above is implemented.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention extracts image information and fuses the image information to obtain feature maps of different scales for feature extraction and processing in the spatial-frequency domain. This design can fully capture the multi-level information in the image, extract local detail features and overall structural features, thereby significantly improving the performance and robustness of the model. Through spatial and frequency domain feature extraction, it is finally used for classification tasks, which can effectively process image data, extract features, and thus improve the performance of the model; Calculating the similarity features in the spatial domain feature according to the feature maps of different scales, dividing the image blocks into tokens of different scales, can effectively capture the spatial similarity between different regions, calculate the similarity between each image region, help the model better understand the structural and texture features in the image, improve the model's perception ability of the image content, and make it perform more accurately when processing complex images; Calculating the similarity features in the frequency domain according to the feature maps of different scales, feature extraction and similarity calculation can be performed in the frequency domain, which can capture details that cannot be directly perceived by the spatial domain features, can more comprehensively understand the content of the image, enable it to handle more complex and diverse image scenarios, and at the same time improve the robustness of the model in different tasks; Fusing the similarity features in the spatial domain and the similarity features in the frequency domain to obtain a feature map for spatial-frequency multi-scale feature similarity measurement. By fusing the feature maps from different scales and branches, a comprehensive capture of complex patterns and details is achieved. Through feature mapping, fusion, and weighting, these modules enhance the model's processing ability for various visual tasks, improve its performance and robustness, and enhance the model's comprehensive ability.

[0017] In summary, by designing a similarity measurement strategy, the present invention significantly improves the accuracy of similarity calculation while reducing the overall computational burden, and proposes an advanced spatial-frequency multi-scale feature fusion technology, which enables the model of the present application to detect forgery signs in the spatial and frequency domains, thereby enriching the content of feature extraction, obtaining better performance, providing better reconstruction results, and maintaining the detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings forming a part of this application are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 is a flowchart of the method according to an embodiment of the present invention; Figure 2 is a system block diagram according to an embodiment of the present invention; Figure 3 is a system processing flowchart according to an embodiment of the present invention; wherein, (a) is the backbone network; (b) is the similarity measurement module in the spatial domain; (c) is the similarity measurement module in the frequency domain; (d) is the spatial-frequency cross-attention module; Figure 4 is a structural block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] The present invention will be described in detail below with reference to the drawings and in conjunction with the embodiments. It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other.

[0020] The following detailed descriptions are all exemplary descriptions, aiming to provide further detailed descriptions of the present invention. Unless otherwise specified, all technical terms used in the present invention have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention.

[0021] See Figure 1 , this application discloses an efficient deepfake detection method based on spatial-frequency multi-scale feature similarity measurement, including: S1: Obtain an image, extract image information and fuse the image information to obtain feature maps of different scales; S2: Calculate the similarity features in the spatial domain features and the similarity features in the frequency domain features respectively according to the feature maps of different scales; S3: Fuse the similarity features in the spatial domain features and the similarity features in the frequency domain features to obtain a feature map of spatial-frequency multi-scale feature similarity measurement; S4: Determine whether the image is forged according to the feature map of spatial-frequency multi-scale feature similarity measurement.

[0022] The core of the present invention is to capture the subtle manipulation traces in different feature scales of video data in the spatial and frequency domains through the similarity measurement within the spatial-frequency multi-scale features. The present invention calculates the similarity between each image block. The core of the similarity measurement is to represent two objects, such as images, features, vectors, etc. as numerical sequences, and then calculate the similarity between them. An empty-frequency domain cross-attention mechanism is designed to fuse the similarity within the extracted spatial-frequency multi-scale features. The deep forgery detection method using the similarity measurement within the features can identify subtle forgery traces and greatly reduce the computational complexity. At the same time, the spatial-frequency domain analysis architecture of the present invention provides a multi-scale and multi-feature perspective for image processing, which can improve the capabilities of feature extraction, analysis, image detection, etc.

[0023] By directly regarding the image as a serialized input and using the similarity measurement method to process the internal relationship of image features at different scales, the image is divided into a series of image patches, and each patch is converted into a vector representation as the input sequence. Then, these vectors are processed by the spatial domain feature internal difference measurement (SIDM) and the frequency domain feature internal difference measurement (FIDM) to capture the difference between the realistic forgery and the real content existing in the image under multi-scale and multi-domain. Finally, the similarity measurement method is used to calculate these features, and finally a similarity feature map containing rich frequency and spatial information is output. Compared with the prior art, the similarity within the spatial-frequency domain features of the image can be obtained with lower model parameters and higher efficiency.

[0024] In some embodiments, the extracting image information and fusing the image information to obtain feature maps of different scales includes: S101: For the obtained image, extract it using the efficientnet-b4 network to obtain image semantic information and image spatial information; S102: Fuse the image semantic information and the image spatial information to obtain feature maps of different scales.

[0025] In some embodiments, the calculating the similarity features in the spatial domain features according to the feature maps of different scales specifically includes: After slicing the feature maps of different scales, they are processed by a channel attention model to obtain query values in the spatial domain; The feature maps of different scales are successively processed by depthwise separable convolution, BN layer processing, and slicing division to obtain key values in the spatial domain; Calculate the similarity features in the spatial domain features based on the query values in the spatial domain and the key values in the spatial domain.

[0026] In some embodiments, calculating the similarity features in the frequency domain features based on the feature maps of different scales specifically includes: Use average pooling to process the feature maps of different scales to obtain a global average image patch; successively perform discrete Fourier transform, flattening, and projection on the global average image patch to obtain global frequency markers; Use average pooling to process the feature maps of different scales to obtain a global average image patch; successively perform discrete Fourier transform and flattening on the global average image patch to obtain local frequency markers, and introduce frequency embeddings into the local frequency markers; Process the global frequency markers, local frequency markers, and frequency embeddings through a learnable filter to obtain query values in the frequency domain and key values in the frequency domain; The key values in the frequency domain are processed by a channel attention module to obtain key features; The key features and the query values in the frequency domain are subjected to a projection operation to obtain the similarity features in the frequency domain features.

[0027] In some embodiments, fusing the similarity features in the spatial domain features and the similarity features in the frequency domain features to obtain a similarity measurement result map specifically includes: S301: After slicing the similarity features in the spatial domain features, obtain a spatial domain value vector; after slicing the similarity features in the frequency domain features, obtain a frequency domain value vector; S302: Process the spatial domain value vector and the frequency domain value vector through a linear embedding layer to obtain query values in the spatial domain, key values in the spatial domain, query values in the frequency domain, and key values in the frequency domain; S303: Fuse the query values in the spatial domain, key values in the spatial domain, query values in the frequency domain, and key values in the frequency domain to obtain a similarity measurement result map.

[0028] In some embodiments, S303 is specifically implemented by the following formula: ( ), =unpatchfy( ), =unpatchfy( ), = ( ) where unpatchfy is an operation that reorganizes the flattened sequence into an image according to the relative spatial positions, which consists of a ReLU activation layer, a 1×1 convolutional layer, and a BN layer; represents the spatio-frequency domain joint self-attention weight matrix; represents the Softmax normalization function that maps the input to a probability distribution; represents the query vector of the frequency domain features; represents the query vector of the spatial domain features; represents the key vector of the frequency domain features; represents the key vector of the spatial domain features; the superscript represents the transpose; represents the feature dimension; represents the concatenation operation of vectors; represents the reconstructed features after self-attention weighting in the frequency domain; represents the spatial domain value vector; represents the reconstructed features after self-attention weighting in the frequency domain; represents the frequency domain value vector; represents the feature map of the spatio-frequency domain multi-scale feature similarity metric, which is input into a binary classifier to discriminate between real and fake faces.

[0029] In some embodiments, determining whether the image is forged based on the feature map of the spatio-frequency domain multi-scale feature similarity metric includes the following steps: Mapping the feature map of the spatio-frequency domain multi-scale feature similarity metric to the label space in the classification task to obtain the unnormalized log odds; Using an activation function on the unnormalized log odds to perform multi-class classification to obtain a predicted label; the activation function is shown in the following formula:

[0030] Judging whether the image is forged according to the predicted label; wherein, if , it is determined to be forged; if , it is determined to be real.

[0031] See Figure 2 , this application also discloses an efficient deepfake detection system based on spatio-frequency domain multi-scale feature similarity metric, including: An image preprocessing unit for obtaining an image, extracting image information and fusing the image information to obtain feature maps of different scales; A feature extraction unit, configured to calculate the similarity features within the spatial domain features and the similarity features within the frequency domain features respectively according to feature maps of different scales; A feature fusion unit, configured to fuse the similarity features within the spatial domain features and the similarity features within the frequency domain features to obtain a feature map of the spatial-frequency multi-scale feature similarity metric; A judgment unit, configured to judge whether the image is forged according to the feature map of the spatial-frequency multi-scale feature similarity metric.

[0032] In some embodiments, the feature extraction unit includes: a similarity metric module within the spatial domain and a similarity metric module within the frequency domain; The similarity metric module within the spatial domain: The similarity metric module within the spatial domain divides the input feature into tokens of different sizes, and then calculates the similarity between each image region by calculating the spatial similarity of different local regions, enhancing the model's understanding of the structural and texture features in the image; The similarity metric module within the frequency domain: The similarity metric module within the frequency domain processes the image divided into blocks of the same size by converting it to the frequency domain, extracts features and calculates the similarity in the frequency domain, which can capture details that are difficult to directly perceive in the spatial domain, significantly enhancing the expressiveness and robustness of the model.

[0033] This application particularly proposes a spatial-frequency multi-scale and feature-internal similarity metric architecture, which can fully exploit the spatial and frequency domain information of the image to enhance the ability of feature extraction and analysis. By dividing the input image into multiple image blocks of different sizes, the similarity of the image blocks is calculated in the spatial domain and the frequency domain respectively. The similarity metric within the spatial domain label includes two paths respectively used to calculate the query value Q and the key value K. On one path, the input feature p After being sliced and input into the channel attention model to generate the query value Q, on the other path, the input feature p After being fed into the depthwise separable convolution, passing through the BN layer, and then sliced and divided to obtain the key value K, the similarity within the spatial domain features can be obtained according to Q and K. The frequency domain similarity metric model first processes the input feature pGlobal average image patches are obtained using average pooling, and then discrete Fourier transform (DCT) is applied to them. The spectrum is flattened and projected through a linear layer to obtain global frequency tokens. Additionally, the input features are first sliced and then subjected to DCT transformation, and flattened into local frequency tokens. Frequency embeddings are introduced to supplement frequency position information. The global frequency tokens, local frequency tokens, and frequency embeddings are fed into a learnable filter to obtain query value Q and key value K in the frequency domain. Among them, after passing through the channel attention module, key value K obtains key features related to the internal difference metric of the features, and the similarity within the frequency domain features is obtained after the projection operation of this feature and query value Q. By combining spatial and frequency information, through the similarity measurement mechanism under different scales and domains, the accuracy of deepfake detection can be greatly improved. In addition, the present invention focuses on the internal differences of features rather than the values of the features themselves, so the number of network parameters is greatly reduced and the computational complexity is low. Through a series of extensive experiments, the effectiveness and efficiency of this method are fully verified. The experimental results show that the present invention significantly outperforms the existing state-of-the-art detection techniques in deepfake detection, providing a more efficient and accurate method for deepfake detection, with broad application prospects and social value.

[0034] In some embodiments, the present invention provides a lightweight detection system for detecting subtle forgery traces hidden in forged images. The image preprocessing unit is the backbone network module; the feature extraction unit includes an intra-spatial similarity measurement module and an intra-frequency similarity measurement module; the feature fusion unit is an empty-frequency domain cross-attention module, and the judgment unit is a classifier module. The specific module descriptions are as follows: Backbone network module: The backbone network module is designed to extract feature maps of different scales through the backbone network for spatial and frequency domain feature extraction, and finally for classification tasks, which can effectively process image data, extract features, and thus improve the performance of the model; Intra-spatial similarity measurement module: The intra-spatial similarity measurement module cuts the input features into tokens of different sizes, and then calculates the similarity between each image region by calculating the spatial similarity of different local regions, improving the model's understanding of the structural and texture features in the image; Intra-frequency similarity measurement module: The intra-frequency similarity measurement module processes the image divided into blocks of the same size by converting it to the frequency domain, extracts features and calculates the similarity in the frequency domain, which can capture details that are difficult to directly perceive in the spatial domain, significantly enhancing the expressiveness and robustness of the model; Empty-frequency domain cross-attention module: The empty-frequency domain cross-attention module fuses the similarity features of the spatial and frequency domains, simultaneously capturing local details in the spatial domain and global features in the frequency domain, which can effectively process forgeries and subtle changes in images, significantly enhancing the comprehensive ability of the model in different scenarios, and improving the overall performance and robustness of the model.

[0035] Classifier module: The classifier module processes the input similarity metric feature map after empty-frequency domain fusion and finally outputs the classification result (main output), the feature map of the fused spatial-frequency domain multi-scale feature similarity metric ( , visualization), and the similarity metric weight (interpretability analysis), classifies the processed image or video features, and determines whether the video is forged.

[0036]

Embodiment

[0037] Refer to Figure 3 (a), the backbone network adopts the efficientnet-b4 network, extracts the outputs of its b2, b3, b5, and b6 modules as feature maps of corresponding scales, sends each feature map into the Feature Pyramid Network (FPN), fuses the semantic information of the high-level feature map with the spatial information of the low-level feature map, thereby generating a feature representation with rich multi-scale information without significantly increasing the computational amount. Through the FPN, four levels of features are obtained in this application p p , Figure 3 , i ,

[0037] , , where \(i = 1, 2, 3, 4\); their sizes are \(c1\times(H / d1)\times(W / d1)\), \(c2\times(H / d2)\times(W / d2)\), \(c3\times(H / d3)\times(W / d3)\) and \(c4\times(H / d4)\times(W / d4)\) respectively. Then, these multi-level features \(p\) are sent to the corresponding spatial-domain similarity metric module and frequency-domain similarity metric module with different block sizes S i and are sent to the corresponding spatial-domain similarity metric module and frequency-domain similarity metric module. Finally, all spatial similarities \(x\) from different scales s and frequency similarities \(x\) f will be fused into a unified representation through the spatio-frequency domain cross-attention module.

[0038] Specifically, the spatial-domain similarity metric weight values and frequency-domain similarity metric feature values at different scales are calculated through Step 2 and Step 3 respectively, fused through Step 4, and finally, through Step 5, it is determined whether the input video is forged.

[0039] Step 2: Design of the spatial-domain similarity metric module.

[0040] Refer to Figure 3 Figure (b). The spatial-domain similarity metric module (SISM) contains two branches, which are used to generate query values Q i and key values K i . One branch is to slice the input feature p i into feature blocks of size s i , and then put them into the channel attention module (CAM) to generate Q ; the other branch is to send p i into the depthwise separable convolution, then into the BatchNormal layer, and then slice it as follows to generate K : = CAM(patchfy( ))(1) = patchfy(SepareableConv2D( ))(2) where patchfy represents slicing the input feature p i into feature blocks of size s iThe slicing operation of the feature blocks is then flattened into tokens. The CAM module uses AvgPool and MaxPool to compress the spatial information of the tokens to highlight those tokens that are more important for measuring internal differences.

[0041] Finally, the present application obtains the feature p i intra-similarity in the spatial domain feature , and fuses it with the last similarity through a 1×1 convolutional layer: : Conv1( (Softmax( )) + ), where is the dimension of the key, used for scaling to avoid too large inner product values. The Softmax operation ensures that all similarity measure weight values are between [0, 1] and sum to 1. The projection consists of a ReLU activation layer, a 1×1 convolutional layer, and a BN layer. f represents the downsampling scale factor, is the dot product. It should be noted that is set to be empty. Since the dot product of two vectors can represent their similarity, the larger the value of the dot product, the more similar they are. By calculating the similarity between tokens at multiple levels, SISM provides a measurement of internal differences in the spatial domain.

[0042] Step 3: Design of the intra-frequency similarity measurement module.

[0043] In addition to the spatial modality, the frequency modality can also provide supplementary information for revealing forgery clues. Referring to Figure 3 (c), the intra-frequency similarity measurement module (FISM) first performs an AvgPool operation on p i to obtain a global average map of size s i , then transposes it to the frequency domain through the discrete cosine transform (DCT) operator D, and flattens and projects it through a linear layer to obtain the global frequency token f g , as follows: (D(AvgPool( p i , s i ))). In addition, FISM can also splice p i into a size of s iThe image patches are transposed to the frequency domain and flattened into local frequency tokens , as follows: D(patchfy( ). To preserve the frequency information, this application introduces frequency embedding (FE) as the position embedding of the frequency embedding: =sin(j / ), =sin(j / ), where j represents the position of the frequency block, and k represents the k-th dimension among the dm feature dimensions. Then, this application adds the frequency embedding to the local frequency token to generate the following and : =CAM( ( + + ) )), = ( + ) ), where, is a learnable filter with a configurable frequency band b, and this application empirically sets b to 2. Finally, this application obtains p i the similarity between the frequency domains , and fuses it with the last similarity through a 1×1 convolutional layer: =Conv1( (Softmax( ))+ ))). Step 4: Design of the spatio-frequency cross-attention module.

[0044] Inspired by the cross-attention structure, this application designs a spatio-frequency cross-attention module (SFCA), as shown in Figure 3 (d), which is a fusion module for cross-fusing the intra-similarity of multi-scale features in the spatial and frequency domains. By extracting the similarity between features at different levels and fusing them layer by layer through four SISM and FISM modules, this application will finally obtain the spatial and frequency similarities ( and ) is input into the SFCA as follows: = patchfy( , ), This application first performs a block processing on the input feature map (the block size is ), and then generates a spatial domain value vector and a frequency domain value vector respectively through a flattening operation. represents the value vector in the spatial domain; represents the value vector in the frequency domain; Then, this application obtains query values and key values through a linear embedding layer as follows: = SQK-Emb(patchfy( )), = FQK-Emb(patchfy( )), The query values and key values from the spatial domain and the frequency domain are concatenated and fused by the following self-attention mechanism: ( ), = unpatchfy( ), = unpatchfy( ), = ( ), where unpatchfy is an operation to reconstruct the flattened sequence into an image according to the relative spatial position, is composed of a ReLU activation layer, a 1×1 convolutional layer, and a BN layer.

[0045] Through this fusion, the model can simultaneously capture the local details and global structure of the image, enhance the perception ability of complex image features, especially in tasks such as deepfake image detection, and can more effectively capture the subtle manipulation traces in the image.

[0046] Now, this application obtains the feature map representing the spatial-frequency multi-scale feature similarity measure of the input image. Finally, is input into a binary classifier to distinguish real and fake faces.

[0047] Step 5: Classifier Design First, the classifier receives the fused spatial-frequency domain multi-scale feature similarity metric feature map passed from step 4. In the present invention, it is used for classification decision-making in the last layer. The role of using the classifier is to map high-dimensional features to the label space in the classification task.

[0048]

[0049] Among them, A is the weight matrix, b is the bias term, and x is the input feature.

[0050] The activation function is applied after the fully connected layer and is used for the output layer in multi-class classification problems, enabling the model to learn complex patterns:

[0051] If , it is determined to be forged; if , it is determined to be genuine. During the training process, the loss function is used to measure the difference between the prediction and the true label:

[0052] Among them, y is the one-hot encoding of the true label, is the probability predicted by the model. Using the gradient calculated by the loss function, if the true sample is misjudged as forged, the network weights are updated through backpropagation:

[0053] The optimization algorithm is used to update the weights according to the gradient to minimize the loss function, where is the learning rate.

[0054]

[0055] Effect example: To evaluate the effectiveness of the video deepfake detection method based on the spatio-frequency domain multi-scale feature internal similarity metric proposed in the present invention, this application is compared with the existing state-of-the-art technologies MAT (CVPR’21), M2TR (ICMR’22), DFTM (TAI’24), and EtD (AAAI’24). Experiments are carried out on the public datasets FFDF and CelebDF-v2, and two metrics, the accuracy score (Acc) and the area under the receiver operating characteristic curve (AUC), are used to evaluate the performance of the algorithms. In addition, the model sizes and FLOPs of different algorithms are also compared, and the comparison results are shown in Table 1. By iterating the same number of times, it can be seen that the model of this application can complete high-quality video deepfake detection with fewer model parameters, and this application has a lower computational complexity and a faster test speed.

[0056] Table 1 Performance comparison of different algorithms

[0057] As shown Figure 4 in the figure, the present invention also provides an electronic device 100 for implementing the efficient deepfake detection method based on spatio-frequency domain multi-scale feature similarity measurement described in any one of the above; The electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0058] The memory 101 can be used to store the computer program 103. The processor 102 realizes the steps of the efficient deepfake detection method based on spatio-frequency domain multi-scale feature similarity measurement by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101.

[0059] The memory 101 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the electronic device 100 (such as audio data, etc.). In addition, the memory 101 can include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0060] The at least one processor 102 can be a central processing unit (CPU), or can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 can be a microprocessor or the processor 102 can also be any conventional processor, etc. The processor 102 is the control center of the electronic device 100 and connects various parts of the entire electronic device 100 through various interfaces and lines.

[0061] The memory 101 in the electronic device 100 stores multiple instructions to implement an efficient deepfake detection method based on spatio-frequency domain multi-scale feature similarity measurement. The processor 102 can execute the multiple instructions to thereby implement: Obtain an image, extract image information and fuse the image information to obtain feature maps of different scales; Calculate the similarity features in the spatial domain features and the similarity features in the frequency domain features respectively according to the feature maps of different scales; Fuse the similarity features in the spatial domain features and the similarity features in the frequency domain features to obtain a feature map of spatial-frequency multi-scale feature similarity measurement; Judge whether the image is forged according to the feature map of spatial-frequency multi-scale feature similarity measurement.

[0062] Embodiment If the modules / units integrated in the electronic device 100 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above embodiment methods of the present invention, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0063] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate for implementing in the process Figure 1 one process or multiple processes and / or blocksFigure 1 means for the functions specified in one or more boxes.

[0065] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to work in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction means that implements the functions specified in one Figure 1 process or processes and / or boxes Figure 1 means for the functions specified in one or more boxes.

[0066] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus, such that a series of operation steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one Figure 1 process or processes and / or boxes Figure 1 means for the functions specified in one or more boxes.

[0067] In the description of this specification, the descriptions with reference to the terms "one embodiment", "example", "specific example", etc. mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: still modifications or equivalent replacements can be made to the specific embodiments of the present invention, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered by the protection scope of the claims of the present invention.

Claims

1. An efficient deepfake detection method based on spatio-frequency domain multi-scale feature similarity measurement, characterized in that, Including: Obtain an image, extract image information and fuse the image information to obtain feature maps of different scales; Calculate the similarity features in the spatial domain features and the similarity features in the frequency domain features respectively according to the feature maps of different scales; Fuse the similarity features in the spatial domain features and the similarity features in the frequency domain features to obtain a feature map of spatial-frequency multi-scale feature similarity measurement; Judge whether the image is forged according to the feature map of spatial-frequency multi-scale feature similarity measurement.

2. An efficient deepfake detection method based on spatio-temporal frequency domain multi-scale feature similarity measurement according to claim 1, characterized in that, The extracting image information and fusing the image information to obtain feature maps of different scales includes: S101: For the obtained image, use the efficientnet-b4 network to extract to obtain image semantic information and image spatial information; S102: Fuse the image semantic information and the image spatial information to obtain feature maps of different scales.

3. An efficient deepfake detection method based on spatio-temporal multi-scale feature similarity measurement according to claim 1, characterized in that The calculating the similarity features in the spatial domain features according to the feature maps of different scales specifically includes: After slicing the feature maps of different scales, use a channel attention model to process to obtain query values in the spatial domain; The feature maps of different scales are successively processed by depthwise separable convolution, BN layer processing and slice division to obtain key values in the spatial domain; Calculate the similarity features in the spatial domain features according to the query values in the spatial domain and the key values in the spatial domain.

4. An efficient deepfake detection method based on spatio-temporal multi-scale feature similarity metric according to claim 1, characterized in that, The calculating the similarity features in the frequency domain features according to the feature maps of different scales specifically includes: Use average pooling to process the feature maps of different scales to obtain a global average image block; the global average image block is successively subjected to discrete Fourier transform, flattening and projection to obtain global frequency markers; Use average pooling to process the feature maps of different scales to obtain a global average image block; the global average image block is successively subjected to discrete Fourier transform and flattening to obtain local frequency markers, and frequency embedding is introduced into the local frequency markers; Process the global frequency markers, local frequency markers and frequency embedding through a learnable filter to obtain query values in the frequency domain and key values in the frequency domain; The key values in the frequency domain are processed by a channel attention module to obtain key features; The key features and the query values in the frequency domain are subjected to a projection operation to obtain the similarity features in the frequency domain features.

5. An efficient deepfake detection method based on spatio-temporal frequency domain multi-scale feature similarity measurement according to claim 1, characterized in that, The fusing the similarity features in the spatial domain features and the similarity features in the frequency domain features to obtain a similarity measurement result map specifically includes: S301: After slicing the similarity features in the spatial domain features, obtain a spatial domain value vector; after slicing the similarity features in the frequency domain features, obtain a frequency domain value vector; S302: Process the spatial domain value vector and the frequency domain value vector through a linear embedding layer to obtain query values in the spatial domain, key values in the spatial domain, query values in the frequency domain and key values in the frequency domain; S303: Fuse the query values in the spatial domain, the key values in the spatial domain, the query values in the frequency domain and the key values in the frequency domain to obtain a similarity measurement result map.

6. An efficient deepfake detection method based on spatio-temporal frequency domain multi-scale feature similarity metric according to claim 5, characterized in that, The S303 is specifically implemented by the following formula: ( ), =unpatchfy( ), =unpatchfy( ), = ( ) where unpatchfy is an operation that reorganizes the flattened sequence into an image according to the relative spatial positions, which consists of a ReLU activation layer, a 1×1 convolutional layer, and a BN layer; represents the spatial-frequency domain joint self-attention weight matrix; represents the Softmax normalization function that maps the input to a probability distribution; represents the query vector of the frequency-domain features; represents the query vector of the spatial-domain features; represents the key vector of the frequency-domain features; represents the key vector of the spatial-domain features; the superscript represents the transpose; represents the feature dimension; represents the concatenation operation of vectors; represents the reconstructed features after self-attention weighting in the frequency domain; represents the spatial-domain value vector; represents the reconstructed features after self-attention weighting in the frequency domain; represents the frequency-domain value vector; represents the feature map of the spatial-frequency domain multi-scale feature similarity metric, which is input to a binary classifier for genuine and fake face discrimination.

7. An efficient deepfake detection method based on spatio-temporal frequency domain multi-scale feature similarity measurement according to claim 1, characterized in that The judging whether the image is forged according to the feature map of spatial-frequency multi-scale feature similarity measurement includes the following steps: Map the feature map of spatial-frequency multi-scale feature similarity measurement to the label space in the classification task to obtain unnormalized log odds; The unnormalized log odds are subjected to an activation function for multi-class classification to obtain a predicted label; the activation function is shown in the following formula: Determine whether the image is forged according to the prediction label; wherein, if , it is determined to be forged; if , it is determined to be authentic.

8. An efficient deepfake detection system based on spatio-temporal multi-scale feature similarity measurement, characterized in that It includes: An image preprocessing unit for acquiring an image, extracting image information and fusing the image information to obtain feature maps of different scales; A feature extraction unit for calculating the similarity features in the spatial domain feature and the similarity features in the frequency domain feature respectively according to the feature maps of different scales; A feature fusion unit for fusing the similarity features in the spatial domain feature and the similarity features in the frequency domain feature to obtain a feature map of the spatial-frequency multi-scale feature similarity metric; A judgment unit for judging whether the image is forged according to the feature map of the spatial-frequency multi-scale feature similarity metric; 9. An electronic device, characterized in that, It includes a processor and a memory, and the processor is configured to execute a computer program stored in the memory to implement the efficient deep fake detection method based on the spatial-frequency domain multi-scale feature similarity metric according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, it implements the efficient deep fake detection method based on the spatial-frequency domain multi-scale feature similarity metric according to any one of claims 1 to 7.

Images