Deepfake image detection method and system based on frequency-enhanced self-attention

Abstract

The application provides a deep fake image processing method and system based on frequency enhanced self-attention. The method comprises: amplifying frequency domain features by using an attention mechanism, and integrating RGB and frequency domain information by using a self-attention structure. Specifically, a convolutional attention module structure is adopted, and channel attention mechanism and spatial attention mechanism are used to enhance the key frequency band and region in the frequency domain respectively. This flexible enhancement of the frequency domain features is more general in deep fake detection. Then, a double-flow network structure is adopted to extract RGB information and frequency domain information. In addition, in order to further promote the collaborative learning of the RGB and frequency domain information, a feature fusion method based on the self-attention structure is adopted, which globally models the two kinds of information, thereby constructing a comprehensive face fake representation. The scheme provided by the application solves the problems that the existing generalization is poor, the effect needs to be enhanced, and the new deep fake images in the Internet cannot be effectively coped with.

Description

Technical Field

[0001] This invention belongs to the field of deepfake image processing and detection, and particularly relates to a deepfake image processing method and system based on frequency-enhanced self-attention. Background Technology

[0002] The rapid advancements in computer hardware and major breakthroughs in deep learning have laid a solid foundation for the rise of artificial intelligence, making it a focus of public attention. Variational autoencoders and generative adversarial networks (GANs), as representatives of generative deep neural networks, have shown particularly outstanding performance in the generation of images, videos, music, and text. In the field of deep learning, face spoofing methods based on these two technologies are known as deepfake technology. These advanced technologies allow ordinary people to easily create realistic fake face images without in-depth professional knowledge and training. If someone uses this technology to create malicious fraudulent images, especially those intended to deceive and mislead the public, it could pose significant social risks. Therefore, it is necessary to develop and promote efficient face spoofing image processing methods to achieve spoofing detection. This technology should not only be able to quickly and accurately identify spoofed content but also possess the ability to combat increasingly sophisticated spoofing techniques.

[0003] Currently, the academic community has conducted relevant research on deepfake image processing. Early studies primarily extracted forgery traces from the original RGB information of images and used these features for deepfake detection. However, with the rapid development and refinement of deepfake technology, the forgery traces in the original RGB domain have weakened, significantly reducing the detection effectiveness of these traditional methods. To address this challenge, recent research has shifted its focus, attempting to combine frequency domain information for deepfake image detection. In one study, the authors observed that deepfake models often need to upsample low-resolution features when synthesizing forged images. This process creates differences between the forged and genuine images in the frequency domain. Based on this conclusion, some studies have combined frequency domain information for deepfake detection, achieving significant breakthroughs. However, current methods often focus on fixed frequency bands or complete regions in the frequency domain. In reality, different forgery techniques and their underlying model structures and parameters vary, resulting in diverse forgery traces. Methods that only focus on fixed frequency bands and entire regions lack flexibility and may be affected by useless information, leading to reduced generalization. Some methods only consider frequency domain information, which ignores certain micro-forgery traces in the RGB domain that might be easier to detect in the original RGB space. Other methods, when integrating frequency domain and RGB information, only consider local relationships between data, which limits their ability to extract detection features.

[0004] Existing technology First, there's FDFL, a method derived from the paper "Frequency-aware discriminative feature learning supervised by single-center loss for face forgery detection." This method proposes a deepfake image forgery detection approach based on a combination of frequency domain and RGB information. The authors developed an adaptive frequency domain feature generation module that mines important frequency bands in the frequency domain in a completely data-driven manner and combines the frequency domain information with the image's RGB information. Simultaneously, they designed a single-center loss that only reduces the intra-class distance of real images, improving the inter-class discriminative distance in the embedding space. The network can learn more discriminative features with less optimization difficulty. The proposed model was trained on the FaceForensics++ HQ dataset and achieved an AUC of 82.4% on Celeb-DF-v2.

[0005] Next is F3Net, a method from the paper *Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues*, which proposes a frequency-domain-based two-stream network. The authors introduce two frequency-domain features. Using three filters, the frequency-domain image is divided into high, mid, and low frequency bands, and an inverse Fourier transform is performed to obtain the frequency-domain perceptual image decomposition features. After performing a sliding-window Fourier transform on the image, filters are used to obtain the local spectrum, and the mean combination is calculated to obtain the local frequency-domain statistical features. The authors also designed a fusion module to input the two features into the two-stream network to obtain the classification result. The proposed model achieves an accuracy of 97.52% and an AUC of 98.10% on the FaceForensics++ HQ dataset.

[0006] Existing technological defects The main drawback of the first method is that it does not focus on important regions in the frequency domain information, and its use of a convolutional neural network structure to integrate frequency domain and RGB information fails to consider the global relationship between the two types of information. This results in insufficient detection performance.

[0007] The main drawback of the second method is that it only uses frequency domain information, ignoring forgery traces in the RGB space, and its proposed local frequency domain statistical features use a fixed filter kernel, resulting in poor flexibility. This limits the detection performance and generalization ability of the method. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention proposes a technical solution for a deep forgery image detection method based on frequency-enhanced self-attention, thereby resolving these issues.

[0009] The first aspect of this invention discloses a method for detecting deepfake images based on frequency-enhanced self-attention, the method comprising: Step S1: Convert the RGB image of the face to obtain frequency features; apply a channel attention mechanism to process the frequency features to obtain channel attention features; apply a spatial attention mechanism to process the channel attention features to obtain frequency domain enhancement features. Step S2: Input the face RGB image and frequency domain enhancement features into two Xception networks with the same structure but independent parameters to obtain RGB features in three stages and frequency domain enhancement extraction features in three stages. Step S3: Apply a self-attention mechanism to fuse the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages to obtain fused features; add the fused features to the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages to obtain RGB fused features and frequency domain fused features. Step S4: Input the RGB fusion feature and the frequency domain fusion feature into the embedding layer respectively to obtain the RGB embedding feature and the frequency domain embedding feature; add the RGB embedding feature and the frequency domain embedding feature to obtain the total fusion feature; input the total fusion feature into the fully connected layer to obtain the image processing result.

[0010] According to the method of the first aspect of the present invention, in step S1, the method of converting the RGB image of a face to obtain frequency features includes: Convert the RGB image of a face from the RGB color space to the grayscale color space. H and W These are the height and width of the image, respectively; using a size of P × P The sliding window segments the grayscale image into H / 2• W / 2 image blocks; then, a two-dimensional discrete cosine transform is applied to each image block; after transformation to the frequency domain, the same frequency components in all image blocks are combined into one channel, while maintaining the relative position of the image blocks in the original image, to obtain frequency features.

[0011] According to a first aspect of the method of the present invention, in step S1, the method of applying a channel attention mechanism to process the frequency features to obtain channel attention features includes: The frequency features are passed through two parallel max pooling and average pooling layers to obtain a first max pooling feature and a first average pooling feature. Then, the first max pooling feature and the first average pooling feature are fed into the same multilayer perceptron and summed to obtain a channel attention feature map. Finally, the channel attention feature map is multiplied element-wise with the frequency features to obtain the channel attention feature.

[0012] According to a first aspect of the present invention, in step S1, the method of applying a spatial attention mechanism to process the channel attention features to obtain frequency domain enhancement features includes: The channel attention features are subjected to max pooling and average pooling operations along the channel direction to obtain a second max pooling feature and a second average pooling feature, respectively. Subsequently, the second max pooling feature and the second average pooling feature are concatenated along the channel direction. Finally, the concatenated second max pooling feature and the second average pooling feature are processed by a convolutional layer with a kernel size of 7×7 to achieve feature fusion and obtain a spatial attention map. The spatial attention map is then multiplied element-wise with the channel attention features to obtain a frequency domain enhancement feature.

[0013] According to the method of the first aspect of the present invention, in step S3, the method of fusing the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages using a self-attention mechanism to obtain the fused features includes: To ensure uniform dimensionality of the features, the RGB features of the three stages and the frequency domain enhanced extracted features of the three stages are average-pooled and downsampled to a fixed 8x8 resolution. Then, the downsampled RGB features of the three stages and the frequency domain enhanced extracted features of the three stages are flattened and stacked together along the channel direction to obtain a feature sequence. Subsequently, a learnable position embedding is added to the feature sequence to obtain a position embedding sequence. Then, the position embedding sequence is passed through multiple self-attention blocks to obtain a fused feature.

[0014] A second aspect of this invention discloses a deepfake image detection system based on frequency-enhanced self-attention, the system comprising: The first processing module is configured to: perform conversion processing on the RGB image of the face to obtain frequency features; apply a channel attention mechanism to process the frequency features to obtain channel attention features; and apply a spatial attention mechanism to process the channel attention features to obtain frequency domain enhancement features. The second processing module is configured to input the face RGB image and frequency domain enhancement features into two Xception networks with the same structure but independent parameters, respectively, to obtain RGB features in three stages and frequency domain enhancement extraction features in three stages. The third processing module is configured to apply a self-attention mechanism to fuse the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages to obtain fused features; and to add the fused features to the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages to obtain RGB fused features and frequency domain fused features. The fourth processing module is configured to input the RGB fusion feature and the frequency domain fusion feature into the embedding layer respectively to obtain the RGB embedding feature and the frequency domain embedding feature; add the RGB embedding feature and the frequency domain embedding feature to obtain the total fusion feature; and input the total fusion feature into the fully connected layer to obtain the image processing result.

[0015] According to the system of the second aspect of the present invention, the method for converting a human face RGB image to obtain frequency features includes: Convert the RGB image of a face from the RGB color space to the grayscale color space. H and W These are the height and width of the image, respectively; using a size of P × P The sliding window segments the grayscale image into H / 2• W / 2 image blocks; then, a two-dimensional discrete cosine transform is applied to each image block; after transformation to the frequency domain, the same frequency components in all image blocks are combined into one channel, while maintaining the relative position of the image blocks in the original image, to obtain frequency features.

[0016] The method for processing the frequency features using the channel attention mechanism to obtain channel attention features includes: The frequency features are passed through two parallel max pooling and average pooling layers to obtain a first max pooling feature and a first average pooling feature. Then, the first max pooling feature and the first average pooling feature are fed into the same multilayer perceptron and summed to obtain a channel attention feature map. Finally, the channel attention feature map is multiplied element-wise with the frequency features to obtain the channel attention feature.

[0017] The method for processing the channel attention features using the applied spatial attention mechanism to obtain frequency domain enhanced features includes: The channel attention features are subjected to max pooling and average pooling operations along the channel direction to obtain a second max pooling feature and a second average pooling feature, respectively. Subsequently, the second max pooling feature and the second average pooling feature are concatenated along the channel direction. Finally, the concatenated second max pooling feature and the second average pooling feature are processed by a convolutional layer with a kernel size of 7×7 to achieve feature fusion and obtain a spatial attention map. The spatial attention map is then multiplied element-wise with the channel attention features to obtain a frequency domain enhancement feature.

[0018] According to the system of the second aspect of the present invention, the method for fusing the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages using a self-attention mechanism to obtain the fused features includes: To ensure uniform dimensionality of the features, the RGB features of the three stages and the frequency domain enhanced extracted features of the three stages are average-pooled and downsampled to a fixed 8x8 resolution. Then, the downsampled RGB features of the three stages and the frequency domain enhanced extracted features of the three stages are flattened and stacked together along the channel direction to obtain a feature sequence. Subsequently, a learnable position embedding is added to the feature sequence to obtain a position embedding sequence. Then, the position embedding sequence is passed through multiple self-attention blocks to obtain a fused feature.

[0019] A third aspect of this invention discloses an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of a deep forgery image processing method based on frequency-enhanced self-attention according to any one of the first aspects of this disclosure.

[0020] A fourth aspect of this invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a deepfake image processing method based on frequency-enhanced self-attention, as described in any of the first aspects of this disclosure.

[0021] In summary, the solution proposed in this invention solves the problems of poor generalization, insufficient effectiveness, and inability to effectively deal with new deepfake images on the Internet. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1A flowchart of a deep forgery image processing method based on frequency-enhanced self-attention according to an embodiment of the present invention; Figure 2 A flowchart illustrating a deep forgery image processing method based on frequency-enhanced self-attention according to an embodiment of the present invention; Figure 3 This is a flowchart of frequency domain feature enhancement according to an embodiment of the present invention; Figure 4 This is a structural diagram of a deep forgery image processing system based on frequency-enhanced self-attention according to an embodiment of the present invention; Figure 5 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] The first aspect of this invention discloses a deep forgery image processing method based on frequency-enhanced self-attention. Figure 1 This is a flowchart of a depth forgery image processing method based on frequency-enhanced self-attention according to an embodiment of the present invention, as shown below. Figures 1-3 As shown, the method includes: Step S1: Convert the RGB image of the face to obtain frequency features; apply a channel attention mechanism to process the frequency features to obtain channel attention features; apply a spatial attention mechanism to process the channel attention features to obtain frequency domain enhancement features. Step S2: Input the face RGB image and frequency domain enhancement features into two Xception networks with the same structure but independent parameters to obtain RGB features in three stages and frequency domain enhancement extraction features in three stages. Step S3: Apply a self-attention mechanism to fuse the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages to obtain fused features; add the fused features to the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages to obtain RGB fused features and frequency domain fused features. Step S4: Input the RGB fusion feature and the frequency domain fusion feature into the embedding layer respectively to obtain the RGB embedding feature and the frequency domain embedding feature; add the RGB embedding feature and the frequency domain embedding feature to obtain the total fusion feature; input the total fusion feature into the fully connected layer to obtain the image processing result.

[0026] In step S1, the RGB image of the face is converted to obtain frequency features; the frequency features are processed by a channel attention mechanism to obtain channel attention features; and the channel attention features are processed by a spatial attention mechanism to obtain frequency domain enhancement features.

[0027] In some embodiments, in step S1, the method for converting the RGB image of a face to obtain frequency features includes: Convert the RGB image of a face from the RGB color space to the grayscale color space. H and W These are the height and width of the image, respectively; using a size of P × P The sliding window segments the grayscale image into H / 2• W / 2 image blocks; then, a two-dimensional discrete cosine transform is applied to each image block; after transformation to the frequency domain, the same frequency components in all image blocks are combined into one channel, while maintaining the relative position of the image blocks in the original image, to obtain frequency features.

[0028] The method for processing the frequency features using the channel attention mechanism to obtain channel attention features includes: The frequency features are passed through two parallel max pooling and average pooling layers to obtain a first max pooling feature and a first average pooling feature. Then, the first max pooling feature and the first average pooling feature are fed into the same multilayer perceptron and summed to obtain a channel attention feature map. Finally, the channel attention feature map is multiplied element-wise with the frequency features to obtain the channel attention feature.

[0029] The method for processing the channel attention features using the applied spatial attention mechanism to obtain frequency domain enhanced features includes: The channel attention features are subjected to max pooling and average pooling operations along the channel direction to obtain a second max pooling feature and a second average pooling feature, respectively. Subsequently, the second max pooling feature and the second average pooling feature are concatenated along the channel direction. Finally, the concatenated second max pooling feature and the second average pooling feature are processed by a convolutional layer with a kernel size of 7×7 to achieve feature fusion and obtain a spatial attention map. The spatial attention map is then multiplied element-wise with the channel attention features to obtain a frequency domain enhancement feature.

[0030] Specifically, any provided image needs to be preprocessed first. First, it is passed through a face detector to detect faces, resulting in RGB images of the faces.

[0031] Channel attention operations enable the model to flexibly focus on important frequency bands in the frequency domain of forged images at different depths, while spatial attention operations focus on spatial regions to explore salient areas. This process can be summarized as follows:

[0032] in Represents channel attention operations. This represents spatial attention operations, while Represents frequency domain enhancement features, Represents frequency characteristics.

[0033] In step S2, the face RGB image and frequency domain enhancement features are respectively input into two Xception networks with the same structure but independent parameters to obtain RGB features in three stages and frequency domain enhancement extraction features in three stages.

[0034] Specifically, deepfake image processing currently faces significant challenges. On the one hand, real images exhibit vastly different features due to various factors such as scene, lighting, and device; on the other hand, technological advancements have led to increasingly diverse features in forged images. To address these challenges, this step employs a dual-stream strategy. Specifically, unlike F3NET and FDFL, this method uses two structurally identical but parameter-independent deep convolutional networks to process frequency domain features and RGB features in parallel, thereby better capturing the variations of these two types of features at different scales and achieving effective processing of forged images. Considering the Xception network's good expressive power and ability to adaptively capture multi-scale features, it is chosen as the backbone network for feature extraction in this step. Based on the previous steps, the obtained RGB image of the face is... and frequency domain enhancement features As input, since the Xception network can be divided into three main stages, each corresponding to a different feature extraction depth and scale, the outputs of the two branches at different stages can be obtained as follows: and ,in .

[0035] In step S3, a self-attention mechanism is applied to fuse the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages to obtain fused features; the fused features are added to the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages to obtain RGB fused features and frequency domain fused features.

[0036] In some embodiments, in step S3, the method for fusing the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages using a self-attention mechanism to obtain the fused features includes: To ensure uniform dimensionality of the features, the RGB features from the three stages and the frequency-enhanced extracted features from the three stages are average-pooled and downsampled to a fixed 8×8 resolution. Then, the downsampled RGB features from the three stages and the frequency-enhanced extracted features from the three stages are flattened and stacked together along the channel direction to obtain a feature sequence. Subsequently, a learnable positional embedding is added to the feature sequence to obtain a positional embedding sequence, enabling the network to capture dependencies between different positions. Then, the positional embedding sequence is passed through multiple self-attention blocks to obtain fused features.

[0037] In some embodiments, before adding the fused feature to the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages in step S3, the method further includes: adjusting and upsampling the fused feature. To match input dimensions .

[0038] Specifically, the self-attention block is defined as follows:

[0039]

[0040] in Representative layer normalization layer Represents bullish attention. This represents a multilayer perceptron.

[0041] In summary, the proposed scheme of this invention employs an attention mechanism to enhance the original frequency information in its frequency domain feature enhancement method based on an attention structure. Compared with traditional methods based on fixed filter kernels, this data-driven approach is more flexible and can specifically enhance frequency bands and regions related to forgery traces in the frequency domain. This not only amplifies key signals related to forgery but also reduces interference from information irrelevant to deep forgery detection, improving the universality of the processing model. The feature extraction method based on a multi-scale dual-stream network structure, compared to methods using only frequency domain or RGB information, ensures that each feature is fully mined and utilized, while employing two structurally identical but parameter-independent deep convolutional networks. This parallel dual-network structure ensures that each feature is better captured, thereby capturing the changes of these two types of features at different scales to achieve effective identification of forged images. The feature fusion module based on a self-attention structure leverages the self-attention mechanism's ability to capture long-range dependencies within a sequence. This allows the fusion module to effectively understand and enhance the global contextual relationship between RGB and frequency domain information compared to fusion modules using convolutional networks, achieving complementarity between the two types of information and thus improving detection performance.

[0042] The method proposed in this invention achieves an accuracy of 98.00% and an AUC of 99.38% on the FaceForensics++ HQ dataset.

[0043] A second aspect of the present invention discloses a deep forgery image processing system based on frequency-enhanced self-attention. Figure 4 This is a structural diagram of a deep forgery image processing system based on frequency-enhanced self-attention according to an embodiment of the present invention; as shown. Figure 4 As shown, the system 100 includes: The first processing module 101 is configured to: perform conversion processing on the RGB image of the face to obtain frequency features; apply a channel attention mechanism to process the frequency features to obtain channel attention features; and apply a spatial attention mechanism to process the channel attention features to obtain frequency domain enhancement features. The second processing module 102 is configured to input the face RGB image and frequency domain enhancement features into two Xception networks with the same structure but independent parameters, respectively, to obtain RGB features in three stages and frequency domain enhancement extraction features in three stages. The third processing module 103 is configured to apply a self-attention mechanism to fuse the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages to obtain fused features; and to add the fused features to the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages to obtain RGB fused features and frequency domain fused features. The fourth processing module 104 is configured to input the RGB fusion feature and the frequency domain fusion feature into the embedding layer respectively to obtain the RGB embedding feature and the frequency domain embedding feature; add the RGB embedding feature and the frequency domain embedding feature to obtain the total fusion feature; and input the total fusion feature into the fully connected layer to obtain the image processing result.

[0044] According to the system of the second aspect of the present invention, the first processing module 101 is specifically configured such that the method for converting the RGB image of a face to obtain frequency features includes: Convert the RGB image of a face from the RGB color space to the grayscale color space. H and W These are the height and width of the image, respectively; using a size of P × P The sliding window segments the grayscale image into H / 2• W / 2 image blocks; then, a two-dimensional discrete cosine transform is applied to each image block; after transformation to the frequency domain, the same frequency components in all image blocks are combined into one channel, while maintaining the relative position of the image blocks in the original image, to obtain frequency features.

[0045] The method for processing the frequency features using the channel attention mechanism to obtain channel attention features includes: The frequency features are passed through two parallel max pooling and average pooling layers to obtain a first max pooling feature and a first average pooling feature. Then, the first max pooling feature and the first average pooling feature are fed into the same multilayer perceptron and summed to obtain a channel attention feature map. Finally, the channel attention feature map is multiplied element-wise with the frequency features to obtain the channel attention feature.

[0046] The method for processing the channel attention features using the applied spatial attention mechanism to obtain frequency domain enhanced features includes: The channel attention features are subjected to max pooling and average pooling operations along the channel direction to obtain a second max pooling feature and a second average pooling feature, respectively. Subsequently, the second max pooling feature and the second average pooling feature are concatenated along the channel direction. Finally, the concatenated second max pooling feature and the second average pooling feature are processed by a convolutional layer with a kernel size of 7×7 to achieve feature fusion and obtain a spatial attention map. The spatial attention map is then multiplied element-wise with the channel attention features to obtain a frequency domain enhancement feature.

[0047] Specifically, any provided image needs to be preprocessed first. First, it is passed through a face detector to detect faces, resulting in RGB images of the faces.

[0048] Channel attention operations enable the model to flexibly focus on important frequency bands in the frequency domain of forged images at different depths, while spatial attention operations focus on spatial regions to explore salient areas. This process can be summarized as follows:

[0049] in Represents channel attention operations. This represents spatial attention operations, while Represents frequency domain enhancement features, Represents frequency characteristics.

[0050] According to the system of the second aspect of the present invention, the second processing module 102 is specifically configured to address the significant challenges currently facing deepfake image processing. On the one hand, real images exhibit vastly different features due to various factors such as scene, lighting, and device; on the other hand, with technological advancements, the features exhibited by forged images are becoming increasingly diverse. To address these challenges, this step employs a dual-stream strategy. Specifically, unlike F3NET and FDFL, this method uses two structurally identical but parameter-independent deep convolutional networks to process frequency domain features and RGB features in parallel, thereby better capturing the changes of these two types of features at different scales to achieve effective processing of forged images. Considering that the Xception network has good expressive power and can adaptively capture multi-scale features, this step selects it as the backbone network for feature extraction. Based on the previous steps, the obtained face RGB image is... and frequency domain enhancement features As input, since the Xception network can be divided into three main stages, each corresponding to a different feature extraction depth and scale, the outputs of the two branches at different stages can be obtained as follows: and ,in .

[0051] According to the system of the second aspect of the present invention, the third processing module 103 is specifically configured such that the method for fusing the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages using a self-attention mechanism to obtain the fused features includes: To ensure uniform dimensionality of the features, the RGB features from the three stages and the frequency-enhanced extracted features from the three stages are average-pooled and downsampled to a fixed 8×8 resolution. Then, the downsampled RGB features from the three stages and the frequency-enhanced extracted features from the three stages are flattened and stacked together along the channel direction to obtain a feature sequence. Subsequently, a learnable positional embedding is added to the feature sequence to obtain a positional embedding sequence, enabling the network to capture dependencies between different positions. Then, the positional embedding sequence is passed through multiple self-attention blocks to obtain fused features.

[0052] In some embodiments, before adding the fused feature to the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages in step S3, the method further includes: adjusting and upsampling the fused feature. To match input dimensions .

[0053] Specifically, the self-attention block is defined as follows:

[0054]

[0055] in Representative layer normalization layer Represents bullish attention. This represents a multilayer perceptron.

[0056] A third aspect of this invention discloses an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of a deep forgery image processing method based on frequency-enhanced self-attention, as disclosed in any of the first aspects of this invention.

[0057] Figure 5 This is a structural diagram of an electronic device according to an embodiment of the present invention, such as... Figure 5As shown, the electronic device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, Near Field Communication (NFC), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0058] Those skilled in the art will understand that Figure 5 The structure shown is merely a structural diagram of the part related to the technical solution of this disclosure and does not constitute a limitation on the electronic device to which the solution of this application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0059] A fourth aspect of this invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a deepfake image processing method based on frequency-enhanced self-attention, as described in any of the first aspects of this invention.

[0060] Please note that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for detecting deepfake images based on frequency-enhanced self-attention, characterized in that, The method includes: Step S1: Convert the RGB image of the face to obtain frequency features; apply a channel attention mechanism to process the frequency features to obtain channel attention features; apply a spatial attention mechanism to process the channel attention features to obtain frequency domain enhancement features. Step S2: Input the face RGB image and frequency domain enhancement features into two Xception networks with the same structure but independent parameters to obtain RGB features in three stages and frequency domain enhancement extraction features in three stages. Step S3: Apply a self-attention mechanism to fuse the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages to obtain fused features; add the fused features to the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages to obtain RGB fused features and frequency domain fused features. In step S3, the method for fusing the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages using a self-attention mechanism to obtain the fused features includes: To ensure uniform dimensionality of the features, the RGB features from the three stages and the frequency-enhanced extracted features from the three stages are average-pooled and downsampled to a fixed 8x8 resolution. Then, the downsampled RGB features from the three stages and the frequency-enhanced extracted features from the three stages are flattened and stacked along the channel direction to obtain a feature sequence. Subsequently, a learnable positional embedding is added to the feature sequence to obtain a positional embedding sequence. Finally, the positional embedding sequence is passed through multiple self-attention blocks to obtain a fused feature. Step S4: Input the RGB fusion feature and the frequency domain fusion feature into the embedding layer respectively to obtain the RGB embedding feature and the frequency domain embedding feature; add the RGB embedding feature and the frequency domain embedding feature to obtain the total fusion feature; input the total fusion feature into the fully connected layer to obtain the image processing result.

2. The method for detecting deep forged images based on frequency-enhanced self-attention according to claim 1, characterized in that, In step S1, the method for converting the RGB image of a face to obtain frequency features includes: Convert the RGB image of a face from the RGB color space to the grayscale color space. H and W These are the height and width of the image, respectively; using a size of P × P The sliding window divides the grayscale image into H / 2• W / 2 image blocks; then, a two-dimensional discrete cosine transform is applied to each image block; after transformation to the frequency domain, the same frequency components in all image blocks are combined into one channel, while maintaining the relative position of the image blocks in the original image, to obtain frequency features.

3. The method for detecting deep forged images based on frequency-enhanced self-attention according to claim 1, characterized in that, In step S1, the method for processing the frequency features using the channel attention mechanism to obtain channel attention features includes: The frequency features are passed through two parallel max pooling and average pooling layers to obtain a first max pooling feature and a first average pooling feature. Then, the first max pooling feature and the first average pooling feature are fed into the same multilayer perceptron and summed to obtain a channel attention feature map. Finally, the channel attention feature map is multiplied element-wise with the frequency features to obtain the channel attention feature.

4. The method for detecting deep forged images based on frequency-enhanced self-attention according to claim 1, characterized in that, In step S1, the method for detecting the channel attention features using a spatial attention mechanism to obtain frequency domain enhancement features includes: The channel attention features are subjected to max pooling and average pooling operations along the channel direction to obtain a second max pooling feature and a second average pooling feature, respectively. Subsequently, the second max pooling feature and the second average pooling feature are concatenated along the channel direction. Finally, the concatenated second max pooling feature and the second average pooling feature are processed by a convolutional layer with a kernel size of 7×7 to achieve feature fusion and obtain a spatial attention map. The spatial attention map is then multiplied element-wise with the channel attention features to obtain a frequency domain enhancement feature.

5. A deepfake image detection system based on frequency-enhanced self-attention, characterized in that, The system includes: The first processing module is configured to: perform conversion processing on the RGB image of the face to obtain frequency features; apply a channel attention mechanism to process the frequency features to obtain channel attention features; and apply a spatial attention mechanism to process the channel attention features to obtain frequency domain enhancement features. The second processing module is configured to input the face RGB image and frequency domain enhancement features into two Xception networks with the same structure but independent parameters, respectively, to obtain RGB features in three stages and frequency domain enhancement extraction features in three stages. The third processing module is configured to apply a self-attention mechanism to fuse the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages to obtain fused features; and to add the fused features to the RGB features of the three stages and the frequency domain enhancement extraction features of the three stages to obtain RGB fused features and frequency domain fused features. The method for fusing the RGB features from the three stages and the frequency domain enhancement extraction features from the three stages using a self-attention mechanism to obtain the fused features includes: To ensure uniform dimensionality of the features, the RGB features from the three stages and the frequency-enhanced extracted features from the three stages are average-pooled and downsampled to a fixed 8x8 resolution. Then, the downsampled RGB features from the three stages and the frequency-enhanced extracted features from the three stages are flattened and stacked along the channel direction to obtain a feature sequence. Subsequently, a learnable positional embedding is added to the feature sequence to obtain a positional embedding sequence. Finally, the positional embedding sequence is passed through multiple self-attention blocks to obtain a fused feature. The fourth processing module is configured to input the RGB fusion feature and the frequency domain fusion feature into the embedding layer respectively to obtain the RGB embedding feature and the frequency domain embedding feature; add the RGB embedding feature and the frequency domain embedding feature to obtain the total fusion feature; and input the total fusion feature into the fully connected layer to obtain the image processing result.

6. A deep forgery image detection system based on frequency-enhanced self-attention according to claim 5, characterized in that, The method for converting and processing RGB images of faces to obtain frequency features includes: Convert the RGB image of a face from the RGB color space to the grayscale color space. H and W These are the height and width of the image, respectively; using a size of P × P The sliding window divides the grayscale image into H / 2• W Two image blocks are generated; then, a two-dimensional discrete cosine transform is applied to each image block; after transformation to the frequency domain, the same frequency components in all image blocks are combined into one channel, while maintaining the relative positions of the image blocks in the original image, to obtain the frequency features; The method for processing the frequency features using the channel attention mechanism to obtain channel attention features includes: The frequency features are passed through two parallel max-pooling and average-pooling layers to obtain a first max-pooling feature and a first average-pooling feature. Then, the first max-pooling feature and the first average-pooling feature are fed into the same multilayer perceptron and summed to obtain a channel attention feature map. Finally, the channel attention feature map is multiplied element-wise with the frequency features to obtain the channel attention feature. The method for processing the channel attention features using the applied spatial attention mechanism to obtain frequency domain enhanced features includes: The channel attention features are subjected to max pooling and average pooling operations along the channel direction to obtain a second max pooling feature and a second average pooling feature, respectively. Subsequently, the second max pooling feature and the second average pooling feature are concatenated along the channel direction. Finally, the concatenated second max pooling feature and the second average pooling feature are processed by a convolutional layer with a kernel size of 7×7 to achieve feature fusion and obtain a spatial attention map. The spatial attention map is then multiplied element-wise with the channel attention features to obtain a frequency domain enhancement feature.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the deep forgery image processing method based on frequency enhancement self-attention as described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the deep forgery image processing method based on frequency-enhanced self-attention as described in any one of claims 1 to 4.

Images