Cross-dimensional image fusion adaptive deep counterfeiting discrimination method

Abstract

The invention relates to a cross-dimensional image fusion adaptive deep counterfeiting discrimination method, which belongs to the technical field of deep counterfeiting detection and comprises the following steps of: extracting image color features capable of representing an overall mode of an original image from a spatial domain; two different frequency domain features are extracted. The method comprises the following steps: firstly, directly extracting shallow low-frequency features from an original input image, and expanding an original rgb image to a frequency domain by using the shallow low-frequency features; and the other feature is a depth frequency domain feature, extracting a residual image of the input image, and extracting a high-frequency feature by using a convolution extractor. Fusing shallow low frequency features and deep frequency domain features by using an adaptive feature fusion module formed by gating convolution, so that features of different branches are fully expressed in a classification stage; the method has the beneficial effects that a large number of experiments are carried out on a large number of data sets, and experimental results prove the effectiveness and robustness of the method.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep fake detection, and specifically relates to a cross-dimensional image fusion adaptive deep fake identification method. Background Art

[0002] Deepfake, or AI face swapping, has been a hot topic on the Internet in recent years. Deepfake is an intelligent processing technology that uses artificial intelligence technology to forge digital content such as audio, images or videos, usually referring to the tampering of facial areas, which can imitate a specific person or make a specific person look like doing a specific thing, and can achieve a level of forgery and authenticity for human visual effects.

[0003] With the development of deep learning technology (Generative Adversarial Networks, GAN) and automatically generated anti-network encoders (Autoencoder), deep data-driven forgery generation technology leaves almost no trace in the image. Technology is often a double-edged sword. While technology enriches the entertainment methods of the invention and facilitates the life of the invention, it may also quietly affect the invention. Criminals use deep fake technology to generate defamatory images to defame others, generate false news images, and forge the speeches of public figures, greatly endangering personal reputation, social trust and national stability. Therefore, the present invention starts with the most harmful image forgery technology, face forgery, and studies preventive measures that can quickly and accurately detect forged images. The present invention is called Deepfake Detection.

[0004] The image or video content generated by deep fakes usually contains human faces, with faces as the main content, and some multimedia information is generated based on faces for the purpose of slander, defamation and rumor.

[0005] Since the term deepfake emerged in 2018, deepfake detection technology has also seen very rapid development as people have become more aware of the harm caused by deepfakes and relevant laws have been introduced to limit the spread of deepfake content. Although these methods can achieve 90% or higher accuracy on publicly available datasets, they usually have the following problems: First, most methods try to find visual artifacts left on images during deepfake synthesis, but the images synthesized by the generative adversarial network have little change compared to the original image pattern and can even be repaired, especially for the color pattern of the image; second, in real scenes, most images transmitted on the Internet have been compressed twice, and many classifiers with high detection accuracy for original resolution images will significantly reduce their accuracy when applied to recompressed images. In addition, although there are many methods that can consider the frequency domain features of images, humans also have some defects, that is, they all use fixed filters to extract frequency domain features. These methods are often insufficient to deal with variable fake image generation and GAN network generators, and are easily attacked by new methods. Summary of the invention

[0006] In order to solve the above problems, the present invention provides a cross-dimensional shadow fusion adaptive deep fake identification method, which includes a spatial domain feature extraction module and two frequency domain feature extraction modules, and uses an adaptive cross-domain fusion module to fuse the features of each module; the method includes the following steps:

[0007] S1, extracting image color features that can represent the overall pattern of the original image from the spatial domain;

[0008] S2. Take two different frequency domain features. One is the shallow low-frequency feature, which is directly extracted from the original input image through the shallow 2 / 5 layer low-frequency feature, and the original RGB image is expanded to the frequency domain through the shallow low-frequency feature; the other is the deep frequency domain feature, which extracts the residual map of the input image and uses the convolution extractor to extract the high-frequency features. The residual transformation map of the image represents the fused edge information of the face in the original image, reducing the influence of the chromatic aberration of the smooth part of the image.

[0009] S3. An adaptive feature fusion module composed of gated convolution is used to fuse shallow low-frequency features and deep frequency domain features, so that the features of different branches can be fully expressed in the classification stage.

[0010] Further, step S1 is as follows: The feature extraction stage is divided into two branches, a shallow composite feature extraction branch and a deep frequency domain feature extraction branch. The model uses the ResNet network as the basic backbone.

[0011] First, a convolutional neural network is used to extract the color gamut features of the image, and Dobesy wavelet transform is selected as the filtering method. Dobesy wavelet transform uses a short-time window to analyze long signals and adapts to the local smooth changes of the image by finding a tightly supported orthogonal point to achieve the purpose of scaling.

[0012] The Dobesy wavelet transform consists of an integer L and a series of wavelet filter coefficients {a i , i=0,1,2,...,L-1}, and the following two relations are stipulated:

[0013]

[0014] Where ρ(x) is the amplitude function, ψ(x) is the mother function of wavelet transform, L is the order of Dobesy wavelet transform, and x is the step size in the time domain. As shown in the formula, there are L orthogonal conditions in the wavelet transform, and the wavelet coefficient a is obtained. l , the frequency domain expansion A(w) with orthogonality can be expressed as:

[0015]

[0016] Where C is a constant term, and the fourth-order coefficient h(l) can be obtained as follows:

[0017]

[0018] After fusing the Dobesy-Baud symbol and color features, they are fed into a convolutional neural network to obtain a composite feature map. In the composite stage, simple size stitching is used.

[0019] Furthermore, a residual map is used as a basic input, the residual map is obtained through a filter, and a deep frequency domain feature extraction method is used to obtain deep frequency domain information.

[0020] The frequency domain features extracted from the original image combined with the color can reflect the facial details, while the deep frequency domain features extracted from the residual image reflect the edge information of the facial image.

[0021] First, a residual map reflecting the mixed boundary information is extracted, and the residual map is mapped from the original narrow channel information to a high-dimensional space, and a neural network is used to extract deep frequency domain features;

[0022] The noise residual is expressed as follows:

[0023]

[0024] Among them, X ij It represents the pixel value of the currently calculated pixel point, N ij represents the neighborhood pixels, It is defined in N ij cX ij The predicted value; c is the residual order, which is a variable and changes according to the filter type. The three selected filters are: local linear predictor, 3*3 translation-invariant linear pixel predictor, and a filter based on the Nelder-Nead algorithm with a kernel of 5*5.

[0025] In addition, the two feature maps of shallow low-frequency features and deep frequency domain features have the same output dimensions. Assuming that the dimensions are C*H*W, gated convolution is used to ensure the adaptive nature of the fusion module. In gated convolution, a learnable dynamic feature selection mechanism is adopted, and all positions of each layer of each channel are sorted one by one. In both the spatial and frequency domains, visual artifacts are always fixed on a fixed number of adjacent pixels. Gated convolution is used to capture and locate these key pixels. In addition, the two feature maps mentioned above are used to locate the most discriminative areas of the face. Then, half of them are used as gates to limit the weights of feature points, and the other half are used as features. Sigmoid and relu are used as activation functions to limit their results, respectively. Finally, a point-to-point matrix multiplication operation is performed to obtain the following formula:

[0026]

[0027] Output y,x Represents the output of gated convolution, and the sigmoid expression is: The Relu expression is: Relu(x) = max(0,x), C represents the mapping dimension, W represents the model parameter; the fusion layer adaptive feature extraction module has two branches of input F mixture The output of the composite feature extraction module, F residual Represents the output of the residual feature extraction module, which comes from the previous composite feature extraction and residual feature extraction modules respectively.

[0028] Beneficial effects of the present invention:

[0029] Compared with the prior art, the cross-dimensional image fusion adaptive deep fake identification method of the present invention has shown significant beneficial effects in the field of image processing and computer vision, which are specifically reflected in the following aspects:

[0030] Enhance the accuracy of color pattern recognition: By cleverly combining the characteristics of the spatiotemporal domain and the frequency domain, the present invention can capture the color distribution pattern in the image more comprehensively and meticulously. Directly analyzing the color arrangement in the spatial domain, while using shallow frequency domain features in the frequency domain to capture the basic texture and color contour of the image, and using deep frequency domain features to mine deeper color change information from the self-injury difference map, this cross-domain extraction strategy significantly improves the accuracy and meticulousness of color pattern recognition.

[0031] Improve the comprehensiveness and robustness of feature extraction: The innovation of this invention lies in extracting shallow frequency domain and deep frequency domain features from the original input image and the self-injury difference map respectively. This dual path design not only enriches the diversity of feature sets, but also enhances the system's adaptability to changes in different types of images. Whether it is the overall hue change of the image or the local subtle difference, it can be effectively captured and converted into feature information that is valuable for classification decisions, thereby improving the robustness and generalization ability of the entire system.

[0032] Optimizing feature fusion strategy: The introduced adaptive feature fusion module is a highlight of the present invention. This module can automatically adjust the weights of different branch features according to the characteristics of the input data to achieve the optimal combination of features. This dynamic fusion mechanism not only ensures the dominant position of important features in classification decisions, but also promotes the complementary effect between features, further improving the accuracy and efficiency of classification.

[0033] Wide applicability and practicality: The present invention has been fully verified on multiple large-scale data sets, and the experimental results consistently show its excellent performance. This means that the method is not only applicable to standard image processing tasks, but can also be flexibly applied to more complex and varied practical application scenarios, such as medical image analysis, remote sensing image recognition, security monitoring and other fields, showing strong practical application potential and value.

[0034] Promoting technological progress and industrial upgrading: The successful application of this invention not only provides a new idea and method for image processing technology, but also is expected to promote technological progress and industrial upgrading in related fields. By improving the accuracy and efficiency of image analysis, it provides strong support for the development of cutting-edge technologies such as artificial intelligence and machine vision, thereby promoting the rapid development of industries such as intelligent manufacturing, smart cities, and smart medical care.

[0035] In summary, the present invention has shown significant beneficial effects in improving color pattern recognition capabilities, enhancing the comprehensiveness and robustness of feature extraction, optimizing feature fusion strategies, expanding application areas, and promoting technological progress, and has important scientific research value and socio-economic value. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below in combination with the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0037] Figure 1 It is a principle block diagram of the overall algorithm of the present invention;

[0038] Figure 2 A comparison diagram of color and frequency domain features of a real image and a forged image of the present invention;

[0039] Figure 3 is a schematic diagram of a filter for extracting a residual graph of the present invention;

[0040] Figure 4 It is a schematic diagram of the adaptive fusion module of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Figure 1-4 The cross-dimensional shadow fusion adaptive deep fake identification method is further explained.

[0042] like Figure 1 As shown, a cross-dimensional shadow fusion adaptive deep fake identification method includes the following steps:

[0043] It includes a spatial domain feature extraction part and two frequency domain feature extraction modules, and uses an adaptive cross-domain fusion module to fuse the features of each part; it includes the following steps:

[0044] S1, extracting image color features that can represent the overall pattern of the original image from the spatial domain;

[0045] S2, two different frequency domain features are extracted. One is a shallow low-frequency feature, which is directly extracted from the original input image and used to expand the original RGB image to the frequency domain; the other is a deep frequency domain feature, which is extracted from the input image residual map and uses a convolution extractor to extract high-frequency features. The residual map attenuates the smooth part of the image and emphasizes the edge information of the image.

[0046] S3. An adaptive feature fusion module composed of gated convolution is used to fuse shallow low-frequency features and deep frequency domain features, so that the features of different branches can be fully expressed in the classification stage.

[0047] The deepfake methods proposed in recent years have very strict image post-processing engineering, including blurring and pixel interpolation of blended boundaries, making the fusion boundaries almost invisible. Especially in the RGB color domain, which is the spatial domain directly observed by the human eye, most deepfake detection methods focus on this area. However, Figure 2As shown, the present invention finds that although there is almost no difference between the real image and the fake image in the color domain, there are significant differences in their statistical characteristics when they are mapped to the frequency domain, especially when the high-frequency region reflects the fusion information of the image tampering. Therefore, the face forgery detection method of this paper focuses on the fusion of color domain and frequency domain features. The present invention uses a filter with dynamic characteristics to extract the frequency domain features of the input image, and then uses a combination of three filters to obtain the image residual map, and then extracts the deep frequency domain features. In addition, the present invention also uses an adaptive feature fusion module composed of gated convolution to fuse the two features.

[0048] Specifically, step S1 is as follows: In the feature extraction stage, the model is divided into two branches, a shallow composite feature extraction branch and a deep frequency domain feature extraction branch. The model uses the ResNet network as the basic backbone.

[0049] First, a convolutional neural network is used to extract the color domain features of the image, and Dobesy wavelet transform is selected as the filtering mode. Dobesy wavelet transform uses a short time window to analyze long signals and is strictly supported by positive cross scaling to adapt to local and smooth changes.

[0050] The Dobesy wavelet transform consists of an integer L and a series of wavelet filter coefficients {a i , i=0,1,2,...,L-1}, and the following two relations are stipulated:

[0051]

[0052] Where ρ(x) is the amplitude function, ψ(x) is the mother function of wavelet transform, L is the order of Dobesy wavelet transform, and x is the step size in the time domain. As shown in the formula, there are L orthogonal conditions in the wavelet transform, and the wavelet coefficient a is obtained. l , the frequency domain expansion A(w) with orthogonality can be expressed as:

[0053]

[0054] Where C is a constant term, and the fourth-order coefficient h(l) can be obtained as follows:

[0055]

[0056] After fusing the Dobesy-Baud symbol and color features, they are fed into a convolutional neural network to obtain a composite feature map. In the composite stage, simple size stitching is used.

[0057] The residual image is used as the basic input, the residual image is obtained through the filter, and the deep frequency domain information is obtained by the deep frequency domain feature extraction method. The deep frequency domain and shallow frequency domain features mentioned in the present invention should be distinguished from the high frequency and low frequency features. The former is divided according to the hidden space level position of the feature extraction, while the latter is distinguished according to the frequency band range.

[0058] The frequency domain features extracted from the original image combined with the color can reflect the facial details, while the deep frequency domain features extracted from the residual image reflect the edge information of the facial image.

[0059] Firstly, a residual map reflecting the mixed boundary information is extracted, the residual map is mapped from the original narrow channel information to a high-dimensional space, and a neural network is used to extract deep frequency domain features.

[0060] First, the present invention uses Figure 3 The filters shown are used to capture different types of dependencies between neighboring pixels. The advantage of using a residual map is that the image content is greatly compressed, resulting in a more compatible and more general image representation. The following formula represents the noisy residual:

[0061]

[0062] Among them, X ij It represents the pixel value of the pixel point currently calculated by the present invention, N ij represents the neighborhood pixels, It is defined in N ij cX ij The predicted value. c is the residual order, which is a variable and changes according to the filter type. The three selected filters are: a local linear predictor, a 3*3 translation-invariant linear pixel predictor, and a filter based on the NelderNead ​​algorithm with a kernel of 5*5. Experiments show that the combination of these three filters has the best performance for the face task of the present invention.

[0063] In addition, the present invention truncates the calculated residual value to limit the residual range, and quantizes to make the residual more sensitive to image edges and discontinuities. Based on the kernel size, the present invention selects quantization coefficients of 4, 12, and 2. The present invention uses HardTanh to perform truncation calculations to constrain the residual between 0 and 1.

[0064] Among them, the shallow low-frequency features and deep frequency domain features have the same output dimension, which is assumed to be C*H*W. In order to fuse the two parts and achieve higher deepfake detection performance, the present invention proposes an adaptive feature fusion module with the following structure: Figure 4 shown.

[0065] Gated convolution is used to ensure the adaptive nature of the fusion module. In gated convolution, a learnable dynamic feature selection mechanism is used to select all positions of each layer of each channel, while traditional convolution only considers each pixel as equivalent. In forgery detection, the generated image is subjected to a series of preprocessing such as cropping and affine transformation. In the end, the image is represented by the facial area occupying the vast majority (more than 90%) of the image pixel area. In addition, regardless of whether the present invention is adjacent in the spatial domain or the frequency domain, the visual artifacts are always fixed to a fixed number of adjacent pixels.

[0066] The present invention uses gated convolution to capture and locate these key pixels. In addition, the present invention also uses the above two feature maps to locate the most discriminative areas on the face. First, the C input channels are mapped to a 2C-dimensional feature hidden space, and then half of them are used as gates to limit the weights of feature points, and the other half are used as features. Sigmoid and relu are used as activation functions to limit their results, respectively. Finally, a point-to-point matrix multiplication operation is performed to obtain the following formula:

[0067]

[0068] Output y,x Represents the output of gated convolution, and the sigmoid expression is: The Relu expression is: ReLu(x) = max(0,x), C represents the mapping dimension, and W represents the model parameter. The fusion layer adaptive feature extraction module has two branches of input F as shown in the figure. mixture represents the output of the composite feature extraction module, F residual Represents the output of the residual feature extraction module, which come from the previous composite feature extraction and residual feature extraction modules respectively.

[0069] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A cross-dimensional image fusion adaptive deep fake identification method, which uses a spatial domain feature extraction module and two frequency domain feature extraction modules, and uses an adaptive cross-domain fusion module to fuse the features of each module; It is characterized in that Here are the steps: S1, extracting image color features that can represent the overall pattern of the original image from the spatial domain; The feature extraction stage is divided into two branches, shallow composite feature extraction branch and deep frequency domain feature extraction branch. The model uses the ResNet network as the basic backbone; S2. Take two different frequency domain features, one is shallow low-frequency features, which are directly extracted from the original input image through shallow 2 / 5 layer low-frequency features, and the original RGB image is expanded to the frequency domain through shallow low-frequency features; the other is deep frequency domain features, extracting the residual map of the input image, using the convolution extractor to extract high-frequency features, the residual transformation map of the image represents the fusion edge information of the face in the original image, and reducing the influence of chromatic aberration of the smooth part of the image; the residual map is used as the basic input, the residual map is obtained through the filter, and the deep frequency domain feature extraction method is used to obtain the deep frequency domain information; S3. An adaptive feature fusion module composed of gated convolution is used to fuse shallow low-frequency features and deep frequency domain features, so that the features of different branches can be fully expressed in the classification stage.

2. The cross-dimensional shadow-fusion adaptive deep fake identification method according to claim 1, It is characterized in that Step S1 is as follows: Firstly, the color gamut features of the image are extracted using a convolutional neural network, and Dobesy wavelet transform is selected as the filtering method. Dobesy wavelet transform uses a short-time window to analyze long signals and adapts to the local smooth changes of the image by finding a tightly supported orthogonal point to achieve the purpose of scaling. The Dobesi wavelet transform consists of an integer L and a series of wavelet filter coefficients {a i , i=0,1,2,...,L-1}, and the following two relations are stipulated: Where ρ(x) is the amplitude function, ψ(x) is the mother function of wavelet transform, L is the order of Dobesy wavelet transform, and x is the step size in the time domain. As shown in the formula, there are L orthogonal conditions in the wavelet transform, and the wavelet coefficient a is obtained. l , the frequency domain expansion A(w) with orthogonality can be expressed as: Where C is a constant term, and the fourth-order coefficient h(l) can be obtained as follows: After the Dobesy-Baud symbol and color features are fused, they are input into the convolutional neural network to obtain a composite feature map; in the composite stage, simple size stitching is used.

3. The cross-dimensional shadow-fusion adaptive deep fake identification method according to claim 1, It is characterized in that The frequency domain features extracted from the original image combined with the color can reflect the facial details, while the deep frequency domain features extracted from the residual image reflect the edge information of the facial image; First, a residual map reflecting the mixed boundary information is extracted, and the residual map is mapped from the original narrow channel information to a high-dimensional space, and a neural network is used to extract deep frequency domain features; The noise residual is expressed as follows: Among them, X ij It represents the pixel value of the currently calculated pixel point, N ij Represents the neighborhood pixels, which are defined in N ij cX ij The predicted value; c is the residual order, which is a variable and changes according to the filter type. The three selected filters are: local linear predictor, 3*3 translation-invariant linear pixel predictor, and a filter based on the Nelder-Nead algorithm with a kernel of 5*5.

4. The cross-dimensional shadow-fusion adaptive deep fake identification method according to claim 1, It is characterized in that The two feature maps of shallow low-frequency features and deep frequency domain features have the same output dimension. Assuming that the dimensions are C*H*W, gated convolution is used to ensure the adaptive nature of the fusion module. In gated convolution, a learnable dynamic feature selection mechanism is adopted, and all positions of each layer of each channel are sorted one by one. In the spatial domain and frequency domain, visual artifacts are always fixed on a fixed number of adjacent pixels. Gated convolution is used to capture and locate these key pixels. In addition, the two feature maps mentioned above are used to locate the most discriminative areas in the face. Then, half of them are used as gates to limit the weights of feature points, and the other half are used as features; Sigmoid and relu are used as activation functions to limit their results; finally, a point-to-point matrix multiplication operation is performed to obtain the following formula: Output y,x Represents the output of gated convolution, and the sigmoid expression is: The Relu expression is: Relu(x) = max(0,x), C represents the mapping dimension, W represents the model parameter; the fusion layer adaptive feature extraction module has two branches of input F mixture The output of the composite feature extraction module, F residual Represents the output of the residual feature extraction module, which comes from the previous composite feature extraction and residual feature extraction modules respectively.

Images