A face forgery detection method based on prior collaboration

Abstract

The present invention discloses a face forgery detection method based on prior collaboration, which belongs to the field of image recognition of artificial intelligence. It combines frequency domain, structured information and spatial distance multidimensional prior knowledge to work together to complete face forgery detection. The method specifically includes the following steps: Step 1, obtain a face image, and use a resonance-based frequency dynamic tuning method to enhance and extract the frequency domain modal information of the face image; Step 2, use a structure compression aggregation network to extract and fuse multimodal features with structural information; Step 3, use a ViT network based on a hypersurface attention mechanism to model long-range dependencies for multimodal features and extract deep features with deep semantic information; Step 4, use an attention mechanism to aggregate the deep features, and obtain the detection result of the input image through a classifier containing a fully connected layer. The present invention can efficiently and accurately detect face images created by forgery.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition based on artificial intelligence, and in particular relates to a method for detecting forged faces based on prior collaboration. Background Art

[0002] Face forgery refers to the use of deep learning or computer vision technology to generate realistic fake face images or videos. Early face forgery detection methods used classic convolutional neural networks (such as ResNet50) to extract spatial features from images, thereby achieving accurate detection of face forgeries. However, with the rapid development of deep generative models, these methods have become difficult to handle improved realistic deep forgeries. Some studies have enhanced the network's learning ability by designing more complex network structures to obtain more universal forgery features. However, these technologies mainly face the following problems in practical applications:

[0003] First, images and videos uploaded to social media are often compressed, which not only blurs many forged features in the image but also introduces new, confusing features, disrupting the detection model's discriminative ability. Therefore, existing technologies are still insufficient in dealing with compression. Second, because forgeries can occur in any area of ​​the face, traditional methods typically focus on extracting a single forged texture. This limitation makes it difficult for detection models to cope with diverse forgeries. Third, the complex network structure of existing methods makes it difficult to effectively deploy them on resource-constrained mobile devices such as smartphones. Summary of the Invention

[0004] In order to solve the above problems, the present invention proposes a face forgery detection method based on prior collaboration, which combines multi-dimensional prior knowledge such as frequency domain, structured information and spatial distance to work together, and uses efficient feature extraction and aggregation technology to enhance the accuracy and robustness of forgery detection, while reducing the computational complexity, and can operate efficiently in resource-limited environments.

[0005] The technical solutions of the present invention are as follows:

[0006] A face forgery detection method based on prior collaboration combines frequency domain, structured information, and multi-dimensional spatial distance prior knowledge to work together to complete face forgery detection. The method specifically includes the following steps:

[0007] Step 1: Acquire a facial image and use a resonance-based frequency dynamic tuning method to enhance and extract frequency domain modal information of the facial image;

[0008] Step 2: Use the structure compression aggregation network to extract and fuse multimodal features with structural information;

[0009] Step 3: Use the ViT network based on the hypersurface attention mechanism to model long-range dependencies for multimodal features and extract deep features with deep semantic information;

[0010] Step 4: Use the attention mechanism to aggregate the deep features and obtain the detection results of the input image through a classifier containing a fully connected layer.

[0011] Furthermore, in step 1, the resonance-based frequency dynamic tuning method uses a Riemann resonance filter and an orthogonal divergence filter, and the specific process is as follows:

[0012] Step 1.1: Use the Riemann resonance filter to amplify the edge information of the face image and suppress the aliasing features. The specific process is as follows:

[0013] Step 1.1.1. The Riemann resonance filter uses the Riemann resonance operator as the filter kernel and provides high-frequency oscillation weights at different positions of the filter kernel to achieve efficient filtering of the feature map pixel values. The weight function of the Riemann resonance operator is defined as:

[0014]

[0015] Where R(·) is the weight function; r1 and r2 are different independent variables of the weight function;

[0016] Step 1.1.2: Use the Riemann resonance filter based on the Riemann resonance operator to amplify the edges of the input deep fake face image:

[0017]

[0018] Where I is the input deep fake face image; I r is a deep fake face image processed by the Riemann resonance filter; i and j are the indexes of the face image in the horizontal and vertical directions respectively; m0 and m1 are the indexes of the Riemann resonance operator in the horizontal and vertical directions respectively;

[0019] Step 1.1.3: Normalize and suppress confusing features:

[0020]

[0021] Among them, I sr is a normalized image; max(·) and min(·) represent the maximum and minimum operations respectively;

[0022] Step 1.2: Use an orthogonal divergence filter to obtain the high-frequency information of the image; the specific process is:

[0023] Step 1.2.1, calculate the orthogonal divergence kernel; the original function of the orthogonal divergence kernel is the Laplace operator, which is:

[0024]

[0025] in, is the Laplace operator; f is the Laplace function; f(x,y) is the Laplace function at coordinate (x,y); x and y are the indices of the Laplace function in the horizontal and vertical directions respectively;

[0026] The Laplace operators in two directions are approximated by the finite difference method to obtain the orthogonal divergent kernel, which is calculated as follows:

[0027]

[0028] Among them, f x (x,y) and f y (x,y) are the second-order partial derivatives of the Laplace function with respect to x and y respectively;

[0029] Step 1.2.2: Calculate the high-frequency feature map to generate a comprehensive high-pass response image; the calculation formula is:

[0030]

[0031] I f =I x +I y ;

[0032] Among them, I x and I y Respectively represent I sr Under the action of the orthogonal divergent filter, the high-frequency image is obtained along the horizontal and vertical directions; c0 and c1 are the indexes in the horizontal and vertical directions, respectively, used to traverse the horizontal divergent kernel L x and vertically divergent kernel L y ;I f Represents an image with high-frequency information obtained by a variant of the Laplacian operator.

[0033] Furthermore, the specific process of step 2 is as follows:

[0034] Step 2.1: Input deep fake face image I and image with high frequency information I f Embedded into the graph structure; I and I f The embedding network of adopts the same network structure design, but the parameters are independent; the specific process is as follows:

[0035] Step 2.1.1. Use the convolution module to extract features from the image and generate spatial and frequency domain features respectively. Each convolution module consists of a convolution layer, a batch normalization layer, and an activation function.

[0036] F s =ReLU(BN(Conv(I)));

[0037] F f =ReLU(BN(Conv(I f )));

[0038] Among them, F s 、F f are spatial domain features and frequency domain features respectively; ReLU(·) is the ReLU activation function; BN(·) is the batch normalization layer; Conv(·) is the convolution layer in the convolution module;

[0039] Step 2.1.2: Map the spatial domain features and frequency domain features into the graph structure respectively. The specific process is as follows:

[0040] First, the spatial domain features and frequency domain features are converted into graph nodes respectively. The specific calculation formula is:

[0041]

[0042] in, are spatial domain nodes and frequency domain nodes respectively; Flatten(·) is the flattening operation; Conv2D(·) is the convolution layer in the graph structure;

[0043] Next, generate the adjacency matrix for the graph structure; the calculation formula of the adjacency matrix is:

[0044]

[0045] Among them, E s and E f Represents airspace nodes and frequency domain nodes The adjacency matrix of s and D f They are (A s +T) and (A f +T); w and h are respectively (A s +T) or (A f +T) the number of rows and columns; N represents the number of nodes in the airspace or frequency domain node The number of s and A f are different random number matrices; T is the identity matrix; Diag(·) represents a diagonal matrix;

[0046] Finally, we get the airspace map and frequency domain plots

[0047] Step 2.2: Fusion of spatial domain nodes and frequency domain nodes; use graph neural network to model the spatial domain graph G s and frequency domain graph G f The complex structural information of , the specific process is:

[0048]

[0049] in, They represent the spatial domain nodes and frequency domain nodes after the k+1th layer of graph convolution respectively; They represent the spatial domain nodes and frequency domain nodes after the k-th layer of graph convolution respectively; w k and b k Represent the weight and offset of the node mapping parameters after the k-th layer graph convolution; BN k represents the batch normalization layer corresponding to the k-th layer of graph convolution;

[0050] After completing the feature extraction of the spatial domain graph and the frequency domain graph, the two features are initially fused through the connection operation to obtain the initial fusion feature. Then, the spatial domain features and the frequency domain features are deeply fused through graph convolution. The calculation formula is:

[0051]

[0052] V k+1 =ReLU(BN k (E s +E f )·(W k ·V k +b k ))),k=0,1;

[0053] Among them, V 0 is a fusion node; concat(·) is a connection operation; Respectively represent the spatial domain nodes and frequency domain nodes after the second layer of graph convolution; V k+1 and V k They represent the multimodal nodes after the k+1th layer of graph convolution and the kth layer of graph convolution respectively;

[0054] Finally, we get a multimodal graph G(V 2 ,E s +E f );

[0055] Step 2.3: Use the topological association aggregation mechanism to aggregate the multimodal graph. The specific process is as follows:

[0056] Step 2.3.1: Aggregate the multimodal nodes to obtain the aggregated multimodal feature V′. The calculation formula is:

[0057] V′=AvgPool(V 2 );

[0058] Among them, V 2 Represents the multimodal node before aggregation; AvgPool; (·) represents the average pooling operation;

[0059] Step 2.3.2: Aggregate the adjacency matrix to obtain the aggregated adjacency matrix. The specific process is as follows:

[0060] Define the minimum coordinate of the original adjacency matrix corresponding to the point with coordinate (u,v) in the aggregated adjacency matrix as (u base ,v base ), the calculation formula is:

[0061]

[0062] Among them, u base (·) and v base (·) represents the horizontal starting index and vertical starting index of the node set mapped by the aggregation window; u and v are the horizontal and vertical indices respectively; H0 and H1 are the horizontal length and vertical length of the aggregation window respectively; is the rounding down operation; % is the modulo operation;

[0063] Calculate (u base ,v base ) is the mathematical expectation of the original adjacency matrix area with the minimum coordinate, and the edges of the original adjacency matrix area are comprehensively counted to obtain the aggregated adjacency matrix; the calculation formula is:

[0064]

[0065] P0,P1~Uniform(0,H1-1);

[0066] Q0,Q1~Uniform(0,H2-1);

[0067] Where E is the original adjacency matrix; the edges of the corresponding area in E are comprehensively counted to obtain the aggregated adjacency matrix E′; E′ uv is the element in row u and column v of E′, is the mathematical expectation; Uniform(0,H1-1) and Uniform(0,H2-1) represent random variables uniformly distributed on the interval [0,H1-1] and the interval [0,H2-1] respectively; P0 and P1 are both values ​​randomly selected from the uniform distribution Uniform(0,H1-1); Q0 and Q1 are both values ​​randomly selected from the uniform distribution Uniform(0,H2-1).

[0068] Furthermore, the specific process of step 3 is as follows:

[0069] Step 3.1: Convert the multimodal features and the aggregated adjacency matrix into the input of the ViT network.

[0070] First, the aggregated adjacency matrix is ​​concatenated with its mean, and the position encoding vector is obtained through linear mapping. The calculation formula is:

[0071] pos=Linear(concat(Mean(E′),E′));

[0072] Among them, pos is the position encoding vector; Mean(·) and concat(·) are the average operation and splicing operation respectively; Linear(·) is the linear mapping operation;

[0073] The multimodal feature V′ is directly used as the local token of the subsequent ViT network input, and the mean of the multimodal feature V′ is used as the global token of the subsequent ViT network input; the position encoding vector pos is added to the local token and the global token, and the calculation formula is:

[0074] F emb =concat(Mean(V′),V′)+pos;

[0075] Among them, F emb It is the input feature of the subsequent ViT;

[0076] Step 3.2: Calculate the attention attenuation matrix. The calculation formula is:

[0077]

[0078] in, is the value of the attention decay matrix in row t1 and column t2, where row t1 corresponds to token t1 and column t2 corresponds to token t2; represents the Euclidean distance between token t1 and token t2; d m is the maximum Euclidean distance between tokens; cos(·) is the cosine operation; are the horizontal and vertical coordinates of token t1 respectively; are the horizontal and vertical coordinates of token t2 respectively;

[0079] Step 3.3, the ViT network contains several layers of ViT blocks, and the calculation formula is:

[0080]

[0081] z l =MLP l (LN l (z′ l ))+z′ l ,l=1,2,…,L;

[0082] Among them, A l is the attention score obtained by using the hypersurface attention mechanism for the l-th layer of ViT block; Softmax(·) is the Softmax activation function; L is the maximum number of layers of ViT block; and denote the query matrix, key matrix and value matrix of the ViT block at layer l respectively; is the transposed symbol; D is the attention attenuation matrix; z l ′ represents the intermediate output feature of the ViT block at layer l; z l is the final output feature of the ViT block in the lth layer; LN l and MLP l Represent the linear normalization layer and multi-layer perceptron of the l-th layer ViT block respectively;

[0083] The final extracted deep feature F with deep semantic information d Calculated as:

[0084] F d =LN L (z L );

[0085] Among them, LN L is the linear normalization layer of the Lth layer ViT block; z L is the final output feature of the L-th layer ViT block.

[0086] Furthermore, the specific process of step 4 is as follows:

[0087] Step 4.1: Use the attention mechanism to aggregate local tokens and concatenate them with global tokens to obtain multi-scale features. The specific process is as follows:

[0088] Step 4.1.1. Calculate the attention score of the local token:

[0089] A local =Softmax(Linear(F d [1:]));

[0090] Among them, A local is the attention score of the local token; Linear(·) is the feature mapping operation;

[0091] Step 4.1.2, calculate the final local feature F local :

[0092] F local =A local ·F d ;

[0093] Step 4.1.3, calculate F local Concatenate with the global token in the feature dimension to obtain the final multi-scale feature F ms , the calculation formula is:

[0094] F ms =concat(F local ,F d [0]);

[0095] Step 4.2: Use a classifier containing a fully connected layer to classify the multi-scale features and classify the results as real or fake. The specific process is as follows:

[0096] Step 4.2.1. Use the fully connected layer to calculate the unnormalized scores of multi-scale features:

[0097]

[0098] in, Indicates the classification category, represents the forged category, represents the true category; Representation category The normalized score of Representation category The corresponding weight matrix of the fully connected layer; Representation category The bias of the corresponding fully connected layer;

[0099] Step 4.2.2: Use the Softmax activation function to convert the unnormalized scores into corresponding probabilities, and select the category with the highest probability as the output category determined by the model. The probability calculation formula of the Softmax activation function is as follows:

[0100]

[0101] Among them, P represents the probability of different categories; are the normalized scores of the fake and real categories respectively.

[0102] The beneficial technical effects brought about by the present invention are:

[0103] (1) Achieves efficient and comprehensive frequency domain modal capture. By introducing a resonance-based frequency dynamic tuning method, the high-frequency information of the image is effectively enhanced and extracted, significantly compensating for the forged features lost or blurred during the image compression process. It also effectively filters out the confusing features introduced by the compression process, more accurately capturing the frequency domain modal information related to face forgery.

[0104] (2) Improved ability to capture multi-scale forgery features. By introducing a structured compression aggregation network and a hypersurface-based attention mechanism, we effectively model the structural information and long-range dependencies between features to capture forgery features at different scales. This design significantly improves the model's sensitivity to forgery traces of different sizes and locations, thereby enabling comprehensive detection of complex forgery scenarios.

[0105] (3) Achieves lightweight, high-precision detection of face forgeries. This invention integrates prior knowledge from multiple fields (frequency domain, structured, and spatial distance), enabling the model to achieve high-precision, lightweight face forgery detection without relying on complex network structures. This is particularly suitable for resource-constrained mobile devices and practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] Figure 1 This is a flow chart of the face forgery detection method based on prior collaboration of the present invention.

[0107] Figure 2 Schematic diagram of the discrete form of the Riemann resonance operator and the orthogonal divergence kernel in the present invention.

[0108] Figure 3 This is the ROC curve of the present invention on the FaceForensics++ dataset C23 version.

[0109] Figure 4 This is the ROC curve of the present invention on the FaceForensics++ dataset C40 version. DETAILED DESCRIPTION

[0110] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0111] The present invention extracts and enhances effective high-frequency information related to forgery information in facial images by designing a resonance-based frequency dynamic tuning method. Then, a structural compression aggregation network is used to deeply fuse and compress the spatial and frequency domain features to obtain multimodal features with structural information. The multimodal features are modeled through a ViT network based on a hypersurface attention mechanism to model the dependencies between different features and extract global and local features at the same time. Finally, all features are aggregated and a linear classification layer is used to obtain the prediction result of the input image. The present invention deeply mines forgery traces at different positions and scales, and adopts a lightweight design. While improving the accuracy and robustness of facial forgery detection, it can be effectively deployed on devices with limited computing resources.

[0112] like Figure 1 As shown, a face forgery detection method based on prior collaboration includes the following steps:

[0113] Step 1: Obtain a facial image and use a resonance-based frequency dynamic tuning method to enhance and extract frequency domain modal information. The resonance-based frequency dynamic tuning method uses a Riemann resonance filter and an orthogonal divergence filter to amplify the structural details produced by the face forgery. The specific process is as follows:

[0114] Step 1.1: Use the Riemann resonance filter to amplify the edge information of the face image and suppress the confusing features.

[0115] The Riemann function is a periodic function with a specific frequency response characteristic. When the input of the Riemann function approaches zero, its value will oscillate rapidly. This characteristic manifests itself as a dynamic adjustment of the high-frequency components during the filtering process, achieving adaptive matching of the input and playing a role in resonance selection. The Riemann resonance filter proposed in the present invention uses the Riemann resonance operator as the filter kernel, and provides high-frequency oscillation weights at different positions of the filter kernel to achieve efficient filtering processing of the feature map pixel values. The weight function of the Riemann resonance operator is defined as:

[0116]

[0117] Where R(·) is the weight function; r1 and r2 are different independent variables of the weight function.

[0118] To reduce storage space, compression algorithms tend to discard high-frequency details while retaining low-frequency information, which may lead to the enhancement of smooth areas or irrelevant features. The periodicity of the Riemann resonance operator amplifies the high-frequency components of the feature map, helping to emphasize high-frequency details such as edges and textures in the image. In addition, the nonlinear response of the Riemann resonance operator to the eigenvalues ​​suppresses the features in the low-frequency area and suppresses the confusing features introduced during the compression process. A Riemann resonance filter based on the Riemann resonance operator is used to amplify the edges of the input deep fake face image:

[0119]

[0120] Where I is the input deep fake face image; I r is a deep fake face image processed by the Riemann resonance filter; i and j are the indexes of the face image in the horizontal and vertical directions respectively; m0 and m1 are the indexes of the Riemann resonance operator in the horizontal and vertical directions respectively, which are used to traverse the Riemann resonance operator; I(m0+i,m1+j) represents the pixel value of I at the coordinate (m0+i,m1+j); I r (i,j) represents I r The pixel value at coordinate (i, j).

[0121] like Figure 2 As shown, the Riemann resonance operator will have a larger response in the area where the input is close to zero, and may generate high-frequency oscillating eigenvalues. In order to avoid the problem of high-amplitude noise introduced by nonlinear oscillation characteristics, which leads to an excessively large distribution range, the present invention combines the normalization step to smooth the eigenvalue distribution, suppress possible abnormal responses, make the filtered image more able to reflect the real edge details, and reduce the negative impact of noise interference on the model. Normalized image I sr Defined as:

[0122]

[0123] Among them, max(·) and min(·) represent the maximum value and minimum value operations respectively.

[0124] Step 1.2: Use an orthogonal divergence filter to obtain the high-frequency information of the image. The specific steps are:

[0125] Step 1.2.1. Calculate the orthogonal divergence kernel.

[0126] The original function of the orthogonal divergence kernel is the two-dimensional Laplace operator, which is defined from the sum of second-order derivatives. In two-dimensional continuous space, the Laplace operator can be written as:

[0127]

[0128] in, is the Laplace operator; f is the Laplace function; f(x,y) is the Laplace function at coordinates (x,y); x and y are the indices of the Laplace function in the horizontal and vertical directions, respectively.

[0129] The design of high-pass filters usually utilizes the isotropic property and the principle of edge enhancement. The response of the filter is independent of direction and depends only on the relative relationship between the pixel position and its neighborhood. The Laplace operator can be regarded as being independent of the direction of the neighborhood around the pixel. The present invention decomposes the Laplace operator into two independent operators in the horizontal and vertical directions, which are applied to the horizontal and vertical directions respectively to detect and enhance high-frequency components. In image processing, the Laplace operator in two directions is approximated by the finite difference method, and an orthogonal divergent kernel can be obtained. The calculation formula is:

[0130]

[0131] Among them, f x (x,y) and f y (x,y) are the second-order partial derivatives of the Laplace function with respect to x and y, respectively.

[0132] The orthogonal divergence kernel measures how strongly the Laplace function at coordinate (x,y) changes relative to its one-dimensional neighborhood. The orthogonal divergence kernel is isotropic because its calculation does not depend on the direction, but only on the local distribution of pixel values.

[0133] Step 1.2.2: Calculate the high-frequency feature map to generate a comprehensive high-pass response image.

[0134] like Figure 2 As shown, the convolution kernel corresponding to the orthogonal divergent kernel is the horizontal divergent kernel L x and vertically divergent kernel L y . x and L y Respectively with I sr Convolution is performed in both horizontal and vertical directions to independently extract the horizontal and vertical changes in the image. Finally, the linear superposition method is used to obtain the final image with high-frequency information. The calculation formula is:

[0135]

[0136] I f =I x +I y ;

[0137] Among them, I x and I y Respectively represent I sr Under the action of the orthogonal divergent filter, the high-frequency image is obtained along the horizontal and vertical directions; c0 and c1 are the indexes in the horizontal and vertical directions, respectively, used to traverse the horizontal divergent kernel L x and vertically divergent kernel L y ;I x (i, j) and I y(i,j) represent I x and I y The pixel value at coordinate (i, j); I sr (c0+1,c1+1) represents I sr The pixel value at coordinate (c0+1,c1+1); L x (i+c0,j+c1) and L y (i+c0,j+c1) represent L x and L y The pixel value at coordinate (i+c0,j+c1); I f Represents an image with high-frequency information obtained by a variant of the Laplacian operator.

[0138] Step 2: Use a structured compression aggregation network to extract and fuse multimodal features with structural information. By embedding spatial and frequency domain features into the graph structure, it helps to handle complex relationships and structures. In addition, a topological association aggregation mechanism is used to sparsely represent nodes and edges, reducing computational complexity and highlighting important node features. The specific process is as follows:

[0139] Step 2.1: Input deep fake face image I and image with high frequency information I f Embedded into the graph structure. I and I f The embedding network of adopts the same network structure design, but the parameters are independent. The specific process is as follows:

[0140] Step 2.1.1: Use a convolutional module to extract features from the image, generating spatial and frequency domain features. Each convolutional module consists of a convolutional layer, a batch normalization layer, and an activation function. Residual connections are not introduced in the convolutional module to simplify the network structure and enhance the ability to extract local features.

[0141] F s =ReLU(BN(Conv(I)));

[0142] F f =ReLU(BN(Conv(I f )));

[0143] Among them, F s 、F f are spatial domain features and frequency domain features respectively; ReLU(·) is the ReLU activation function; BN(·) is the batch normalization layer; Conv(·) is the convolution layer in the convolution module.

[0144] Step 2.1.2: Map the spatial and frequency domain features into graph structures. Spatial and frequency domain features have different distributions and characteristics. By embedding them into independent graph structures, we can design specialized graph convolution operations for each feature, improve the pertinence of feature processing, and avoid information loss or confusion caused by direct fusion. The specific steps are as follows:

[0145] The spatial domain features and frequency domain features are converted into graph nodes respectively. The specific calculation formula is:

[0146]

[0147] in, are spatial domain nodes and frequency domain nodes respectively; Flatten(·) is the flattening operation; Conv2D(·) is the convolution layer in the graph structure;

[0148] Next, generate the adjacency matrix for the spatial domain nodes and frequency domain nodes. The calculation formula of the adjacency matrix is:

[0149]

[0150] Among them, E s and E f Represents airspace nodes and frequency domain nodes The adjacency matrix of s and D f They are (A s +T) and (A f +T) is used to balance the feature propagation between nodes; w and h are (A s +T) or (A f +T) the number of rows and columns; N represents the number of nodes in the airspace or frequency domain node The number of s and A f are different random number matrices consisting of 0 and 1; T is the identity matrix. Diag(·) represents a diagonal matrix whose diagonal elements consist of the vectors or values ​​in the brackets and the rest of the elements are zero.

[0151] Finally, we get the airspace map and frequency domain plots

[0152] Step 2.2: Fusion of spatial domain nodes and frequency domain nodes.

[0153] Graph structures have natural noise resistance. Through flexible edge weight modeling and feature aggregation, the interference of noise and redundant features on the model can be reduced. When processing compression, low resolution or image quality degradation, graph neural networks can extract forged features more stably. Therefore, the present invention uses graph neural networks to model the spatial graph G s and frequency domain graph G f The complex structural information of , the specific process is:

[0154]

[0155] in, Respectively represent the spatial domain nodes and frequency domain nodes after the k+1th layer of graph convolution; Respectively represent the spatial domain nodes and frequency domain nodes after the k-th layer of graph convolution; W k and b k Represent the weight and offset of the node mapping parameters after the k-th layer graph convolution. BN k Represents the batch normalization layer corresponding to the k-th layer of graph convolution.

[0156] Spatial domain features are more suitable for capturing local forgery details, while frequency domain features can identify global forgery traces. The combination of the two helps to deal with complex forgery scenarios and cover more forgery patterns. Therefore, after completing the feature extraction of the spatial domain graph and the frequency domain graph, the present invention performs a preliminary fusion of the two features through a concatenation operation to obtain the initial fusion feature. Then, the spatial domain features and the frequency domain features are deeply fused through graph convolution. The calculation formula is:

[0157]

[0158] V k+1 =ReLU9BN k (E s +E f )·(W k ·V k +b k ))),k=0,1;

[0159] Among them, V 0 is a fusion node; concat(·) is a connection operation; Respectively represent the spatial domain nodes and frequency domain nodes after the second layer of graph convolution; V k+1 and V k They represent the multimodal nodes after the k+1th layer of graph convolution and the kth layer of graph convolution respectively;

[0160] Finally, we get a multimodal graph G(V 2 ,E s +E f ).

[0161] Step 2.3: Use the topological association aggregation mechanism to 2 ,E s +E f ) for polymerization.

[0162] The present invention implements a graph aggregation mechanism based on locality, which uses the correlation of the graph topology structure to reflect the local characteristics within the region and achieves dimensionality reduction by reducing the number of nodes and aggregating information.

[0163] Step 2.3.1: Aggregate the multimodal nodes to obtain aggregated multimodal features. The graph nodes are composed of the convolutional features of the image. The spatial position of each node is designed to follow the two-dimensional grid layout of the image. The node index corresponds to the physical position of the image block. Therefore, the present invention uses aggregated nodes to represent the nodes of a specific area, which is equivalent to using the statistics of these features to represent the local area. The multimodal feature V′ obtained after aggregation is calculated as:

[0164] V′=AvgPool(V 2 );

[0165] Among them, V 2 represents the multimodal node before aggregation; AvgPool(·) represents the average pooling operation.

[0166] Step 2.3.2: Aggregate the adjacency matrix to obtain an aggregated adjacency matrix. Similar to node aggregation, for an aggregated region, the weights of all edges can reflect the connectivity and local structural characteristics of the region to a certain extent. The present invention uses the mathematical expectation of all edges in the window to summarize the edge characteristics of the local region. By considering the weights of all sub-edges, it is possible to avoid the excessive influence of a single edge on the result and provide more robust adjacency information. The specific calculation steps are:

[0167] Define the minimum coordinate of the original adjacency matrix corresponding to the point with coordinate (u,v) in the aggregated adjacency matrix as (u base ,v base ), the calculation formula is:

[0168]

[0169] Among them, u base (·) and v base (·) represents the horizontal starting index and vertical starting index of the node set mapped by the aggregation window; u and v are the horizontal and vertical indices respectively; H0 and H1 are the horizontal length and vertical length of the aggregation window respectively; is the rounding down operation; % is the modulo operation.

[0170] Calculate (u base ,v base) is the mathematical expectation of the original adjacency matrix area with the minimum coordinate, and the edges of the original adjacency matrix area are comprehensively counted to obtain the aggregated adjacency matrix; the calculation formula is:

[0171]

[0172] P0,P1~Uniform(0,H1-1);

[0173] Q0,Q1~Uniform(0,H2-1);

[0174] Among them, E is the original adjacency matrix; the edges of the corresponding area in E are comprehensively counted to obtain the aggregated adjacency matrix E′. The aggregated adjacency matrix can correctly reflect the connectivity and integrity of the corresponding area of ​​the original graph; E′ uv is the element in row u and column v of E′, is the mathematical expectation; Uniform(0,H1-1) and Uniform(0,H2-1) represent random variables uniformly distributed on the interval [0,H1-1] and the interval [0,H2-1] respectively; P0 and P1 are both values ​​randomly selected from the uniform distribution Uniform(0,H1-1); Q0 and Q1 are both values ​​randomly selected from the uniform distribution Uniform(0,H2-1).

[0175] By using the topological association aggregation mechanism to aggregate the fused nodes and adjacency matrix, the computational burden is further reduced, and the multimodal feature V′ that integrates the frequency domain and spatial domain information and the corresponding aggregated adjacency matrix E′ are obtained.

[0176] Step 3: Use the ViT network based on the hypersurface attention mechanism to model long-range spatial dependencies for multimodal features and extract deep features with deep semantic information.

[0177] This paper introduces a hypersurface attention mechanism, which adjusts the weights in the self-attention mechanism by considering the hypersurface structure of the high-dimensional space where facial features are located. The specific steps are as follows:

[0178] Step 3.1: Convert the multimodal features and the aggregated adjacency matrix into the input of the ViT network.

[0179] First, concatenate the aggregated adjacency matrix with its mean, and obtain the position encoding vector through linear mapping. The calculation formula is:

[0180] pos=Linear(concat(Mean(E′),E′));

[0181] Among them, pos is the position encoding vector; Mean(·) and concat(·) are the average operation and concatenation operation respectively; Linear(·) is the linear mapping operation used to map the adjacency matrix to the position vector.

[0182] Due to the similarity of the data structure, the multimodal feature V′ will be directly used as the local token of the subsequent ViT network input, and its mean will be used as the global token of the subsequent ViT network input. In addition, the present invention also adds a position encoding vector pos to the local token and the global token to provide them with position information. The calculation formula is:

[0183] F emb =concaat(Mean(V′),V′)+pos;

[0184] Among them, F emb is the input feature of the subsequent ViT network; Mean(·) and concat(·) are the average operation and connection operation respectively.

[0185] Step 3.2: Calculate the attention decay matrix.

[0186] This paper treats tokens as hypersurfaces embedded in a high-dimensional space. By introducing a nonlinear attenuation mechanism based on spatial distance, the attention weights between tokens are determined not only by their semantic similarity but also by their relative positions in space. In the image context, each token is a uniquely positioned two-dimensional coordinate in a plane, so the nonlinear attention attenuation matrix is ​​calculated as:

[0187]

[0188] in, is the value of the attention decay matrix in row t1 and column t2, where row t1 corresponds to token t1 and column t2 corresponds to token t2; represents the Euclidean distance between token t1 and token t2; d m is the maximum Euclidean distance between tokens; cos(·) is the cosine operation; are the horizontal and vertical coordinates of token t1 respectively; are the horizontal and vertical coordinates of token t2 respectively.

[0189] All local tokens share the attention decay matrix, and the attention decay matrix of the global token is always equal to 1. The attention decay matrix calculates the decay factor for each pair of tokens in space. This decay factor allows these tokens to still conform to the semantics and structure of the hypersurface under the representation of two-dimensional space, so as to better capture the nonlinear relationship and spatial dependency between features.

[0190] Step 3.3: Use the ViT network based on the hypersurface attention mechanism to model long-range dependencies between multimodal features, which helps reason about relationships between image content. The ViT network contains several layers of ViT blocks. The hypersurface attention mechanism can connect all tokens while ensuring that interactions between neighboring tokens are more prominent, avoiding the risk of overfitting long-range correlations. The calculation formula is:

[0191]

[0192] z l =MLP l (LN l (z l ′))+z l ′,l=1,2,…,L;

[0193] Among them, A l is the attention score obtained by using the hypersurface attention mechanism for the l-th layer ViT block; Softmax(·) is the Softmax activation function; L is the maximum number of layers of the ViT block, and in this invention, L=6. and They represent the query matrix, key matrix, and value matrix of the ViT block at the first layer respectively; T is the transpose symbol; D is the attention decay matrix; z l ′ represents the intermediate output feature of the ViT block at layer l; z l LN is the final output feature of the ViT block in the lth layer. l and MLP l Represent the linear normalization layer and multi-layer perceptron of the lth layer ViT block respectively. The final extracted deep feature F with deep semantic information d Calculated as:

[0194] F d =LN L (z L );

[0195] Among them, LN L is the linear normalization layer of the Lth layer ViT block; z L is the final output feature of the L-th layer ViT block.

[0196] Step 4: Use the attention mechanism to aggregate the deep features and concatenate them with the global tokens, and obtain the detection results of the input image through a classifier containing a fully connected layer. The specific process is:

[0197] Step 4.1: Use the attention mechanism to aggregate local tokens and concatenate them with global tokens to obtain multi-scale features.

[0198] Features at different scales may have different importance for a task. By generating an attention score matrix for local tokens, the contribution of features at different scales can be dynamically adjusted, thereby more specifically aggregating the most effective features. This mechanism helps improve the model's discriminative ability, especially when dealing with multi-scale features. The calculation process is as follows:

[0199] A local =Softmax(Linear(F d [1:]));

[0200] Among them, A local is the attention score of the local token; Linear(·) is the feature mapping operation.

[0201] Weighted summation is a lightweight and efficient fusion method that can avoid the feature redundancy problem that may be caused by direct connection and reduce the computational cost of the model. The final local feature F local The calculation process is as follows:

[0202] F local =A local F d ;

[0203] The ViT network models the long-range dependencies between tokens, where the global token has rich global information. local Concatenate with the global token in the feature dimension to obtain the final multi-scale feature F ms , the calculation formula is:

[0204] F ms =concat(F local ,F d [0]);

[0205] Step 4.2: Use a classifier containing a fully connected layer to discriminate and classify the multi-scale features, and the result is either real or fake. The present invention constructs a binary classifier to map the multi-scale features to the target category, and obtain the results of the two categories of fake or real. This binary classifier uses a fully connected layer and a softmax activation function to make category judgments; the specific process is as follows:

[0206] Step 4.2.1. Use the fully connected layer to calculate the unnormalized score of the multi-scale feature. The fully connected layer will project the dimension of the multi-scale feature to the category dimension to be classified. In this invention, the category dimension of classification is 2. The formula for the fully connected layer projection is as follows:

[0207]

[0208] in, Indicates the classification category, represents the forged category, represents the true category; Representation category The normalized score of F ms Represent multi-scale features; Representation category The corresponding weight matrix of the fully connected layer; Representation category The corresponding fully connected layer bias.

[0209] Step 4.2.2: Determine the category based on the unnormalized score to obtain the classification result of the model of the present invention. Use the Softmax activation function to convert the unnormalized score into the corresponding probability, and select the category with the larger probability as the output category determined by the model of the present invention. The probability calculation formula of the Softmax activation function is as follows:

[0210]

[0211] Among them, P represents the probability of different categories; are the normalized scores of the fake and real categories respectively.

[0212] The face forgery detection method based on prior collaboration provided by the present invention achieves high-accuracy identification of compressed face images by designing high-frequency feature extraction, multimodal feature fusion and multi-scale feature aggregation methods.

[0213] In order to demonstrate the feasibility and superiority of the present invention, the following comparative experiments were conducted: ViT was selected as the baseline for comparison of experimental results, and the evaluation indicators included accuracy (ACC), sensitivity curve (ROC curve), and area under the sensitivity curve (AUC).

[0214] ACC is used to measure the proportion of samples correctly classified by the model on the test set. It is the most intuitive evaluation indicator. The specific calculation formula is:

[0215]

[0216] Among them, ACC is the accuracy value.

[0217] The ROC curve (also known as the sensitivity curve) uses the true positive rate (TPR) as the vertical axis and the false positive rate (FPR) as the horizontal axis. It can intuitively evaluate the classification performance of the compressed face image identification classifier. The specific calculation formula is:

[0218]

[0219] Among them, TPR is the true positive rate; FPR is the false positive rate; TP stands for true positive, that is, the number of samples whose actual label is positive and the predicted result is also positive; FP stands for false positive, that is, the number of samples whose actual label is negative and the wrong prediction is positive; TN stands for true negative, that is, the number of samples whose actual label is negative and the predicted result is also negative; FN stands for false negative, that is, the number of samples whose actual label is positive and the wrong prediction is negative.

[0220] AUC represents the area under the ROC curve, which is a numerical value used to evaluate classification ability.

[0221] Performance evaluation was conducted on the FaceForensics++ dataset, which contains 1,000 original videos collected from YouTube and 4,000 fake videos forged using four forgery algorithms: Deepfakes, FaceSwap, Face2Face, and NeuralTextures. Deepfakes and FaceSwap are used to exchange faces between two different individuals, while Face2Face and NeuralTextures are used to achieve face reproduction. FF++ includes two compression versions: C23 and C40. The present invention divides it into 30 frames at fixed intervals and takes the average of the 30-frame prediction results as the video prediction result. The comparison results with ViT in terms of accuracy and area under the sensitivity curve are shown in Table 1 below:

[0222] Table 1 Comparison results of C23 version and C40 version in different methods

[0223]

[0224] From the comparison results, it can be seen that the higher the compression rate and the lower the image quality, the better the performance improvement of the present invention. On the C23 high-quality compressed version, the performance of the present invention is close to that of ViT, and the ACC is improved by 1.49%. However, the number of parameters of the invention is only 60% of that of ViT, which means that the present invention can also run efficiently on devices with limited hardware resources, greatly reducing the deployment cost. The floating-point operation volume is less than 30% of ViT, which makes the model highly usable in embedded devices and mobile terminals, and provides a new choice for low-power, high-efficiency application scenarios. On the C40 low-quality compressed version, ACC is improved by 5.03% and AUC is improved by 3.60%, demonstrating the excellent robustness of the present invention, which can accurately identify facial images with severe compression damage. This significant performance improvement and resource saving highlight the huge advantages of the present invention in practical applications. The ROC curves of the present invention on the C23 and C40 versions of the FaceForensics++ dataset are shown as follows: Figure 3 、 Figure 4 As shown, through Figure 3 、 Figure 4The excellent classification performance of the present invention is intuitively demonstrated.

[0225] Compared to existing mainstream detection methods, this invention not only pioneers new methodological innovations in theory but also demonstrates its superiority in practical performance. This dual improvement in performance and efficiency promises widespread application in future large-scale facial image recognition scenarios. This marks a new milestone in facial image recognition technology and provides a powerful reference for research and development in related fields.

[0226] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.

Claims

1. A face forgery detection method based on prior collaboration, characterized in that: Combining frequency domain, structured information, and multi-dimensional prior knowledge of spatial distance to work together to complete face forgery detection, specifically including the following steps: Step 1: Acquire a facial image and use a resonance-based frequency dynamic tuning method to enhance and extract frequency domain modal information of the facial image. The resonance-based frequency dynamic tuning method uses a Riemann resonance filter and an orthogonal divergence filter. The Riemann resonance filter is used to amplify edge information of the facial image and suppress aliasing features. The orthogonal divergence filter is used to obtain high-frequency information of the image. Step 2: Use the structure compression aggregation network to extract and fuse multimodal features with structural information; Step 3: Use the ViT network based on the hypersurface attention mechanism to model long-range dependencies for multimodal features and extract deep features with deep semantic information; Step 4: Use the attention mechanism to aggregate the deep features and obtain the detection results of the input image through a classifier containing a fully connected layer.

2. The face forgery detection method based on prior collaboration according to claim 1, characterized in that: The specific process of step 1 is: Step 1.1: Use the Riemann resonance filter to amplify the edge information of the face image and suppress the aliasing features. The specific process is as follows: Step 1.1.

1. The Riemann resonance filter uses the Riemann resonance operator as the filter kernel and provides high-frequency oscillation weights at different positions of the filter kernel to achieve efficient filtering of the feature map pixel values. The weight function of the Riemann resonance operator is defined as: Where R(·) is the weight function; r1 and r2 are different independent variables of the weight function; Step 1.1.2: Use the Riemann resonance filter based on the Riemann resonance operator to amplify the edges of the input deep fake face image: Where I is the input deep fake face image; I r is a deep fake face image processed by the Riemann resonance filter; i and j are the indexes of the face image in the horizontal and vertical directions respectively; m0 and m1 are the indexes of the Riemann resonance operator in the horizontal and vertical directions respectively; Step 1.1.3: Normalize and suppress confusing features: Among them, I sr is a normalized image; max(·) and min(·) represent the maximum and minimum operations respectively; Step 1.2: Use an orthogonal divergence filter to obtain the high-frequency information of the image; the specific process is: Step 1.2.1, calculate the orthogonal divergence kernel; the original function of the orthogonal divergence kernel is the Laplace operator, which is: in, is the Laplace operator; f is the Laplace function; f(x,y) is the Laplace function at coordinate (x,y); x and y are the indices of the Laplace function in the horizontal and vertical directions respectively; The Laplace operators in two directions are approximated by the finite difference method to obtain the orthogonal divergent kernel, which is calculated as follows: Among them, f x (x,y) and f y (x,y) are the second-order partial derivatives of the Laplace function with respect to x and y respectively; Step 1.2.2: Calculate the high-frequency feature map to generate a comprehensive high-pass response image; the calculation formula is: I f =I x +I y ; Among them, I x and I y Respectively represent I sr Under the action of the orthogonal divergent filter, the high-frequency image is obtained along the horizontal and vertical directions; c0 and c1 are the indexes in the horizontal and vertical directions, respectively, used to traverse the horizontal divergent kernel L x and vertically divergent kernel L y ;I f Represents an image with high-frequency information obtained by a variant of the Laplacian operator.

3. The face forgery detection method based on prior collaboration according to claim 2 is characterized in that: The specific process of step 2 is: Step 2.1: Input deep fake face image I and image with high frequency information I f Embedded into the graph structure; I and I f The embedding network of adopts the same network structure design, but the parameters are independent; the specific process is as follows: Step 2.1.

1. Use the convolution module to extract features from the image and generate spatial and frequency domain features respectively. Each convolution module consists of a convolution layer, a batch normalization layer, and an activation function. F s =ReLU(BN(Conv(I))); F f =ReLU(BN(Conv(I f ))); Among them, F s 、F f are spatial domain features and frequency domain features respectively; ReLU(·) is the ReLU activation function; BN(·) is the batch normalization layer; Conv(·) is the convolution layer in the convolution module; Step 2.1.2: Map the spatial domain features and frequency domain features into the graph structure respectively. The specific process is as follows: First, the spatial domain features and frequency domain features are converted into graph nodes respectively. The specific calculation formula is: in, are spatial domain nodes and frequency domain nodes respectively; Flatten(·) is the flattening operation; Conv2D(·) is the convolution layer in the graph structure; Next, generate the adjacency matrix for the graph structure; the calculation formula of the adjacency matrix is: Among them, E s and E f Represents airspace nodes and frequency domain nodes The adjacency matrix of s and D f They are (A s +T) and (A f +T); w and h are respectively (A s +T) or (A f +T) the number of rows and columns; N represents the number of nodes in the airspace or frequency domain node The number of s and A f are different random number matrices; T is the identity matrix; Diag(·) represents a diagonal matrix; Finally, we get the airspace map and frequency domain plots Step 2.2: Fusion of spatial domain nodes and frequency domain nodes; use graph neural network to model the spatial domain graph G s and frequency domain graph G f The complex structural information of , the specific process is: in, They represent the spatial domain nodes and frequency domain nodes after the k+1th layer of graph convolution respectively; Respectively represent the spatial domain nodes and frequency domain nodes after the k-th layer of graph convolution; W k and b k Represent the weight and offset of the node mapping parameters after the k-th layer graph convolution; BN k represents the batch normalization layer corresponding to the k-th layer of graph convolution; After completing the feature extraction of the spatial domain graph and the frequency domain graph, the two features are initially fused through the connection operation to obtain the initial fusion feature. Then, the spatial domain features and the frequency domain features are deeply fused through graph convolution. The calculation formula is: V k+1 (ReLU(BN). k (E s +E f )·(W k ·V k +b k ))),k0,1 Among them, V 0 is a fusion node; concat(·) is a connection operation; Respectively represent the spatial domain nodes and frequency domain nodes after the second layer of graph convolution; V k+1 and V k They represent the multimodal nodes after the k+1th layer of graph convolution and the kth layer of graph convolution respectively; Finally, we get a multimodal graph G(V 2 ,E s +E f ); Step 2.3: Use the topological association aggregation mechanism to aggregate the multimodal graph. The specific process is as follows: Step 2.3.1: Aggregate the multimodal nodes to obtain the aggregated multimodal feature V′. The calculation formula is: V′=AvgPool(V 2 ); Among them, V 2 represents the multimodal node before aggregation; AvgPool(·) represents the average pooling operation; Step 2.3.2: Aggregate the adjacency matrix to obtain the aggregated adjacency matrix. The specific process is as follows: Define the minimum coordinate of the original adjacency matrix corresponding to the point with coordinate (u,v) in the aggregated adjacency matrix as (u base ,v base ), the calculation formula is: Among them, u base (·) and v base (·) represents the horizontal starting index and vertical starting index of the node set mapped by the aggregation window; u and v are the horizontal and vertical indices respectively; H0 and H1 are the horizontal length and vertical length of the aggregation window respectively; is the rounding down operation; % is the modulo operation; Calculate (u base ,v base ) is the mathematical expectation of the original adjacency matrix area with the minimum coordinate, and the edges of the original adjacency matrix area are comprehensively counted to obtain the aggregated adjacency matrix; the calculation formula is: P0,P1~Uniform(0,H1-1); Q0,Q1~Uniform(0,H2-1); Where E is the original adjacency matrix; the edges of the corresponding area in E are comprehensively counted to obtain the aggregated adjacency matrix E′; E′ uv is the element in row u and column v of E′, is the mathematical expectation; Uniform(0,H1-1) and Uniform(0,H2-1) represent random variables uniformly distributed on the interval [0,H1-1] and the interval [0,H2-1] respectively; P0 and P1 are both values ​​randomly selected from the uniform distribution Uniform(0,H1-1); Q0 and Q1 are both values ​​randomly selected from the uniform distribution Uniform(0,H2-1).

4. The face forgery detection method based on prior collaboration according to claim 3 is characterized in that: The specific process of step 3 is as follows: Step 3.1: Convert the multimodal features and the aggregated adjacency matrix into the input of the ViT network. First, the aggregated adjacency matrix is ​​concatenated with its mean, and the position encoding vector is obtained through linear mapping. The calculation formula is: pos=Linear(concat(Mean(E′),E′)); Among them, pos is the position encoding vector; Mean(·) and concat(·) are the average operation and splicing operation respectively; Linear(·) is the linear mapping operation; The multimodal feature V′ is directly used as the local token of the subsequent ViT network input, and the mean of the multimodal feature V′ is used as the global token of the subsequent ViT network input; the position encoding vector pos is added to the local token and the global token, and the calculation formula is: F emb =concat(Mean(V′),V′)+pos; Among them, F emb It is the input feature of the subsequent ViT; Step 3.2: Calculate the attention attenuation matrix. The calculation formula is: in, is the value of the attention decay matrix in row t1 and column t2, where row t1 corresponds to token t1 and column t2 corresponds to token t2; represents the Euclidean distance between token t1 and token t2; d m is the maximum Euclidean distance between tokens; cos(·) is the cosine operation; are the horizontal and vertical coordinates of token t1 respectively; are the horizontal and vertical coordinates of token t2 respectively; Step 3.3, the ViT network contains several layers of ViT blocks, and the calculation formula is: With l =MLP l (LN l (With' l ))+z' l ,l=1,2,…,L; Among them, A l is the attention score obtained by using the hypersurface attention mechanism for the l-th layer of ViT block; Softmax(·) is the Softmax activation function; L is the maximum number of layers of ViT block; and denote the query matrix, key matrix and value matrix of the ViT block at layer l respectively; is the transposed symbol; D is the attention attenuation matrix; z ′ l represents the intermediate output feature of the ViT block at layer l; z' l is the final output feature of the ViT block in the lth layer; LN l and MLP l Represent the linear normalization layer and multi-layer perceptron of the l-th layer ViT block respectively; The final extracted deep feature F with deep semantic information d Calculated as: F d =LN L (z L ); Among them, LN L is the linear normalization layer of the Lth layer ViT block; z L is the final output feature of the L-th layer ViT block.

5. The face forgery detection method based on prior collaboration according to claim 4 is characterized in that: The specific process of step 4 is as follows: Step 4.1: Use the attention mechanism to aggregate local tokens and concatenate them with global tokens to obtain multi-scale features. The specific process is as follows: Step 4.1.

1. Calculate the attention score of the local token: A local =Softmax(Linear(F d [1:])); Among them, A local is the attention score of the local token; Linear(·) is the feature mapping operation; Step 4.1.2, calculate the final local feature F local : F local =A local ·F d ; Step 4.1.3, calculate F local Concatenate with the global token in the feature dimension to obtain the final multi-scale feature F ms , the calculation formula is: F ms =concat(F local ,F d [0]); Step 4.2: Use a classifier containing a fully connected layer to classify the multi-scale features and classify the results as real or fake. The specific process is as follows: Step 4.2.

1. Use the fully connected layer to calculate the unnormalized scores of multi-scale features: in, Indicates the classification category, represents the forged category, represents the true category; Representation category The normalized score of Representation category The corresponding weight matrix of the fully connected layer; Representation category The bias of the corresponding fully connected layer; Step 4.2.2: Use the Softmax activation function to convert the unnormalized scores into corresponding probabilities, and select the category with the highest probability as the output category determined by the model. The probability calculation formula of the Softmax activation function is as follows: Among them, P represents the probability of different categories; are the normalized scores of the fake and real categories respectively.

Images