Deep forgery detection method based on rank adaptation network
By using a deep forgery detection method based on rank-adaptive networks, the rank of the feature space is dynamically adjusted and multi-scale fusion is performed, which solves the problem of insufficient cross-domain generalization ability in existing technologies and achieves efficient identification and stable detection of unknown forgery methods.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG UNIV OF TECH
- Filing Date
- 2025-12-26
- Publication Date
- 2026-04-17
AI Technical Summary
Existing deepfake detection methods lack cross-domain generalization ability and cannot effectively distinguish and extract subtle features of different forgery methods, especially in complex backgrounds where they are insufficient in recognizing unknown forgery methods.
A deep forgery detection method based on rank-adaptive networks is adopted. Initial feature extraction is performed through CLIP-ViT backbone network. Combined with dynamic rank adjustment module, multi-scale attention module and classifier, the feature space rank is dynamically adjusted and projected, and multi-granularity forgery traces are aggregated to achieve refined multi-scale fusion feature extraction and binary classification decision.
It improves the model's generalization performance and robustness against unknown forgery techniques, significantly enhances the accuracy of identifying emerging forgery methods, and strengthens the capabilities of content security auditing, digital forensics, and cybersecurity protection.
Smart Images
Figure CN121884087A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of forgery detection technology, specifically relating to a deep forgery detection method based on rank-adaptive networks. Background Technology
[0002] With the rapid development of deepfake technology and the widespread adoption of synthetic media, the authenticity of digital content has been seriously threatened. This technology not only exacerbates the risk of spreading misinformation but also brings about problems such as privacy violations, identity theft, and evidence tampering. Therefore, deepfake detection has become a key technology for ensuring cybersecurity and digital forensics. Existing deepfake detection methods mainly rely on models based on backbone networks such as Xception and Contrastive Language-Image Pretraining (CLIP). These methods typically perform well when the training and testing data come from the same forgery generation techniques. However, as forgery generation techniques continue to evolve, the generalization ability of these methods has significantly decreased, making them unable to effectively cope with unknown or emerging forgery methods.
[0003] To address the aforementioned issue of insufficient cross-domain generalization, researchers have proposed various methods to enhance the generalization ability of detection models. For example, they have introduced a low-rank adaptation strategy to improve feature representation, or used a multi-task decoupling framework to improve the model's adaptability to different forgery methods. However, while these methods improve the model's generalization to some extent, they still cannot effectively distinguish and extract subtle features from different forgery methods.
[0004] In recent years, deep learning-based deepfake detection technology has gained widespread attention. These technologies automatically learn to extract deep features from images through deep neural networks, thereby achieving more accurate forgery identification. Nevertheless, existing models still face challenges in fully utilizing spatial information and avoiding overfitting, especially in complex contexts, where better handling of unknown forgery details remains a crucial issue that needs to be addressed. Summary of the Invention
[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a deep forgery detection method based on rank-adaptive networks, which can effectively improve the generalization ability on unknown datasets while maintaining the detection competitiveness within the domain, and also take into account a certain degree of anti-interference robustness.
[0006] To achieve the above objectives, this invention provides a deepfake detection method based on rank-adaptive networks, comprising the following steps: S1. Initial feature extraction is performed on the input image using the CLIP-ViT backbone network; S2. Construct a dynamic rank adjustment module that includes a dynamic rank controller, a learnable basis matrix, and a dynamic rank fusion unit. Input the initial features into the dynamic rank adjustment module. Through the dynamic rank controller, the learnable basis matrix, and dynamic rank fusion, the rank of the feature space is adaptively adjusted and projected to output rank-enhanced features. S3. Feed the rank-enhanced features into the multi-scale attention module. Through the parallel global attention branch, multi-scale local attention branch and detail enhancement branch, aggregate multi-granularity forgery traces and output the refined multi-scale fusion features. S4. Input the multi-scale fused features into the classifier to complete the binary classification decision of image authenticity.
[0007] As a preferred embodiment of the present invention, in S2, after the initial features are input to the dynamic rank adjustment module, the complexity of the initial features is first analyzed by the dynamic rank controller. The dynamic rank controller includes a global average pooling layer, a dimension reduction convolutional layer, a ReLU activation function and a Sigmoid function in sequence. After the initial features pass through the dynamic rank adjustment module, a dynamic rank ratio is output. Subsequently, a dynamic projection matrix is constructed based on the learnable basis matrix containing the left and right basis matrices: the initial features are flattened, the actual projection rank is calculated according to the dynamic rank ratio, the value of the actual projection rank is r, the first r columns of the left basis matrix and the first r rows of the right basis matrix are truncated according to the actual projection rank, and the dynamic projection matrix is obtained by multiplying them; the initial features are then channel-projected using the dynamic projection matrix to obtain the projected features. Finally, in the dynamic rank fusion unit, local spatial detail features are extracted from the initial features using convolutional layers. Through the dynamic rank fusion operation, the projected features and local spatial detail features are added element-wise to output rank-enhanced features.
[0008] As a preferred embodiment of the present invention, the process of outputting the dynamic rank ratio is as follows: ; In the formula, k represents the numerical value of the dynamic rank ratio; Represents the Sigmoid function; Represents the ReLU activation function; This represents a dimension reduction convolution operation; Indicates a global average pooling layer; Indicates the initial characteristics.
[0009] As a preferred embodiment of the present invention, the process of calculating the actual projected rank based on the dynamic rank ratio is as follows: ; In the formula, This indicates the numerical value of the set rank upper limit; This indicates the numerical value of the set lower limit of the rank; The process of obtaining the dynamic projection matrix is as follows: ; In the formula, Represents the dynamic projection matrix; This represents the left basis matrix after the actual projected rank is truncated; This represents the right basis matrix after the actual projected rank is truncated; The process of obtaining projection features is as follows: ; In the formula, Indicates projection features; Indicates the initial features after flattening; The process of outputting rank-enhanced features is as follows: ; In the formula, Indicates rank-enhanced features; This represents a 3×3 convolutional layer.
[0010] As a preferred embodiment of the present invention, in S3, in the global attention branch, the rank enhancement feature is sequentially passed through adaptive global average pooling, two convolutional layers and the Sigmoid function to generate global attention weights, wherein the ReLU activation function is used between the two convolutional layers to output global attention features; In the multi-scale local attention branch, the rank enhancement features are processed through three parallel pathways. Each pathway performs average pooling at different scales, two convolutional layers, and upsampling operations. The ReLU activation function is used between the two convolutional layers. The outputs of the three pathways are averaged to obtain the multi-scale local attention features. In the detail enhancement branch, the rank enhancement features are passed sequentially through a convolutional layer, a ReLU activation function, another convolutional layer, and a sigmoid function to generate detail-level attention weights and output the detail enhancement features. The global attention features, multi-scale local attention features, and detail enhancement features are added together, and the fusion result is weighted and fused with the rank enhancement features to output the refined multi-scale fusion features.
[0011] As a preferred embodiment of the present invention, the process of outputting global attention features in the global attention branch is as follows: ; In the formula, G represents the global attention feature; Represents the Sigmoid function; Indicates a convolutional layer; Represents the ReLU activation function; Indicates a global average pooling layer; Indicates rank-enhanced features; In the multi-scale local attention branch, the process of obtaining multi-scale local attention features is as follows: ; In the formula, This represents the local attention weight corresponding to the i-th scale; This represents the average pooling layer corresponding to the i-th scale; Indicates upsampling; Indicates the upsampling ,include , , L represents multi-scale local attention features; In the detail enhancement branch, the process of outputting detail enhancement features is as follows: ; In the formula, D represents the detail enhancement feature; The process of outputting refined multi-scale fusion features is as follows: ; In the formula, This represents the multi-scale fusion feature.
[0012] In a preferred embodiment of the present invention, in step S4, the classifier includes a 1×1 convolution, batch normalization, ReLU activation function, Dropout, fully connected layer, and Sigmoid function arranged sequentially. The process of making a binary classification decision on the authenticity of an image is as follows: ; In the formula, Indicates compression features; Represents the ReLU activation function; This indicates a batch normalization operation; This represents a 1×1 convolutional layer; Indicates multi-scale fusion features; This indicates the characteristics after random deactivation; Indicates a random deactivation layer; P represents the probability value of a forged image; The sigmoid function is represented by FC; FC represents a fully connected layer. Y represents the binary classification decision threshold; Y represents the binary classification result, Y=1 represents a fake image, and Y=0 represents a real image.
[0013] A deepfake detection device based on rank-adaptive networks includes: One or more processors; Memory, used to store one or more computer programs; Specifically, when one or more programs are executed by one or more processors, the one or more processors execute the methods described above.
[0014] A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.
[0015] The beneficial effects of this invention are: In deepfake detection tasks, this invention effectively combines the advantages of rank adjustment theory and attention mechanism to improve the model's generalization performance and robustness against unknown forgery techniques. The core dual-module structure is designed: the dynamic rank adjustment module adaptively adjusts the rank of the feature space to avoid the model getting stuck in overfitting and low-rank bottlenecks, thereby enhancing its adaptability to unknown forgery patterns; the multi-scale attention module focuses on utilizing the advantages of multi-scale perception to aggregate forgery traces from multiple levels, including global, local and detailed.
[0016] This invention designs a dynamic rank controller and a learnable basis matrix to achieve adaptive projection of the feature space rank. This design enables the model to dynamically select the most suitable representation space based on the input content. A multi-branch attention fusion mechanism is also designed to capture forged features of different granularities in parallel and then weighted and fused these features to optimize the final decision. These designs make this invention valuable in deepfake detection technology, significantly improving the accuracy and stability of identifying emerging forgery methods. It also enhances proactive defense and analysis capabilities in areas such as content security auditing, digital forensics, and network security protection, thus providing key technical support for building a trustworthy digital media environment. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating the principle of this invention; Figure 2 This is an exemplary feature processing flowchart in Embodiment 1 of the present invention. Detailed Implementation
[0018] The embodiments of the present invention will be further described below with reference to the accompanying drawings: Example 1: As Figure 1 and Figure 2 As shown, a deepfake detection method based on rank-adaptive networks includes the following steps: S1. Initial feature extraction is performed on the input image using the CLIP-ViT backbone network; S2. Construct a dynamic rank adjustment module that includes a dynamic rank controller, a learnable basis matrix, and a dynamic rank fusion unit. Input the initial features into the dynamic rank adjustment module. Through the dynamic rank controller, the learnable basis matrix, and dynamic rank fusion, the rank of the feature space is adaptively adjusted and projected to output rank-enhanced features. S3. Feed the rank-enhanced features into the multi-scale attention module. Through the parallel global attention branch, multi-scale local attention branch and detail enhancement branch, aggregate multi-granularity forgery traces and output the refined multi-scale fusion features. S4. Input the multi-scale fused features into the classifier to complete the binary classification decision of image authenticity.
[0019] CLIP-ViT backbone network is an existing technology. Its core design uses a dual-tower structure of image encoder and text encoder to map images and text to the same semantic space, thereby achieving cross-modal feature alignment.
[0020] In S2, after the initial features are input to the dynamic rank adjustment module, the complexity of the initial features is first analyzed by the dynamic rank controller. The dynamic rank controller includes a global average pooling layer, a dimension reduction convolutional layer, a ReLU activation function, and a Sigmoid function. After the initial features pass through the dynamic rank adjustment module, a dynamic rank ratio is output to guide the adaptive selection of the subsequent projected rank. Subsequently, a dynamic projection matrix is constructed based on the learnable basis matrix containing the left and right basis matrices: the initial features are flattened, the actual projection rank is calculated according to the dynamic rank ratio, the value of the actual projection rank is r, the first r columns of the left basis matrix and the first r rows of the right basis matrix are truncated according to the actual projection rank, and the dynamic projection matrix is obtained by multiplying them; the initial features are then channel-projected using the dynamic projection matrix to obtain the projected features. Finally, in the dynamic rank fusion unit, local spatial detail features are extracted from the initial features using convolutional layers. Through the dynamic rank fusion operation, the projected features and local spatial detail features are added element-wise to output rank-enhanced features.
[0021] The left basis matrix and the right basis matrix are two components of the learnable basis matrix. They are parameter matrices that are optimized during model training and are the core basic components for constructing dynamic projection matrices. Typically, the dimension of the left basis matrix is "original number of feature channels × maximum rank", and the dimension of the right basis matrix is "maximum rank × original number of feature channels".
[0022] The process of outputting the dynamic rank ratio is as follows: ; In the formula, k represents the numerical value of the dynamic rank ratio; Represents the Sigmoid function; Represents the ReLU activation function; This represents a dimension reduction convolution operation; Indicates a global average pooling layer; Indicates the initial characteristics.
[0023] The process of calculating the actual projected rank based on the dynamic rank ratio is as follows: ; In the formula, This indicates the numerical value of the set rank upper limit; This indicates the numerical value of the set lower limit of the rank; The process of obtaining the dynamic projection matrix is as follows: ; In the formula, Represents the dynamic projection matrix; This represents the left basis matrix after the actual projected rank is truncated; This represents the right basis matrix after the actual projected rank is truncated; The process of obtaining projection features is as follows: ; In the formula, Indicates projection features; Indicates the initial features after flattening; The process of outputting rank-enhanced features is as follows: ; In the formula, Indicates rank-enhanced features; This represents a 3×3 convolutional layer.
[0024] In S3, in the global attention branch, the rank-enhanced features are sequentially passed through adaptive global average pooling, two convolutional layers, and the Sigmoid function to generate global attention weights. The ReLU activation function is used between the two convolutional layers to output global attention features. In the multi-scale local attention branch, the rank enhancement features are processed through three parallel pathways. Each pathway performs average pooling at different scales, two convolutional layers, and upsampling operations. The ReLU activation function is used between the two convolutional layers. The outputs of the three pathways are averaged to obtain the multi-scale local attention features. In the detail enhancement branch, the rank enhancement features are passed sequentially through a convolutional layer, a ReLU activation function, another convolutional layer, and a sigmoid function to generate detail-level attention weights and output the detail enhancement features. The global attention features, multi-scale local attention features, and detail enhancement features are added together, and the fusion result is weighted and fused with the rank enhancement features to output the refined multi-scale fusion features.
[0025] In the global attention branch, the process of outputting global attention features is as follows: ; In the formula, G represents the global attention feature; Represents the Sigmoid function; Indicates a convolutional layer; Represents the ReLU activation function; Indicates a global average pooling layer; Indicates rank-enhanced features; For G, G can also be connected via a broadcast mechanism. To interact, extend the shape logic of G with Dimensions consistent, and then with By performing element-wise multiplication, we ultimately obtain features that incorporate global attention.
[0026] In the multi-scale local attention branch, the process of obtaining multi-scale local attention features is as follows: ; In the formula, This represents the local attention weight corresponding to the i-th scale; This represents the average pooling layer corresponding to the i-th scale; Indicates upsampling; Indicates the upsampling ,include , , , which represents the local attention weights corresponding to the 0th, 1st, and 2nd scales after upsampling; L represents the multi-scale local attention feature; In the detail enhancement branch, the process of outputting detail enhancement features is as follows: ; In the formula, D represents the detail enhancement feature; The process of outputting refined multi-scale fusion features is as follows: ; In the formula, This represents the multi-scale fusion feature.
[0027] In S4, the classifier can directly use a fully connected layer, with the following settings: The classifier consists of a 1×1 convolution, batch normalization, ReLU activation function, Dropout, fully connected layer, and sigmoid function arranged sequentially. The process of making a binary classification decision on whether an image is real or fake is as follows: ; In the formula, Indicates compression features; Represents the ReLU activation function; This indicates a batch normalization operation; This represents a 1×1 convolutional layer; Indicates multi-scale fusion features; This indicates the characteristics after random deactivation; Indicates a random deactivation layer; P represents the probability value of a forged image; The sigmoid function is represented by FC; FC represents a fully connected layer. Y represents the binary classification decision threshold; Y represents the binary classification result, Y=1 represents a fake image, and Y=0 represents a real image.
[0028] Compared to a simple fully connected layer, this classifier combines 1×1 convolution to compress features with batch normalization and Dropout, which not only simplifies the feature dimensions efficiently but also improves training stability and generalization ability. It can accurately output the probability of fake images and achieve reliable binary classification decisions.
[0029] Example 2: The difference between this example and Example 1 is that the dynamic rank adjustment module also includes a multi-scale local pooling layer, a feature concatenation unit, and an adaptive weighted fusion layer. That is, the dynamic rank controller sequentially includes a multi-scale local pooling layer, a global average pooling layer, a feature concatenation unit, an adaptive weighted fusion layer, a dimensionality-reduced convolutional layer, a ReLU activation function, and a Sigmoid function. The dynamic rank ratio process becomes: First, local average pooling at three different scales is performed on the initial features to obtain three sets of features with local complexity. , , Simultaneously, global complexity features are extracted from the initial features using global average pooling. ,Will , , , Input feature concatenation units, concatenated along the channel dimension to obtain fused features. ; The adaptive weighted fusion layer generates weight coefficients corresponding to four features through two convolutional layers. A ReLU activation function is used between the two convolutional layers to obtain... , , , Corresponding weight coefficients , , , : ; In the formula, Represents the Sigmoid function; Represents the ReLU activation function; Indicates a convolutional layer; Calculate the weighted fusion features using weighting coefficients: ; In the formula, This represents the complexity characteristics after weighted fusion; Finally, the dynamic rank ratio is obtained through mapping using the Sigmoid function: ; In the formula, k represents the numerical value of the dynamic rank ratio; This represents a dimension reduction convolution operation; This indicates the global average pooling layer.
[0030] By integrating multi-scale local and global feature complexity information, the dynamic rank ratio can be more accurately adapted to regional differences in the initial features, thereby improving the targeting and rationality of dynamic rank adjustment.
[0031] Example 3: A deepfake detection device based on rank-adaptive networks, comprising: One or more processors; Memory, used to store one or more computer programs; When one or more programs are executed by one or more processors, the one or more processors perform the method in Embodiment 1 or Embodiment 2.
[0032] Example 4: A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method in Example 1 or Example 2.
Claims
1. A deepfake detection method based on rank-adaptive network, characterized in that Includes the following steps: S1. Initial feature extraction is performed on the input image using the CLIP-ViT backbone network; S2. Construct a dynamic rank adjustment module that includes a dynamic rank controller, a learnable basis matrix, and a dynamic rank fusion unit. Input the initial features into the dynamic rank adjustment module. Through the dynamic rank controller, the learnable basis matrix, and the dynamic rank fusion, the rank of the feature space is adaptively adjusted and projected to output rank-enhanced features. S3. Feed the rank-enhanced features into the multi-scale attention module. Through the parallel global attention branch, multi-scale local attention branch and detail enhancement branch, aggregate multi-granularity forgery traces and output the refined multi-scale fusion features. S4. Input the multi-scale fused features into the classifier to complete the binary classification decision of image authenticity. 2.The deepfake detection method based on rank-adaptive network according to claim 1, wherein: In S2, after the initial features are input to the dynamic rank adjustment module, the complexity of the initial features is first analyzed by the dynamic rank controller. The dynamic rank controller includes a global average pooling layer, a dimension reduction convolutional layer, a ReLU activation function, and a Sigmoid function in sequence. After the initial features pass through the dynamic rank adjustment module, a dynamic rank ratio is output. Subsequently, a dynamic projection matrix is constructed based on the learnable basis matrix containing the left and right basis matrices: the initial features are flattened, the actual projection rank is calculated according to the dynamic rank ratio, the value of the actual projection rank is r, the first r columns of the left basis matrix and the first r rows of the right basis matrix are truncated according to the actual projection rank, and the dynamic projection matrix is obtained by multiplying them; the initial features are then channel-projected using the dynamic projection matrix to obtain the projected features. Finally, in the dynamic rank fusion unit, local spatial detail features are extracted from the initial features using convolutional layers. Through the dynamic rank fusion operation, the projected features and local spatial detail features are added element-wise to output rank-enhanced features.
3. The method of claim 2, wherein the rank adaptation network is a convolutional neural network. The process of outputting the dynamic rank ratio is as follows: ； In the formula, k represents the numerical size of the dynamic rank ratio; represents a Sigmoid function; represents a ReLU activation function; represents a dimension reduction convolution operation; represents a global average pooling layer; represents initial features.
4. The deepfake detection method based on rank-adaptive networks according to claim 3, characterized in that, The process of calculating the actual projected rank based on the dynamic rank ratio is as follows: ； In the formula, This indicates the numerical value of the set rank upper limit; This indicates the numerical value of the set lower limit of the rank; The process of obtaining the dynamic projection matrix is as follows: ； In the formula, Represents the dynamic projection matrix; This represents the left basis matrix after the actual projected rank is truncated; This represents the right basis matrix after the actual projected rank is truncated; The process of obtaining projection features is as follows: ； In the formula, Indicates projection features; Indicates the initial features after flattening; The process of outputting rank-enhanced features is as follows: ； In the formula, Indicates rank-enhanced features; This represents a 3×3 convolutional layer.
5. The deepfake detection method based on rank-adaptive networks according to claim 1, characterized in that: In S3, in the global attention branch, the rank-enhanced features are sequentially passed through adaptive global average pooling, two convolutional layers, and the Sigmoid function to generate global attention weights. The ReLU activation function is used between the two convolutional layers to output global attention features. In the multi-scale local attention branch, the rank enhancement features are processed through three parallel pathways. Each pathway performs average pooling at different scales, two convolutional layers, and upsampling operations. The ReLU activation function is used between the two convolutional layers. The outputs of the three pathways are averaged to obtain the multi-scale local attention features. In the detail enhancement branch, the rank enhancement features are passed sequentially through a convolutional layer, a ReLU activation function, another convolutional layer, and a sigmoid function to generate detail-level attention weights and output the detail enhancement features. The global attention features, multi-scale local attention features, and detail enhancement features are added together, and the fusion result is weighted and fused with the rank enhancement features to output the refined multi-scale fusion features.
6. The deepfake detection method based on rank-adaptive networks according to claim 5, characterized in that, In the global attention branch, the process of outputting global attention features is as follows: ； In the formula, G represents the global attention feature; Represents the Sigmoid function; Indicates a convolutional layer; Represents the ReLU activation function; Indicates a global average pooling layer; Indicates rank-enhanced features; In the multi-scale local attention branch, the process of obtaining multi-scale local attention features is as follows: ； In the formula, This represents the local attention weight corresponding to the i-th scale; This represents the average pooling layer corresponding to the i-th scale; Indicates upsampling; Indicates the upsampling ,include , , L represents multi-scale local attention features; In the detail enhancement branch, the process of outputting detail enhancement features is as follows: ； In the formula, D represents the detail enhancement feature; The process of outputting refined multi-scale fusion features is as follows: ； In the formula, This represents the multi-scale fusion feature.
7. The deepfake detection method based on rank-adaptive networks according to claim 1, characterized in that: In S4, the classifier includes a 1×1 convolution, batch normalization, ReLU activation function, Dropout, fully connected layer, and Sigmoid function arranged sequentially. The process of making a binary classification decision on the authenticity of an image is as follows: ； In the formula, Indicates compression features; Represents the ReLU activation function; This indicates a batch normalization operation; This represents a 1×1 convolutional layer; Indicates multi-scale fusion features; This indicates the characteristics after random deactivation; Indicates a random deactivation layer; P represents the probability value of a forged image; The sigmoid function is represented by FC; FC represents a fully connected layer. Y represents the binary classification decision threshold; Y represents the binary classification result, Y=1 represents a fake image, and Y=0 represents a real image.
8. A deepfake detection device based on rank-adaptive networks, characterized in that, include: One or more processors; Memory, used to store one or more computer programs; Wherein, when one or more programs are executed by one or more processors, the one or more processors perform the method of any one of claims 1-7.
9. A computer-readable storage medium having executable instructions stored thereon, characterized in that, When the instruction is executed by the processor, it causes the processor to perform the method of any one of claims 1-7.