A multi-category image forgery detection method, device, equipment and medium

Abstract

This invention discloses a method, apparatus, device, and medium for multi-class image forgery detection, relating to the field of image processing technology. The method can identify real faces, forged faces, real anime faces, forged anime faces, real sketch faces, and forged sketch faces. A multi-branch network, by integrating the characteristics of pre-trained models such as EfficientNet, ResNet, and DenseNet, combined with a feature fusion attention layer, significantly improves the classification ability for complex image categories. To construct a high-quality forged sketch dataset, a CycleGAN-based forged image generation framework was implemented. Adversarial loss, cycle consistency loss, and perceptual loss were used to optimize the generation effect, enhancing the diversity and robustness of the dataset. Experimental results show that the proposed model exhibits excellent performance in multi-class classification tasks, with an average accuracy and F1 score of 98.12%, demonstrating excellent generalization ability in both real and forged image classification.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, device, and medium for detecting multi-category image forgery based on a fusion of multi-branch networks and generative adversarial models. Background Technology

[0002] The rapid development of information technology has ushered in the era of self-media, with audio and video as the primary mediums of communication. The rise of deepfake technology has enabled fake videos to spread rapidly and widely on online platforms, posing a serious challenge to national security, social stability, and individual reputation. Traditional image classification methods often struggle to handle complex backgrounds and diverse variations, especially when there are significant differences between categories.

[0003] Deepfake technology utilizes advanced deep learning methods such as autoencoders and generative adversarial networks (GANs) to achieve precise forgery and replacement of faces, generating fake videos that are almost indistinguishable from the real thing. This technology is easily exploited due to its low production threshold, high level of simulation, and strong deceptiveness. Therefore, research on the authentication of deepfake videos and the identification of forgery algorithms is particularly important.

[0004] The current state of research at home and abroad shows that GAN has made significant progress in the field of image synthesis. The GAN model proposed by Goodfellow et al. in 2014 achieved implicit estimation of high-dimensional data distribution through adversarial training of generator and discriminator. This model has been widely used in the field of image synthesis and has achieved great success. In recent years, researchers have proposed a variety of improved GAN models, such as LAPGAN[3], StackGAN, PGGAN, WGAN-GP, BIGGAN and COCO-GAN. These models have improved the resolution and quality of synthesized images through different methods. It is particularly noteworthy that StyleGAN and StyleGAN2 were proposed. They significantly improved the realism and quality of synthesized images by improving the network structure and training methods. However, the improved realism of GAN synthesized images has also brought challenges. Criminals may use GAN technology to maliciously create and spread fake images, which pose a threat to social network security and stability. It is evident that how to effectively distinguish between real natural images and GAN synthesized images has become crucial.

[0005] Currently, the types of fake videos obtained using forgery techniques are increasing, such as fake real faces, fake anime faces, and fake sketch faces. However, current forgery detection models struggle to accurately identify all of these forgeries. For example, due to the stylized characteristics of anime characters, traditional image classification methods are difficult to apply directly. Sketch images, due to the abstract nature of their lines and shadows, place higher demands on classification models.

[0006] Therefore, how to provide an accurate and efficient classification model for achieving accurate and efficient identification of real and fake faces, recognition of anime characters, and recognition of sketch images is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of the above problems, the present invention provides a multi-category image forgery detection method, apparatus, device and medium for overcoming or at least partially solving the above problems.

[0008] This invention provides the following solution:

[0009] A multi-category image forgery detection method, comprising:

[0010] Acquire the image of the face to be detected;

[0011] A data preprocessing network is used to capture the texture and edge information of the face image to be detected to generate a reconstructed face image; the data preprocessing network includes at least a two-dimensional discrete wavelet transform module;

[0012] The reconstructed face image is feature-extracted using a fusion multi-branch module of a feature extraction network, and the features extracted from each branch of the fusion multi-branch module are fused using a feature fusion attention module of the feature extraction network to obtain fused features; the fusion multi-branch module includes mutually independent... Model, Model and Model;

[0013] The fused features are mapped to an intermediate feature space using the fully connected layers of the classification network, and the classification network's... The layer generates a probability distribution of the category to which the face image to be detected belongs, and the category includes any one of real face image, fake face image, real anime face image, fake anime face image, real sketch face image, and fake sketch face image;

[0014] Among them, the The model includes fully connected layers and Dropout layers to enhance generalization ability;

[0015] The The model includes The structure includes an improved fully connected layer and three convolutional layers, wherein the improved fully connected layer includes a non-linear activation function. as well as layer;

[0016] The The model includes an improved classifier, which is used to make the... The features output by the model are the same as those described. Model and the Feature alignment of model output.

[0017] Preferably, the two-dimensional discrete wavelet transform module is used to decompose the face image to be detected into low-frequency approximate components and three high-frequency detail components; and to reconstruct the original image by using inverse wavelet transform to obtain the reconstructed face image.

[0018] Preferably, the data preprocessing network further includes Edge detection algorithm, the The edge detection algorithm is used to enhance the edges of the sketch face image after determining that the face image to be detected is a sketch face image, so as to use the two-dimensional discrete wavelet transform module to capture the texture and edge information of the sketch face image to generate the reconstructed face image.

[0019] Preferably: the Edge detection algorithms include the following steps:

[0020] Apply a Gaussian filter to the image to smooth it and reduce noise;

[0021] use Operators compute images in and Gradient of direction;

[0022] The gradient value of each pixel is checked in the gradient direction. If the gradient value of a pixel is not the maximum value in its neighborhood in the gradient direction, it is suppressed to 0 and determined to be a non-edge.

[0023] Set a high threshold and a low threshold, and mark pixels with gradient magnitudes higher than the high threshold as strong edges, pixels with gradient magnitudes lower than the low threshold as non-edges, and pixels in between as weak edges;

[0024] By using hysteresis tracking, weak edges that are connected to strong edges are joined together to form complete edges.

[0025] Preferably, the feature fusion attention module fuses and optimizes features through nonlinear transformation and attention mechanism to obtain the fused features.

[0026] Preferably, the feature fusion attention module is used to perform the following operations:

[0027] The features extracted from each branch of the fusion multi-branch module are concatenated along the channel dimension to obtain a joint feature vector.

[0028] The joint feature vector is passed through a fully connected layer and The activation function performs a nonlinear transformation;

[0029] Another fully connected layer is used to further reduce the dimensionality, and the final classification result is output.

[0030] Preferably, the fusion process of the feature fusion attention module is represented by the following formula:

[0031]

[0032] In the formula: express The model outputs a feature vector. express The model outputs feature vectors, express The model outputs feature vectors. This refers to a function used to concatenate two or more arrays. This represents the attention mechanism.

[0033] A multi-category image forgery detection apparatus is used to perform the above-described multi-category image forgery detection method, the apparatus comprising:

[0034] The image acquisition unit is used to acquire the image of the face to be detected.

[0035] A preprocessing unit is used to capture the texture and edge information of the face image to be detected using a data preprocessing network to generate a reconstructed face image; the data preprocessing network includes at least a two-dimensional discrete wavelet transform module;

[0036] A multi-branch feature extraction unit is used to extract features from the reconstructed face image using a fusion multi-branch module of the feature extraction network, and to fuse the features extracted from each branch of the fusion multi-branch module using a feature fusion attention module of the feature extraction network to obtain fused features; the fusion multi-branch module includes mutually independent... Model, Model and Model;

[0037] A classification unit is used to map the fused features to an intermediate feature space using the fully connected layers of the classification network, and to utilize the classification network's... The layer generates a probability distribution of the category to which the face image to be detected belongs, and the category includes any one of real face image, fake face image, real anime face image, fake anime face image, real sketch face image, and fake sketch face image;

[0038] Among them, the The model includes fully connected layers and Dropout layers to enhance generalization ability;

[0039] The The model includes The structure includes an improved fully connected layer and three convolutional layers, wherein the improved fully connected layer includes a non-linear activation function. as well as layer;

[0040] The The model includes an improved classifier, which is used to make the... The features output by the model are the same as those described. Model and the Feature alignment of model output.

[0041] A multi-category image forgery detection device, the device comprising a processor and a memory:

[0042] The memory is used to store program code and transmit the program code to the processor;

[0043] The processor is used to execute the above-described multi-category image forgery detection method according to the instructions in the program code.

[0044] A computer-readable storage medium for storing program code for performing the above-described multi-category image forgery detection method.

[0045] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0046] This invention provides a multi-class image forgery detection method, apparatus, device, and medium, capable of recognizing real faces, forged faces, real anime faces, forged anime faces, real sketch faces, and forged sketch faces. A multi-branch network, by integrating the characteristics of pre-trained models such as EfficientNet, ResNet, and DenseNet, and combining them with a feature fusion attention layer, significantly improves the classification ability for complex image categories. To construct a high-quality forged sketch dataset, a forged image generation framework based on CycleGAN is implemented. Adversarial loss, cycle consistency loss, and perceptual loss are used to optimize the generation effect, enhancing the diversity and robustness of the dataset. Experimental results show that the proposed model exhibits excellent performance in multi-class classification tasks, with an average accuracy and F1 score of 98.12%, demonstrating excellent generalization ability in both real and forged image classification.

[0047] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0049] Figure 1 This is a flowchart of a multi-category image forgery detection method provided in an embodiment of the present invention;

[0050] Figure 2 This is a framework diagram of a multi-category image forgery detection method provided in an embodiment of the present invention;

[0051] Figure 3 This is a flowchart of the DWT conversion process provided in an embodiment of the present invention;

[0052] Figure 4a This is a flowchart illustrating the face image extraction process provided in an embodiment of the present invention;

[0053] Figure 4b This is a flowchart illustrating the process of extracting anime character images provided in an embodiment of the present invention;

[0054] Figure 4c This is a flowchart illustrating the sketch image extraction process provided in an embodiment of the present invention;

[0055] Figure 5 This is a multi-branch network architecture diagram provided in an embodiment of the present invention;

[0056] Figure 6 This is a diagram of the EfficientNet network structure provided in an embodiment of the present invention;

[0057] Figure 7 This is a diagram of the ResNet50 network structure provided in an embodiment of the present invention;

[0058] Figure 8 This is a diagram of the DenseNet121 network structure provided in an embodiment of the present invention;

[0059] Figure 9 This is a sample diagram illustrating the dataset provided in an embodiment of the present invention;

[0060] Figure 10 This is the Accuracy trend chart provided in the embodiments of the present invention;

[0061] Figure 11 This is a loss trend chart provided in an embodiment of the present invention;

[0062] Figure 12This is a schematic diagram of a multi-category image forgery detection device provided in an embodiment of the present invention;

[0063] Figure 13 This is a schematic diagram of a multi-category image forgery detection device provided in an embodiment of the present invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0065] See Figure 1 This invention provides a multi-category image forgery detection method, such as... Figure 1 As shown, the method may include:

[0066] S101: Obtain the image of the face to be detected;

[0067] S102: A data preprocessing network is used to capture the texture and edge information of the face image to be detected to generate a reconstructed face image; the data preprocessing network includes at least a two-dimensional discrete wavelet transform module; specifically, the two-dimensional discrete wavelet transform module is used to decompose the face image to be detected into low-frequency approximate components and three high-frequency detail components; and the decomposed low-frequency approximate components and three high-frequency detail components are reconstructed back into the original image through inverse wavelet transform in order to obtain the reconstructed face image.

[0068] To achieve edge enhancement for sketch-type images, this embodiment of the invention further provides that the data preprocessing network includes a Canny edge detection algorithm. The Canny edge detection algorithm enhances the edges of the sketch-type face image after determining that the face image to be detected is a sketch-type face image, so that the texture and edge information of the sketch-type face image can be captured by the two-dimensional discrete wavelet transform module to generate the reconstructed face image. After obtaining the face image to be detected, the type of the face image can be determined. If it is determined to be a sketch-type face image, edge enhancement is performed first, followed by feature extraction. If it is determined to be a non-sketch-type face image, feature extraction can be performed directly.

[0069] Furthermore, the aforementioned Edge detection algorithms include the following steps:

[0070] Apply a Gaussian filter to the image to smooth it and reduce noise;

[0071] use Operators compute images in and Gradient of direction;

[0072] The gradient value of each pixel is checked in the gradient direction. If the gradient value of a pixel is not the maximum value in its neighborhood in the gradient direction, it is suppressed to 0 and determined to be a non-edge.

[0073] Set a high threshold and a low threshold, and mark pixels with gradient magnitudes higher than the high threshold as strong edges, pixels with gradient magnitudes lower than the low threshold as non-edges, and pixels in between as weak edges;

[0074] By using hysteresis tracking, weak edges that are connected to strong edges are joined together to form complete edges.

[0075] S103: The reconstructed face image is feature-extracted using the fusion multi-branch module of the feature extraction network, and the features extracted by each branch of the fusion multi-branch module are fused through the feature fusion attention module of the feature extraction network to obtain fused features; the fusion multi-branch module includes mutually independent... Model, Model and Model;

[0076] In a specific implementation, embodiments of the present invention may provide a feature fusion attention module that fuses and optimizes features through nonlinear transformation and attention mechanisms to obtain the fused features.

[0077] The feature fusion attention module is used to perform the following operations:

[0078] The features extracted from each branch of the fusion multi-branch module are concatenated along the channel dimension to obtain a joint feature vector.

[0079] The joint feature vector is passed through a fully connected layer and The activation function performs a nonlinear transformation;

[0080] Another fully connected layer is used to further reduce the dimensionality, and the final classification result is output.

[0081] The fusion process of the feature fusion attention module is represented by the following formula:

[0082]

[0083] In the formula: express The model outputs a feature vector. express The model outputs feature vectors, express The model outputs feature vectors. This refers to a function used to concatenate two or more arrays. This represents the attention mechanism.

[0084] S104: The fused features are mapped to an intermediate feature space using the fully connected layers of the classification network, and the fused features are then utilized using the classification network's... The layer generates a probability distribution of the category to which the face image to be detected belongs, and the category includes any one of real face image, fake face image, real anime face image, fake anime face image, real sketch face image, and fake sketch face image;

[0085] Among them, the The model includes fully connected layers and Dropout layers to enhance generalization ability;

[0086] The The model includes The structure includes an improved fully connected layer and three convolutional layers, wherein the improved fully connected layer includes a non-linear activation function. as well as layer;

[0087] The The model includes an improved classifier, which is used to make the... The features output by the model are the same as those described. Model and the Feature alignment of model output.

[0088] The multi-category image forgery detection method provided in this invention employs a multi-branch network structure for image classification, offering a more accurate and robust classification model. By introducing residual connections and multi-scale feature fusion techniques, the recognition accuracy of forged images and the generalization ability of the model are significantly improved. Through the constructed multi-branch network model, combined with feature fusion technology, accurate classification of six different image categories can be achieved.

[0089] The method provided by this invention will be described in detail below.

[0090] like Figure 2 As shown, the core objective of the method provided in this embodiment of the invention is to develop a multi-branch network architecture to identify subtle anomalies in forged images, thereby improving the accuracy of forgery detection and its applicability across different datasets. To achieve this objective, a comprehensive system comprising a Discrete Wavelet Transform (DWT) feature extraction module and a feature fusion module is employed.

[0091] 1. Data preprocessing.

[0092] To construct the overall model, this embodiment of the invention builds an image dataset containing six categories: real faces, fake faces, real anime faces, fake anime faces, real sketch faces, and fake sketch faces. The dataset is derived from multiple public datasets and custom datasets to ensure data diversity and representativeness.

[0093] To improve the model's generalization ability, various data augmentation operations were performed on the images, including random horizontal flipping, random rotation, and color jittering. These operations help the model learn more robust feature representations.

[0094] ① Frequency domain feature extraction.

[0095] Two-dimensional discrete wavelet transform (DWT) was applied to images of all categories to deeply analyze their frequency characteristics and extract key frequency domain features. As a highly efficient image analysis tool, DWT can decompose an image into multiple frequency sub-bands, each revealing different aspects of the image.

[0096] 2D-DWT can extract image features at different scales, such as edges, textures, and shapes. By analyzing the coefficients after wavelet transform, edges and important features in the image can be accurately located, which is crucial for image analysis and recognition. Furthermore, Gabor transform utilizes filters to extract image features at different scales and directions, making it particularly suitable for texture recognition.

[0097] See Figure 3 The following are the key steps and mathematical expressions of DWT in image processing:

[0098] Two-dimensional discrete wavelet transform: for a given input image DWT breaks it down into four sub-bands:

[0099] LL (Low-Frequency Component): Represents approximate information about the image, including the main trends and low-frequency content.

[0100] LH (Horizontal High Frequency Component): Captures high-frequency variations in an image in the horizontal direction, such as edges and textures.

[0101] HL (Vertical High Frequency Component): Captures high-frequency variations in the vertical direction of an image.

[0102] HH (Diagonal High Frequency Component): Captures high-frequency variations along the diagonal of an image.

[0103] Mathematically, these components can be calculated using the following formula:

[0104]

[0105]

[0106]

[0107]

[0108] in, and These are the coefficients of the wavelet filter. Indicates the number of layers in the decomposition.

[0109] To match the size of the transformed image to the original image, the DWT result is resized. The resized low-frequency component LL is assigned back to the first channel of the image to preserve the main structural information. The high-frequency components (LH, HL, HH) obtained through DWT decomposition effectively capture the texture and edge information of the image, which is crucial for enhancing the model's ability to recognize sketched images.

[0110] This method not only supports subsequent feature extraction and image processing tasks but also significantly improves the model's ability to recognize and classify different types of images, including sketches. This DWT-based image analysis method provides a new perspective and tool for forged image detection. As shown in the figure, the original image is first decomposed into LL (low-frequency approximation) and three high-frequency detail components using DWT: LH (horizontal detail), HL (vertical detail), and HH (diagonal detail). These components can then be used for further processing. The decomposed sub-band images are reconstructed back into the original image using inverse wavelet transform. The effects of performing discrete wavelet transform on the three types of images are shown below. Figure 4a , Figure 4b , Figure 4c As shown.

[0111] ②Canny is enhanced.

[0112] For edge enhancement of sketch-type images, the Canny edge detection algorithm was employed. The Canny algorithm is a multi-stage edge detection method designed to find the locations of maximum gradient changes in an image, thereby determining the edge positions. The main steps of the algorithm include:

[0113] Gaussian filtering for noise reduction: First, a Gaussian filter is applied to the image to smooth it and reduce noise. The mathematical expression for a Gaussian filter is:

[0114]

[0115] in, The standard deviation, representing the Gaussian distribution, determines the smoothness of the Gaussian filter.

[0116] Calculate image gradient: Use the Sobel operator to calculate the gradient of the image in the x and y directions. Gradient magnitude. and gradient direction Determined by the following formula:

[0117]

[0118]

[0119] in, and They are direction and square The gradient in the direction.

[0120] Non-maximum suppression: Examine the gradient value of each pixel in the gradient direction. If the gradient value of the pixel is not the maximum value in its neighborhood in the gradient direction, suppress it to 0, i.e., non-edge.

[0121] Dual threshold detection: Set a high threshold and a low threshold. Pixels with gradient magnitudes higher than the high threshold are marked as strong edges, pixels with gradient magnitudes lower than the low threshold are marked as non-edges, and pixels in between are marked as weak edges.

[0122] Hysteresis tracking: By using hysteresis tracking, weak edges that are connected to strong edges are connected together to form complete edges.

[0123] For sketch images, the double-threshold detection step of the Canny edge detection algorithm is particularly important because sketch images typically contain clear outlines and lines, features that are crucial for image classification and recognition. The Canny algorithm highlights this edge information, providing key features for subsequent forgery detection.

[0124] To enable the model to effectively perform classification tasks, a technique called label encoding is employed, converting textual category names into numerical labels. This process is a crucial step in machine learning preprocessing, allowing the model to understand and process categorical data in numerical form. Label encoding plays a vital role in transforming abstract concepts into quantifiable data points. The data is given new life; each unique category name is assigned a unique numerical identifier, paving the way for model training and prediction.

[0125] 2. Model architecture.

[0126] like Figure 5 As shown, the model is based on , and Three independent branches were constructed. Each branch was optimized for its specific feature extraction capabilities. It was selected as a branch due to its high computational performance and good classification results; It was chosen because its deep residual structure can effectively alleviate the gradient vanishing problem; It was included in the model because its densely connected structure can improve the efficiency of feature utilization.

[0127] like Figure 6 As shown, Feature extraction, As the first branch of the model, it is known for its efficient feature extraction capabilities and model scaling strategies. Its core idea lies in evenly scaling the network's depth, width, and resolution, thereby optimizing computational efficiency and performance. Specifically, use The module incorporates depthwise separable convolution to reduce the number of parameters and computational cost. Meanwhile, the embodiments of this invention provide... The classifier part was redesigned, replaced with a fully connected layer, and added... Layers are added to enhance generalization capabilities.

[0128] like Figure 7 As shown, Feature extraction, the second branch is based on It effectively alleviates the vanishing gradient problem through residual connections, enabling deeper networks to be trained stably. Adopted The structure, by replacing the traditional two-layer design with three convolutional layers, improves the model's representational power while maintaining a relatively low number of parameters. Simultaneously, for... The fully connected layers were modified to suit the needs of classification tasks, and a non-linear activation function was added. and This adds layers, thereby further enhancing the robustness of the model.

[0129] like Figure 8 As shown, Feature extraction, the third branch uses Its notable feature is that it achieves efficient feature reuse through a dense connection strategy. Each layer is directly connected to all preceding layers, significantly reducing parameter redundancy and improving gradient propagation efficiency. This structure is particularly advantageous when processing small sample data, helping to capture more fine-grained features. In the model implementation, The classifier part was also readjusted to ensure that the output features are aligned with the features of other branches.

[0130] The multi-branch deep learning model provided in this embodiment of the invention employs a feature fusion attention layer to integrate different branches. The extracted features enhance the performance of classification decisions. The core idea of ​​this module is to use an attention mechanism to weight the concatenated multi-branch features, making the classification process focus more on features that contribute to the target task, while suppressing irrelevant or redundant information.

[0131] The advantage of multi-branch networks is that each branch can extract features from different perspectives, such as... It excels at extracting fine-grained features. It has a strong ability to capture hierarchical features, while It can efficiently utilize feature flow. However, simply concatenating the outputs of these branches may not fully exploit the complementarity between features. Therefore, a feature fusion attention layer is introduced, which fuses and optimizes features through nonlinear transformation and attention mechanisms, assigning higher weights to important features.

[0132] The feature fusion attention layer includes the following key steps:

[0133] Feature splicing: Three branches The output feature vectors are respectively , , Concatenate them along the channel dimension to obtain the joint feature vector:

[0134]

[0135] The concatenated feature dimension is 256×3=768.

[0136] Nonlinear Transformation: The joint features undergo a nonlinear transformation through a fully connected layer and a ReLU activation function to learn the high-dimensional relationships between different features.

[0137]

[0138] in, 512 is the weight matrix, and 512 is the dimension of the intermediate hidden layer.

[0139] Attention-weighted: Another fully connected layer is used to further reduce the dimensionality, outputting the final classification result.

[0140]

[0141] in, C represents the number of categories.

[0142] The complete feature fusion process can be summarized as follows:

[0143]

[0144] The feature fusion attention layer effectively integrates features extracted from different branches through feature concatenation and attention weighting, fully leveraging their complementarity. Simultaneously, this module highlights key features and suppresses redundant information, thereby improving the model's classification performance. Its design is simple and flexible, applicable not only to multi-branch networks but also easily transferable to other complex classification tasks, demonstrating strong adaptability and versatility. It aims to maximize the utilization of the characteristics of multi-branch networks, further ensuring the model's classification performance and providing solid theoretical support for subsequent experimental validation.

[0145] In the feature fusion stage, the three branches The output feature vectors are concatenated to form a comprehensive feature representation. This fusion method not only integrates the perspectives of different branches, but also significantly improves the model's ability to distinguish between complex categories, especially in classification tasks involving multimodal data.

[0146] The classifier part consists of a fully connected layer and The system consists of layers, where fully connected layers map the fused features to an intermediate feature space, while... The next layer generates probability distributions for the six categories, completing the final classification task. This design efficiently handles high-dimensional features, ensuring the accuracy and stability of the classification results.

[0147] Training Strategy. To fully optimize model performance and improve generalization ability, this invention adopts an efficient training strategy, covering loss function selection, optimizer configuration, learning rate scheduling, and model evaluation methods, ensuring stable model performance in complex tasks.

[0148] First, the loss function used is cross-entropy loss, which is mathematically expressed as:

[0149]

[0150] in, For the sample size, For the number of categories, and Representing samples respectively In category The true label and predicted probability are compared. This loss function effectively measures the difference between the predicted distribution and the true distribution, providing a reliable optimization objective for classification tasks. The optimizer chosen is AdamW, which combines the fast convergence of the Adam optimizer with the regularization effect of weight decay, effectively suppressing overfitting of model parameters. The specific optimization steps are as follows:

[0151]

[0152] in, For learning rate, and These are the momentum and second moment estimates, respectively. This represents the weight decay coefficient. Secondly, the learning rate scheduling employs a cosine annealing warp restart scheduler, with the following scheduling formula:

[0153]

[0154] in, and These are the minimum and maximum values ​​of the learning rate, respectively. The number of steps in the current period. The learning rate is the period length. By dynamically adjusting the learning rate, this strategy can achieve rapid convergence in the early stages while gradually and finely optimizing the weights in the later stages, thus avoiding getting trapped in local optima.

[0155] Finally, model evaluation was performed using StratifiedKFold cross-validation, which employed a stratified sampling strategy to ensure consistency in the proportion of samples from each class. For each fold of the training and validation process, the model was optimized and tested on both the training and validation sets, ensuring a comprehensive evaluation of model performance. The stratified sampling process can be represented as:

[0156]

[0157] in, For the number of categories, Indicates the first Compromise from the first Samples were extracted from the class. This method was used to verify the stability and robustness of the model under different data distributions.

[0158] In summary, this training strategy effectively balances model performance optimization and generalization ability, laying a solid foundation for subsequent experimental verification and practical applications.

[0159] The experiments were conducted on a workstation equipped with an NVIDIA GeForce RTX 4060 GPU, fully leveraging its powerful parallel computing capabilities to accelerate model training and inference. The software environment included the PyTorch deep learning framework, CUDA 12.5, and cuDNN 7.6.5, ensuring a high degree of compatibility between computational efficiency and hardware performance. To comprehensively evaluate the model's performance, a StratifiedKFold cross-validation strategy was employed to hierarchically partition the dataset, further dividing the data into training and validation sets within each fold. This method maintains a consistent distribution ratio of samples from each class across different subsets of the data, thereby improving the reliability and fairness of the model evaluation results.

[0160] To construct a dataset of fake sketch images, this invention provides an unsupervised image-to-image transformation framework based on CycleGAN, used to convert real face images into sketch-style images. The CycleGAN model establishes a mapping relationship between the source domain (real face) and the target domain (sketch) through adversarial learning between the generator and discriminator, generating high-quality fake sketch images. The generator employs a ResNet residual network structure, combined with channel and spatial attention mechanisms (CBAM) to enhance the expressive power of key features; the discriminator is designed based on PatchGAN, performing fine-grained realism judgments on local image regions to ensure the realism of the generated images in both local details and overall style.

[0161] During training, the model incorporates various loss functions to optimize the generated results. Adversarial loss (GAN Loss) is used to constrain the consistency between the generated image and the true distribution of the target domain.

[0162] in, and These are the generator and the discriminator, respectively. and These represent the images in the source and target domains, respectively. Cycle Consistency Loss constrains the images as they pass through the bidirectional generator. and The conversion can restore the original sample, thereby improving the fidelity of the generated image.

[0163] Furthermore, perceptual loss ensures consistency between the generated and target images at the visual perception level by extracting high-level features; edge loss further enhances the detail representation of the generated image by comparing the edge information of the generated and target images. The comprehensive design of these loss functions effectively guarantees a high degree of consistency in style and detail in the generated forged sketch images.

[0164] The training process employs the Adam optimizer combined with a dynamic learning rate scheduling strategy to alternately update the parameters of the generator and discriminator. In each training round, the generator is responsible for generating fake sketch images and optimizing the generation quality through adversarial loss, cycle consistency loss, perceptual loss, and edge loss; the discriminator, on the other hand, gradually improves its ability to distinguish between real and fake samples, thus enhancing its ability to discriminate the authenticity of the generated images. After multiple rounds of iterative training, the generator and discriminator reach a stable adversarial balance, and the final generated fake sketch images exhibit excellent performance in terms of texture detail, style consistency, and edge features.

[0165] CycleGAN successfully generated high-quality fake sketch images, providing crucial support for data construction in fake detection tasks. This method not only enriches the diversity of datasets but also provides more samples for training fake detection models, significantly enhancing their ability to learn fake features. The introduction of CycleGAN demonstrates the potential of unsupervised generation techniques in complex image transformation tasks and provides a pathway for expanding fake datasets.

[0166] Dataset Construction. To comprehensively validate the model's performance in multi-class classification tasks, this invention uses diverse data sources, covering face images of different styles and types. Specifically, these include the following categories:

[0167] Real Faces vs. Fake Faces: This section of data comes from the FaceForensics++ and ForgeryNet datasets. Real face images are presented with their original attributes, while fake faces are generated using various deepfake techniques, including DeepFakes, Face2Face, and other common forgery methods.

[0168] Real Anime Faces vs. Fake Anime Faces: The anime style dataset is divided into two categories: real and fake. Real anime faces are taken from the high-resolution anime face dataset, while fake anime faces are generated by the AnimeGAN model, providing rich training samples for style transformation of anime images.

[0169] Realistic sketched faces: The realistic sketched face data combines the CUFS dataset from the Chinese University of Hong Kong with related resources provided by Kaggle, covering face images of various sketch styles, providing diverse training data for the model's classification ability in this category.

[0170] Fake Sketch Faces: Fake sketch faces are generated using a self-designed and trained CycleGAN model. CycleGAN converts real face images into sketch styles through unsupervised learning, and utilizes generative adversarial networks and cycle consistency loss to ensure the quality and diversity of generated images, providing high-quality sample support for the fake face category.

[0171] By integrating the above data sources, a diverse dataset was constructed that includes real and fake data, different styles (such as anime and sketches), and various forgery techniques, laying a solid foundation for a comprehensive evaluation of model performance.

[0172] Table 1 Dataset Details

[0173]

[0174] To provide a more intuitive understanding, some representative samples from the dataset used are presented. These samples include real human faces and fake face images created using different generation techniques. Figure 9 As shown, real images use solid borders, while fake images use dashed borders.

[0175] In the experiment, accuracy, precision, recall, and F1 score were used as evaluation metrics to comprehensively reflect the classification performance of the model.

[0176] Accuracy: Measures the proportion of samples correctly classified by the model out of the total number of samples. The formula is as follows:

[0177]

[0178] In this context, TP represents a True Positive, TN represents a True Negative, FP represents a False Positive, and FN represents a False Negative.

[0179] Precision: Measures the proportion of cases where the model predicts a positive class but the actual class is positive. The formula is:

[0180]

[0181] High accuracy means fewer false positives.

[0182] Recall: Measures the proportion of actual positive classes that are correctly predicted as positive by the model. Calculation formula:

[0183]

[0184] A high recall rate means fewer false negatives.

[0185] F1 score: The harmonic mean of precision and recall, which balances precision and recall. The formula is:

[0186]

[0187] During each training and validation cycle, the loss value, accuracy, and F1 score were recorded during the training and testing phases. This comprehensive recording method not only intuitively reflects the model's learning progress but also effectively monitors its generalization ability under different data distributions.

[0188] Furthermore, sensitivity analyses were performed on key hyperparameters, including the effects of learning rate and batch size. The choice of learning rate is crucial to the model's convergence and performance. A lower learning rate provides a stable training process but results in slower convergence; a higher learning rate may lead to training oscillations or overfitting. A cosine annealing learning rate scheduler was used to dynamically adjust the learning rate, while simultaneously selecting... As an initial value, it balances convergence speed and training stability.

[0189] The choice of batch size directly affects the efficiency and stability of model training. Smaller batch sizes help reduce gradient fluctuations during training but are less computationally efficient; while larger batch sizes improve training efficiency to some extent but may reduce the model's ability to capture complex features. After comprehensive analysis, 8 was chosen as the optimal batch size setting, balancing efficiency and model performance.

[0190] After 15 training epochs, the face forgery detection model integrating multi-branch networks and generative adversarial models was thoroughly evaluated. The training and validation curves show that the model performs excellently in terms of accuracy and robustness, demonstrating strong learning capabilities and superior generalization performance.

[0191] During the training phase, the loss curve remains stable, as shown below. Figure 10 , Figure 11 As shown, the initial value was approximately 0.001, and it steadily decreased over several cycles, eventually approaching 0.0008, indicating that the model can efficiently extract key features from the training data. Furthermore, although the validation loss showed a brief peak in the 8th cycle, it quickly recovered to a low level and stabilized, demonstrating the model's adaptability and strong robustness against different data distributions.

[0192] The accuracy curves also demonstrate the model's superior performance. Training accuracy started at 0.996, steadily climbing to a high level of nearly 0.998 within the first 10 epochs, then fluctuating slightly but remaining stable around 0.994. This indicates that the model has extremely high accuracy in learning and representing features during the training phase. Validation accuracy consistently remained around 0.998, showcasing the model's high generalization ability to unseen data. Ultimately, the accuracy stabilized at a level close to the theoretical upper limit, further demonstrating the model's robustness and reliability.

[0193] Overall, the model demonstrates excellent performance in terms of loss value and accuracy, showcasing its advanced capabilities in face forgery detection. Brief fluctuations in the validation loss did not substantially impact overall performance; instead, they highlighted the model's ability to quickly adapt to diverse data distributions. Combining the architectural advantages of multi-branch networks and generative adversarial models, the model not only exhibits significant advantages in learning speed but also achieves leading levels in classification accuracy and generalization performance, providing strong technical support for the field of face forgery detection.

[0194] Model Comparison. A comprehensive performance comparison of multi-branch network models with various mainstream models was conducted, including single-branch models (and multi-branch fusion models). Single-branch models, as classic feature extraction networks, demonstrate high accuracy and stability in deep forgery detection, but their capabilities are limited by the representation of a single feature. In contrast, multi-branch networks achieve deep feature fusion by integrating multiple feature extraction branches and combining the characteristics of different branches (such as the efficiency of EfficientNet, the residual properties of ResNet, and the feature reuse capabilities of DenseNet).

[0195] Table 2. Accuracy results on the FaceForensics++ dataset / %

[0196]

[0197] Based on comparative experiments, the performance of the proposed multi-branch model in a six-class classification task was further evaluated, covering three categories of data: real and fake sketches, anime, and faces. The classification accuracy and F1 score of the model in different categories are shown in Table 3. Overall, the results demonstrate that the model exhibits excellent performance in distinguishing between real and fake data, especially in the anime and sketch categories, where both accuracy and F1 score are near perfect. However, the model's performance slightly decreases in the detection of some complex forgery methods (such as FaceSwap and NeuralTextures), reflecting the diverse challenges posed by different forgery techniques. In summary, these results further validate the effectiveness and robustness of multi-branch networks in enhancing model discrimination through feature fusion, and also indicate future optimization directions.

[0198] Table 3. Accuracy and F1 / % of each dataset on the model.

[0199]

[0200] As can be seen, the method provided in this embodiment of the invention focuses on distinguishing six categories of images: real faces, fake faces, real anime faces, fake anime faces, real sketch faces, and fake sketch faces. Experiments have verified that the model demonstrates excellent performance and strong generalization ability in classification tasks. In the model design, the multi-branch network integrates the characteristics of different pre-trained models, fully utilizing the unique advantages of each branch to effectively improve the classification performance for complex image categories. Furthermore, the introduction of feature fusion strategies and attention mechanisms allows the model to more accurately focus on features crucial to classification decisions, thereby further improving the accuracy and reliability of classification.

[0201] Regarding model robustness, the combination of data augmentation and wavelet transform techniques enables the model to better adapt to different image variations, particularly excelling in processing sketch-style images. Although the multi-branch structure increases computational complexity to some extent, experimental results show that the performance improvement brought about by this computational overhead is significant, especially in classification accuracy and F1 score, demonstrating the feasibility and effectiveness of this design in practical applications.

[0202] This method not only provides a novel multi-branch network structure but also validates its effectiveness in multi-class image classification tasks through extensive experiments, offering a feasible solution for future research on similar problems. Simultaneously, it explores directions for model optimization and generalization capabilities, laying the foundation for designing more efficient and robust image classification models. Although the core task of this method focuses on specific classification scenarios, the proposed network structure and method possess high versatility and cross-domain application potential, and can be extended to more complex image analysis tasks in the future.

[0203] In summary, the multi-class image forgery detection method provided in this invention can recognize real faces, forged faces, real anime faces, forged anime faces, real sketch faces, and forged sketch faces. The multi-branch network, by integrating the characteristics of pre-trained models such as EfficientNet, ResNet, and DenseNet, and combining them with a feature fusion attention layer, significantly improves the classification ability for complex image categories. To construct a high-quality forged sketch dataset, a forged image generation framework based on CycleGAN was implemented. Adversarial loss, cycle consistency loss, and perceptual loss were used to optimize the generation effect, enhancing the diversity and robustness of the dataset. Experimental results show that the proposed model exhibits excellent performance in multi-class classification tasks, with an average accuracy and F1 score of 98.12%, demonstrating excellent generalization ability in both real and forged image classification.

[0204] See Figure 12 The present invention can also provide a multi-category image forgery detection device, such as... Figure 12 As shown, the apparatus for performing the above-described multi-category image forgery detection method may include:

[0205] Image acquisition unit 1201 is used to acquire the face image to be detected;

[0206] Preprocessing unit 1202 is used to capture the texture and edge information of the face image to be detected using a data preprocessing network to generate a reconstructed face image; the data preprocessing network includes at least a two-dimensional discrete wavelet transform module;

[0207] The multi-branch feature extraction unit 1203 is used to extract features from the reconstructed face image using the fusion multi-branch module of the feature extraction network, and to fuse the features extracted by each branch of the fusion multi-branch module through the feature fusion attention module of the feature extraction network to obtain fused features; the fusion multi-branch module includes mutually independent... Model, Model and Model;

[0208] Classification unit 1204 is used to map the fused features to an intermediate feature space using the fully connected layer of the classification network, and to utilize the classification network's... The layer generates a probability distribution of the category to which the face image to be detected belongs, and the category includes any one of real face image, fake face image, real anime face image, fake anime face image, real sketch face image, and fake sketch face image;

[0209] Among them, the The model includes fully connected layers and Dropout layers to enhance generalization ability;

[0210] The The model includes The structure includes an improved fully connected layer and three convolutional layers, wherein the improved fully connected layer includes a non-linear activation function. as well as layer;

[0211] The The model includes an improved classifier, which is used to make the... The features output by the model are the same as those described. Model and the Feature alignment of model output.

[0212] This invention can also provide a multi-category image forgery detection device, the device including a processor and a memory:

[0213] The memory is used to store program code and transmit the program code to the processor;

[0214] The processor is used to execute the steps of the multi-category image forgery detection method described above according to the instructions in the program code.

[0215] like Figure 13 As shown in the figure, an embodiment of the present invention provides a multi-category image forgery detection device, which may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 all communicate with each other through the communication bus 13.

[0216] In this embodiment of the invention, the processor 10 may be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0217] The processor 10 can call programs stored in the memory 11. Specifically, the processor 10 can execute operations in the embodiments of the multi-category image forgery detection method.

[0218] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment of the invention, the memory 11 stores at least a program for implementing the following functions:

[0219] Acquire the image of the face to be detected;

[0220] A data preprocessing network is used to capture the texture and edge information of the face image to be detected to generate a reconstructed face image; the data preprocessing network includes at least a two-dimensional discrete wavelet transform module;

[0221] The reconstructed face image is feature-extracted using a fusion multi-branch module of a feature extraction network, and the features extracted from each branch of the fusion multi-branch module are fused using a feature fusion attention module of the feature extraction network to obtain fused features; the fusion multi-branch module includes mutually independent... Model, Model and Model;

[0222] The fused features are mapped to an intermediate feature space using the fully connected layers of the classification network, and the classification network's... The layer generates a probability distribution of the category to which the face image to be detected belongs, and the category includes any one of real face image, fake face image, real anime face image, fake anime face image, real sketch face image, and fake sketch face image;

[0223] Among them, the The model includes fully connected layers and Dropout layers to enhance generalization ability;

[0224] The The model includes The structure includes an improved fully connected layer and three convolutional layers, wherein the improved fully connected layer includes a non-linear activation function. as well as layer;

[0225] The The model includes an improved classifier, which is used to make the... The features output by the model are the same as those described. Model and the Feature alignment of model output.

[0226] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0227] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0228] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0229] Of course, it should be noted that, Figure 13 The structure shown does not constitute a limitation on the multi-category image forgery detection device in the embodiments of the present invention. In practical applications, the multi-category image forgery detection device may include more than Figure 13 More or fewer components as shown, or combinations of certain components.

[0230] This invention can also provide a computer-readable storage medium for storing program code for executing the steps of the multi-category image forgery detection method described above.

[0231] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0232] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0233] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0234] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A multi-category image forgery detection method, characterized in that, include: Acquire the image of the face to be detected; A data preprocessing network is used to capture the texture and edge information of the face image to be detected to generate a reconstructed face image; The data preprocessing network includes at least a two-dimensional discrete wavelet transform module and... Edge detection algorithm, the An edge detection algorithm is used to enhance the edges of the sketch face image after determining that the face image to be detected is a sketch face image, so as to capture the texture and edge information of the sketch face image using the two-dimensional discrete wavelet transform module to generate the reconstructed face image; the two-dimensional discrete wavelet transform module is used to decompose the face image to be detected into low-frequency approximate components and three high-frequency detail components; and to reconstruct the decomposed low-frequency approximate components and three high-frequency detail components back to the original image through inverse wavelet transform, so as to obtain the reconstructed face image; The reconstructed face image is feature extracted using the fusion multi-branch module of the feature extraction network, and the features extracted by each branch of the fusion multi-branch module are fused to obtain fused features through the feature fusion attention module of the feature extraction network. The fusion multi-branch module includes mutually independent modules. Model, Model and Model; The fused features are mapped to an intermediate feature space using the fully connected layers of the classification network, and the classification network's... The layer generates a probability distribution of the category to which the face image to be detected belongs, and the category includes any one of real face image, fake face image, real anime face image, fake anime face image, real sketch face image, and fake sketch face image; Among them, the The model includes fully connected layers and Dropout layers to enhance generalization ability; The The model includes The structure includes an improved fully connected layer and three convolutional layers, wherein the improved fully connected layer includes a non-linear activation function. as well as layer; The The model includes an improved classifier, which is used to make the... The features output by the model are the same as those described. Model and the Feature alignment of model output; The feature fusion attention module fuses and optimizes features through nonlinear transformation and attention mechanisms to obtain the fused features; the feature fusion attention module is used to perform the following operations: The features extracted from each branch of the fusion multi-branch module are concatenated along the channel dimension to obtain a joint feature vector; The joint feature vector is passed through a fully connected layer and The activation function performs a nonlinear transformation; Another fully connected layer is used to further reduce the dimensionality, and the final classification result is output. The fusion process of the feature fusion attention module is represented by the following formula: In the formula: express The model outputs feature vectors. express The model outputs feature vectors, express The model outputs feature vectors. This refers to a function used to concatenate two or more arrays. This represents the attention mechanism.

2. The multi-category image forgery detection method according to claim 1, characterized in that, The Edge detection algorithms include the following steps: Apply a Gaussian filter to the image to smooth it and reduce noise; use Operators compute images in and Gradient of direction; The gradient value of each pixel is checked in the gradient direction. If the gradient value of a pixel is not the maximum value in its neighborhood in the gradient direction, it is suppressed to 0 and determined to be a non-edge. Set a high threshold and a low threshold, and mark pixels with gradient magnitudes higher than the high threshold as strong edges, pixels with gradient magnitudes lower than the low threshold as non-edges, and pixels in between as weak edges; By using hysteresis tracking, weak edges that are connected to strong edges are joined together to form complete edges.

3. A multi-category image forgery detection device, characterized in that, The apparatus for performing the multi-category image forgery detection method according to any one of claims 1-2, the apparatus comprising: The image acquisition unit is used to acquire the image of the face to be detected. A preprocessing unit is used to capture the texture and edge information of the face image to be detected using a data preprocessing network to generate a reconstructed face image; the data preprocessing network includes at least a two-dimensional discrete wavelet transform module; A multi-branch feature extraction unit is used to extract features from the reconstructed face image using a fusion multi-branch module of the feature extraction network, and to fuse the features extracted from each branch of the fusion multi-branch module using a feature fusion attention module of the feature extraction network to obtain fused features; the fusion multi-branch module includes mutually independent... Model, Model and Model; A classification unit is used to map the fused features to an intermediate feature space using the fully connected layers of the classification network, and to utilize the classification network's... The layer generates a probability distribution of the category to which the face image to be detected belongs, and the category includes any one of real face image, fake face image, real anime face image, fake anime face image, real sketch face image, and fake sketch face image; Among them, the The model includes fully connected layers and Dropout layers to enhance generalization ability; The The model includes The structure includes an improved fully connected layer and three convolutional layers, wherein the improved fully connected layer includes a non-linear activation function. as well as layer; The The model includes an improved classifier, which is used to make the... The features output by the model are the same as those described. Model and the Feature alignment of model output.

4. A multi-category image forgery detection device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the multi-category image forgery detection method according to any one of claims 1-2 according to the instructions in the program code.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the multi-category image forgery detection method according to any one of claims 1-2.

Images