Image forgery detection method and device based on multi-view representation preprocessing

Abstract

The invention relates to an image forgery detection method and device based on multi-view representation preprocessing, and is applied to the technical field of computer vision, and the method comprises the steps: processing a to-be-detected image through N preprocessing modes, and obtaining N groups of to-be-detected features; inputting the N groups of to-be-detected features into a multi-view counterfeit detection model, and performing feature extraction on the N groups of to-be-detected features to obtain N groups of to-be-detected feature vectors; routing the N groups of to-be-detected feature vectors to N perspective expert networks by using a gating network in the hybrid expert module, and outputting N groups of to-be-detected optimized feature vectors; respectively routing N groups of to-be-detected feature vectors output by the target feature extraction layer and N groups of to-be-detected optimized feature vectors output by the first M-1 hybrid expert modules to L high-dimensional feature expert networks by using a gating network in the last hybrid expert module, and outputting a fused to-be-detected feature vector; and by extracting and processing the classification mark, outputting a counterfeit category. And the detection precision can be improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image forgery detection method and apparatus based on multi-view representation preprocessing. Background Technology

[0002] With the development of deep learning technology in the field of computer vision, the Visual Transformer (ViT) has gradually replaced the traditional convolutional neural network as the core architecture for image classification tasks due to its powerful global feature modeling capabilities.

[0003] For example, in the field of forgery detection, images can be preprocessed before being input into a visual Transformer for forgery detection. However, a single preprocessing method cannot adapt to the feature differences of different types of forgery data, resulting in low detection accuracy. For instance, normalized preprocessing can only identify semantically different forgeries such as traditional face swapping through brightness calibration, and its performance is poor in identifying frequency / trace differences in forgeries from novel AIGC (Artificial Intelligence Generated Content) diffusion models. While pixel remapping preprocessing can capture pixel-level forgery traces, it easily loses the semantic association anomalies of traditional forgeries, leading to missed detections. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides an image forgery detection method, apparatus, electronic device, storage medium, and computer program product based on multi-view characterization preprocessing.

[0005] According to a first aspect of this application, a method for training an image forgery detection model based on multi-view representation preprocessing is provided, comprising: The image to be detected is acquired, and the image is processed using N preprocessing methods to obtain N sets of features to be detected; N is an integer greater than 1. The N sets of features to be detected are input into a pre-trained multi-view forgery detection model constructed based on a visual feature extraction network, M hybrid expert modules, and a classification head; wherein, the first M-1 hybrid expert modules are located after the M-1 feature extraction layers of the visual feature extraction network; the last hybrid expert module is located after all feature extraction layers and before the classification head; The visual feature extraction network is used to extract features from the N sets of features to be detected, resulting in N sets of feature vectors to be detected. For each of the first M-1 hybrid expert modules, the gating network in the hybrid expert module is used to route the N sets of feature vectors to be detected input to the hybrid expert module to the corresponding N viewpoint expert networks in the hybrid expert module, and output the N sets of optimized feature vectors to be detected after optimization by the N viewpoint expert networks. The gated network in the last hybrid expert module routes the N sets of target feature vectors output by the target feature extraction layer and the N sets of optimized target feature vectors output by the first M-1 hybrid expert modules to L high-dimensional feature expert networks in the hybrid expert module, respectively, and outputs a fused target feature vector after being fused by the high-dimensional feature expert networks; the target feature extraction layer is one or more of the other feature extraction layers besides the M-1 feature extraction layers; M and L are both integers greater than 1 and less than K; K is the total number of all feature extraction layers; The classification label is extracted from the fused feature vector to be detected, and the classification head is used to process the classification label to output the fake category.

[0006] Optionally, the method further includes: Obtain multiple sample images and the actual forgery category corresponding to each sample image; For the N+1th preprocessing method, add the perspective expert network corresponding to the N+1th preprocessing method to each of the first M-1 hybrid expert modules in the multi-view forgery detection model to obtain an updated multi-view forgery detection model. Each sample image is processed using N+1 preprocessing methods to obtain N+1 sets of preprocessed features; The N+1 preprocessed features are input into the updated multi-view forgery detection model, and the visual feature extraction network is used to extract features from the N+1 preprocessed features to obtain N+1 feature vectors. For each of the first M-1 hybrid expert modules, a gating network is used to route the N+1 sets of feature vectors input to the hybrid expert module to the corresponding N+1 view expert networks, and output the N+1 sets of optimized feature vectors after optimization by the view expert networks. The gated network in the last hybrid expert module routes the N+1 sets of feature vectors output by the target feature extraction layer and the N+1 sets of optimized feature vectors output by the first M-1 hybrid expert modules to the L high-dimensional feature expert networks in the hybrid expert module, and outputs the fused feature vector after being fused by the high-dimensional feature expert networks. The classification label is extracted from the fused feature vector, and the classification label is processed using the classification head to output the predicted fake category; The loss value is calculated based on the actual forgery category and the predicted forgery category of the sample image. The network parameters of the viewpoint expert network corresponding to the N+1th preprocessing method are updated based on the loss value, and other network parameters are frozen to generate the updated multi-view forgery detection model after training.

[0007] Optionally, each of the first M-1 hybrid expert modules includes: N low-rank adaptive (LoRA) fine-tuning structures, with each of the N LoRA fine-tuning structures corresponding to one of the N viewpoint expert networks. The LoRA fine-tuning structures are used to fine-tune the network parameters of the corresponding viewpoint expert network.

[0008] Optionally, N is 4, and the process of processing a single sample image using N preprocessing methods includes: Normalize individual sample images; Perform pixel remapping on a single sample image; Perform a Fast Fourier Transform on a single sample image; Perform data augmentation on a single sample image.

[0009] Optionally, the M-1 feature extraction layers are located in the middle of all feature extraction layers in the visual feature extraction network.

[0010] According to a second aspect of this application, an image forgery detection apparatus based on multi-view characterization preprocessing is provided, comprising: The image acquisition module is used to acquire the image to be detected; The image preprocessing module is used to process the image to be detected using N preprocessing methods to obtain N sets of features to be detected; N is an integer greater than 1. The feature input module is used to input the N sets of features to be detected into a pre-trained multi-view forgery detection model constructed based on a visual feature extraction network, M hybrid expert modules, and a classification head; wherein, the first M-1 hybrid expert modules are located after the M-1 feature extraction layers of the visual feature extraction network; the last hybrid expert module is located after all feature extraction layers and before the classification head; The feature extraction module is used to extract features from the N sets of features to be detected using a visual feature extraction network, so as to obtain N sets of feature vectors to be detected. The feature optimization module is used to route the N sets of feature vectors to be detected input to the hybrid expert module to the corresponding N viewpoint expert networks in the hybrid expert module for each of the first M-1 hybrid expert modules, and output the N sets of optimized feature vectors to be detected after optimization by the N viewpoint expert networks. The feature fusion module is used to route the N sets of target feature vectors output by the target feature extraction layer and the N sets of optimized target feature vectors output by the first M-1 hybrid expert modules to L high-dimensional feature expert networks in the hybrid expert module, respectively, and output a fused target feature vector after being fused by the high-dimensional feature expert networks. The target feature extraction layer is one or more feature extraction layers other than the M-1 feature extraction layers. M and L are both integers greater than 1 and less than K. K is the total number of all feature extraction layers. The forgery category output module is used to extract classification labels from the fused feature vector to be detected, process the classification labels using the classification head, and output the forgery category.

[0011] Optionally, the device further includes: The training data acquisition module is used to acquire multiple sample images and the actual forgery category corresponding to each sample image; The model structure update module is used to add the perspective expert network corresponding to the N+1th preprocessing method to each of the first M-1 hybrid expert modules in the multi-view forgery detection model for the N+1th preprocessing method, so as to obtain an updated multi-view forgery detection model. The model update module is used to process a single sample image using N+1 preprocessing methods to obtain N+1 sets of preprocessed features; the N+1 sets of preprocessed features are input into the updated multi-view forgery detection model, and the visual feature extraction network is used to extract features from the N+1 sets of preprocessed features to obtain N+1 sets of feature vectors. For each of the first M-1 hybrid expert modules, a gating network is used to route the N+1 sets of feature vectors input to the hybrid expert module to the corresponding N+1 viewpoint expert networks, outputting N+1 sets of optimized feature vectors after optimization by the viewpoint expert networks. The gating network in the last hybrid expert module routes the N+1 sets of feature vectors output from the target feature extraction layer and the N+1 sets of optimized feature vectors output from the first M-1 hybrid expert modules to L high-dimensional feature expert networks in the hybrid expert module, outputting a fused feature vector fused by the high-dimensional feature expert networks. Classification labels are extracted from the fused feature vector, and the classification labels are processed using a classification head to output a predicted forgery category. A loss value is calculated based on the actual forgery category and the predicted forgery category of the sample image. The network parameters of the viewpoint expert network corresponding to the N+1th preprocessing method are updated based on the loss value, and other network parameters are frozen to generate a trained and updated multi-view forgery detection model.

[0012] Optionally, each of the first M-1 hybrid expert modules includes: N LoRA fine-tuning structures, each LoRA fine-tuning structure corresponding to one of the N viewpoint expert networks, and the LoRA fine-tuning structure is used to fine-tune the network parameters of the corresponding viewpoint expert network.

[0013] Optionally, N is 4, and the image preprocessing module is specifically used to normalize a single sample image; remap pixels in a single sample image; perform fast Fourier transform on a single sample image; and perform data augmentation on a single sample image to obtain 4 sets of features to be detected.

[0014] Optionally, the M-1 feature extraction layers are located in the middle of all feature extraction layers in the visual feature extraction network.

[0015] According to a third aspect of this application, an electronic device is provided, comprising: a processor configured to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the method described in the first aspect.

[0016] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0017] According to a fifth aspect of this application, a computer program product is provided that, when the computer program product is run on a computer, causes the computer to perform the method described in the first aspect.

[0018] The technical solution provided in this application has the following advantages compared with the prior art: Multiple preprocessing methods are used to process the images to be detected in parallel, resulting in multi-view representation features. Through multi-dimensional feature complementarity, the limitations of a single preprocessing method are fundamentally overcome, providing comprehensive and highly discriminative input features for subsequent deep networks. Hybrid expert modules are added after multiple feature extraction layers of the visual feature extraction network. Each hybrid expert module has a viewpoint expert network corresponding to a preprocessing method, allowing each viewpoint expert network to deepen and refine features from a specific perspective, effectively avoiding mutual inhibition between features of different dimensions and significantly improving feature discriminative power. Hybrid expert modules are added after all feature extraction layers. These modules can perform cross-layer, macro-level expert routing, enabling high-level fusion and comprehensive judgment of forgery clues at the decision-making level. The cooperation of these two levels of hybrid expert modules ensures proper processing of multi-view representation features and achieves feature abstraction and integration across forgery types at a high level. This avoids feature interference and modality bias problems that may occur when multi-view representation features enter the backbone of the visual feature extraction network, enhancing the model's generalization ability and decision robustness. It is evident that the dynamic routing enhancement system of multi-view representation preprocessing and two-level hybrid expert modules constitutes a tightly coupled, progressively layered feature understanding system, which improves feature utilization efficiency and model robustness. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of an image forgery detection method in an embodiment of this application; Figure 2 This is a schematic diagram of the image forgery detection architecture in an embodiment of this application; Figure 3 This is a flowchart of a model update method in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an image forgery detection device in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device in an embodiment of this application. Detailed Implementation

[0022] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0024] With the development of technology, forgery techniques have continuously evolved, progressing from traditional image editing software-based forgery to AI-based forgery. Different forgery methods require different data processing methods for detection. For example, early face-swapping techniques exhibited significant semantic differences, making them easily identifiable by traditional methods. However, the latest forgery techniques (such as generative forgery based on the AIGC diffusion model) show minimal semantic differences; their core differences lie in frequency features and pixel-level forgery traces. Existing solutions, relying on a single data processing method, struggle to adapt to different types of forgery patterns simultaneously.

[0025] To address this technical problem, embodiments of this application provide an image forgery detection model training method, an image forgery detection method, an apparatus, an electronic device, a storage medium, and a computer program product. Through a deep collaborative mechanism between multi-view representation and dynamic expert networks, it achieves end-to-end adaptive feature optimization from the data source to the high-level representation, in order to cope with increasingly complex and rapidly evolving forgery technologies.

[0026] See Figure 1 , Figure 1 This is a flowchart of an image forgery detection model training method in an embodiment of this application, which may include the following steps: Step S102: Obtain the image to be detected, and process the image to be detected using N preprocessing methods to obtain N sets of features to be detected.

[0027] The image to be detected can be, for example, an RGB image. N is an integer greater than 1, representing the features extracted from the image from different dimensions using various preprocessing methods. Optionally, N is 4. The following four preprocessing methods can be used to process the image to be detected: The image to be detected is normalized: through standardization (e.g., mapping to [0,1]), non-critical semantic interferences such as illumination and contrast are eliminated, stabilizing the basic features in the spatial domain. This primarily serves the identification of traditional semantically different forgeries (such as crude face swaps), providing the model with stable brightness and color benchmarks.

[0028] Pixel remapping is performed on the image to be detected: a special nonlinear mapping transformation is applied to the pixel values, scrambling the monotonic arrangement of pixel values ​​(0-255), thereby "converting" the originally smooth low-frequency semantic structure into a high-frequency signal while preserving local correlations and avoiding severe information loss. Within the high-frequency band, the distribution of generated traces is easier to fit than complex semantic distributions, prompting the detector to complete the classification task based on the generated traces. Pixel remapping, by introducing nonlinear transformations and channel recombination, actively strips away high-level semantic information of the image (such as object category and identity features), forcing the model to focus on low-level pixel-level geometric consistency, local texture anomalies, and gradient discontinuities. This branch is key to capturing novel "trace-difference type" forgeries (such as structural flaws in objects generated by AIGC).

[0029] Fast Fourier Transform (FFT) is applied to the image to be detected: this transforms the image to the frequency domain, allowing for direct analysis and extraction of high-frequency texture details and spectral distribution anomalies. Generative models (especially diffusion models) often leave specific spectral traces or uneven energy distribution in the frequency domain. This primarily serves the identification of novel "frequency-difference" forgeries.

[0030] Data augmentation is performed on the images to be detected: through a lossy reconstruction process, local artifacts and structural distortions introduced by the generation process that are imperceptible at the original scale are actively amplified and highlighted, providing the model with enhanced forgery signals. For example, a downsampling-upsampling reconstruction method can be used. Through a controlled "blur-reconstruction" loop, some natural image content (including some high-frequency noise) that is useless for detection can be effectively filtered out. At the same time, the "algorithmic traces" hidden in the image and introduced by the generation process are nonlinearly amplified and reshaped, making them more prominent and separable in the residual domain.

[0031] By fusing the four preprocessing methods mentioned above in parallel, the model simultaneously possesses: a stable semantic baseline, sensitivity to pixel-level artifacts, insight into frequency domain anomalies, and the ability to amplify subtle artifacts. This multi-dimensional feature complementarity mechanism fundamentally resolves the inherent contradiction that a single preprocessing method cannot simultaneously address different types of forgery patterns, providing subsequent deep networks with comprehensive and highly discriminative input features.

[0032] Step S104: Input N sets of features to be detected into a pre-trained multi-view forgery detection model constructed based on a visual feature extraction network, M hybrid expert modules and a classification head.

[0033] The multi-view forgery detection model is constructed based on a visual feature extraction network, M hybrid expert modules, and a classification head. The first M-1 hybrid expert modules are located after each of the M-1 feature extraction layers of the visual feature extraction network; that is, a hybrid expert module is added after each of the M-1 feature extraction layers. Specifically, the hybrid expert modules are located after the feedforward neural network of each feature extraction layer. These first M-1 hybrid expert modules are called intra-layer hybrid expert modules. The last hybrid expert module is located after all feature extraction layers and before the classification head; this single hybrid expert module is called the global hybrid expert module.

[0034] In some embodiments, the visual feature extraction network can be the embedding layer and encoder layer in a visual Transformer, and the classification head can be the classification head in a visual Transformer. That is, the multi-view forgery detection model can be built based on a visual Transformer and M hybrid expert modules. For example, for the four preprocessing methods mentioned above, the corresponding forgery detection architecture diagrams can be found in [reference needed]. Figure 2 In some embodiments, the visual feature extraction network can be a ResNet network, etc., and the classification head can be configured according to business requirements.

[0035] Compared to the earlier feature extraction layers, the middle feature extraction layers can extract better feature information. Furthermore, a hybrid expert module is added after all feature extraction layers. Therefore, the M-1 feature extraction layers can be located in the middle of all feature extraction layers in the visual feature extraction network. In this way, the first M-1 hybrid expert modules can optimize based on the already good feature information, resulting in better optimized features. For example, if the total number of feature extraction layers is 12 and M is 7, then M-1 (i.e., 6) feature extraction layers can be the 4th to 9th feature extraction layers of the visual feature extraction network.

[0036] It should be noted that when training a multi-view forgery detection model, the training data can be directly used to update all network parameters in the model, ultimately generating the multi-view forgery detection model. Alternatively, expert networks corresponding to individual preprocessing methods can be added sequentially, training the network parameters of one branch of the expert network each time, ultimately generating the multi-view forgery detection model. For example, for the four preprocessing branches mentioned above, an expert network for any branch can be added first. During the initial training, the shared network parameters and the network parameters of that expert network are trained. Then, an expert network for the second branch can be added. During training, the network parameters of the first branch's expert network can be frozen or not, and the network parameters of the second branch's expert network and the shared network parameters can be updated. This process continues until the network parameters of all expert networks are trained.

[0037] Step S106: Use a visual feature extraction network to extract features from N sets of features to be detected, and obtain N sets of feature vectors to be detected.

[0038] Each feature extraction layer of the visual feature extraction network includes a self-attention mechanism and a feedforward neural network. After each set of features to be detected is processed by the self-attention mechanism and feedforward neural network of each feature extraction layer, a corresponding set of Q (query vector), V (value vector), and K (key vector) is output. The QVK vector is the feature vector to be detected.

[0039] Step S108: For each of the first M-1 hybrid expert modules, the N sets of feature vectors to be detected input to the hybrid expert module are routed to the corresponding N viewpoint expert networks in the hybrid expert module using the gating network in the hybrid expert module, and the N sets of optimized feature vectors to be detected are output after optimization by the N viewpoint expert networks.

[0040] In this embodiment, each group of features to be detected contains a corresponding preprocessing method label, and the feature vector to be detected after feature extraction also contains the corresponding preprocessing method label. The gating network dynamically determines the core discriminative attributes of each group of QVKs (such as semantic stability, pixel anomaly, frequency domain feature strength, etc.) based on the feature distribution characteristics of each group of QVKs and the corresponding preprocessing method label, and routes them to the most suitable viewpoint expert network. Each viewpoint expert network can be a lightweight feedforward neural network or convolutional module, responsible for deepening and refining features from a specific perspective. The gating network dynamically allocates weights according to the current features, achieving adaptive fusion at the feature level.

[0041] By mapping N sets of features to be detected to N viewpoint expert networks, the heterogeneous features provided by the preprocessing are divided and refined by different viewpoint expert networks within the visual feature extraction network layer. This effectively avoids mutual inhibition of features of different dimensions in the same feedforward neural network and significantly improves feature discrimination.

[0042] For each layer's hybrid expert module, the N sets of optimized feature vectors to be detected, optimized by N viewpoint expert networks, are further input into the next feature extraction layer. Features are then extracted again by the next feature extraction layer, and further optimized by the next hybrid expert module. Through continuous iteration, the discriminative power of each branch's features is continuously strengthened, while avoiding mutual interference between features from different modalities.

[0043] Optionally, each of the first M-1 hybrid expert modules may include N LoRA fine-tuning structures, each corresponding one-to-one with one of the N viewpoint expert networks. The LoRA fine-tuning structures are used to fine-tune the network parameters of the corresponding viewpoint expert network. That is, each viewpoint expert network is equipped with an independent LoRA fine-tuning structure, enabling it to quickly fine-tune its weights with minimal parameter increments (low-rank matrices), thereby specializing in optimizing one or more modal features. For example, one expert, after LoRA fine-tuning, may be better at handling frequency domain features introduced by the FFT branch, while another may be better at handling pixel anomalies introduced by pixel mapping.

[0044] Step S110: Using the gated network in the last hybrid expert module, the N sets of target feature vectors output by the target feature extraction layer and the N sets of optimized target feature vectors output by the first M-1 hybrid expert modules are routed to the L high-dimensional feature expert networks in the hybrid expert module, respectively, and the fused target feature vector after being fused by the high-dimensional feature expert networks is output.

[0045] The last hybrid expert module, the global hybrid expert module, can receive N sets of feature vectors to be detected from the target feature extraction layer and N sets of optimized feature vectors to be detected from the outputs of the first M-1 hybrid expert modules. The target feature extraction layer can be one or more of the other feature extraction layers besides the M-1 feature extraction layers; M and L are both integers greater than 1 and less than K; K is the total number of all feature extraction layers. For example, the target feature extraction layer can be one or more of the last feature extraction layers, or one or more feature extraction layers close to the M-1 feature extraction layers, etc.

[0046] The global hybrid expert module can perform cross-layer, macro-level expert routing, discovering and fusing common high-order features and specific discriminative information exhibited by "traditional semantic forgery" and "novel frequency / trace forgery" throughout the deep evolution of the network. Examples include the correlation patterns between semantic layer forgery features and frequency domain anomaly features, and the complementary information between pixel-level artifacts and enhancement artifacts. The L high-dimensional feature expert networks can include attention mechanisms or deeper multilayer perceptrons, and the gating network can assign weights to features at each layer and from each perspective.

[0047] By combining the global hybrid expert module with the intra-layer hybrid expert module, the multi-perspective representation features are properly handled, and cross-forgery type feature abstraction and integration are achieved at the high level, which greatly enhances the model's generalization ability and decision robustness.

[0048] Step S112: Extract classification labels from the fused feature vector to be detected, process the classification labels using the classification head, and output the fake category.

[0049] CLS Tokens are extracted from the fused feature vectors to be detected output by the global hybrid expert module, and the forgery category is output through a classification head (e.g., fully connected layer + Softmax).

[0050] The image forgery detection method in this application systematically integrates feature extractors serving different forgery types at the input end through multi-view representation preprocessing collaboration, thereby overcoming the limitations of a single preprocessing method. Through an intra-layer hybrid expert module, the model can dynamically allocate computing resources to refine the multi-view representation features. The global hybrid expert module then completes high-level semantic fusion and decision-making. This two-level hybrid expert module deepens and refines multi-view representation features from specific perspectives, and achieves cross-forgery type feature abstraction and integration at a high level, improving the classification accuracy and model robustness for various types of forged data. This two-level hybrid expert system and multi-view representation preprocessing constitute a tightly coupled, progressively layered feature understanding system, achieving a leap in comprehensive detection accuracy for both traditional semantic and novel frequency / trace-based forgeries, thus meeting the urgent need for "all-type, high-reliability" detection in security, content moderation, and other scenarios.

[0051] The embodiments of this application can extend the multi-view forgery detection model. For new data preprocessing methods, corresponding expert networks can be dynamically added to the trained multi-view forgery detection model. See also... Figure 3 , Figure 3 This is a flowchart of a model update method in an embodiment of this application, including the following steps: Step S302: Obtain multiple sample images and the actual forgery category corresponding to each sample image.

[0052] The sample image and its corresponding actual forgery category are used as input data and label data, respectively, for subsequent model training. Depending on the actual scenario, there can be multiple actual forgery categories, such as face replacement and generative synthesis. Each sample image corresponds to one of these actual forgery categories.

[0053] Step S304: For the N+1th preprocessing method, add the perspective expert network corresponding to the N+1th preprocessing method to each of the first M-1 hybrid expert modules in the multi-view forgery detection model to obtain the updated multi-view forgery detection model.

[0054] For the new preprocessing method, other structures in the multi-view forgery detection model can remain unchanged, and only the view expert network corresponding to the new preprocessing method can be added to the hybrid expert module in each layer.

[0055] Step S306: Process each sample image using N+1 preprocessing methods to obtain N+1 sets of preprocessed features.

[0056] Step S308: Input the N+1 sets of preprocessed features into the updated multi-view forgery detection model, and use the visual feature extraction network to extract features from the N+1 sets of preprocessed features to obtain N+1 sets of feature vectors.

[0057] Step S310: For each of the first M-1 hybrid expert modules, a gating network is used to route the N+1 sets of feature vectors input to the hybrid expert module to the corresponding N+1 view expert networks, and the N+1 sets of optimized feature vectors after optimization by the view expert networks are output.

[0058] Step S312: Using the gated network in the last hybrid expert module, the N+1 sets of feature vectors output by the target feature extraction layer and the N+1 sets of optimized feature vectors output by the first M-1 hybrid expert modules are routed to the L high-dimensional feature expert networks in the hybrid expert module, respectively, and the fused feature vector after being fused by the high-dimensional feature expert networks is output.

[0059] Step S314: Extract classification labels from the fused feature vector, process the classification labels using the classification head, and output the predicted fake category.

[0060] Step S316: Calculate the loss value based on the actual forgery category and the predicted forgery category of the sample image, update the network parameters of the viewpoint expert network corresponding to the N+1th preprocessing method based on the loss value, freeze other network parameters, and generate the updated multi-view forgery detection model after training.

[0061] During training, only the network parameters of the viewpoint expert network corresponding to the new preprocessing method need to be fine-tuned, while other network parameters are frozen, thereby improving the efficiency of model training. It can be seen that the multi-view forgery detection model in this embodiment can be dynamically expanded according to the preprocessing methods applicable to different forgery categories. When a new forgery category appears and a new preprocessing method applies to it, the corresponding viewpoint expert network can be directly added to the multi-view forgery detection model, demonstrating strong practicality.

[0062] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0063] This application also provides an image forgery detection device based on multi-view characterization preprocessing, see [link to relevant documentation]. Figure 4 The image forgery detection device 400 based on multi-view characterization preprocessing includes: Image acquisition module 402 is used to acquire the image to be detected; The image preprocessing module 404 is used to process the image to be detected using N preprocessing methods to obtain N sets of features to be detected; N is an integer greater than 1. The feature input module 406 is used to input N sets of features to be detected into a pre-trained multi-view forgery detection model constructed based on a visual feature extraction network, M hybrid expert modules, and a classification head; wherein, the first M-1 hybrid expert modules are located after the M-1 feature extraction layers of the visual feature extraction network; the last hybrid expert module is located after all feature extraction layers and before the classification head; The feature extraction module 408 is used to extract features from N sets of features to be detected using a visual feature extraction network, and obtain N sets of feature vectors to be detected. The feature optimization module 410 is used to route the N sets of feature vectors to be detected input to the hybrid expert module to the corresponding N viewpoint expert networks in the hybrid expert module for each of the first M-1 hybrid expert modules, and output the N sets of optimized feature vectors to be detected after optimization by the N viewpoint expert networks. The feature fusion module 412 is used to route the N sets of detectable feature vectors output by the target feature extraction layer and the N sets of optimized detectable feature vectors output by the first M-1 hybrid expert modules to L high-dimensional feature expert networks in the hybrid expert module, respectively, using the gated network in the last hybrid expert module, and outputting a fused detectable feature vector after fusion by the high-dimensional feature expert networks; the target feature extraction layer is one or more of the other feature extraction layers besides the M-1 feature extraction layers; M and L are both integers greater than 1 and less than K; K is the total number of all feature extraction layers; The forgery category output module 414 is used to extract classification labels from the fused feature vector to be detected, process the classification labels using the classification head, and output the forgery category.

[0064] Optionally, the image forgery detection device 400 based on multi-view characterization preprocessing further includes: The training data acquisition module is used to acquire multiple sample images and the actual forgery category corresponding to each sample image; The model structure update module is used to add the perspective expert network corresponding to the N+1th preprocessing method to each of the first M-1 hybrid expert modules in the multi-view forgery detection model for the N+1th preprocessing method, so as to obtain the updated multi-view forgery detection model. The model update module is used to process a single sample image using N+1 preprocessing methods to obtain N+1 sets of preprocessed features. The N+1 sets of preprocessed features are then input into the updated multi-view forgery detection model, and the visual feature extraction network is used to extract features from the N+1 sets of preprocessed features to obtain N+1 sets of feature vectors. For each of the first M-1 hybrid expert modules, a gating network is used to route the N+1 sets of feature vectors input to the hybrid expert module to the corresponding N+1 viewpoint expert networks, outputting N+1 sets of optimized feature vectors after optimization by the viewpoint expert networks. The gating network in the last hybrid expert module routes the N+1 sets of feature vectors output from the target feature extraction layer and the N+1 sets of optimized feature vectors output from the first M-1 hybrid expert modules to L high-dimensional feature expert networks in the hybrid expert module, outputting a fused feature vector after fusion by the high-dimensional feature expert networks. Classification labels are extracted from the fused feature vectors, and the classification heads are used to process the classification labels to output the predicted forgery category. A loss value is calculated based on the actual forgery category and the predicted forgery category of the sample image. Based on the loss value, the network parameters of the viewpoint expert network corresponding to the N+1th preprocessing method are updated, and other network parameters are frozen, generating a trained and updated multi-view forgery detection model.

[0065] Optionally, each of the first M-1 hybrid expert modules includes: N LoRA fine-tuning structures, each of which corresponds one-to-one with one of the N viewpoint expert networks. The LoRA fine-tuning structures are used to fine-tune the network parameters of the corresponding viewpoint expert network.

[0066] Optionally, N is 4, and the image preprocessing module 404 is specifically used to perform normalization processing on a single sample image; pixel remapping on a single sample image; fast Fourier transform on a single sample image; and data augmentation processing on a single sample image to obtain 4 sets of features to be detected.

[0067] Optionally, the M-1 feature extraction layers are located in the middle of all feature extraction layers in the visual feature extraction network.

[0068] The specific details of each module or unit in the above-mentioned device have been described in detail in the corresponding methods, so they will not be repeated here.

[0069] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0070] This application also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the above-described image forgery detection method based on multi-view characterization preprocessing.

[0071] Reference Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device in an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0072] like Figure 5 As shown, the electronic device may include: a processor 502, a communication interface 504, a memory 506, and a communication bus 508.

[0073] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508.

[0074] Communication interface 504 is used to communicate with other electronic devices or servers.

[0075] The processor 502 is used to execute program 510, specifically the relevant steps in the above method embodiments.

[0076] Specifically, program 510 may include program code that includes computer operation instructions.

[0077] Processor 502 may be a central processing unit, a specific integrated circuit, or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0078] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0079] Specifically, program 510 can be used to cause processor 502 to execute the steps in the above method embodiments.

[0080] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0081] In this embodiment of the application, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the above-described image forgery detection method based on multi-view representation preprocessing.

[0082] It should be noted that the computer-readable storage medium shown in this application can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0083] In this embodiment of the application, a computer program product is also provided, which, when run on a computer, causes the computer to execute the above-described image forgery detection method based on multi-view representation preprocessing.

[0084] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0085] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image forgery detection method based on multi-view representation preprocessing, characterized in that, include: The image to be detected is acquired, and the image is processed using N preprocessing methods to obtain N sets of features to be detected. N is an integer greater than 1; The N sets of features to be detected are input into a pre-trained multi-view forgery detection model constructed based on a visual feature extraction network, M hybrid expert modules, and a classification head; wherein, the first M-1 hybrid expert modules are located after the M-1 feature extraction layers of the visual feature extraction network; the last hybrid expert module is located after all feature extraction layers and before the classification head; The visual feature extraction network is used to extract features from the N sets of features to be detected, resulting in N sets of feature vectors to be detected. For each of the first M-1 hybrid expert modules, the gating network in the hybrid expert module is used to route the N sets of feature vectors to be detected input to the hybrid expert module to the corresponding N viewpoint expert networks in the hybrid expert module, and output the N sets of optimized feature vectors to be detected after optimization by the N viewpoint expert networks. The gated network in the last hybrid expert module routes the N sets of target feature vectors output by the target feature extraction layer and the N sets of optimized target feature vectors output by the first M-1 hybrid expert modules to L high-dimensional feature expert networks in the hybrid expert module, respectively, and outputs a fused target feature vector after being fused by the high-dimensional feature expert networks; the target feature extraction layer is one or more of the other feature extraction layers besides the M-1 feature extraction layers; M and L are both integers greater than 1 and less than K; K is the total number of all feature extraction layers; The classification label is extracted from the fused feature vector to be detected, and the classification head is used to process the classification label to output the fake category.

2. The method according to claim 1, characterized in that, The method further includes: Obtain multiple sample images and the actual forgery category corresponding to each sample image; For the N+1th preprocessing method, add the perspective expert network corresponding to the N+1th preprocessing method to each of the first M-1 hybrid expert modules in the multi-view forgery detection model to obtain an updated multi-view forgery detection model. Each sample image is processed using N+1 preprocessing methods to obtain N+1 sets of preprocessed features; The N+1 preprocessed features are input into the updated multi-view forgery detection model, and the visual feature extraction network is used to extract features from the N+1 preprocessed features to obtain N+1 feature vectors. For each of the first M-1 hybrid expert modules, a gating network is used to route the N+1 sets of feature vectors input to the hybrid expert module to the corresponding N+1 view expert networks, and output the N+1 sets of optimized feature vectors after optimization by the view expert networks. The gated network in the last hybrid expert module routes the N+1 sets of feature vectors output by the target feature extraction layer and the N+1 sets of optimized feature vectors output by the first M-1 hybrid expert modules to the L high-dimensional feature expert networks in the hybrid expert module, and outputs the fused feature vector after being fused by the high-dimensional feature expert networks. The classification label is extracted from the fused feature vector, and the classification label is processed using the classification head to output the predicted fake category; The loss value is calculated based on the actual forgery category and the predicted forgery category of the sample image. The network parameters of the viewpoint expert network corresponding to the N+1th preprocessing method are updated based on the loss value, and other network parameters are frozen to generate the updated multi-view forgery detection model after training.

3. The method according to claim 1, characterized in that, Each of the first M-1 hybrid expert modules includes: N low-rank adaptive LoRA fine-tuning structures, with each of the N LoRA fine-tuning structures corresponding to one of the N viewpoint expert networks. The LoRA fine-tuning structures are used to fine-tune the network parameters of the corresponding viewpoint expert networks.

4. The method according to claim 1, characterized in that, When N is 4, the process of processing a single sample image using N preprocessing methods includes: Normalize individual sample images; Perform pixel remapping on a single sample image; Perform a Fast Fourier Transform on a single sample image; Perform data augmentation on a single sample image.

5. The method according to claim 1, characterized in that, The M-1 feature extraction layers are located in the middle of all feature extraction layers in the visual feature extraction network.

6. An image forgery detection device based on multi-view characterization preprocessing, characterized in that, include: The image acquisition module is used to acquire the image to be detected; The image preprocessing module is used to process the image to be detected using N preprocessing methods to obtain N sets of features to be detected. N is an integer greater than 1; The feature input module is used to input the N sets of features to be detected into a pre-trained multi-view forgery detection model constructed based on a visual feature extraction network, M hybrid expert modules, and a classification head; wherein, the first M-1 hybrid expert modules are located after the M-1 feature extraction layers of the visual feature extraction network; the last hybrid expert module is located after all feature extraction layers and before the classification head; The feature extraction module is used to extract features from the N sets of features to be detected using a visual feature extraction network, so as to obtain N sets of feature vectors to be detected. The feature optimization module is used to route the N sets of feature vectors to be detected input to the hybrid expert module to the corresponding N viewpoint expert networks in the hybrid expert module for each of the first M-1 hybrid expert modules, and output the N sets of optimized feature vectors to be detected after optimization by the N viewpoint expert networks. The feature fusion module is used to route the N sets of target feature vectors output by the target feature extraction layer and the N sets of optimized target feature vectors output by the first M-1 hybrid expert modules to L high-dimensional feature expert networks in the hybrid expert module, respectively, and output a fused target feature vector after being fused by the high-dimensional feature expert networks. The target feature extraction layer is one or more feature extraction layers other than the M-1 feature extraction layers. M and L are both integers greater than 1 and less than K. K is the total number of all feature extraction layers. The forgery category output module is used to extract classification labels from the fused feature vector to be detected, process the classification labels using the classification head, and output the forgery category.

7. The apparatus according to claim 6, characterized in that, The device further includes: The training data acquisition module is used to acquire multiple sample images and the actual forgery category corresponding to each sample image; The model structure update module is used to add the perspective expert network corresponding to the N+1th preprocessing method to each of the first M-1 hybrid expert modules in the multi-view forgery detection model for the N+1th preprocessing method, so as to obtain an updated multi-view forgery detection model. The model update module is used to process a single sample image using N+1 preprocessing methods to obtain N+1 sets of preprocessed features; the N+1 sets of preprocessed features are input into the updated multi-view forgery detection model, and the visual feature extraction network is used to extract features from the N+1 sets of preprocessed features to obtain N+1 sets of feature vectors. For each of the first M-1 hybrid expert modules, a gating network is used to route the N+1 sets of feature vectors input to the hybrid expert module to the corresponding N+1 viewpoint expert networks, outputting N+1 sets of optimized feature vectors after optimization by the viewpoint expert networks. The gating network in the last hybrid expert module routes the N+1 sets of feature vectors output from the target feature extraction layer and the N+1 sets of optimized feature vectors output from the first M-1 hybrid expert modules to L high-dimensional feature expert networks in the hybrid expert module, outputting a fused feature vector fused by the high-dimensional feature expert networks. Classification labels are extracted from the fused feature vector, and the classification labels are processed using a classification head to output a predicted forgery category. A loss value is calculated based on the actual forgery category and the predicted forgery category of the sample image. The network parameters of the viewpoint expert network corresponding to the N+1th preprocessing method are updated based on the loss value, and other network parameters are frozen to generate a trained and updated multi-view forgery detection model.

8. The apparatus according to claim 6, characterized in that, Each of the first M-1 hybrid expert modules includes: N low-rank adaptive LoRA fine-tuning structures, with each of the N LoRA fine-tuning structures corresponding to one of the N viewpoint expert networks. The LoRA fine-tuning structures are used to fine-tune the network parameters of the corresponding viewpoint expert networks.

9. The apparatus according to claim 6, characterized in that, When N is 4, the image preprocessing module is specifically used to normalize a single sample image; remap pixels in a single sample image; perform fast Fourier transform on a single sample image; and perform data augmentation on a single sample image to obtain 4 sets of features to be detected.

10. The apparatus according to claim 6, characterized in that, The M-1 feature extraction layers are located in the middle of all feature extraction layers in the visual feature extraction network.

Images