Image forgery detection method and system based on multi-expert model decision

Abstract

The invention discloses an image forgery detection method and system based on multi-expert model decision, and belongs to the technical field of image forgery detection. The method comprises the following steps: acquiring a multi-field normal image, generating forged images based on different forging types and forging degrees, grouping the forged images, pre-training a pre-screening model and a multi-expert model capable of predicting the forging probability by using the normal image and each group of forged images, and focusing on different features or forging traces of the image by each expert model; training an image forgery detection network comprising an image coding module, a gating calculation module, a pre-screening model and a multi-expert model by using the normal image and the forgery image; during training, parameters of the pre-screening model and the multi-expert model are frozen; and identifying whether the input image is forged based on the trained image forging detection network. According to the method, multi-dimensional detection of the forged image can be realized, a result is comprehensively output through a weighted fusion decision mechanism, the advantages of different models are fully utilized, and the detection precision and efficiency of the forged image are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image forgery detection, and in particular to an image forgery detection method and system based on multi-expert model decision-making. Background Art

[0002] With the development of generative artificial intelligence technology, deepfakes are increasingly used in the fields of images and videos, and the means of forgery are becoming more and more complex. Traditional methods of forged content detection show low robustness and accuracy in the face of increasingly complex and diverse forgery methods.

[0003] Specifically, existing methods usually rely on a single detection model, which makes it difficult to fully identify different forgery features. Especially in complex scenes, the model tends to learn simple and obvious forgery features, while ignoring the tiny forgery traces left by complex generation models. This limitation makes traditional methods prone to misjudgment or omission when detecting unknown or complex forgery content.

[0004] Existing multi-task learning methods usually use task sharing and task branching to perform forgery detection. They design a shared underlying feature extraction module and set up independent task branches for each subtask to process different forgery detection tasks in parallel. The specific steps are as follows: First, all tasks share an underlying feature extraction network to extract common features of images or videos; second, each task processes specific forgery features, such as the details of forged images, the location of forged areas, and the identification of global forgery patterns, through a dedicated branch network; third, by designing a fusion loss function, the losses of multiple tasks are weighted and fused, and the parameters of the network are jointly optimized so that each task can be fully learned in the entire network; finally, the fusion optimization is performed. Although this method can significantly improve the detection performance, there may be information conflicts and interference between tasks, and the optimization process is relatively complex. There are also the following problems:

[0005] 1) Insufficient generalization ability: The detection model has insufficient generalization ability when facing unknown counterfeiting techniques, and it is difficult to cope with complex and changeable counterfeiting methods in practical applications.

[0006] 2) Lack of robustness: When faced with common disturbances such as image compression and noise, the detection accuracy of existing forgery detection technologies drops significantly, showing poor robustness.

[0007] 3) Lack of explainability: Traditional forgery detection methods mostly rely on complex deep learning models and lack a clear explanation of the basis for model judgment. Summary of the invention

[0008] In response to the above technical problems, the present invention proposes an image forgery detection method and system based on multi-expert model decision-making. Multiple expert models are combined through a gating mechanism. Each model focuses on different features or forgery traces of the image to achieve multi-dimensional detection. The results are comprehensively output through a weighted fusion decision mechanism to improve detection accuracy and efficiency, and solve the problems of traditional methods in unknown algorithm detection, compression distortion and forgery tracing.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0010] In a first aspect, the present invention provides an image forgery detection method based on multi-expert model decision making, comprising the following steps:

[0011] (1) obtaining normal images in multiple fields, generating and grouping forged images based on different forgery types and forgery degrees, and using normal images and groups of forged images to pre-train a pre-screening model and a multi-expert model that can predict the probability of forgery. The multi-expert model includes a typical forgery trace detection model based on the EfficientNet network, a complex refinement trace detection model based on the ViT network, and a local forgery trace detection model based on the Xception network;

[0012] (2) Using normal images and forged images, an image forgery detection network is trained, which includes an image encoding module, a gated calculation module, a pre-screening model, and a multi-expert model. During training, the parameters of the pre-screening model and the multi-expert model are frozen.

[0013] (3) Based on the trained image forgery detection network, identify whether the input image is forged, including:

[0014] The image encoding module is used to extract image features; the pre-screening model uses the image features as input to identify whether it is a forged image. If so, the judgment ends; if not, the gated calculation module uses the image features as input to generate a probability vector with the same dimension as the number of multiple expert models as the weight value of each expert model; the expert model with non-zero weight value is screened and the forgery probability is independently judged using the image features as input, and the forgery probability values ​​output by each expert model are fused based on the non-zero weight value, and the input image with a total forgery probability value higher than the threshold is identified as a forged image.

[0015] Furthermore, it also includes an image preprocessing process, and the preprocessing includes standardization, denoising and enhancement.

[0016] Furthermore, in step (1), generating a forged image based on different forgery types and forgery degrees includes:

[0017] The types of forgery include texture forgery, boundary forgery, brightness forgery, facial feature forgery, expression forgery, lighting forgery and synthetic splicing forgery;

[0018] The forgery degree refers to the total disturbance degree of a normal image under different forgery types of disturbances.

[0019] Furthermore, in step (1), the method for grouping forged images includes:

[0020] (1-1) Obtaining the forgery type and forgery degree corresponding to the forged image. If the forgery type corresponding to the forged image includes one or more of texture forgery, boundary forgery, brightness forgery, and synthetic splicing forgery, and the forgery degree is higher than a threshold, the forgery image is classified into the first forged image group; otherwise, executing step (1-2);

[0021] (1-2) If the forgery type corresponding to the forged image includes one or more of texture forgery, boundary forgery, brightness forgery, and synthetic splicing forgery, and the degree of forgery is not higher than the threshold; or the forgery type is selected from any one of facial feature forgery, expression forgery, and lighting forgery, and the degree of forgery is higher than the threshold; then it is classified into the second forged image group; otherwise, step (1-3) is executed;

[0022] (1-3) If the forgery type corresponding to the forged image is selected from facial feature forgery, expression forgery, and lighting forgery, and the degree of forgery is not higher than the threshold, then the image is classified into the third forged image group;

[0023] (1-4) Selecting images whose forged regions are local regions from the third forged image group, and the selected images constitute a fourth forged image group.

[0024] Furthermore, in step (1), the combination of the first forged image group and several normal images is used as a pre-screening model training set, the combination of the second forged image group and several normal images is used as a training set for the EfficientNet network, the combination of the third forged image group and several normal images is used as a training set for the ViT network, and the combination of the fourth forged image group and several normal images is used as a training set for the Xception network.

[0025] Furthermore, the pre-screening model adopts a model based on a CNN network, and the gating calculation module adopts a fully connected network.

[0026] Furthermore, the method also includes a visualization step of the image forgery detection process, including:

[0027] Display the heat map corresponding to the input image;

[0028] As well as, the forgery probability, weight distribution, total forgery probability and final recognition result independently judged by each expert model are displayed.

[0029] In a second aspect, the present invention provides an image forgery detection system based on multi-expert model decision-making, which is used to implement the above-mentioned image forgery detection method based on multi-expert model decision-making.

[0030] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image forgery detection method based on multi-expert model decision-making as described above is implemented.

[0031] In a fourth aspect, the present invention provides a computer electronic device, comprising a memory and a processor;

[0032] The memory is used to store computer programs;

[0033] The processor is used to implement the above-mentioned image forgery detection method based on multi-expert model decision when executing the computer program.

[0034] The beneficial effects of the present invention are:

[0035] The present invention establishes a multi-expert fusion decision-making mechanism, dynamically selects an adapted expert model to judge based on the input image features, and each expert model focuses on different data features or forgery traces of the image. The multi-dimensional analysis based on the expert model can give full play to the advantages of each expert model and improve the recognition ability of different image forgery methods, which is especially suitable for complex forgery scenarios; the present invention can not only identify known forgery methods, but also deal with unknown forgery algorithms, effectively improve the generalization ability of the detection model, and reduce the risk of failure. In addition, the pre-screening model is used to preliminarily screen the input image to identify simple forged images, which not only meets the high real-time requirements of simple images, but also meets the high precision requirements of complex images, ensuring the accuracy and efficiency of image forgery detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the framework of the image forgery detection method based on multi-expert model decision-making shown in an embodiment of the present invention.

[0037] Figure 2 4 is a flowchart of an image forgery detection method based on multi-expert model decision-making according to an embodiment of the present invention.

[0038] Figure 3 is a schematic diagram of an image forgery detection system based on multi-expert model decision-making shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described and illustrated below in conjunction with specific embodiments. The embodiments are merely exemplary of the present disclosure and do not define the scope of limitation. The technical features of each embodiment of the present invention may be combined accordingly without conflicting with each other.

[0040] The accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0041] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0042] Figure 1 The figure shows a framework diagram of an image forgery detection method based on multi-expert model decision-making proposed by the present invention. A pre-screening model and a multi-expert model are introduced. The pre-screening model realizes rapid identification of obvious forgeries, and each expert model focuses on different data features or forgery traces of the image; for image features that are not identified as forged by the pre-screening model, the weights are further calculated by the gating mechanism and sent to each expert model with non-zero weight for independent detection. Finally, the detection results of each expert model are fused and output based on the weight value, and the visualization results are output in multiple dimensions at the same time, which enhances the interpretability and transparency of the image forgery detection process.

[0043] Specifically, Figure 2 As shown, the present invention proposes an image forgery detection method based on multi-expert model decision-making, which mainly includes the following steps:

[0044] S1, obtain normal images in multiple fields, generate and group forged images based on different forgery types and forgery degrees.

[0045] In this embodiment, normal images are selected from public data sets such as ImageNet, COCO, and FFHQ, covering images in multiple fields such as natural scenes, faces, and objects, and normal images are preprocessed, such as standardization, denoising, and enhancement. For example: unify the image resolution (such as 256x256) and standardize the image to remove low-quality image samples; remove unnecessary noise in the image by filtering, wavelet transform, or Fourier transform; enhance the image, such as rotating, scaling, and cropping, to simulate the transformation of forged content that may occur in the real world, so as to improve the robustness of the model to different inputs, so that the system can still maintain high accuracy and reliability when facing various interferences and disturbances in the real environment.

[0046] The preprocessed normal images are perturbed with different forgery types and degrees to generate a large number of forged images.

[0047] In a specific implementation of the present invention, the types of forgery include but are not limited to texture forgery, boundary forgery, brightness forgery, facial feature forgery, expression forgery, lighting forgery and synthetic splicing forgery; the degree of forgery refers to the total degree of disturbance of a normal image under different types of forgery disturbances.

[0048] Among them, texture forgery refers to modifying local texture (such as adding noise, blurring), which may cause the texture of the local area to be inconsistent with the surrounding area (such as being too smooth or repetitive). For example, the texture of the forged area may not match the real background. Boundary forgery refers to adding unnatural transitions at the image boundary, which may cause unnatural jagged edges at the boundary. Brightness forgery refers to adjusting the local brightness. The brightness of the forged area may be inconsistent with the surrounding area. For example, the local brightness is too high or too low, which looks unnatural. Facial feature forgery refers to forgery by modifying facial features (such as eyes, nose, and mouth), such as DeepFake and other technologies, which may be difficult to distinguish with the naked eye. Expression forgery refers to modifying facial expressions (such as changing a smile to a serious one), which may look natural, especially small modifications, such as expression forgery generated by GAN, which may not have obvious traces. Lighting forgery refers to adjusting the lighting of the image. If handled properly, it will look very natural, especially when the forged lighting direction is consistent with the scene, and it is difficult for the naked eye to detect abnormalities. Synthetic splicing forgery refers to splicing different images together, which may cause lighting, color or perspective inconsistencies, such as inconsistent lighting directions between the spliced ​​person and the background. These forgery types can be implemented using image processing tools or deep learning models. One or more forgery types can be used to generate forged images of varying degrees. Here, the degree of forgery of multiple forgery types, that is, the total disturbance degree, refers to the average of the intensities of each forgery type. The definition of the disturbance degree of an image is usually based on the degree of difference between the image and the original image. Image quality assessment indicators (such as PSNR and SSIM) can be used to measure the difference between the forged image and the original image to evaluate the degree of forgery of each forgery type.

[0049] For the generated forged images, a detailed data index table is established to record the corresponding forgery type and degree of forgery, which is convenient for subsequent grouping and research and analysis of forged images in different groups, and provides support for training models that focus on different data features or forgery traces of images.

[0050] Taking the above-mentioned forgery types as an example, texture forgery, boundary forgery, brightness forgery, and synthetic splicing forgery are easy to identify, while facial feature forgery, expression forgery, and lighting forgery are not easy to identify. Combined with the degree of forgery of each forgery type, the large number of forged images generated above are divided into four categories.

[0051] In a specific implementation of the present invention, the method for grouping forged images includes:

[0052] S1-1, obtaining the forgery type and forgery degree corresponding to the forged image. If the forgery type corresponding to the forged image includes one or more of texture forgery, boundary forgery, brightness forgery, and synthetic splicing forgery, and the forgery degree is higher than a threshold, the forgery image is classified into the first forged image group; otherwise, S1-2 is executed;

[0053] S1-2, if the forgery type corresponding to the forged image includes one or more of texture forgery, boundary forgery, brightness forgery, and synthetic splicing forgery, and the degree of forgery is not higher than the threshold; or the forgery type is selected from any one of facial feature forgery, expression forgery, and lighting forgery, and the degree of forgery is higher than the threshold; then it is classified into the second forged image group; otherwise, S1-3 is executed;

[0054] S1-3, if the forgery type corresponding to the forged image is selected from facial feature forgery, expression forgery, and lighting forgery, and the degree of forgery is not higher than the threshold, then the forged image is classified into the third forged image group;

[0055] S1-4, selecting images whose forged areas are local areas from the third forged image group, and the selected images constitute a fourth forged image group.

[0056] In practical applications, the data features and forgery traces presented by images of different forgery types and degrees are significantly different. Simple image forgery may only involve basic image editing operations, with relatively simple changes in data features and obvious forgery traces; while complex image forgery may use a combination of multiple advanced technologies to deeply process the image, making the data features highly complex and the forgery traces hidden and difficult to find. If a single model is used to process images of all difficulties, the model is easily disturbed by images of different difficulties during the learning process, and it is difficult to accurately capture the key features of various types of images, resulting in a decrease in the generalization ability and accuracy of the model. By dividing the forged images into four categories according to the degree of forgery type and using them to train four models respectively, the aim is to allow each model to focus on learning the unique data features or forgery traces of images in a specific difficulty range. In this way, each model can perform deep optimization for images of a specific difficulty, avoid interference between images of different difficulty levels, and more effectively extract and learn the key information of the corresponding difficulty images, thereby improving the model's recognition ability and accuracy for images of different difficulty levels.

[0057] S2, uses normal images and groups of forged images to pre-train a pre-screening model and a multi-expert model that can predict the probability of forgery.

[0058] The present invention designs and deploys multiple expert models, each of which is optimized for different types of forgery features or forgery algorithms, to ensure that the system can make comprehensive and accurate judgments when faced with various forgeries. The construction of expert models not only involves the selection of different model architectures (such as EfficientNet, ViT, and Xception), but also includes how to effectively configure and train these models in practical applications to ensure that they can work in a complementary and collaborative manner.

[0059] S2-1, Design of the pre-screening model.

[0060] The pre-screening model only needs to be able to preliminarily screen simple forged images. For example, the CNN-based MobileNet network is a lightweight deep learning model that can provide efficient image recognition and processing capabilities for resource-constrained environments. The model is trained using a combination of the first forged image group and the same number of randomly sampled normal images as a training set, which can quickly extract high-level features and perform preliminary classification of forged images and normal images.

[0061] Through simple preliminary judgment, the system can effectively avoid excessive processing of real data, thereby improving efficiency.

[0062] S2-2, Design of multi-expert model.

[0063] The multi-expert model should focus on different types of forgery features, using a typical forgery trace detection model based on the EfficientNet network, a complex refinement trace detection model based on the ViT network, and a local forgery trace detection model based on the Xception network. Multiple independent expert models detect specific forgery content, and each expert model outputs a confidence level.

[0064] In addition, each expert model adopts a difficult sample replay strategy, which ensures that the system can effectively adapt to these new forgery features by replaying and retraining the difficult-to-detect samples in the model first, thereby continuously improving the detection ability. This strategy enables the expert model to continuously adjust and improve its detection ability, maintain efficient and accurate forged content identification, and ensure the long-term effectiveness and robustness of the system.

[0065] The three expert models in this embodiment are described below.

[0066] (1) A typical forgery trace detection model based on the EfficientNet network.

[0067] The core goal of the model is to identify some common low-level features in forged images, such as unnatural boundaries, inconsistent brightness, broken textures, lighting, color or perspective inconsistencies caused by synthetic splicing, etc. introduced by common deep forgery techniques, as well as forgeries generated by conventional deep learning models (such as GAN networks), which generally involve forgeries of facial features, expression features, lighting features, etc. The model outputs detection results by analyzing typical forgery features introduced by conventional forgery techniques in the image.

[0068] The typical forgery trace detection model in the present invention adopts the EfficientNet network, which proposes a composite scaling method to adjust the depth by composite coefficients. ,width and resolution Make coordinated adjustments to improve model performance:

[0069]

[0070]

[0071]

[0072] in, , , is the scaling factor, satisfying , is the scaling level. By balancing the scaling factors, the EfficientNet network can significantly improve the representation capability of the network model while maintaining a low computational cost.

[0073] The typical forgery trace detection model is designed with light weight and high efficiency as the core, and adopts a compound scaling strategy to optimize the computational efficiency. The model is trained using a combination of the second forged image group and the same number of randomly sampled normal images as the training set. The expert model can maintain high detection accuracy in large-scale forgery data detection while reducing computational costs.

[0074] (2) Complex refinement trace detection model based on ViT network.

[0075] The complex refined data detection model in the present invention mainly detects images with more concealed traces of forgery, generally facial feature forgery, expression forgery, lighting forgery, etc. The high-precision forgery model uses image enhancement, noise removal, color correction and other technologies to adjust the lighting, color, texture, etc. There are no obvious traces of splicing, which is more natural and difficult to detect visually. The texture, color and lighting of the forged image are more refined.

[0076] In this paper, Vision Transformer (ViT) is used to build a complex refined data detection model, and the self-attention mechanism is used to model the global features of the image. ViT is an image processing method based on the Transformer architecture. Specifically, the input image is divided into several small blocks, each of which is , the total size of the image is , so the image is divided into Each small block will be flattened and mapped to a D-dimensional embedding space through linear embedding:

[0077]

[0078] in, It is image blocks, is the embedding matrix, is the embedding vector of the block. In order to preserve the spatial information in the image, ViT adds position encoding to the embedding vector , and get the final input sequence:

[0079]

[0080] in, It is The embedding vector of the image patch.

[0081] Finally, the self-attention mechanism is used to calculate the similarity between each image block and weighted sum it to obtain the global context information:

[0082]

[0083] in, is the query vector, is the key vector, is a vector of values, is the dimension of the key vector. In this way, ViT can effectively capture long-range dependencies and focus more on capturing the global features of images or videos, thereby detecting complex and refined forged content.

[0084] The model is trained using a combination of the third forged image group and the same number of randomly sampled normal images as the training set.

[0085] (3) Local forgery trace detection model based on Xception network.

[0086] The local forgery trace detection model in the present invention focuses on the forgery traces in local areas of the image, such as forgeries in the face, eyes, mouth and other areas, and aims to identify those subtle forgery features that are difficult to detect, such as local facial feature forgeries, expression forgeries, lighting forgeries, etc.

[0087] The local forgery trace detection model in the present invention adopts the Xception network as the core backbone network.

[0088] Unlike traditional convolutional neural networks, Xception uses a deep separable convolution structure, which splits the standard convolution operation into two stages: one is deep convolution, which only performs convolution on each input channel; the other is point-by-point convolution, which performs convolution across channels. Such a decomposition can significantly reduce computational complexity and improve the ability to learn local features. Local forgery traces are usually small regional changes in the image, and deep separable convolution can effectively capture these changes. This design reduces the number of parameters and improves computational efficiency while retaining high sensitivity to local features, especially when dealing with small-scale forgery features. The expert model is trained using a combination of the fourth forgery image group and the same number of randomly sampled normal images as a training set, which can provide high accuracy in local forgery detection.

[0089] S3, uses normal images and forged images to train an image forgery detection network including an image encoding module, a gated calculation module, a pre-screening model and a multi-expert model; during training, the parameters of the pre-screening model and the multi-expert model are frozen.

[0090] In a specific implementation of the present invention, the image encoding module can be implemented using VGG16 or ResNet50, which is used to extract image features using images as input. The gated calculation module uses a fully connected network, which uses image features as input, generates a probability vector with the same dimension as the number of multiple expert models as the weight value of each expert model, selects expert models with non-zero weight values, and independently determines the forgery probability using image features as input, fuses the forgery probability values ​​output by each expert model based on the non-zero weight value, and identifies the input image with a total forgery probability value higher than a threshold as a forged image. Due to the diversity of forged content, in general, the weight value of each expert model is a non-zero result.

[0091] Assuming the expert model described above , , The probability values ​​of forged detection output are , , , the corresponding weight calculated by the gate is , , , the system obtains the final detection result through weighted fusion :

[0092]

[0093] The gated calculation module flexibly adjusts the weight of each expert model according to the input image features, ensures that the appropriate expert model is selected to analyze the characteristics of the data according to the input image features, and automatically assigns the highest weight to the expert model that can most accurately process the current counterfeit content. The present invention generates the final counterfeit judgment result by weighted fusion of the detection results of multiple expert models, balancing the contributions of different expert models, and dealing with the judgment inconsistency between models.

[0094] S4, identifying whether the input image is forged based on the trained image forgery detection network.

[0095] The workflow is: the input image uses the image encoding module to extract image features, and then sends it to the pre-screening model, using a simple detection algorithm to quickly screen out obvious forged data and identify whether it is a forged image. If so, the judgment ends; if not, the gated calculation module uses the image features as input, generates a probability vector with the same dimension as the number of multiple expert models as the weight value of each expert model, and distributes the image features to the appropriate expert model through the gating mechanism. Each expert model in the expert model module extracts features and performs forgery detection on the input image features, independently determines the forgery probability, and fuses the forgery probability values ​​output by each expert model based on non-zero weight values, and identifies the input image with a total forgery probability value higher than the threshold as a forged image.

[0096] In a specific implementation of the present invention, a visualization output module can also be introduced into the overall process, and the comprehensive results are sent to the visualization output module to display the confidence of the independent judgment of each expert model, the weight distribution and the final fusion decision process, and provide visualization results such as forged area positioning. In this embodiment, the importance or significance of certain areas in the image is visualized in the form of a heat map generated by methods such as Grad-CAM. The forged area of ​​the image usually shows anomalies in the heat map, for example: low significance or abnormal significance distribution, and the heat map distribution of the forged area may be inconsistent with other areas. By displaying the final detection results to the user through a visual interface, outputs in the form of forged area positioning, detection reports, etc. are provided, and the interpretability of the decision chain is presented at the same time, helping users understand the decision process.

[0097] Each model in the present invention focuses on images of a specific difficulty, and can more deeply explore and learn the unique data features and forgery traces of the corresponding images. Compared with a single model learning all images, the feature extraction is more accurate. Training models separately for images of different difficulty levels can effectively reduce interference in the model learning process, optimize the training effect of the model, and thus significantly improve the performance of the model in the image recognition tasks of their respective difficulty levels. The good performance of each model on images of a specific difficulty enables the entire system to have stronger generalization capabilities when facing images of different difficulty levels, and can more accurately identify and process various complex actual scenes. The present invention has high flexibility and scalability. If new difficulty level images or forgery techniques appear in the future, the system's functions can be easily expanded by adding new model categories and training data to adapt to changing needs.

[0098] Based on the same inventive concept, an image forgery detection system based on multi-expert model decision-making is also provided in the present embodiment, and the system is used to implement the above-mentioned embodiment. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the system described in the following embodiments is preferably implemented in software, it is also possible to implement hardware, or a combination of software and hardware.

[0099] An image forgery detection system based on multi-expert model decision making, such as Figure 3 As shown, including:

[0100] An image sample generation module, which is used to obtain normal images in multiple fields, generate forged images based on different forgery types and forgery degrees, and group them;

[0101] A one-time training module, which is used to pre-train a pre-screening model and a multi-expert model capable of predicting the probability of forgery using normal images and each group of forged images, wherein the multi-expert model includes a typical forgery trace detection model based on an EfficientNet network, a complex refinement trace detection model based on a ViT network, and a local forgery trace detection model based on an Xception network;

[0102] A secondary training module, which is used to train an image forgery detection network including an image encoding module, a gated calculation module, a pre-screening model and a multi-expert model using normal images and forged images; during training, the parameters of the pre-screening model and the multi-expert model are frozen;

[0103] The image forgery identification module is used to identify whether the input image is forged based on the trained image forgery detection network, including: using the image encoding module to extract image features; the pre-screening model uses the image features as input to identify whether it is a forged image, and if so, the judgment ends; if not, the gated calculation module uses the image features as input to generate a probability vector with the same dimension as the number of multiple expert models as the weight value of each expert model; the expert model with non-zero weight value is screened and the forgery probability is independently judged using the image features as input, and the forgery probability values ​​output by each expert model are fused based on the non-zero weight value, and the input image with a total forgery probability value higher than the threshold is identified as a forged image.

[0104] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0105] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0106] It should also be noted that the image forgery detection method based on multi-expert model decision in the above embodiment can essentially be executed by a computer program. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the method provided in the above embodiment, which includes a memory and a processor;

[0107] The memory is used to store computer programs;

[0108] The processor is used to implement the image forgery detection method based on multi-expert model decision in the above embodiment when executing the computer program.

[0109] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0110] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the method provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the image forgery detection method based on multi-expert model decision-making in the above embodiment can be implemented.

[0111] It is understandable that the above storage medium may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk storage. The storage medium may also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc., which may store program codes.

[0112] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0113] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0114] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. An image forgery detection method based on multi-expert model decision making, characterized in that: The following steps are involved: (1) obtaining normal images in multiple fields, generating and grouping forged images based on different forgery types and forgery degrees, and using normal images and groups of forged images to pre-train a pre-screening model and a multi-expert model that can predict the probability of forgery. The multi-expert model includes a typical forgery trace detection model based on the EfficientNet network, a complex refinement trace detection model based on the ViT network, and a local forgery trace detection model based on the Xception network; (2) Using normal images and forged images, an image forgery detection network is trained, which includes an image encoding module, a gated calculation module, a pre-screening model, and a multi-expert model; During training, the parameters of the pre-screening model and the multi-expert model are frozen; (3) Based on the trained image forgery detection network, identify whether the input image is forged, including: Extract image features using image coding module; The pre-screening model takes image features as input to identify whether it is a forged image. If so, the judgment ends; if not, the gated calculation module takes image features as input to generate a probability vector with the same dimension as the number of multiple expert models as the weight value of each expert model; the expert model with non-zero weight value is screened and the forgery probability is independently judged with image features as input, and the forgery probability values ​​output by each expert model are fused based on the non-zero weight value, and the input image with a total forgery probability value higher than the threshold is identified as a forged image.

2. The image forgery detection method based on multi-expert model decision making according to claim 1 is characterized in that: Step (1) also includes an image preprocessing process, wherein the preprocessing includes standardization, denoising and enhancement.

3. The image forgery detection method based on multi-expert model decision making according to claim 1 is characterized in that: In step (1), generating a forged image based on different forgery types and forgery degrees includes: The types of forgery include texture forgery, boundary forgery, brightness forgery, facial feature forgery, expression forgery, lighting forgery and synthetic splicing forgery; The forgery degree refers to the total disturbance degree of a normal image under different forgery types of disturbances.

4. The image forgery detection method based on multi-expert model decision making according to claim 3 is characterized in that: In step (1), the method for grouping forged images includes: (1-1) Obtaining the forgery type and forgery degree corresponding to the forged image. If the forgery type corresponding to the forged image includes one or more of texture forgery, boundary forgery, brightness forgery, and synthetic splicing forgery, and the forgery degree is higher than a threshold, the forgery image is classified into the first forged image group; otherwise, executing step (1-2); (1-2) If the forgery type corresponding to the forged image includes one or more of texture forgery, boundary forgery, brightness forgery, and synthetic splicing forgery, and the degree of forgery is not higher than the threshold; or the forgery type is selected from any one of facial feature forgery, expression forgery, and lighting forgery, and the degree of forgery is higher than the threshold; then it is classified into the second forged image group; otherwise, step (1-3) is executed; (1-3) If the forgery type corresponding to the forged image is selected from facial feature forgery, expression forgery, and lighting forgery, and the degree of forgery is not higher than the threshold, then the image is classified into the third forged image group; (1-4) Selecting images whose forged regions are local regions from the third forged image group, and the selected images constitute a fourth forged image group.

5. The image forgery detection method based on multi-expert model decision making according to claim 3 is characterized in that: In step (1), the combination of the first forged image group and several normal images is used as the pre-screening model training set, the combination of the second forged image group and several normal images is used as the training set of the EfficientNet network, the combination of the third forged image group and several normal images is used as the training set of the ViT network, and the combination of the fourth forged image group and several normal images is used as the training set of the Xception network.

6. The image forgery detection method based on multi-expert model decision making according to claim 1 or 5, characterized in that: The pre-screening model adopts a model based on a CNN network, and the gating calculation module adopts a fully connected network.

7. The image forgery detection method based on multi-expert model decision making according to claim 1 is characterized in that: It also includes visualization steps of the image forgery detection process, including: Display the heat map corresponding to the input image; As well as, the forgery probability, weight distribution, total forgery probability and final recognition result independently judged by each expert model are displayed.

8. An image forgery detection system based on multi-expert model decision-making, used to implement the method described in claim 1; characterized in that: The system comprises: An image sample generation module, which is used to obtain normal images in multiple fields, generate forged images based on different forgery types and forgery degrees, and group them; A one-time training module, which is used to pre-train a pre-screening model and a multi-expert model capable of predicting the probability of forgery using normal images and each group of forged images, wherein the multi-expert model includes a typical forgery trace detection model based on an EfficientNet network, a complex refinement trace detection model based on a ViT network, and a local forgery trace detection model based on an Xception network; A secondary training module, which is used to train an image forgery detection network including an image encoding module, a gated calculation module, a pre-screening model and a multi-expert model using normal images and forged images; during training, the parameters of the pre-screening model and the multi-expert model are frozen; The image forgery identification module is used to identify whether the input image is forged based on the trained image forgery detection network, including: using the image encoding module to extract image features; the pre-screening model uses the image features as input to identify whether it is a forged image, and if so, the judgment ends; if not, the gated calculation module uses the image features as input to generate a probability vector with the same dimension as the number of multiple expert models as the weight value of each expert model; the expert model with non-zero weight value is screened and the forgery probability is independently judged using the image features as input, and the forgery probability values ​​output by each expert model are fused based on the non-zero weight value, and the input image with a total forgery probability value higher than the threshold is identified as a forged image.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the image forgery detection method based on multi-expert model decision-making according to any one of claims 1 to 7 is implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the image forgery detection method based on multi-expert model decision-making according to any one of claims 1 to 7 when executing the computer program.

Images