Image tampering positioning method and system based on image generation large model, terminal and storage medium
By constructing an image tampering localization network based on a high-pass filter, a multimodal fusion encoder, and a large image generation model, the problem of insufficient model localization capability under small-scale training sets is solved, and accurate localization and robustness improvement of complex tampered images are achieved.
Patent Information
- Application Number
- CN202510871632.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-26
- Publication Date
- 2025-11-18
AI Technical Summary
Existing technologies have weak localization capabilities for image tampering localization models on small training sets, and lack robustness and generalization, making it difficult to adapt to complex and unknown tampered images.
An image tampering localization network model based on a preset high-pass filter, a multimodal fusion encoder, and a large-scale image generation model network is constructed. The model is trained using a training dataset, and the tampered areas are identified and located through feature extraction and localization decoding modules.
It improves the ability to locate image tampering regions in low-resource scenarios, enhances the robustness and generalization of the model, and can accurately locate tampering regions in complex tampered images.
Smart Images
Figure CN120976508A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimedia information security technology, and in particular to an image tampering location method, system, terminal, and computer-readable storage medium based on a large image generation model. Background Technology
[0002] With the rapid evolution of deep learning technology and the widespread availability of image editing tools, generating highly realistic forgeries has become easy. Today, ordinary users can easily manipulate images using only everyday devices and user-friendly software. While this technological advancement has brought unprecedented convenience and creativity, its misuse has also raised serious social risks. For example, altered images circulating on social media can mislead public opinion and exacerbate social conflicts; in legal and judicial settings, the reliability of digital images as evidence is also threatened, affecting judicial fairness.
[0003] To combat the ever-evolving image manipulation techniques, researchers in the field of multimedia forensics are continuously exploring more accurate and robust image forgery detection methods. Although deep learning-based approaches have made significant progress in recent years, performing well on standard datasets, their generalization ability in real-world, complex environments remains challenging. This is primarily due to the high diversity of real-world forged images: different manipulation tools, post-processing operations, and scene styles are combined in various ways, making it difficult to learn universal manipulation patterns from a limited number of samples. While some works have attempted to expand the training set to enhance the model's coverage, the high-quality construction of forged images relies heavily on manual operations, severely limiting the efficiency and quality of building large-scale datasets.
[0004] Therefore, current deep learning-based image tampering detection technologies generally suffer from weak generalization ability and insufficient robustness under limited training data conditions. How to improve the model's adaptability to complex and unknown tampering under small sample conditions has become a core technical challenge that urgently needs to be solved in the field of image forensics.
[0005] Therefore, existing technologies still need to be improved and developed. Summary of the Invention
[0006] The main objective of this invention is to provide an image tampering localization method, system, terminal, and computer-readable storage medium based on a large image generation model. This aims to solve the problem that in the prior art, when an image tampering localization model is trained using a small-scale training set, the model has weak localization ability for complex tampered images and insufficient robustness and generalization.
[0007] To achieve the above objectives, the present invention provides an image tampering localization method based on a large image generation model, the method comprising the following steps:
[0008] An image tampering localization network model is constructed based on a preset high-pass filter, a preset multimodal fusion encoder, and a preset large-scale image generation model network.
[0009] The image tampering localization network model is trained using the training dataset to obtain a trained image tampering localization network model;
[0010] The image to be tested is input into the trained image tampering localization network model, which identifies and locates the tampered areas in the image to be tested and outputs the tampered area localization result.
[0011] Optionally, in the image tampering localization method based on a large image generation model, the image tampering localization network model includes a first input branch, a second input branch, and a localization output; the preset high-pass filter and the preset multimodal fusion encoder are connected in series as the first input branch, the encoder of the preset large image generation model network is the second input branch, and the decoder of the preset large image generation model network is the localization output.
[0012] The preset image generation large model network is a network with an encoder and decoder structure that can perform image generation tasks.
[0013] The preset multimodal fusion encoder includes multiple sequentially connected feature extraction blocks. The encoder and decoder of the preset image generation large model network each include one feature extraction block. The feature extraction block is a neural network composed of multiple convolutional kernels. The feature extraction block is a transformer block that supports window attention and residual connections.
[0014] The image tampering localization network model includes a feature extraction module and a localization decoding module; the preset high-pass filter, the preset multimodal fusion encoder, and the encoder of the preset image generation large model network are combined to form the feature extraction module; the decoder of the image generation large model network constitutes the localization decoding module.
[0015] Optionally, in the image tampering localization method based on a large image generation model, the feature extraction block includes a local attention feature extraction block and a global attention feature extraction block; the local attention feature extraction block is a feature extraction block using local window attention, and the size of the local window is a preset patch size in the image tampering localization network model; the global attention feature extraction block is a feature extraction block using global window attention, and the size of the global window is the image size input to the image tampering localization network model.
[0016] The number of feature extraction blocks is an integer multiple of 3, and the feature extraction blocks are connected in a group of 3, wherein the 3 feature extraction blocks are two local attention feature extraction blocks and one global attention feature extraction block.
[0017] Optionally, in the image tampering localization method based on a large image generation model, the preset multimodal fusion encoder includes an SRM convolutional kernel, a latent variable mapping network, and a tampering feature latent variable mapping network.
[0018] Optionally, the image tampering localization method based on a large image generation model, wherein training the image tampering localization network model using a training dataset to obtain a trained image tampering localization network model specifically includes:
[0019] Obtain the tampered images and their corresponding tamper-marked images from the CatNet dataset as the training dataset;
[0020] The preset multimodal fusion encoder in the image tampering localization network model is trained using the training dataset, so that the preset multimodal fusion encoder performs supervised learning under the training dataset, learns the tampering traces left when editing images, and enables the preset multimodal fusion encoder to recognize the tampering traces in the image.
[0021] Optionally, the image tampering localization method based on a large image generation model, wherein training the preset multimodal fusion encoder in the image tampering localization network model using the training dataset specifically includes:
[0022] Freeze the weights corresponding to the three convolution kernels in the pre-loaded preset high-pass filter;
[0023] Freeze the weights corresponding to the feature extraction blocks in the encoder and decoder of the pre-loaded preset image generation large model network;
[0024] The training dataset is input into the image tampering localization network model for training to update the weights of the non-frozen regions, thereby obtaining the trained image tampering localization network model.
[0025] The weights corresponding to the feature extraction blocks are the weights obtained by training on the preset dataset of the preset image generation large model network, and the non-frozen region is the latent variable mapping network and the tampered feature latent variable mapping network in the preset multimodal fusion encoder.
[0026] Optionally, in the image tampering localization method based on a large image generation model, the preset high-pass filter includes multiple convolutional kernels of the same size, and each convolutional kernel is used to perform different spatial frequency response processing on the input image.
[0027] Furthermore, to achieve the above objectives, the present invention also provides an image tampering localization system based on a large-scale image generation model, wherein the image tampering localization system based on a large-scale image generation model includes:
[0028] The model building module is used to build an image tampering localization network model based on a preset high-pass filter, a preset multimodal fusion encoder, and a preset large image generation model network.
[0029] The model training module is used to train the image tampering localization network model using the training dataset to obtain the trained image tampering localization network model.
[0030] The tampering region localization module is used to input the image to be tested into the trained image tampering localization network model. The trained image tampering localization network model identifies and locates the tampered region of the image to be tested and outputs the tampering region localization result.
[0031] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an image tampering location program based on an image generation large model stored in the memory and executable on the processor, wherein when the image tampering location program based on the image generation large model is executed by the processor, it implements the steps of the image tampering location method based on the image generation large model as described above.
[0032] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image tampering location program based on an image generation large model, and the image tampering location program based on the image generation large model, when executed by a processor, implements the steps of the image tampering location method based on the image generation large model as described above.
[0033] In this invention, an image tampering localization network model is constructed based on a preset high-pass filter, a preset multimodal fusion encoder, and a preset large-scale image generation model network. The image tampering localization network model is trained using a training dataset to obtain a trained image tampering localization network model. The image to be tested is input into the trained image tampering localization network model, which identifies and locates the tampered regions in the image and outputs the tampered region localization result. This invention fully leverages the rich semantic information contained in the large-scale image generation model, and by utilizing its excellent generalization ability and robustness, effectively improves the accurate localization capability of image tampering regions in low-resource scenarios, thus enhancing the localization performance of the image tampering localization network model for tampered regions in tampered images. Attached Figure Description
[0034] Figure 1 This is a flowchart of a preferred embodiment of the image tampering localization method based on a large image generation model of the present invention;
[0035] Figure 2 This is a preferred embodiment of the image tampering localization method based on a large image generation model of the present invention, in which SRM is used as the convolution kernel parameter diagram of a preset high-pass filter;
[0036] Figure 3 This is a schematic diagram of the training process of the image tampering localization network model in a preferred embodiment of the image tampering localization method based on a large image generation model of the present invention;
[0037] Figure 4 This is a detailed structural diagram of the latent variable mapping network in a preferred embodiment of the image tampering localization method based on a large image generation model of the present invention;
[0038] Figure 5 This is a detailed structural diagram of the tamper feature latent variable mapping network in a preferred embodiment of the image tampering localization method based on a large image generation model of the present invention;
[0039] Figure 6 This is a schematic diagram illustrating the image tampering localization performance of the image tampering localization network model of the present invention and other image tampering localization networks on Coverage and Cocoglide tampered images in a preferred embodiment of the image tampering localization method based on the image generation large model of the present invention.
[0040] Figure 7 This is a schematic diagram illustrating the performance of the image tampering localization network model of the present invention and other image tampering localization networks in tampering region localization on PS-Boundary tampered images in a preferred embodiment of the image tampering localization method based on a large image generation model of the present invention.
[0041] Figure 8This is a schematic diagram illustrating the image tampering localization performance of the image tampering localization network model of the present invention and other image tampering localization networks on a Coverage tampered image with added Gaussian noise in a preferred embodiment of the image tampering localization method based on a large image generation model of the present invention.
[0042] Figure 9 This is a structural diagram of a preferred embodiment of the image tampering localization system based on a large image generation model of the present invention;
[0043] Figure 10 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation
[0044] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0045] The method of this invention fully explores the rich semantic information contained in the large image generation model. With its excellent generalization ability and robustness, it effectively improves the ability to accurately locate the image tampered area in low-resource scenarios, thereby overcoming the shortcomings of existing technologies in poor performance under small sample conditions.
[0046] The preferred embodiment of the image tampering localization method based on a large image generation model of the present invention, such as... Figure 1 As shown, the image tampering localization method based on a large image generation model includes the following steps:
[0047] Step S10: Construct an image tampering localization network model based on a preset high-pass filter, a preset multimodal fusion encoder, and a preset image generation large model network.
[0048] Specifically, an image tampering localization network model is constructed based on a preset high-pass filter, a preset multimodal fusion encoder, and a preset image generation large model network. The model has two inputs and one output, that is, the image tampering localization network model includes a first input branch, a second input branch, and a localization output. The preset high-pass filter and the preset multimodal fusion encoder are connected in series as the first input branch, the encoder of the preset image generation large model network is the second input branch, and the decoder of the preset image generation large model network is the localization output.
[0049] The preset high-pass filter includes multiple convolutional kernels of the same size (5×5 pixels), each kernel processing the input image with different spatial frequency responses. The preset image generation large model network is an encoder-decoder structure network capable of performing image generation tasks. The preset multimodal fusion encoder includes multiple sequentially connected feature extraction blocks. Both the encoder and decoder of the preset image generation large model network include one feature extraction block, which is a neural network composed of multiple convolutional kernels and a transformer block supporting window attention and residual connections. The output of the feature extraction block is the result obtained by adding the input processed by the window attention mechanism to the original input through residual connections, and then processing it through a multilayer perceptron. The encoder output of the preset image generation large model network is fused with the output of the preset multimodal fusion encoder. The preset high-pass filter and the preset multimodal fusion encoder are connected in series to mainly extract high-frequency features, while the encoder of the preset image generation large model network mainly extracts target features.
[0050] The image tampering localization network model includes a feature extraction module and a localization decoding module; the preset high-pass filter, the preset multimodal fusion encoder, and the encoder of the preset image generation large model network are combined to form the feature extraction module; the decoder of the image generation large model network constitutes the localization decoding module; the localization decoding module is connected to the feature extraction module, and the localization decoding module is used to generate the result of tampering area localization.
[0051] The feature extraction block includes a local attention feature extraction block and a global attention feature extraction block; the local attention feature extraction block is a feature extraction block using local window attention, and the size of the local window is the preset patch size in the image tampering localization network model; the global attention feature extraction block is a feature extraction block using global window attention, and the size of the global window is the image size input to the image tampering localization network model.
[0052] In this embodiment, the number of feature extraction blocks is an integer multiple of 3, and the feature extraction blocks are connected in a sequential manner in groups of 3. The 3 feature extraction blocks are, in turn, two local attention feature extraction blocks and one global attention feature extraction block. That is, every group of 3 feature extraction blocks consists of two local attention feature extraction blocks and one global attention feature extraction block.
[0053] For example, using the three convolutional kernels in SRM (Steganalysis Rich Model Filters) as preset high-pass filters, an image tampering localization network model is constructed using the encoder and decoder of the SD3 (Stable Diffusion V3) image generation large model. The convolutional kernel parameters of SRM are as follows: Figure 2 As shown, these three convolutional kernels are used to extract different types of residual information in the image: the upper left filter enhances medium-scale edge changes and is used to capture local structural anomalies; the upper right filter extracts stronger high-frequency residuals, which is suitable for detecting compression artifacts and detail tampering; the lower filter is a horizontal first-order difference, used to enhance horizontal edges.
[0054] Step S20: Train the image tampering localization network model using the training dataset to obtain the trained image tampering localization network model.
[0055] Specifically, such as Figure 3 The diagram illustrates the training process of the image tampering localization network model. The preset multimodal fusion encoder includes an SRM convolutional kernel (SRM Filter), a latent variable mapping network (Latent Mapping Model), and a tampering feature latent variable mapping network (Forgery Latent Mapping Model). m Z represents the latent variable after the SD3 encoder transforms the true location result of the tampered image. i Z represents the latent variable after the tampered image is converted by the SD3 encoder. f This represents the latent variables after the tampered image has been transformed by the SRM convolution kernel and the tampered feature latent variable mapping network. L represents the latent variables after multimodal fusion. lm Z represents m and The calculated loss value, L loc This represents the loss value calculated from the predicted tampered location result and the actual tampered location result.
[0056] Figure 4 express Figure 3 Detailed structure diagram of the latent variable mapping network. Figure 5 for Figure 3 The detailed structure diagram of the tampered feature latent variable mapping network is shown. The tampered feature latent variable mapping network is the feature extraction block of the preset multimodal fusion encoder. The SRM convolution kernel, the tampered feature latent variable mapping network and the latent variable mapping network together constitute the preset multimodal fusion encoder. The encoder of the SD3 image generation large model includes one feature extraction block.
[0057] In this embodiment, the tampered images and their corresponding tampered marked images in the CatNet dataset are used as training samples (i.e., training dataset) and input into the image tampering localization network model for training. In other words, the training dataset is used to pre-train the preset multimodal fusion encoder.
[0058] Understandably, the image tampering localization network model based on the pre-set image generation large model network, with the help of knowledge learned from the image generation large model, and trained on the pre-set multimodal fusion encoder branch through the CatNet image tampering localization dataset (which contains the tampered image and its corresponding tampering-marked image training dataset), that is, the image tampering localization network model performs supervised learning under training samples, learns the tampering traces left when editing images, so that the multimodal fusion encoder can effectively identify tampering traces in images, thus obtaining a network model with pixel-level tampering region localization capability.
[0059] It should be noted that the proposal of large-scale image generation models has brought new inspiration to deep learning-based image generation tasks. Deep learning models trained on large-scale datasets with large-scale parameters and complex structures can exhibit robustness and generalization far exceeding that of conventional models. In the field of medical image segmentation, large-scale models have also been applied to downstream tasks of image segmentation through fine-tuning and other means.
[0060] Specifically, the training process is as follows: freeze the weights corresponding to the three convolutional kernels in the pre-loaded preset high-pass filter; freeze the weights corresponding to the feature extraction blocks in the encoder and decoder of the pre-loaded preset image generation large model network; input the training dataset into the image tampering localization network model for training to update the weights of the non-frozen regions, and obtain the trained image tampering localization network model; wherein, the weights corresponding to the feature extraction blocks are the weights trained on the preset dataset (ImageNet dataset and CC12M dataset) of the preset image generation large model network, and the non-frozen regions are the latent variable mapping network and the tampering feature latent variable mapping network in the preset multimodal fusion encoder.
[0061] Understandably, after the image tampering localization network model is constructed, the weights obtained by training the large image generation model on the image generation dataset are loaded onto the large model encoder and decoder. These weights are the weights of the feature extraction blocks in the large model encoder and decoder, and the weights of these feature extraction blocks are frozen. In the image tampering localization network model, the region where the pre-trained weights of the large model image generation model are loaded onto the large model encoder and decoder is the frozen region. Then, the tampered images and their corresponding tampering-marked images in the training dataset are used as training samples and input into the image tampering localization network model to update the weights of the non-frozen regions. The resulting network with pixel-level tampering region localization capability is used as the trained image tampering localization network model.
[0062] For example, in training the image tampering localization network model, PyTorch 1.8.0 can be used to implement the proposed large-model-based image tampering localization and detection method of this invention, and all experiments are conducted on an NVIDIA A100 GPU (a type of graphics card). PyTorch is an open-source Python machine learning library. Publicly available tampered images and their corresponding tampering-marked images are used as training samples. The training samples are then randomly divided into training and validation sets at a 9:1 ratio, and a publicly available dataset is used as the test set. The test set contains no duplicate images from the training set, and the tampered images in both sets may have been modified using different tampering tools. The test set includes tampered images without image rendering and tampered images with image rendering. During the training and testing phases, the input image resolution can be scaled to 512×512, using the Adam optimizer, a batch size of 4, and a learning rate of 1×10⁻⁶. -4 The number of epochs during the training phase can be set to 200. Tamper localization uses F1 as the performance evaluation metric, while tamper detection uses accuracy as the performance evaluation metric.
[0063] Step S30: Input the image to be tested into the trained image tampering localization network model. The trained image tampering localization network model identifies and locates the tampered areas of the image to be tested and outputs the tampered area localization result.
[0064] Specifically, the image tampering localization network model is trained using a training dataset. After obtaining the trained model, the images to be tested are divided into three categories: 1. Natural scene tampering images; 2. Book cover tampering images; 3. Natural scene tampering images after post-processing (Gaussian noise is added). The images to be tested are input into the trained model, and the detection output is obtained to identify the tampered regions, thus achieving accurate localization between the tampered and original images.
[0065] As can be seen, in this embodiment of the invention, by using the tampered images and their corresponding tampered marked images in the training dataset as training samples, and then using these training samples to train the image tampering localization network model constructed based on a preset high-pass filter, a preset multimodal fusion encoder, and a preset large-scale image generation network, a network model capable of accurately locating the tampered region on the tampered image is obtained. That is, the trained image tampering localization network model can accurately locate the editing behavior on the tampered image. With the help of the generalization and robustness of the large-scale network model, the generalization and robustness of the image tampering localization network model for tampered images can be greatly improved. In other words, the trained image tampering localization network model exhibits excellent performance on tampered images in different scenarios and after different post-processing operations, playing an important role in improving the image tampering localization network model's performance in tampering image localization and overcoming the current research in the field of image tampering localization which lacks robustness and generalization.
[0066] By comparing the image tampering localization method based on a large image generation model proposed in this invention with current tampering localization methods, it can be verified that the method proposed in this invention has better performance than current tampering localization methods. This embodiment uses the deep learning models TruFor and CatNetv2, which have excellent performance in the field of forensics, and tests them using Coverage and NIST16 datasets, which, like the CatNet dataset, depict tampered images in natural scenes. The tampering localization test results are shown in Table 1:
[0067] Table 1: Localization performance of the present invention and different models
[0068] Model Coverage NIST16 CatNetv2 0.381 0.301 TroFor 0.525 0.362 This invention 0.586 0.503
[0069] As shown in Table 1, the method proposed in this invention demonstrates superior localization performance for tampered images compared to current leading deep learning models TruFor and CatNetv2. For comparisons of localization results for some samples, please refer to [reference needed]. Figure 6 As shown.
[0070] This embodiment uses the text image dataset PS-Boundary for testing. The tamper location test results are shown in Table 2:
[0071] Table 2: Generalization performance of the present invention and different models
[0072] Model F1 CatNetv2 0.433 TroFor 0.473 This invention 0.593
[0073] As shown in Table 2, the method proposed in this invention demonstrates superior generalization performance for text-tampered images compared to current leading deep learning models TruFor and CatNetv2. For comparisons of localization results for some samples, please refer to [link to table]. Figure 7 As shown.
[0074] This embodiment uses TruFor, a deep learning model that performs well in the field of forensics. Using the tampered image dataset Coverage, Gaussian noise was added to the images. The added Gaussian noise had a mean of 0 and variances of 0.1, 0.3, and 0.5, respectively. Testing was then conducted, and the tamper location test results are shown in Table 3.
[0075] Table 3: Localization robustness of the present invention and different models
[0076] Model 0.1 0.3 0.5 TroFor 0.357 0.331 0.131 This invention 0.581 0.580 0.580
[0077] As shown in Table 3, the method proposed in this invention exhibits less performance degradation compared to the current excellent deep learning model TruFor, and demonstrates superior robustness in localization of tampered images. See Table 3 for comparisons of localization results for some samples. Figure 8 As shown.
[0078] This invention utilizes a training dataset to train an image tampering localization network model constructed based on a preset high-pass filter, a preset multimodal fusion encoder, and a preset large model network. This results in a network model capable of accurately locating tampered regions in a tampered image. In other words, the trained image tampering localization network model can accurately locate editing actions on a tampered image while retaining the rich semantic information learned by the large image generation model network. This improves the image tampering localization network model's performance in locating tampered images and significantly enhances its generalization and robustness. This invention plays a crucial role in overcoming the weaknesses in tampering localization performance, generalization, and robustness of current image tampering localization methods.
[0079] Furthermore, such as Figure 9 As shown, based on the above-mentioned image tampering localization method based on a large-scale image generation model, the present invention also provides an image tampering localization system based on a large-scale image generation model, wherein the image tampering localization system based on a large-scale image generation model includes:
[0080] Model building module 11 is used to build an image tampering localization network model based on a preset high-pass filter, a preset multimodal fusion encoder and a preset image generation large model network;
[0081] Model training module 12 is used to train the image tampering localization network model using the training dataset to obtain a trained image tampering localization network model.
[0082] The tampering region localization module 13 is used to input the image to be tested into the trained image tampering localization network model, the trained image tampering localization network model identifies and locates the tampered region of the image to be tested, and outputs the tampering region localization result.
[0083] Furthermore, such as Figure 10 As shown, based on the above-described image tampering localization method and system based on a large image generation model, the present invention also provides a terminal, the terminal comprising:
[0084] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.
[0085] When the processor 502 executes the program, it implements the image tampering location method based on the large image generation model provided in the above embodiments.
[0086] Furthermore, the terminal also includes:
[0087] Communication interface 503 is used for communication between memory 501 and processor 502.
[0088] The memory 501 is used to store computer programs that can run on the processor 502.
[0089] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0090] If the memory 501, processor 502, and communication interface 503 are implemented independently, they can be interconnected via a bus to communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in the diagram, but this does not imply that there is only one bus or one type of bus.
[0091] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.
[0092] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.
[0093] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image tampering location program based on an image generation large model, and the image tampering location program based on the image generation large model, when executed by a processor, implements the steps of the image tampering location method based on the image generation large model as described above.
[0094] In summary, this invention provides an image tampering localization method, system, terminal, and computer-readable storage medium based on a large image generation model. The method includes: constructing an image tampering localization network model based on a preset high-pass filter, a preset multimodal fusion encoder, and a preset large image generation model network; training the image tampering localization network model using a training dataset to obtain a trained image tampering localization network model; inputting a test image into the trained image tampering localization network model, whereby the trained image tampering localization network model identifies and locates the tampered region in the test image and outputs the tampered region localization result. This invention fully leverages the rich image semantic information contained in the large image generation model, and by utilizing its excellent generalization ability and robustness, effectively improves the accurate localization capability of image tampering regions in low-resource scenarios, thus enhancing the localization performance of the image tampering localization network model for tampered regions in tampered images.
[0095] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.
[0096] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.
[0097] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A method for locating image tampering based on a large-scale image generation model, characterized in that, The image tampering localization method based on a large image generation model includes: An image tampering localization network model is constructed based on a preset high-pass filter, a preset multimodal fusion encoder, and a preset large-scale image generation model network. The image tampering localization network model is trained using the training dataset to obtain a trained image tampering localization network model; The image to be tested is input into the trained image tampering localization network model, which identifies and locates the tampered areas in the image to be tested and outputs the tampered area localization result.
2. The image tampering localization method based on a large image generation model according to claim 1, characterized in that, The image tampering localization network model includes a first input branch, a second input branch, and a localization output; the preset high-pass filter and the preset multimodal fusion encoder are connected in series as the first input branch, the encoder of the preset image generation large model network is the second input branch, and the decoder of the preset image generation large model network is the localization output; The preset image generation large model network is a network with an encoder and decoder structure that can perform image generation tasks. The preset multimodal fusion encoder includes multiple sequentially connected feature extraction blocks. The encoder and decoder of the preset image generation large model network each include one feature extraction block. The feature extraction block is a neural network composed of multiple convolutional kernels. The feature extraction block is a transformer block that supports window attention and residual connections. The image tampering localization network model includes a feature extraction module and a localization decoding module; the preset high-pass filter, the preset multimodal fusion encoder, and the encoder of the preset image generation large model network are combined to form the feature extraction module; the decoder of the image generation large model network constitutes the localization decoding module.
3. The image tampering localization method based on a large image generation model according to claim 2, characterized in that, The feature extraction block includes a local attention feature extraction block and a global attention feature extraction block; the local attention feature extraction block is a feature extraction block using local window attention, and the size of the local window is the preset patch size in the image tampering localization network model; the global attention feature extraction block is a feature extraction block using global window attention, and the size of the global window is the image size input to the image tampering localization network model; The number of feature extraction blocks is an integer multiple of 3, and the feature extraction blocks are connected in a group of 3, wherein the 3 feature extraction blocks are two local attention feature extraction blocks and one global attention feature extraction block.
4. The image tampering localization method based on a large image generation model according to claim 3, characterized in that, The preset multimodal fusion encoder includes an SRM convolution kernel, a latent variable mapping network, and a tampered feature latent variable mapping network.
5. The image tampering localization method based on a large image generation model according to claim 4, characterized in that, The step of training the image tampering localization network model using a training dataset to obtain a trained image tampering localization network model specifically includes: Obtain the tampered images and their corresponding tamper-marked images from the CatNet dataset as the training dataset; The preset multimodal fusion encoder in the image tampering localization network model is trained using the training dataset, so that the preset multimodal fusion encoder performs supervised learning under the training dataset, learns the tampering traces left when editing images, and enables the preset multimodal fusion encoder to recognize the tampering traces in the image.
6. The image tampering localization method based on a large image generation model according to claim 5, characterized in that, The step of training the preset multimodal fusion encoder in the image tampering localization network model using the training dataset specifically includes: Freeze the weights corresponding to the three convolution kernels in the pre-loaded preset high-pass filter; Freeze the weights corresponding to the feature extraction blocks in the encoder and decoder of the pre-loaded preset image generation large model network; The training dataset is input into the image tampering localization network model for training to update the weights of the non-frozen regions, thereby obtaining the trained image tampering localization network model. The weights corresponding to the feature extraction blocks are the weights obtained by training on the preset dataset of the preset image generation large model network, and the non-frozen region is the latent variable mapping network and the tampered feature latent variable mapping network in the preset multimodal fusion encoder.
7. The image tampering localization method based on a large image generation model according to claim 1, characterized in that, The preset high-pass filter includes multiple convolution kernels, all of which are of the same size. Each convolution kernel is used to perform different spatial frequency response processing on the input image.
8. An image tampering localization system based on a large image generation model, characterized in that, The image tampering localization system based on a large image generation model includes: The model building module is used to build an image tampering localization network model based on a preset high-pass filter, a preset multimodal fusion encoder, and a preset large image generation model network. The model training module is used to train the image tampering localization network model using the training dataset to obtain the trained image tampering localization network model. The tampering region localization module is used to input the image to be tested into the trained image tampering localization network model. The trained image tampering localization network model identifies and locates the tampered region of the image to be tested and outputs the tampering region localization result.
9. A terminal, characterized in that, The terminal includes: a memory, a processor, and an image tampering location program based on an image generation model stored in the memory and executable on the processor. When the image tampering location program based on an image generation model is executed by the processor, it implements the steps of the image tampering location method based on an image generation model as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image tampering localization program based on a large image generation model, which, when executed by a processor, implements the steps of the image tampering localization method based on a large image generation model as described in any one of claims 1-7.