Image tampering positioning and detecting method and device based on large model and medium

Abstract

The invention discloses an image tampering positioning and detecting method and device based on a large model, and a medium. The method comprises the following steps: constructing an image tampering positioning and detecting network model based on a preset high-pass filter, a preset feature fusion encoder, a preset large model network and a preset full-connection network; training the image tampering positioning and detection network model by using the training data set to obtain a trained image tampering positioning and detection network model; inputting a to-be-detected image into the trained image tampering positioning and detection network model, obtaining positioning output, and obtaining a tampering region positioning result; and inputting a to-be-detected image into the trained image tampering positioning and detection network model, obtaining detection output, and obtaining a detection result used for reflecting whether the to-be-detected image is tampered or not. The image tampering localization and detection network model obtained through training can improve the tampering area localization and tampering image detection performance of the tampering image, and has excellent generalization and robustness.

Description

Technical Field

[0001] The present invention relates to the field of multimedia information security technology, and in particular to a large model-based image tampering positioning and detection method, device and medium. Background Art

[0002] With the rapid development of deep learning and the widespread availability of manipulation tools, creating realistic fake images has become easy for the general public. Tasks that once required professional skills and advanced software are now easily accomplished through a variety of user-friendly applications and everyday devices. While these technologies have promoted convenience and creativity, their misuse can have serious social consequences. Maliciously manipulated images, especially those shared on social media, can incite unrest and threaten social stability. Moreover, in the legal context, the credibility of digital images is crucial, and manipulated evidence can jeopardize the administration of justice.

[0003] Experts in the field of multimedia forensics are constantly working to develop advanced methods for locating and detecting image forgeries to meet the increasingly complex challenges of image tampering. Although these methods have shown significant performance in specific scenarios, they still face many challenges. Real tampered images are extremely complex, covering a variety of scenarios, tampering tools, and post-processing operations, which makes it extremely difficult to mathematically model their tampering traces. Existing methods usually try to cover the real situation as much as possible by increasing the diversity of training sets. Models trained on small-scale training sets often lack sufficient robustness and generalization performance when faced with unseen data or tampered images that have undergone multiple post-processing operations. However, building a large-scale training set faces major challenges. Since the production of tampered images is highly dependent on manual operations, the entire process is extremely time-consuming and labor-intensive, which limits the scale and quality of training data.

[0004] In summary, the existing deep learning-based image tampering localization and detection technology has poor generalization and robustness in tampered image localization and detection when the training set is relatively small.

[0005] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method, device and medium for locating and detecting image tampering based on a large model in view of the above-mentioned defects of the prior art, aiming to solve the problems in the existing image tampering locating and detection technology that the tampered image positioning and detection performance is weak, the robustness and generalization are not strong when training with a small-scale training set. The technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the present invention provides a method for locating and detecting image tampering based on a large model, wherein the method comprises:

[0008] An image forgery localization and detection network model is constructed based on a preset high-pass filter, a preset feature fusion encoder, a preset large model network, and a preset fully connected network. The image forgery localization and detection network model includes a localization output and a detection output;

[0009] The image forgery localization and detection network model is trained using a training data set to obtain a trained image forgery localization and detection network model;

[0010] The image to be tested is input into the trained image forgery localization and detection network model, and the localization output of the trained image forgery localization and detection network model is obtained to identify and locate the forgery area of the image to be tested, and a forgery area localization result is obtained;

[0011] The image to be tested is input into the trained image forgery localization and detection network model, and the detection output of the trained image forgery localization and detection network model is obtained to determine whether the image to be tested has been forged, and a detection result reflecting whether the image to be tested has been forged is obtained.

[0012] In one implementation, the preset high-pass filter includes a plurality of convolution kernels, the size of each convolution kernel is the same, the size of each convolution kernel is 5×5 pixels, and each convolution kernel is used to perform different spatial frequency response processing on the input image.

[0013] In one implementation, the preset large model network is a network that combines an encoder and a decoder and can perform image segmentation tasks. The preset feature fusion encoder is an encoder pre-trained in the network that combines an encoder and a decoder and can perform image segmentation tasks. The preset high-pass filter is connected in series with the feature fusion encoder, and the preset feature fusion encoder is connected to the encoder in the preset large model network for extracting target features.

[0014] In one implementation, the preset feature fusion encoder includes a plurality of sequentially connected feature extraction blocks, each of the feature extraction blocks is connected in parallel with the corresponding feature extraction block inside the encoder in the preset large model network, and the feature extraction block is a transformer block that supports window attention and residual connection;

[0015] Moreover, the feature extraction block includes a local attention feature extraction block and a global attention feature extraction block;

[0016] Among them, the local attention feature extraction block is a feature extraction block using local window attention, and the window size of the local window attention is the preset patch size in the image forgery localization and detection network model;

[0017] The global attention feature extraction block is a feature extraction block using global window attention, and the window size of the global window attention is the size of the image to be tested input into the image forgery localization and detection network model.

[0018] In one implementation, the number of the feature extraction blocks is an integer multiple of three, and the connection mode of the sequentially connected feature extraction blocks is that three feature extraction blocks form a group, and the three feature extraction blocks are two local attention feature extraction blocks and one global attention feature extraction block in sequence.

[0019] In one implementation, the preset fully connected network includes a flattening layer and one or more fully connected layers. The flattening layer and the fully connected layers are connected in series. The flattening layer takes multi-dimensional data as input and outputs one-dimensional data. The fully connected layer is composed of multiple neurons. If there are multiple fully connected layers, each neuron is connected to all neurons in the previous layer, forming a dense connection structure. The preset fully connected network is connected in series with the preset feature fusion encoder. The localization output is the output of the decoder of the preset large model network, and the detection output is the output of the preset fully connected network.

[0020] In one implementation, training the image forgery localization and detection network model using the training data set to obtain a trained image forgery localization and detection network model includes:

[0021] Freezing the weights corresponding to three convolutional kernels in the pre-loaded preset high-pass filter;

[0022] Freezing the weights corresponding to the feature extraction blocks of the encoder and decoder in the pre-loaded preset large model network;

[0023] Inputting the training data set into the image forgery localization and detection network model for training to update the weights in the unfrozen area, obtaining a trained image forgery localization and detection network model;

[0024] Wherein, the weights corresponding to the feature extraction blocks are the weights obtained by training the preset large model network on the image segmentation data set, and the unfrozen area is the preset feature fusion encoder and the preset fully connected network.

[0025] In a second aspect, an embodiment of the present invention further provides an image forgery localization and detection device based on a large model, wherein the device includes:

[0026] A model construction module for constructing an image forgery localization and detection network model based on a preset high-pass filter, a preset feature fusion encoder, a preset large model network, and a preset fully connected network. The image forgery localization and detection network model includes a localization output and a detection output;

[0027] A model training module, configured to train the image forgery localization and detection network model by using a training data set, so as to obtain a trained image forgery localization and detection network model;

[0028] A forgery localization module, configured to input a to-be-detected image into the trained image forgery localization and detection network model, and obtain a localization output of the trained image forgery localization and detection network model, so as to identify and locate the forgery area of the to-be-detected image, and obtain a forgery area localization result;

[0029] An image detection module, configured to input a to-be-detected image into the trained image forgery localization and detection network model, and obtain a detection output of the trained image forgery localization and detection network model, so as to determine whether the to-be-detected image has been forged, and obtain a detection result for reflecting whether the to-be-detected image has been forged.

[0030] In a third aspect, an embodiment of the present invention further provides a terminal. The terminal includes a memory, a processor, and an image forgery localization and detection program based on a large model stored in the memory and executable on the processor. When the processor executes the image forgery localization and detection program based on the large model, the steps of the image forgery localization and detection method based on the large model in any one of the above solutions are implemented.

[0031] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium. An image forgery localization and detection program based on a large model is stored on the computer-readable storage medium. When the image forgery localization and detection program based on the large model is executed by a processor, the steps of the image forgery localization and detection method based on the large model in any one of the above solutions are implemented.

[0032] Beneficial effects: Compared with the prior art, the present invention provides an image forgery localization and detection method based on a large model. First, an image forgery localization and detection network model is constructed based on a preset high-pass filter, a preset feature fusion encoder, a preset large model network, and a preset fully connected network. The image forgery localization and detection network model includes a localization output and a detection output. Then, the image forgery localization and detection network model is trained by using a training data set to obtain a trained image forgery localization and detection network model. Next, a to-be-detected image is input into the trained image forgery localization and detection network model, and a localization output of the trained image forgery localization and detection network model is obtained to identify and locate the forgery area of the to-be-detected image, and a forgery area localization result is obtained. Finally, a to-be-detected image is input into the trained image forgery localization and detection network model, and a detection output of the trained image forgery localization and detection network model is obtained to determine whether the to-be-detected image has been forged, and a detection result for reflecting whether the to-be-detected image has been forged is obtained.

[0033] The present invention uses a training data set to train an image tampering localization and detection network model constructed based on a preset high-pass filter, a preset feature fusion encoder, a preset large model network, and a preset fully connected network, so as to obtain a network model that can accurately locate the tampered area on the tampered image and accurately determine whether the image has been tampered with. That is, the trained image tampering localization network model can achieve accurate localization of the editing behavior on the tampered image and accurate judgment of whether the image has been edited, while retaining the original image segmentation performance of the large model network. Furthermore, it improves the localization and detection performance of the image tampering localization and detection network model for tampered images, and at the same time can greatly improve the generalization and robustness of the image tampering localization and detection network model for tampered images, playing an important role in overcoming the weak localization and detection performance, weak generalization, and weak robustness of the current methods in the field of image tampering localization. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flowchart of a preferred embodiment of the method for image tampering localization and detection based on a large model provided by an embodiment of the present invention.

[0035] Figure 2 It is a convolution kernel parameter diagram of a high-pass filter using SRM (Steganalysis Rich Model Filters) in the present invention.

[0036] Figure 3 It is a schematic diagram of an image tampering localization and detection network model based on a large model encoder using the SAM (Segment Anything Model) image segmentation large model as the large model network in the present invention.

[0037] Figure 4 It is a schematic diagram of the tampered area localization performance of the image tampering localization network model of the present application and other image tampering localization networks on Coverage and Cocoglide tampered images in the present invention.

[0038] Figure 5 It is a schematic diagram of the tampered area localization performance of the image tampering localization network model of the present application and other image tampering localization networks on PS-Boundary tampered images in the present invention.

[0039] Figure 6 It is a schematic diagram of the tampered area localization performance of the image tampering localization network model of the present application and other image tampering localization networks on Coverage tampered images with added Gaussian noise in the present invention.

[0040] Figure 7Schematic diagram of the architecture of the image forgery localization and detection device based on a large model provided by an embodiment of the present invention.

[0041] Figure 8 Principle block diagram of the terminal provided by an embodiment of the present invention. Detailed implementation manners

[0042] To make the objectives, technical solutions and effects of the present invention clearer and more definite, the following further describes the present invention in detail with reference to the accompanying drawings and by way of examples. It should be understood that the specific examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] The flowcharts shown in the accompanying drawings are only illustrative examples, and do not necessarily include all the contents, operations or steps, nor do they necessarily need to be executed in the described order. For example, some operations or steps can also be decomposed, combined or partially merged, so the actual execution order may be changed according to the actual situation.

[0044] It should be understood that the terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0045] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, terms such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and roles. For example, the first control information and the second control information are only used to distinguish different control information, and do not limit their sequence.

[0046] Those skilled in the art can understand that the terms "first", "second", etc. do not limit the quantity and execution order, and the terms "first", "second", etc. do not necessarily mean different.

[0047] It should also be understood that the term " / and" as used in the specification of the present invention and the appended claims refers to any combination and all possible combinations of one or more of the related listed items, and includes these combinations.

[0048] To solve the problems of the prior art, an embodiment of the present invention may first construct an image tampering localization and detection network model based on a preset high-pass filter, a preset feature fusion encoder, a preset large model network, and a preset fully connected network. The image tampering localization and detection network model includes a localization output and a detection output. Then, use a training data set to train the image tampering localization and detection network model to obtain a trained image tampering localization and detection network model. Next, input the image to be tested into the trained image tampering localization and detection network model, and obtain the localization output of the trained image tampering localization and detection network model to identify and locate the tampered area of the image to be tested, and obtain a tampered area localization result. Finally, input the image to be tested into the trained image tampering localization and detection network model, and obtain the detection output of the trained image tampering localization and detection network model to determine whether the image to be tested has been tampered with, and obtain a detection result reflecting whether the image to be tested has been tampered with.

[0049] The method for image tampering localization and detection based on a large model in this embodiment can be applied to a terminal, and the terminal is an intelligent product terminal such as a computer, a smart phone, and a smart TV. In this embodiment, as Figure 1 shown, the method for image tampering localization and detection based on a large model includes the following steps:

[0050] Step S100, construct an image tampering localization and detection network model based on a preset high-pass filter, a preset feature fusion encoder, a preset large model network, and a preset fully connected network. The image tampering localization and detection network model includes a localization output and a detection output.

[0051] In this embodiment, a dual-input and dual-output image tampering localization and detection network model based on a preset high-pass filter, a preset feature fusion encoder, a preset large model network, and a preset fully connected network is constructed. That is to say, a dual-input and dual-output image tampering localization and detection network model based on a large model network is constructed by combining a high-pass filter, a pre-trained encoder and decoder in a preset large model network, and a fully connected network.

[0052] The image tampering localization and detection network model has two input branches, namely: 1. A preset high-pass filter and a preset feature fusion encoder are connected in series as one branch; 2. The encoder of the preset large model network is a branch. The image tampering localization and detection network model has two output branches, namely: 1. The decoder of the preset large model network is used as the positioning output; 2. The preset fully connected network is used as the detection output. Among them, the preset large model network is a network that can perform image segmentation tasks in combination with an encoder and a decoder, and the preset feature fusion encoder is a pre-trained encoder in the network that can perform image segmentation tasks based on the encoder and the decoder. The preset high-pass filter is connected in series with the feature fusion encoder, and the preset feature fusion encoder is connected to the encoder in the preset large model network for extracting target features. And the preset high-pass filter and the preset feature fusion encoder are connected in series to mainly extract high-frequency features, and the preset large model encoder mainly extracts target features, that is, the high-pass filter, the feature fusion encoder and the preset large model network encoder are combined to form a feature extraction module in the image tampering positioning network model, and the decoder of the preset large model network constitutes a positioning decoding module in the image tampering positioning network model. The positioning decoding module is connected to the feature extraction module, and the positioning decoding module is used to generate the result of tampering area positioning. The fully connected network constitutes the detection decoding module in the image tampering positioning network model, the detection decoding module is linked to the feature extraction module, and the detection decoding module is used to generate the result of whether the image has been tampered with.

[0053] The preset large model encoder of this embodiment includes multiple feature extraction blocks connected in sequence, and the feature extraction block is a transformer block that supports window attention and residual connection; wherein the output of the feature extraction block is the result of the input being processed by the window attention mechanism and added to the original input through a residual connection, and then processed by a multi-layer perceptron.

[0054] The feature extraction block of this embodiment includes a local attention feature extraction block and a global attention feature extraction block. Among them, the local attention feature extraction block is a feature extraction block using local window attention, and the window size of the local window attention is the preset patch size in the image tampering positioning and detection network model. The global attention feature extraction block is a feature extraction block using global window attention, and the window size of the global window attention is the size of the image to be tested input into the image tampering positioning and detection network model. The number of feature extraction blocks can be an integer multiple of 3, and the way in which the feature extraction blocks are connected in sequence can be three feature extraction blocks as a group, and the three feature extraction blocks are two local attention feature extraction blocks and one global attention feature extraction block in sequence, that is, every three feature extraction blocks are a group, which are two local attention feature extraction blocks and one global attention feature extraction block in sequence.

[0055] For example, take the three convolutional kernels in SRM (Steganalysis Rich Model Filters) as high-pass filters, use the encoder and decoder of the large SAM (Segment Anything Model) image segmentation model, and construct an image tampering localization and detection network model using a single-layer fully connected network. Among them, the convolutional kernel parameters of SRM are as Figure 2 described, and the image tampering localization and detection network model is as Figure 3 described, and the encoder of the large SAM image segmentation model includes 12 feature extraction blocks, and the output of each feature extraction block is added to the output of the corresponding extraction block in the feature fusion encoder.

[0056] Step S200: Use the training dataset to train the image tampering localization and detection network model to obtain a trained image tampering localization and detection network model.

[0057] In this embodiment, the tampered images and their corresponding tampering marked images in the CatNet dataset are used as the training dataset and input into the image tampering localization and detection network model for training, that is, the feature fusion encoder and the fully connected network are pre-trained using the training dataset.

[0058] It can be understood that the image tampering localization and detection network model based on the preset large model network, with the knowledge learned from the large image segmentation model, and the feature fusion encoder branch is trained through the CatNet image tampering localization dataset (a training dataset containing tampered images and their corresponding tampering marked images). That is, the image tampering localization and detection network model performs supervised learning under the training samples, learns the tampering traces left when editing pictures, so that the feature fusion encoder can effectively identify the tampering traces in the image, and thus obtains a network model with the ability to locate pixel-level tampering regions and the ability to distinguish whether a single image has been tampered with.

[0059] The proposed large image segmentation model in this embodiment brings new inspiration to the image segmentation task based on deep learning. The deep learning model trained on a large-scale dataset with large-scale parameters and complex structures can exhibit far stronger robustness and generalization ability than conventional models. In the field of medical image segmentation, the large model has also been applied to downstream tasks of image segmentation through means such as fine-tuning.

[0060] Specifically, in this embodiment, the weights corresponding to three convolution kernels in the pre-loaded preset high-pass filter are frozen; the weights corresponding to the feature extraction blocks of the encoder and decoder in the pre-loaded preset large model network are frozen; the training data set is input into the image forgery localization and detection network model for training to update the weights of the unfrozen regions, obtaining a trained image forgery localization and detection network model; wherein, the weights corresponding to the feature extraction blocks are the weights obtained by training the preset large model network on the image segmentation data set, and the unfrozen regions are the preset feature fusion encoder and the preset fully connected network.

[0061] It can be understood that after the image forgery localization and detection network model is constructed, the weights obtained by training the image segmentation large model on the image segmentation data set are loaded into the encoder and decoder of the preset large model network, that is, the weights of the feature extraction blocks in the large model encoder and decoder, and the weights of the feature extraction blocks are frozen. Among them, in the image forgery localization and detection network model, the region where the weights of the image segmentation large model pre-trained weights are loaded into the encoder and decoder of the preset large model network is the frozen region. Furthermore, the forged images and their corresponding forged marked images in the training data set are used as training samples and input into the image forgery localization and detection network model for training to update the weights of the unfrozen regions, obtaining a network with the ability to locate pixel-level forgery regions and the ability to identify whether a single image has been forged, and using it as a trained forgery localization and detection network.

[0062] In practical applications, in the process of training the image forgery localization and detection network model in this embodiment, PyTorch 1.8.0 can be used to implement the proposed large model-based image forgery localization and detection method of this application, and all experiments are carried out on an NVIDIA V100 GPU (a type of graphics card). Among them, PyTorch is an open-source Python machine learning library. The publicly available forged images and their corresponding forged marked images are used as training samples, and then the training samples are randomly divided into a training set and a validation set according to a ratio of 9:1, and the publicly available data set is used as the test set. There are no duplicate images in the test set and the training set, and the forged images in the test set and the training set may be forged by different forgery tools. The test set includes forged images without using image painting and forged images using image painting. In the training and testing phases, the input image resolution can be scaled to 1024×1024, the Adam optimizer is used, the batch size is 16, the learning rate is set to 1×10-4, the epochs in the training phase can be set to 200, and F1 is used as the evaluation index for the model performance of forgery localization, and accuracy is used as the evaluation index for the model performance of forgery detection.

[0063] Step S300: Input the image to be tested into the trained image forgery localization and detection network model, and obtain the localization output of the trained image forgery localization and detection network model to identify and locate the forgery area of the image to be tested, so as to obtain the forgery area localization result.

[0064] In this embodiment, after training the image forgery localization and detection network model with the training data set to obtain the trained image forgery localization and detection network model, the image to be tested is divided into three categories, namely: 1. Natural scene forgery image; 2. Book cover forgery image; 3. Natural scene forgery image after post-processing operations (i.e., adding Gaussian noise). Input the image to be tested into the trained image forgery localization and detection network model, and obtain the localization output to identify and locate the forgery area of the image to be tested, so as to obtain the forgery area localization result.

[0065] Step S400: Input the image to be tested into the trained image forgery localization and detection network model, and obtain the detection output of the trained image forgery localization and detection network model to determine whether the image to be tested has undergone forgery operations, so as to obtain the detection result reflecting whether the image to be tested has been forged.

[0066] In this embodiment, after training the image forgery localization and detection network model with the training data set to obtain the trained image forgery localization and detection network model. Input the image to be tested into the trained image forgery localization and detection network model, and obtain the detection output to determine whether the image to be tested has undergone forgery operations, so as to obtain the detection result of whether the image to be tested is a forged image, thus realizing the accurate localization of forged images and original images.

[0067] It can be seen that in the embodiment of the present invention, by using the forged images and their corresponding forged mark images in the training data set as training samples, and then using the training samples to train the image forgery localization and detection network model constructed based on a preset high-pass filter, a preset feature fusion encoder, a preset large model network, and a preset fully connected network, a network model that can accurately locate the forgery area on the forged image and accurately determine whether a single image has undergone forgery operations can be obtained. That is, the trained image forgery localization network model can realize the accurate localization and detection of the editing behavior on the forged image. With the generalization and robustness of the preset large model network, the generalization and robustness of the image forgery localization and detection network model for forged images can be greatly improved. That is, the trained image forgery localization and detection network model shows excellent performance on forged images in different scenarios and after different post-processing operations, and plays an important role in improving the localization and detection performance of the image forgery localization and detection network model for forged images and overcoming the weak robustness and generalization in the current research field of image forgery localization and detection.

[0068] By comparing the method for image forgery localization and detection based on large models proposed in this embodiment with the current forgery localization and detection methods, it can be verified that the method proposed by the present invention has better performance than the current forgery localization methods. In this embodiment, the deep learning models TruFor and CatNetv2, which perform excellently in the field of forensics, are adopted, and the forgery image datasets Coverage and Cocoglide, which are of the same natural scene as the CatNet dataset, are used for testing. The forgery localization test results are shown in Table 1, and the forgery detection test results are shown in Table 2.

[0069] Table 1

[0070] Model Coverage Cocoglide CatNetv2 0.381 0.434 TroFor 0.600 0.523 The present invention 0.627 0.537

[0071] Table 2

[0072] Model Coverage Cocoglide CatNetv2 0.635 0.580 TroFor 0.680 0.639 The present invention 0.720 0.943

[0073] As can be seen from Table 1 and Table 2, the method proposed by the present invention shows more excellent localization and detection performance for forged images compared with the current excellent deep learning models TruFor and CatNetv2. The localization results of some samples are compared as Figure 4 shown. In this embodiment, the text image dataset PS-Boundary is used for testing, and the forgery localization test results are shown in Table 3.

[0074] Table 3

[0075] Model F1 CatNetv2 0.433 TroFor 0.473 The present invention 0.506

[0076] As can be seen from Table 3 above, the method proposed by the present invention shows more excellent generalization performance for text forged images compared with the current excellent deep learning models TruFor and CatNetv2. The localization results of some samples are compared as Figure 5 shown.

[0077] In this embodiment, the deep learning model TruFor, which performs excellently in the field of forensics, is adopted. Gaussian noise is added to the images in the forgery image dataset Coverage. The mean of the added Gaussian noise is 0, and the variances are 0.1, 0.3, and 0.5 respectively. Then the test is carried out, and the forgery localization test results are shown in Table 4.

[0078] Table 4

[0079] Model 0.1 0.3 0.5 TroFor 0.357 0.331 0.131 The present invention 0.51 0.509 0.368

[0080] As can be seen from Table 4 above, the method proposed by the present invention shows less performance degradation compared with the current excellent deep learning model TruFor, and shows more excellent localization robustness performance for forged images. The localization results of some samples are compared as Figure 6 shown.

[0081] Based on the above embodiments, the present invention further provides an image tampering localization and detection device based on a large model, as Figure 7 shown in. The device includes: a model construction module 10, a model training module 20, a tampering localization module 30, and an image detection module 40. Specifically, the model construction module 10 is used to construct an image tampering localization and detection network model based on a preset high-pass filter, a preset feature fusion encoder, a preset large model network, and a preset fully connected network. The image tampering localization and detection network model includes a localization output and a detection output. The model training module 20 is used to train the image tampering localization and detection network model using a training data set to obtain a trained image tampering localization and detection network model. The tampering localization module 30 is used to input a to-be-detected image into the trained image tampering localization and detection network model, obtain the localization output of the trained image tampering localization and detection network model, and identify and locate the tampered area of the to-be-detected image to obtain a tampered area localization result. The image detection module 40 is used to input a to-be-detected image into the trained image tampering localization and detection network model, obtain the detection output of the trained image tampering localization and detection network model, and determine whether the to-be-detected image has been tampered with to obtain a detection result reflecting whether the to-be-detected image has been tampered with.

[0082] In the image tampering localization and detection device based on a large model in this embodiment, the working principles of each module are the same as those of each step in the above method embodiment, and will not be elaborated here.

[0083] Each module in the above image tampering localization and detection system based on a large model can be implemented in whole or in part by software, hardware, and their combinations. The above modules can be embedded in the processor in the terminal in hardware form or be independent of the processor, or can be stored in the memory in the terminal in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0084] Based on the above embodiments, the present invention further provides a terminal. The principle block diagram of the terminal can be as Figure 8 shown in. The terminal may include one or more processors 100( Figure 8 only one is shown in), a memory 101, and a computer program 102 stored in the memory 101 and executable on one or more processors 100.

[0085] In one embodiment, the so-called processor 100 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0086] In one embodiment, the memory 101 may be an internal storage unit of the electronic device, such as the hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 101 may also include both the internal storage unit and the external storage device of the electronic device. The memory 101 is used to store computer programs and other programs and data required by the terminal. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0087] Those skilled in the art can understand that Figure 8 the principle block diagram shown in

[0088] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, operational database, or other medium used in the embodiments provided by the present invention can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or equivalently replace some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image forgery localization and detection method based on a large model, characterized in that, The method includes: Constructing an image tampering localization and detection network model based on a preset high-pass filter, a preset feature fusion encoder, a preset large model network, and a preset fully connected network, where the image tampering localization and detection network model includes a localization output and a detection output; Training the image tampering localization and detection network model using a training data set to obtain a trained image tampering localization and detection network model; Inputting the image to be tested into the trained image tampering localization and detection network model, and obtaining the localization output of the trained image tampering localization and detection network model to identify and locate the tampered area of the image to be tested, so as to obtain a tampered area localization result; Inputting the image to be tested into the trained image tampering localization and detection network model, and obtaining the detection output of the trained image tampering localization and detection network model to determine whether the image to be tested has been tampered, so as to obtain a detection result reflecting whether the image to be tested has been tampered; 2. The method for locating and detecting image tampering based on a large model according to claim 1, wherein The preset high-pass filter includes multiple convolutional kernels, the size of each convolutional kernel is the same, the size of each convolutional kernel is 5×5 pixels, and each convolutional kernel is used to perform different spatial frequency response processing on the input image; 3. The method for image forgery localization and detection based on large models according to claim 1 or 2, characterized in that The preset large model network is a network that combines an encoder and a decoder and can perform image segmentation tasks. The preset feature fusion encoder is an encoder pre-trained in the network that combines an encoder and a decoder and can perform image segmentation tasks. The preset high-pass filter is connected in series with the feature fusion encoder, and the preset feature fusion encoder is connected to the encoder in the preset large model network for extracting target features; 4. The method for image forgery localization and detection based on a large model according to claim 3, characterized in that, The preset feature fusion encoder includes multiple sequentially connected feature extraction blocks, each of the feature extraction blocks is connected in parallel with the corresponding feature extraction block inside the encoder in the preset large model network, and the feature extraction block is a transformer block that supports window attention and residual connection; Moreover, the feature extraction block includes a local attention feature extraction block and a global attention feature extraction block; Among them, the local attention feature extraction block is a feature extraction block using local window attention, and the window size of the local window attention is the preset patch size in the image tampering localization and detection network model; The global attention feature extraction block is a feature extraction block using global window attention, and the window size of the global window attention is the size of the image to be tested input into the image tampering localization and detection network model; 5. The method for image forgery localization and detection based on large models according to claim 4, characterized in that The number of the feature extraction blocks is an integer multiple of three, and the sequential connection mode of the feature extraction blocks is that three of the feature extraction blocks form a group, and the three feature extraction blocks are sequentially two local attention feature extraction blocks and one global attention feature extraction block.

6. The method for image forgery localization and detection based on large models according to any one of claims 1-5, characterized in that, The preset fully-connected network includes a flattening layer and one or more fully-connected layers. The flattening layer is connected in series with the fully-connected layers. The flattening layer takes multi-dimensional data as input and outputs one-dimensional data. The fully-connected layer consists of multiple neurons. If there are multiple fully-connected layers, each neuron is connected to all neurons in the previous layer, forming a dense connection structure. The preset fully-connected network is connected in series with the preset feature fusion encoder. The positioning output is the output of the decoder of the preset large model network, and the detection output is the output of the preset fully-connected network.

7. The method for image forgery localization and detection based on a large model according to any one of claims 1-6, characterized in that, Training the image tampering localization and detection network model using the training data set to obtain a trained image tampering localization and detection network model includes: Freezing the weights corresponding to three convolutional kernels in the pre-loaded preset high-pass filter; Freezing the weights corresponding to the feature extraction blocks of the encoder and decoder in the pre-loaded preset large model network; Inputting the training data set into the image tampering localization and detection network model for training to update the weights in the unfrozen regions, obtaining a trained image tampering localization and detection network model; Wherein, the weights corresponding to the feature extraction blocks are the weights obtained by training the preset large model network on the image segmentation data set, and the unfrozen regions are the preset feature fusion encoder and the preset fully-connected network.

8. An image forgery localization and detection device based on a large model, characterized in that, The device includes: A model construction module for constructing an image tampering localization and detection network model based on a preset high-pass filter, a preset feature fusion encoder, a preset large model network, and a preset fully-connected network. The image tampering localization and detection network model includes a positioning output and a detection output; A model training module for training the image tampering localization and detection network model using the training data set to obtain a trained image tampering localization and detection network model; A tampering localization module for inputting the image to be tested into the trained image tampering localization and detection network model, obtaining the positioning output of the trained image tampering localization and detection network model, to identify and locate the tampered area of the image to be tested, obtaining a tampered area localization result; An image detection module for inputting the image to be tested into the trained image tampering localization and detection network model, obtaining the detection output of the trained image tampering localization and detection network model, to determine whether the image to be tested has undergone a tampering operation, obtaining a detection result for reflecting whether the image to be tested has been tampered.

9. A terminal, characterized in that, The terminal includes a memory, a processor, and an image tampering localization and detection program based on a large model stored in the memory and executable on the processor. When the processor executes the image tampering localization and detection program based on the large model, the steps of the image tampering localization and detection method according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, An image tampering localization and detection program based on a large model is stored on the computer-readable storage medium. When the image tampering localization and detection program based on the large model is executed by the processor, the steps of the image tampering localization and detection method according to any one of claims 1-7 are implemented.

Images