Image forgery detection method and system based on large model, terminal and storage medium

Abstract

The invention discloses an image forgery detection method and system based on a large model, a terminal and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining a to-be-detected image; obtaining an authenticity category judgment result corresponding to the to-be-detected image, an explanation text for the authenticity category judgment result and forged area indication information corresponding to the to-be-detected image through a trained multi-modal detection large model; according to the to-be-detected image and the forged area indication information, determining a pixel-level forged area positioning mask corresponding to the to-be-detected image through a trained pixel-level segmentation model; and generating a counterfeit detection result corresponding to the to-be-detected image according to the authenticity category judgment result, the explanation text and the pixel-level counterfeit region positioning mask. Thus, the generated counterfeit detection result has interpretability, pixel-level accurate indication for the counterfeit area in the image can be realized, and the image counterfeit detection effect can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a large model-based image forgery detection method and system, a terminal and a storage medium. BACKGROUND

[0002] With the rapid development of generative artificial intelligence technology and new image local editing technologies such as diffusion models, digital image authenticity identification is facing unprecedented challenges. Although existing image authenticity detection tools have certain explainability and positioning ability in traditional forgery image detection, there are still key technical bottlenecks when facing AI controllable editing images. For example, the existing image authenticity detection tools usually output a true or false judgment result for the whole image, directly indicating that the image is a real image or a fake image, which has weak explainability and cannot accurately indicate the forged area in the image, which is not conducive to improving the image forgery detection effect.

[0003] Therefore, the related art still needs to be improved and developed. SUMMARY

[0004] The main purpose of the present application is to provide a large model-based image forgery detection method and system, a terminal and a storage medium, which aims to solve the technical problem that the related art usually outputs a true or false judgment result for the whole image, which has weak explainability and cannot accurately indicate the forged area in the image, thus not conducive to improving the image forgery detection effect.

[0005] In order to achieve the above purpose, the first aspect of the present application provides a large model-based image forgery detection method, wherein the large model-based image forgery detection method comprises:

[0006] obtaining a to-be-detected image;

[0007] obtaining, by a trained multi-modal detection large model, a true or false category judgment result corresponding to the to-be-detected image, an explanation text corresponding to the true or false category judgment result, and forged area indication information corresponding to the to-be-detected image;

[0008] determining, by a trained pixel-level segmentation model, a pixel-level forged area positioning mask corresponding to the to-be-detected image according to the to-be-detected image and the forged area indication information;

[0009] generating a forgery detection result corresponding to the to-be-detected image according to the true or false category judgment result, the explanation text and the pixel-level forged area positioning mask.

[0010] Optionally, the forged area indication information comprises a bounding box feature and a forgery trace feature.

[0011] The bounding box feature is used to indicate the boundary position of the rectangular forged region in the to-be-detected image.

[0012] The forged trace feature is information of a corresponding position of a hidden layer of the multi-modal detection large model.

[0013] Optionally, the pixel-level forged region positioning mask corresponding to the to-be-detected image is determined according to the to-be-detected image and the forged region indication information by using a trained pixel-level segmentation model.

[0014] According to the bounding box feature and the forged trace feature, sparse features for representing boundary box coordinates and dense features for representing forged trace information are obtained by using a prompt word encoder of the pixel-level segmentation model.

[0015] The pixel-level forged region positioning mask corresponding to the to-be-detected image is determined according to the sparse features, the dense features, and the to-be-detected image by using the pixel-level segmentation model.

[0016] Optionally, the pixel-level forged region positioning mask corresponding to the to-be-detected image is determined according to the sparse features, the dense features, and the to-be-detected image by using the pixel-level segmentation model, and the method comprises the following steps of:

[0017] The bounding box feature and the forged trace feature are subjected to linear layer mapping processing.

[0018] The mapped bounding box feature and the sparse features are spliced to obtain first spliced features, and the mapped forged trace feature and the dense features are spliced to obtain second spliced features.

[0019] The first spliced features and the second spliced features are input into a decoder of the pixel-level segmentation model to obtain the pixel-level forged region positioning mask.

[0020] Optionally, the resolution of the pixel-level forged region positioning mask is the same as the resolution of the to-be-detected image.

[0021] Each pixel value in the pixel-level forged region positioning mask is between 0 and 1, and the pixel value is used to represent the editing intensity of a corresponding pixel point in the to-be-detected image.

[0022] Optionally, the training data corresponding to the multi-modal detection large model comprises a forged image.

[0023] The forged image is obtained by editing a real image based on an image editing instruction by using an image editor.

[0024] The image editing instruction is generated by a preset large language model based on prompt information input by a target object.

[0025] Optionally, the generating, according to the authenticity category judgment result, the explanation text, and the pixel-level fake region positioning mask, of a fake detection result corresponding to the to-be-detected image comprises:

[0026] Obtaining a target output data format;

[0027] Based on the target output data format, the authenticity category judgment result, the explanation text, and the pixel-level fake region positioning mask, a fake detection result corresponding to the to-be-detected image is generated and outputted.

[0028] The second aspect of the present application provides a large model-based image fake detection system, wherein the large model-based image fake detection system comprises:

[0029] A data acquisition module is configured to acquire a to-be-detected image;

[0030] A first processing module is configured to acquire, through a trained multi-modal detection large model, an authenticity category judgment result corresponding to the to-be-detected image, an explanation text corresponding to the authenticity category judgment result, and fake region indication information corresponding to the to-be-detected image;

[0031] A second processing module is configured to determine, through a trained pixel-level segmentation model, a pixel-level fake region positioning mask corresponding to the to-be-detected image according to the to-be-detected image and the fake region indication information;

[0032] A result generation module is configured to generate, according to the authenticity category judgment result, the explanation text, and the pixel-level fake region positioning mask, a fake detection result corresponding to the to-be-detected image.

[0033] The third aspect of the present application provides a terminal, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of any one of the large model-based image fake detection methods.

[0034] The fourth aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the steps of any one of the large model-based image fake detection methods.

[0035] As can be seen from the above, in the scheme, an image to be detected is acquired; a true or false category judgment result corresponding to the image to be detected is acquired through a trained multi-modal detection large model, an explanation text for the true or false category judgment result, and a fake area indication information corresponding to the image to be detected; a pixel-level fake area positioning mask corresponding to the image to be detected is determined through a trained pixel-level segmentation model according to the image to be detected and the fake area indication information; and a fake detection result corresponding to the image to be detected is generated according to the true or false category judgment result, the explanation text, and the pixel-level fake area positioning mask.

[0036] Compared with the prior art, in the image fake detection method based on a large model provided in the application, the image to be detected is processed in combination with the trained multi-modal detection large model and the trained pixel-level segmentation model. Specifically, the multi-modal detection large model can output a true or false category judgment result, an explanation text, and fake area indication information. Further, a more accurate pixel-level fake area positioning mask is generated through the pixel-level segmentation model in combination with the fake area indication information, so as to realize pixel-level accurate indication of the fake area. Finally, the fake detection result is generated based on the true or false category judgment result, the explanation text, and the pixel-level fake area positioning mask. Compared with the prior art, the fake detection result generated in the application has explainability, and can realize pixel-level accurate indication of the fake area in the image, which is beneficial to improving the image fake detection effect. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Figure 1 is a flowchart of an image fake detection method based on a large model provided by an embodiment of the application;

[0039] Figure 2 is a structure diagram of the flow scheme provided by an embodiment of the application;

[0040] Figure 3 is a specific process diagram of automatic construction of training data provided by an embodiment of the application;

[0041] Figure 4 is a specific training process diagram of a pixel-level segmentation model provided by an embodiment of the application;

[0042] Figure 5is a specific training process schematic diagram of a multi-modal detection large model provided by an embodiment of the present application;

[0043] Figure 6 is a component module schematic diagram of an image forgery detection system based on a large model provided by an embodiment of the present application;

[0044] Figure 7 is an internal structure principle block diagram of a terminal provided by an embodiment of the present application. DETAILED DESCRIPTION

[0045] In the following description, for the purposes of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the application. However, it will be apparent to those skilled in the art that the application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known devices, circuits, and methods are omitted so as not to obscure the description of the application with unnecessary detail.

[0046] It should be understood that the term "comprises" when used in this specification and accompanying claims, indicates the presence of the stated features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0047] It should also be understood that the terms used in the specification and the appended claims are intended to describe particular embodiments and do not intend to limit the present application. As used in the specification and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0048] It should further be understood that the term "and / or" as used in the specification and the appended claims, means any one or more of the associated listed items, as well as all possible combinations of the items.

[0049] As used in the specification and the appended claims, the term "if' can be construed to mean "when" or "once," or "in response to a determination" or "in response to a classification," depending on the context. Similarly, the phrase "if it is determined" or "if it is classified" can be construed to mean "once it is determined" or "in response to the determination," or "once it is classified" or "in response to the classification," depending on the context.

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0051] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0052] With the development of generative artificial intelligence technology, especially novel image local editing techniques that combine multimodal large models and diffusion models, the field of digital image authentication faces severe challenges. This type of technology allows users to modify specific areas of an image using natural language commands, such as "removing people from the image" or "changing a daytime scene to nighttime." The results generated by this AI local editing technology are visually highly realistic; the edited area blends well with the original background in terms of lighting, texture, noise, and other underlying features, making the forged result visually highly realistic and difficult for the human eye to distinguish.

[0053] A deeper challenge lies in the fact that this technology fundamentally alters the underlying model of digital tampering. Although the editing intent is semantically local, the generation process of the diffusion model often leads to slight but widespread perturbations in the pixel values ​​of the entire image. This characteristic directly challenges the core assumption of traditional forgery detection methods—the existence of a clearly defined, segmentable tampered region. Therefore, simple pixel difference calculations can no longer reliably generate annotations for the edited region, and traditional binarization masks are insufficient to accurately define the diffuse effects produced by this new type of editing technology. This necessitates a shift in identification methods from "locating clear boundaries" to "identifying local semantic changes under global perturbations," placing entirely new demands on digital image detection technology.

[0054] Detection methods based on underlying physical traces use deep learning models to analyze tampering traces introduced by image editing operations. These traces include, but are not limited to, block artifacts introduced by compression algorithms, differences in underlying noise, and edge artifacts. However, when faced with forged content generated by deep generation techniques such as diffusion models, the local inconsistencies relied upon by these traditional methods may become insignificant because these techniques can learn and reconstruct local textures and noise patterns highly similar to the surrounding environment during generation. This leads to a sharp decline in detection performance and a large number of false negatives.

[0055] Detection methods based on high-level forgery traces attempt to use multimodal large-scale model analysis to discover inconsistencies at the semantic level of forgery. For example, they might detect whether the direction of an object's shadow contradicts the position of the light source, or determine whether the appearance of an object conforms to scene logic. However, when the forged content itself possesses a high degree of logical plausibility, the detection basis of this method weakens significantly. Furthermore, such methods still face risks of insufficient accuracy in logical judgments, misjudgments, or the generation of inaccurate logical connections; therefore, the reliability of their detection conclusions needs further improvement.

[0056] Therefore, despite the performance improvements in current image authenticity detection technologies, numerous challenges remain for novel AI-based image local editing detection techniques. Existing technologies exhibit limitations when dealing with the authenticity and location of novel forged images characterized by highly realistic low-level features and highly consistent high-level semantics. Analysis methods based on low-level physical traces struggle to detect traces smoothed out by advanced generative models, while high-level semantic analysis methods fail due to the logical self-consistency of the forged content. Thus, existing technologies face the following core problems when dealing with such novel forged images:

[0057] (1) The diffuse forgery effect leads to inaccurate localization: Current local image editing instructions are semantically only targeted at specific regions, but the generation model will cause slight perturbations of pixels throughout the entire image in order to ensure visual consistency. The proposed pixel difference presents noise throughout the entire image, making it difficult to generate a clear and effective localization mask.

[0058] (2) Lack of high-quality and diverse training data: Currently, there is a lack of training datasets that can systematically simulate new forgery techniques, have accurate pixel-level annotations, cover diverse adversarial scenarios, and can be automatically acquired and generated. This greatly limits the training effect, robustness, and adaptability of existing detection models to the evolution of image detection forgery techniques.

[0059] (3) Lack of effective collaborative mechanism in detection framework: Existing technologies lack a unified detection framework that can deeply collaborate between low-level visual artifact analysis and high-level semantic contradiction judgment. Most current methods only apply or superficially fuse the two approaches, resulting in insufficient accuracy and generalization performance when facing new and diverse image forgery techniques.

[0060] To address at least one of the aforementioned technical problems, this application's solution utilizes a fully automated generation pipeline to acquire a large number of high-quality AI-edited images; trains a multimodal forgery detection and localization model using these high-quality images; acquires an image to be detected and, through the trained multimodal detection model, obtains the authenticity category judgment result corresponding to the image to be detected, an explanatory text for the authenticity category judgment result, and forgery region indication information corresponding to the image to be detected; based on the image to be detected and the forgery region indication information, determines the pixel-level forgery region localization mask corresponding to the image to be detected using a trained pixel-level segmentation model; and generates a forgery detection result corresponding to the image to be detected based on the authenticity category judgment result, the explanatory text, and the pixel-level forgery region localization mask.

[0061] Compared with existing technologies, the image forgery detection method based on a large model provided in this application processes the image to be detected by combining a trained multimodal detection large model and a trained pixel-level segmentation model. Specifically, the multimodal detection large model can output the authenticity category judgment result, explanatory text, and forgery region indication information. Furthermore, the forgery region indication information is combined with the pixel-level segmentation model to generate a more accurate pixel-level forgery region localization mask, thereby achieving pixel-level precise indication of the forgery region. The final forgery detection result is generated based on the authenticity category judgment result, explanatory text, and pixel-level forgery region localization mask. Compared with existing technologies, the forgery detection result generated in this application is interpretable and can achieve pixel-level precise indication of forgery regions in the image, which is beneficial to improving the image forgery detection effect.

[0062] like Figure 1 As shown in the embodiments of this application, an image forgery detection method based on a large model is provided. Specifically, the method includes the following steps:

[0063] Step S100: Obtain the image to be detected;

[0064] Step S200: Using the trained multimodal detection model, obtain the authenticity category judgment result of the above-mentioned image to be detected, the explanatory text of the authenticity category judgment result, and the forgery region indication information of the above-mentioned image to be detected.

[0065] Step S300: Based on the image to be detected and the forged region indication information, determine the pixel-level forged region localization mask corresponding to the image to be detected using the trained pixel-level segmentation model.

[0066] Step S400: Based on the above-mentioned authenticity category judgment result, the above-mentioned explanatory text, and the above-mentioned pixel-level forgery region location mask, generate the forgery detection result corresponding to the above-mentioned image to be detected.

[0067] The images to be detected are those that need to be forged to determine whether they have been altered or forged. It should be noted that these images may have been edited by AI. The explanatory text above explains the results of the authenticity classification of the images to be detected. It should be noted that the explanatory text is in natural language; for example, in one application scenario, if the authenticity classification result is "fake image," the corresponding explanatory text would be "The image contains obvious visual distortion and color distortion."

[0068] Therefore, in this application, the image to be detected is processed by combining a trained multimodal detection model and a trained pixel-level segmentation model. Specifically, the multimodal detection model can output a true / false category judgment result, explanatory text, and forgery region indication information. Furthermore, the forgery region indication information is combined with the pixel-level segmentation model to generate a more accurate pixel-level forgery region localization mask, achieving pixel-level precise indication of the forgery region. The final forgery detection result is generated based on the true / false category judgment result, explanatory text, and pixel-level forgery region localization mask. Compared with existing technologies, the forgery detection result generated in this application is interpretable and can achieve pixel-level precise indication of forgery regions in the image, which is beneficial to improving the image forgery detection effect.

[0069] Specifically, the aforementioned forged area indication information includes bounding box features and forgery trace features;

[0070] The aforementioned bounding box features are used to indicate the boundary positions of the rectangular forged regions in the image to be detected.

[0071] The forgery traces mentioned above are information about the corresponding locations in the hidden layers of the large multimodal detection model.

[0072] Specifically, the aforementioned forgery traces are the forgery traces corresponding to the aforementioned rectangular forgery area used to distinguish authenticity.

[0073] It should be noted that the forged region indicated by the aforementioned bounding box features is an approximate rectangular area, not the exact boundary of the forged region. Therefore, in this embodiment, the forged region is further precisely segmented based on a pixel-level segmentation model to improve the accuracy of the forged region indication.

[0074] Specifically, based on the image to be detected and the forged region indication information, the pixel-level forged region localization mask corresponding to the image to be detected is determined using a trained pixel-level segmentation model, including:

[0075] Based on the bounding box features and forgery trace features mentioned above, the prompt word encoder of the pixel-level segmentation model is used for encoding to obtain sparse features for representing bounding box coordinates and dense features for representing forgery trace information.

[0076] Based on the above sparse features, the above dense features, and the above image to be detected, the pixel-level forgery region localization mask corresponding to the above image to be detected is determined by the above pixel-level segmentation model.

[0077] The process of determining the pixel-level forgery region localization mask corresponding to the image to be detected based on the sparse features, dense features, and the image to be detected using the pixel-level segmentation model includes:

[0078] Linear layer mapping is applied to the bounding box features and forgery trace features described above.

[0079] The mapped bounding box features are concatenated with the above sparse features to obtain the first concatenated feature, and the mapped forgery trace features are concatenated with the above dense features to obtain the second concatenated feature.

[0080] The first and second splicing features are input into the decoder of the pixel-level segmentation model to obtain the pixel-level fake region localization mask.

[0081] It should be further explained that after the image is processed by the multimodal detection model, forgery trace features, bounding box features, forgery discrimination features, and forgery explanation features are obtained. The forgery features are input into the linear layer of the model, outputting binary labels for true / false classification (i.e., the true / false category judgment result). The forgery explanation features are input into the logits layer of the model, outputting human-readable explanatory text explaining the forgery traces (i.e., the explanation text). The pixel-level forgery region localization mask is obtained through a pixel-level segmentation model. Finally, based on the true / false category judgment result, the explanation text, and the pixel-level forgery region localization mask, the forgery detection result corresponding to the image to be detected is generated. Specifically, the resolution of the aforementioned pixel-level forgery region localization mask is the same as the resolution of the image to be detected.

[0082] Each pixel value in the aforementioned pixel-level fake region location mask is between 0 and 1, and these pixel values ​​are used to characterize the editing intensity of the corresponding pixel in the aforementioned image to be detected.

[0083] Thus, in this embodiment, indicating the forged region based on a pixel-level forged region location mask that matches the actual boundary of the forged region can achieve more accurate detection results. The output mask of the pixel-level segmentation model is a pixel-by-pixel prediction result, and its boundary is completely determined by the actual shape of the forged region, which may be any complex contour. For example, if the forged region is a cloud, the mask will precisely fit the shape of the cloud. Moreover, the pixel value of the pixel-level forged region location mask corresponds to the value range [0, 1]. The pixel value is continuous within the value range, rather than a simple binary classification, which can achieve a better description of the forged region. This pixel value can represent the editing intensity corresponding to the pixel point, and can also indicate the probability that the pixel point is forged. For example, a pixel value of 0 means that the editing intensity corresponding to the pixel point is 0, and therefore the corresponding forgery probability is 0.

[0084] It should be noted that the training data for the aforementioned large-scale multimodal detection model includes forged images;

[0085] The aforementioned forged images were obtained by editing real images using an image editor based on image editing commands;

[0086] The aforementioned image editing instructions are generated by a preset large language model based on the prompts input from the target object.

[0087] Thus, in this embodiment of the application, a method for automatically generating training data is provided for the training data required for model training. This can solve the problem of the lack of high-quality and diverse training data in the prior art, which is conducive to improving the model training effect and thus improving the detection effect of the trained model.

[0088] It should be noted that the fake images used during the training of the pixel-level segmentation model can be generated in the same way as the fake images included in the training data of the large multimodal detection model, which will not be elaborated here.

[0089] Furthermore, based on the aforementioned authenticity category judgment results, the aforementioned explanatory text, and the aforementioned pixel-level forgery region localization mask, the forgery detection result corresponding to the aforementioned image to be detected is generated, including:

[0090] Obtain the target output data format;

[0091] Based on the above target output data format, according to the above authenticity category judgment results, the above explanatory text, and the above pixel-level forgery region location mask, the forgery detection result corresponding to the above image to be detected is generated and output.

[0092] In this embodiment, the above-mentioned image forgery detection method based on a large model is further described in detail based on a specific application scenario. It should be noted that this embodiment specifically describes the process from the generation of model training data, model training, to the use of the model.

[0093] Figure 2 This is a flowchart illustrating the process structure provided in the embodiments of this application. Figure 3 This is a schematic diagram illustrating a specific process for automatically constructing training data according to an embodiment of this application, such as... Figure 2 and Figure 3 As shown in the embodiment of this application, training data is first automatically generated. For details, please refer to... Figure 2 and Figure 3 The process shown generates image editing instructions for the original image based on a preset large language model; based on the generated image editing instructions, the original image is edited using an extensible image editor to automatically generate an edited image; based on the original image and the edited image, a quantitative forgery mask image of the edited area and editing intensity is generated.

[0094] The specific method for generating a quantitative forgery mask image is as follows:

[0095] A first difference map is calculated between the original image and the generated forged image (i.e., the edited image), which represents the changes in the underlying physical features between the two; a second difference map is calculated between the original image and the forged image, which quantifies the differences in the high-level semantic features between the two; and a quantitative forgery mask is generated by fusing the first difference map and the second difference map.

[0096] The aforementioned quantitative forgery mask serves as a supervisory signal for subsequent model training. Furthermore, coarse-grained semantic labels (including bounding boxes, true / false labels, and interpretable text describing the differences between true and false labels) are derived from the aforementioned quantitative forgery mask to support the separate training of large multimodal detection models. The aforementioned original images are not only used in this data generation process but also serve as positive class labels during subsequent model training.

[0097] The aforementioned large language model is an existing closed-source model (GPT-4o) used to generate diverse image editing instructions. The first difference map was obtained by calculating the pixel-level differences between the original image and the forged image, both within the same perceptually uniform color space. The perceptually uniform color space used is that of the International Commission on Illumination (CIE). The LAB color space. The second difference map mentioned above is obtained by comparing the deep features of the original image and the forged image using a deep perceptual similarity model. The deep perceptual similarity model mentioned above is an existing model, such as the Learned Perceptual Image Patch Similarity (LPIPS) model.

[0098] Furthermore, based on the automatically generated data mentioned above, a corresponding training dataset is constructed to train the model, specifically training the pixel-level segmentation model and the multimodal detection model. The training dataset includes the aforementioned forged image, the aforementioned quantitative forgery mask, and coarse-grained semantic labels derived from the mask. The forged image serves as the training input, the aforementioned quantitative forgery mask serves as the supervision label for the pixel-level segmentation model, and the aforementioned coarse-grained semantic labels serve as the supervision label for the multimodal detection model.

[0099] Specifically, in this embodiment, a pre-defined large language model is used to automatically generate diverse and complex image editing instructions based on a pre-defined theme or rules. Specifically, the large language model receives an input prompt template, such as "Generate a local editing instruction for [image theme], including color transformation, object generation or replacement, with an editing area size of [small / medium / large], ensuring instruction diversity and coverage of adversarial scenarios." During the generation process, the model outputs a diverse set of instructions, such as "Add a red hat to the car in the center of the image" or "Change the background sky from blue to sunset orange." In one embodiment, the large language model is a GPT-4o model, and the generated editing instructions include editing guidance and target editing areas. The number of instructions is set to 10 per original image to achieve diverse editing coverage.

[0100] Subsequently, the aforementioned image editing instructions drive one or more controllable image editors to perform batch editing operations on the input set of real images to generate fake images. The set of real images can be derived from the publicly available dataset MSCOCO. The controllable image editor implements batch editing based on a Generative Adversarial Network (GAN) or a diffusion model, such as a variant of the Stable Diffusion model. The input is the original image and editing instructions, and the output is the edited fake image. In one embodiment, the controllable image editor is the GPT-4o-image tool, with editing parameters including a guidance scale (7.5) and steps (50) to ensure high fidelity. During batch processing, multiple images can be processed in parallel to generate a dataset of locally edited images containing different editing methods (e.g., object addition, deletion, transformation, etc.), edited content, and edited objects.

[0101] Quantitative Forgery Mask Generation: By capturing editing traces from two dimensions, the differences between the original image and the forged image are analyzed to construct a quantitative forgery mask. Specifically, this includes: obtaining a first difference map (bottom-level pixel differences); converting the original image and its corresponding forged image to a perceptually uniform color space. In one embodiment, this color space is the Lab color space defined by the International Commission on Illumination (ICI) (the L channel represents luminance, and the a / b channels represent color contrast). After conversion, the pixel-level absolute difference between the two images in the aforementioned color space is calculated to quantify the bottom-level pixel changes, highlighting the diffuse forgery effect caused by the generative model, such as slight pixel perturbations throughout the image. The final first difference map is generated as shown in the following formula:

[0102] P(x,y)=|Lab(Original (x,y) )-Lab(Forged (x,y) )|;

[0103] Where P(x,y) represents the pixel-level difference, Original (x,y) Forged represents the pixel value at the corresponding coordinate point in the original image. (x,y) This represents the pixel value at the corresponding coordinate point in the edited image, and Lab represents the color space conversion method.

[0104] Obtain the second difference map (high-level semantic difference). Using a pre-trained deep-aware similarity model, extract deep features from the original and forged images respectively. In one embodiment, the model is a learned perceptual image patch similarity model (LPIPS) based on a VGG-16 backbone network, extracting multi-layer features. Quantify the high-level semantic difference, such as object consistency or texture inconsistency, by calculating the cosine distance or L2 norm difference between the two deep features, obtaining the second difference map as shown in the following formula:

[0105] L(x,y)=LPIPS(Original (x,y) Forged (x,y) );

[0106] Here, L(x,y) represents the semantic difference, and LPIPS represents the perceptual image patch similarity model.

[0107] Difference Fusion and Normalization: To ensure that the final generated image considers both differences from the first and second difference maps mentioned above, we use the following formula for weighted fusion to obtain the fused difference map:

[0108] f(P(x,y),L(x,y))=α·P(x,y)+(1-α)L(x,y);

[0109] Here, parameter α refers to the model weights, and P(x,y) and L(x,y) represent the pixel differences and semantic differences between the real and edited images, respectively. The formulas are shown below:

[0110]

[0111] Where k represents the "steepness" of the control curve, and k is set to 10. d0 represents the threshold and offset, and is set to 0.5.

[0112] To ensure that the fused difference map conforms to the physical meaning of the detected differences in the image and to avoid noise interference, we consider performing intensity fusion only on areas where pixel differences exist. The final quantitative forgery mask is generated as shown in the following formula:

[0113]

[0114] Among them, T p This represents the threshold for the difference in pixel values. To avoid slight perturbation errors, the threshold is set to 5 in this embodiment. T(x,y) represents the target editing area that guides the image editor. The generated mask can be represented as a heatmap or grayscale image and stored in PNG format with the same resolution as the original image.

[0115] Multimodal data construction: The real images, synthetic forged images, generated quantitative forgery masks, and text descriptions of the differences in image editing (which can be generated by the large language model or another specialized model) mentioned above are combined into a complete multimodal data unit. Multiple of these data units constitute a complete training dataset for training the subsequent detection model.

[0116] Furthermore, the aforementioned generated dataset is used for model training, and during the inference phase, the input image is analyzed through the collaborative work of the pixel-level segmentation model and the multimodal detection model.

[0117] Figure 4 This is a schematic diagram illustrating the specific training process of a pixel-level segmentation model provided in an embodiment of this application. Figure 5 This is a schematic diagram illustrating the specific training process of a large multimodal detection model provided in an embodiment of this application. For example... Figure 4As shown, during the training (fine-tuning) of the pixel-level segmentation model, a forged image and coarse-grained location cues (i.e., bounding box coordinates extended from the aforementioned quantitative forgery mask) are input to the pixel-level segmentation model. The aforementioned quantitative forgery mask is used as a supervision signal, and the model is trained using Dice loss and cross-entropy loss to output fine-grained segmentation results that accurately match the forged regions. Training parameters: learning rate 1e-5, batch size 16, 20 epochs. After fine-tuning, the model can identify forged regions with fine-grained precision, improving localization accuracy. The trained pixel-level segmentation model outputs pixel-level forged region segmentation results; the pixel-level segmentation model can use existing models, such as the second-generation arbitrary segmentation (SAM2, Segment Anything Model 2) model.

[0118] like Figure 5 As shown, during the training (fine-tuning) of the multimodal detection large model, real and fake images (including real and fake images) are input into the multimodal detection large model, and coarse-grained semantic labels are derived from the quantitative forgery mask, including bounding boxes, real and fake labels, and interpretable text describing the differences between real and fake images (e.g., natural language sentences describing the type, location, and impact of tampering, such as "The forged region in the bottom-left corner exhibits unnatural texture inconsistencies due to splicing."). These serve as supervisory signals to train the multimodal detection large model to output preliminary analysis results, including real and fake judgment labels, interpretable text, coarse-grained localization information presented in the form of bounding boxes, and semantic guidance features. The multimodal detection large model adopts an existing model, such as the Qwen2.5-VL multimodal large model, which supports image-text input and includes a visual transducer (ViT) and a language decoder. Its multimodal characteristics are reflected in its ability to process the joint visual images and language text.

[0119] The pixel-level segmentation model and the multimodal detection model are trained independently as collaborative but independent models; the original image is not used as training input, but only for the data generation process.

[0120] During fine-tuning, multiple question-and-answer (QA) pairs are first introduced as additional supervision or cue signals to guide the model in capturing fine-grained differences between real and fake images. These QA pairs simulate the human reasoning process for image forgery. For example, a typical QA pair includes the question, "Does a fake region exist in the image? If so, describe its location, type, and explanation of the difference." and the answer, "A fake region exists, located at the bottom left of the image, exhibiting inconsistent texture due to splicing, with bounding box coordinates of [x=100, y=200, w=50, h=30], and the label is fake." These QA pairs generate text embedding sequences using the built-in tokenizer serializer of Qwen2.5 Vision-Language. This sequence not only provides contextual cues but also establishes direct associations with subsequent special tokens: for example, the location description in the QA guides the [BBOX] token to learn the bounding box representation, while the difference interpretation assists the [MASK] token in capturing pixel-level forgery traces, thereby strengthening the supervisory role of the QA signal on visual tokens in the Transformer attention mechanism.

[0121] Next, the input image is fed into the Vision Transformer (ViT) within the large model to extract image pixel features (patch embeddings). ViT divides the image into 196 patches, each patch embedding with a dimension of 768. Furthermore, this method introduces special tokens to expand the input sequence: one bounding box identifier ([BBOX] token, used to regress forgery region coordinates), 32 mask identifiers ([MASK] tokens, used for multi-instance forgery information indication and pixel-level supervision), one classification identifier ([CLS] token, used for overall real / fake classification), and 32 expert feature identifiers ([EXP] tokens, used to inject external forgery detection knowledge). These tokens are all learnable embedding vectors with the same dimensionality as the patch embeddings (768 dimensions), and their initial values ​​are obtained through random initialization (e.g., Gaussian distribution) and progressively optimized during fine-tuning. These tokens, together with image pixel features, form a 2^62-bit embedding sequence (196 patches + 1 [BBOX] + 32 [MASK] + 1 [CLS] + 32 [EXP]), which is then directly concatenated with the text embedding sequence generated by QA to form a unified input sequence, supplemented by positional encoding to preserve sequence structure. The tokens are designed to facilitate efficient location and extraction of corresponding features; for example, relevant representations can be directly accessed through predefined positional indices without traversing the entire sequence, thus improving computational efficiency and model interpretability. More importantly, after concatenation, the QA text embedding sequence interacts with special tokens through a cross-attention mechanism: the QA question guides the model to focus on the classification and regression tasks of the [CLS] and [BBOX] tokens, while the QA answer provides supervisory signals to help optimize the representation of forgery differences by the [MASK] and [EXP] tokens.

[0122] To further uncover underlying forgery traces in images, this scheme maps the features extracted by the expert model through a fully connected layer (FC) and injects them into the embedding vector positions pointed to by [EXP]tokens. This replacement mechanism achieves seamless integration of external knowledge, enhancing the model's sensitivity to tampering clues. In a specific embodiment, we use the penultimate layer output feature vector of the forensic expert model (NPR, Non-Photorealistic Rendering-based forgery detector) as input, projects it to 768 dimensions through the FC layer, and directly overwrites the initial embedding of [EXP]tokens, thereby injecting a professional forgery detection representation. This complements the QA signal: the expert features reinforce the "differential interpretation" mentioned in the QA answer, further guiding the tokens to learn fine-grained forgery patterns.

[0123] Finally, the constructed input sequence is fed into the Qwen 2.5-VL large model and propagated forward through multiple Transformer blocks. The model's visual-language fusion module (based on cross-attention and MLP layers) handles multimodal interactions, fusing visual features with QA text prompts to generate the final output. Specifically, the output includes: (1) bounding box coordinates, extracted from the final representation of the [BBOX] token through a linear regression head; (2) true / false labels, generated from the representation of the [CLS] token through a classification head to produce binary predictions; and (3) interpretable text of true / false differences, generated by autoregression using a language model head (LM head) based on the output of the entire sequence (especially the representation at the end of the sequence). This text generation process is similar to the natural language output mechanism of the Qwen model: the hidden state of the Transformer is fed into the LM head, predicting the sequence token by token to generate a narrative explanation similar to a QA answer (e.g., "The fake region is located at the bottom left, and the texture is inconsistent"). This mechanism ensures that the output is not only structured (coordinates and labels) but also interpretable in natural language, directly guided by QA prompts.

[0124] Fine-tuning employs an end-to-end optimization strategy, with gradient propagation involved throughout the entire process from QA serialization to output generation. The loss functions include: (1) Binary Cross-Entropy Loss (BCE) for supervising real and fake label classification, based on the prediction of [CLS] tokens and the calculation of ground-truth labels; (2) L1 loss for bounding box regression, ensuring the accuracy of coordinate prediction for [BBOX] tokens; (3) Mean Squared Error Loss (MSE) for mask supervision, optimizing the representation of [MASK] tokens for pixel-level tampering detection; and (4) Cross-Entropy Loss (CE) for text generation, calculating sequence-level likelihood based on ground-truth interpretable text (derived from QA answers) to supervise the natural language output of the LM head. The total loss is a weighted sum of the above components (weights can be optimized through hyperparameter search). Training uses the AdamW optimizer with a learning rate of 1e-4, a batch size of 16, and 10 epochs.

[0125] When detecting the test image, the image to be tested is input into a trained multimodal detection model to obtain preliminary analysis results containing at least one semantic guidance feature. Specifically, this includes the true / false category judgment result corresponding to the test image, explanatory text for the true / false category judgment result, and forgery region indication information corresponding to the test image. The explanatory text serves as the semantic guidance feature, and the forgery region indication information serves as coarse-grained localization information. The test image and its corresponding semantic guidance feature are then input into the pixel-level segmentation model to generate a pixel-level forgery region localization mask for the test image. It should be noted that the testing process does not involve the training dataset or the original image; it only performs collaborative inference based on the trained model. During the testing process, the model training effect is judged based on the detection results and the label data corresponding to the test image. It should be further noted that when using the model for image forgery detection, the same processing procedure as during model testing can be referred to.

[0126] In some specific application scenarios, during the model usage phase, the following collaborative reasoning process is executed for a single image to be detected: Preliminary analysis: The image to be detected (resolution 512x512) is input into the fine-tuned multimodal detection model mentioned above, and its output results (binary label: true / false), interpretable text (e.g., "There is semantic inconsistency in the sky area of ​​the image, suspected editing traces"), as well as the bounding box localization identifier ([BBOX] output coordinates) and the fake localization identifier ([MASK] output semantic embedding).

[0127] Feature guidance: Based on the acquired bounding box localization identifiers and forgery localization identifiers, feature embeddings corresponding to the predicted forgery regions are extracted from the last layer feature map of the internal visual transducer (ViT) of the aforementioned multimodal detection large model (e.g., region features cropped by RoI pooling, dimension [1, 768]). The bounding box features and forgery trace features are combined with the cue word encoder of the pixel-level segmentation model through their respective linear layers (dimension reduced to 256) to form sparse features (bounding box coordinates) and dense features (semantic embeddings).

[0128] Refined segmentation: The extracted features are embedded as high-level semantic prompts and input into the fine-tuned pixel-level segmentation model. The pixel-level segmentation model uses this high-level semantic prompt to analyze the image to be detected and outputs a final, high-precision pixel-level fake region localization mask (resolution matches the input data, value range [0, 1]).

[0129] Results aggregation: Integrate the acquired authenticity identification results, interpretable text, and refined localization masks to form a comprehensive, multi-dimensional image detection and analysis report. For example, it can use serialized (JSON) format, including labels, text, mask images, and bounding boxes.

[0130] In one application scenario, the above solution is implemented by constructing a closed-loop data generation and multimodal training architecture, which includes an automated fake data generation module and a multimodal collaborative detection module. The automated fake data generation module can automatically generate a dataset of locally edited images containing edited text information and edit masks.

[0131] The generation pipeline of this module primarily utilizes a large language model to generate diverse image editing instructions for multi-scale editable objects of varying sizes. To ensure the accuracy and diversity of these instructions, a keyword-based editing preset prompt engineering is introduced. Its structure revolves around two dimensions: object size (e.g., large, medium, small) and editing method (content dragging, color transformation, generation, scaling, removal, movement). This systematically guides the model to output instructions that meet specific needs (e.g., "replace the apple in the image with a blue basketball"). Subsequently, based on the editing inspiration provided by the large model (including the editable object, editing process, start position, and end position), the pipeline drives multiple controllable image editors based on Generative Adversarial Networks (GANs) or diffusion models to perform batch editing, thereby generating locally edited images with varying editing methods, content, and objects.

[0132] The evaluation pipeline of this module automatically assesses the quality of the forged images using both quantitative and qualitative analysis methods, retaining only the edited images that meet the required quality standards. Specifically, the quantitative evaluation includes: calculating the similarity score between the edited image and the editing guidance text using a Contrastive Language–Image Pretraining (CLIP) model; and evaluating the perceptual similarity between the edited image and the real image using a 1-LPIPS model (higher scores indicate smaller low-level visual differences). The qualitative evaluation involves subjective analysis, utilizing a large model (GPT-4o) to replace manual subjective evaluation of the image editing quality. The large model outputs a generation quality score (e.g., assessing naturalness, coherence, and detail richness, with scores ranging from 0 to 1). Subsequently, the quantitative and qualitative indicators are analyzed together, and the large model (GPT-4o) determines whether regeneration is necessary. If regeneration is deemed unnecessary, textual information is output, including image content descriptions, image difference comparisons, and descriptions of editing traces. Finally, the automated fake data generation module obtains a mask image using the aforementioned quantitative fake mask generation method, and concatenates image differences and editing traces to form a textual modal content description of the image. This description, combined with genuine and fake markers, constitutes the supervision information for training the model. The aforementioned multimodal collaborative detection module employs a dual-model collaborative architecture of a pixel-level segmentation model and a large-scale multimodal detection model. The pixel-level segmentation model is used for fine-grained localization, while the large-scale multimodal detection model is used for global semantic understanding and coarse-grained localization. The model is trained using the dataset generated by the aforementioned automated fake data generation module.

[0133] During the model testing and usage phase, the image to be tested is received as input and processed by a trained multimodal detection model and a pixel-level segmentation model to obtain the final forgery detection result.

[0134] In this application, the above technical solution realizes a closed-loop process from data generation to detection and analysis. The above quantitative forgery mask provides a supervision signal, and the above multimodal collaborative detection framework guides the analysis of low-level pixels through high-level semantics, thereby realizing the identification of inconsistencies between low-level physical and high-level semantics.

[0135] Specifically, based on automated data generation, a large-scale dataset containing forged images and corresponding quantitative forgery masks can be generated. This solves the problem of the lack of high-quality, fine-grained labeled data in existing technologies, providing data support for the training of detection models. The "quantitative forgery mask" generated by fusing low-level physical differences and high-level semantic differences can describe the boundaries and strength of forgeries more accurately than traditional binary masks. Using this mask as a supervisory signal for training helps improve the accuracy of the detection model's localization. Semantic guidance features are generated using the global semantic analysis capabilities of the multimodal detection model, and these features guide the pixel-level segmentation model for fine-grained localization. This "coarse-to-fine" collaborative mechanism effectively combines the advantages of macroscopic semantic judgment and microscopic pixel localization, improving the detection capability for complex forgery scenarios. When outputting detection results, natural language analysis text can be generated simultaneously to explain the basis of the judgment. This makes the detection process no longer a "black box," improving the system's credibility and practical value.

[0136] like Figure 6 As shown, corresponding to the above-described image forgery detection method based on large models, this application embodiment also provides an image forgery detection system based on large models, the above-described image forgery detection system based on large models includes:

[0137] Data acquisition module 610 is used to acquire the image to be detected;

[0138] The first processing module 620 is used to obtain the authenticity category judgment result of the above-mentioned image to be detected, the explanatory text of the authenticity category judgment result, and the forgery region indication information of the above-mentioned image to be detected through the trained multimodal detection big model.

[0139] The second processing module 630 is used to determine the pixel-level fake region localization mask corresponding to the image to be detected by using a trained pixel-level segmentation model based on the image to be detected and the fake region indication information.

[0140] The result generation module 640 is used to generate the forgery detection result corresponding to the above-mentioned image to be detected based on the above-mentioned true / false category judgment result, the above-mentioned explanatory text and the above-mentioned pixel-level forgery region positioning mask.

[0141] Thus, for the image to be detected, a pre-trained multimodal detection model and a pre-trained pixel-level segmentation model are combined for processing. Specifically, the multimodal detection model can output a true / false category judgment result, explanatory text, and forgery region indication information. Furthermore, the forgery region indication information is combined with the pixel-level segmentation model to generate a more accurate pixel-level forgery region localization mask, achieving pixel-level precise indication of the forgery region. The final forgery detection result is generated based on the true / false category judgment result, explanatory text, and pixel-level forgery region localization mask. Compared with existing technologies, the forgery detection result generated in this application is interpretable and can achieve pixel-level precise indication of forgery regions in the image, which is beneficial to improving the image forgery detection effect.

[0142] It should be noted that the specific structure and implementation of the above-mentioned large-model-based image forgery detection system and its various modules or units can be referred to the corresponding descriptions in the above method embodiments, and will not be repeated here.

[0143] It should be noted that the division of the modules in the above-mentioned large-model-based image forgery detection system is not unique and is not intended as a specific limitation.

[0144] Based on the above embodiments, this application also provides a terminal, the principle block diagram of which can be as follows: Figure 7 As shown. The aforementioned terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps of any of the aforementioned image forgery detection methods based on a large model. The display screen can be a liquid crystal display (LCD) or an e-ink display.

[0145] Those skilled in the art will understand that Figure 7 The block diagram shown is only a partial structural diagram related to the solution of this application and does not constitute a limitation on the terminal on which the solution of this application is applied. The specific terminal may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0146] In one embodiment, a terminal is provided, the terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of any of the large-model-based image forgery detection methods provided in the embodiments of this application.

[0147] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the large-model-based image forgery detection methods provided in this application.

[0148] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0149] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0150] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0151] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0152] In the embodiments provided in this application, it should be understood that the disclosed systems / terminal devices and methods can be implemented in other ways. For example, the system / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units described above is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0153] If the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0154] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions are not in essence a departure from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for detecting image forgery based on a large model, characterized in that, The method includes: Acquire the image to be detected; The system obtains the authenticity category judgment result of the image to be detected, the explanatory text of the authenticity category judgment result, and the forgery region indication information of the image to be detected through the trained multimodal detection model. Based on the image to be detected and the forged region indication information, the pixel-level forged region localization mask corresponding to the image to be detected is determined by a trained pixel-level segmentation model. Based on the authenticity category judgment result, the explanatory text, and the pixel-level forgery region location mask, a forgery detection result corresponding to the image to be detected is generated.

2. The image forgery detection method based on a large model according to claim 1, characterized in that, The forged region indication information includes bounding box features and forgery trace features; The bounding box feature is used to indicate the boundary position of the rectangular forged region in the image to be detected; The forgery trace features are information about the corresponding positions of the hidden layers of the large model in the multimodal detection.

3. The image forgery detection method based on a large model according to claim 2, characterized in that, The step of determining the pixel-level forgery region localization mask corresponding to the image to be detected based on the image to be detected and the forgery region indication information, using a trained pixel-level segmentation model, includes: Based on the bounding box features and the forgery trace features, the pixel-level segmentation model's prompt word encoder performs encoding processing to obtain sparse features for representing bounding box coordinates and dense features for representing forgery trace information. Based on the sparse features, the dense features, and the image to be detected, the pixel-level forgery region localization mask corresponding to the image to be detected is determined by the pixel-level segmentation model.

4. The image forgery detection method based on a large model according to claim 3, characterized in that, The step of determining the pixel-level forgery region localization mask corresponding to the image to be detected using the pixel-level segmentation model based on the sparse features, the dense features, and the image to be detected includes: Linear layer mapping is performed on the bounding box features and the forgery trace features; The mapped bounding box features are concatenated with the sparse features to obtain the first concatenated feature, and the mapped forgery trace features are concatenated with the dense features to obtain the second concatenated feature; The first and second splicing features are input into the decoder of the pixel-level segmentation model to obtain the pixel-level fake region localization mask.

5. The image forgery detection method based on a large model according to claim 3, characterized in that, The resolution of the pixel-level fake region location mask is the same as the resolution of the image to be detected; Each pixel value in the pixel-level fake region location mask is between 0 and 1, and the pixel value is used to characterize the editing intensity of the corresponding pixel in the image to be detected.

6. The image forgery detection method based on a large model according to claim 1, characterized in that, The training data corresponding to the large multimodal detection model includes forged images; The forged image was obtained by editing a real image using an image editor based on image editing instructions; The image editing instructions are generated by a preset large language model based on the prompt information input by the target object.

7. The image forgery detection method based on a large model according to any one of claims 1 to 6, characterized in that, The step of generating a forgery detection result corresponding to the image to be detected based on the authenticity category judgment result, the explanatory text, and the pixel-level forgery region localization mask includes: Obtain the target output data format; Based on the target output data format, according to the authenticity category judgment result, the explanatory text, and the pixel-level forgery region location mask, the forgery detection result corresponding to the image to be detected is generated and output.

8. An image forgery detection system based on a large model, characterized in that, The system includes: The data acquisition module is used to acquire the image to be detected; The first processing module is used to obtain the authenticity category judgment result of the image to be detected, the explanatory text of the authenticity category judgment result, and the forgery region indication information of the image to be detected through a trained multimodal detection model. The second processing module is used to determine the pixel-level fake region localization mask corresponding to the image to be detected by using a trained pixel-level segmentation model based on the image to be detected and the fake region indication information. The result generation module is used to generate a forgery detection result corresponding to the image to be detected based on the authenticity category judgment result, the explanatory text, and the pixel-level forgery region location mask.

9. A terminal, characterized in that, The terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the steps of the image forgery detection method based on a large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image forgery detection method based on a large model as described in any one of claims 1 to 7.

Images