Tampered image generation method, device and equipment based on multi-modal model

Abstract

The invention discloses a tampered image generation method, device and equipment based on a multi-modal model. The method comprises the following steps: acquiring an original image and a first mask image of the original image; screening a first object mask according to the original image and the first mask image to obtain a target area; performing semantic classification on the target area through a multi-model cooperation method to obtain a target candidate object category; through an LLaVA model, according to the original image and the target candidate object category, generating a tampering prompt word; and according to the tampering prompt word, tampering the target area to obtain a target tampered image. The authenticity of the generated tampered image can be improved, and the method can be widely applied to the technical field of image processing.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a method, device, and equipment for generating tampered images based on a multi-modal model. Background Art

[0002] With the rapid development of generative AI technology, the capabilities of image editing, synthesis, and tampering have been greatly improved. These models can achieve high-quality local tampering of images and generate visually realistic content, greatly increasing the difficulty of tampering detection. The existing GRE (Generated Region Editing Dataset) dataset is an image dataset for local tampering using AIGC (Artificial Intelligence Generated Content). However, the editing prompts for generating the GRE dataset are generated by a fixed template, lacking multi-modal visual semantic understanding; post-processing is added after generating the GRE dataset, covering up the tampering traces. In addition, most of the images in the GRE dataset have a small resolution; GRE uses a dense point grid based on SAM (Segment Anything Model) for region selection, unable to preferentially select key semantic regions, resulting in blind selection of tampering regions and weakening the realism of the images; the GRE dataset lacks true image - tampered image - mask Figure 3 meta-data pairs, which is not conducive to subsequent model training. Summary of the Invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide a method, device, and equipment for generating tampered images based on a multi-modal model, in order to solve at least one of the problems in the prior art. The present invention can improve the authenticity of the generated tampered images.

[0004] To achieve the above object, on the one hand, an embodiment of the present invention provides a method for generating tampered images based on a multi-modal model, the method comprising:

[0005] Obtain an original image and a first mask image of the original image;

[0006] According to the original image and the first mask image, screen the first object mask to obtain a target region;

[0007] Through a multi-model collaborative method, perform semantic classification on the target region to obtain target candidate object categories;

[0008] Through the LLaVA model, generate a tampering prompt according to the original image and the target candidate object categories;

[0009] According to the tampering prompt, tamper with the target region to obtain a target tampered image.

[0010] In some embodiments, acquiring the original image and the first mask image of the original image comprises the following steps:

[0011] Obtaining the original image through a high-resolution image dataset;

[0012] Parsing the original image and the JSON file corresponding to the original image to obtain object segmentation information;

[0013] The first mask image is generated according to the object segmentation information.

[0014] In some embodiments, screening the first object mask according to the original image and the first mask image to obtain the target area includes the following steps:

[0015] acquiring a first mask area of the first object mask according to the first mask image;

[0016] Performing hierarchical screening on the first object mask according to a ratio of the first mask area to the original image to obtain a plurality of initial regions;

[0017] performing de-overlapping processing on the plurality of initial regions to obtain a plurality of intermediate regions;

[0018] The second object mask of each of the middle regions is screened to obtain a candidate object mask and the target region.

[0019] In some embodiments, performing de-overlapping on the plurality of initial regions to obtain a plurality of intermediate regions comprises the following steps:

[0020] Obtaining an intersection-over-union (IoU) of the first object mask in the initial region;

[0021] Obtaining the first object mask whose intersection-over-union ratio is greater than a preset threshold to obtain a third object mask;

[0022] Obtaining a second mask area of the third object mask;

[0023] The third object mask corresponding to the largest second mask area is retained to obtain the second object mask and the middle area.

[0024] In some embodiments, performing semantic classification on the target area using a multi-model collaborative method to obtain a candidate target object category includes the following steps:

[0025] Obtain the first candidate category through the OneFormer model;

[0026] Obtain the second candidate category through the BLIP model;

[0027] Using the CLIP model, perform semantic feature matching on the target region according to the first candidate category and the second candidate category to obtain a third candidate category;

[0028] Using the CLIPSeg model, perform semantic segmentation on the target region according to the third candidate category to obtain a pixel-level candidate object category;

[0029] Obtain the pixel-level candidate object category with the largest proportion in the target region to obtain the target candidate object category.

[0030] In some embodiments, the step of generating a tampering prompt word using the LLaVA model according to the original image and the target candidate object category includes the following steps:

[0031] Annotate the first mask image according to the target candidate object category to obtain a second mask image;

[0032] Preset a question sentence according to the candidate object index of the second mask image and the target candidate object category;

[0033] Input the original image and the second mask image into the LLaVA model, and generate the tampering prompt word according to the question sentence.

[0034] In some embodiments, the step of tampering the target region according to the tampering prompt word to obtain a target tampered image includes the following steps:

[0035] Input the original image and the first mask image into the Stable Diffusion XL inpainting model, and tamper with the target region according to the tampering prompt word to obtain the target tampered image.

[0036] To achieve the above object, another aspect of the embodiments of the present invention provides a tampered image generation device based on a multi-modal model, the device includes:

[0037] A first module, configured to obtain an original image and a first mask image of the original image;

[0038] A second module, configured to screen a first object mask according to the original image and the first mask image to obtain a target region;

[0039] A third module, configured to perform semantic classification on the target region through a multi-model collaboration method to obtain a target candidate object category;

[0040] A fourth module, configured to generate a tampering prompt word according to the original image and the target candidate object category by using the LLaVA model;

[0041] A fifth module, configured to tamper with the target area according to the tampering prompt word to obtain a target tampered image.

[0042] To achieve the above object, on the other hand, an embodiment of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the foregoing method for generating a tampered image based on a multi-modal model.

[0043] To achieve the above object, on the other hand, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the foregoing method for generating a tampered image based on a multi-modal model.

[0044] To achieve the above object, on the other hand, an embodiment of the present invention provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, the computer device executes the foregoing method for generating a tampered image based on a multi-modal model.

[0045] The embodiments of the present invention at least include the following beneficial effects: The present invention provides a method, device and equipment for generating a tampered image based on a multi-modal model. The solution obtains an original image and a first mask image of the original image; filters a first object mask according to the original image and the first mask image to obtain a target area; performs semantic classification on the target area by a multi-model collaboration method to obtain a target candidate object category; generates a tampering prompt word according to the original image and the target candidate object category by using the LLaVA model; tamper with the target area according to the tampering prompt word to obtain a target tampered image. The present invention can improve the authenticity of the generated tampered image. Description of the Drawings

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to these drawings.

[0047] Figure 1It is a flowchart of the steps of a method for generating tampered images based on a multimodal model provided by an embodiment of the present invention;

[0048] Figure 2 It is an overall framework diagram of a method for generating tampered images based on a multimodal model provided by an embodiment of the present invention;

[0049] Figure 3 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0050] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention detailed in the appended claims.

[0051] It should be noted that although functional module division is performed in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the system or the flowchart. The terms "first / S100" and "second / S200" in the specification, claims and the above-mentioned drawings can be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information can also be called the second information, and similarly, the second information can also be called the first information. Depending on the context, the words "if" and "when" as used herein can be interpreted as "when...", "when...", or "in response to determining".

[0052] The terms "at least one", "multiple", "each", "any one", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any one refers to any one of the multiple.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0054] Before elaborating on the embodiments of the present invention in detail, some nouns and terms involved in the embodiments of the present invention are first explained, and the nouns and terms involved in the embodiments of the present invention are applicable to the following explanations.

[0055] Image tampering: Image tampering refers to the modification, synthesis, or forgery of an original image through technical means (such as copy-move, splicing, filling, AI generation, etc.) to change its content, appearance, or authenticity, usually serving a specific purpose (such as entertainment, information hiding, etc.).

[0056] Image tampering detection: Image tampering detection refers to the identification and analysis of abnormal traces in an image through technical means (such as machine learning, deep learning methods, etc.) to determine whether it has been artificially modified or synthesized, thereby verifying the authenticity of the image.

[0057] Image tampering localization: In this task, effective algorithms and models need to be developed. Taking the image to be tested as input, a probability map of the same size as the input is output accordingly. The magnitude of the value of each element represents the likelihood of the pixel at the corresponding position being tampered with. After thresholding this probability map, a binary image (mask) (black and white image) is obtained as the output result. It can be seen that image tampering localization is essentially a pixel-level binary classification problem.

[0058] Deep learning: Originating from the research of artificial neural networks. By combining low-level features to form more abstract high-level representation attribute categories or features to discover the distributed feature representations of data.

[0059] Multimodal model: A multimodal model is a type of model that combines natural language processing capabilities with the ability to understand and generate data of other modalities (such as vision, audio, etc.). These models integrate various types of inputs and outputs such as text, images, and sounds to provide a richer and more natural interaction experience. The core advantage of multimodal models is their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, a multimodal model can analyze a picture and generate descriptive text, or generate a corresponding image according to a text description. This cross-modal understanding and generation ability make multimodal large language models (MLLMs) have broad application prospects in multiple fields (such as autonomous driving, intelligent assistants, content recommendation systems, education and training, etc.).

[0060] Contrastive Language-Image Pretraining (CLIP), used for cross-modal matching of images and texts.

[0061] CLIPSeg is a method that combines the CLIP model and image segmentation technology for text-prompted image segmentation. It allows users to specify the image regions to be segmented through natural language descriptions, thus enabling flexible and efficient image segmentation tasks.

[0062] The Bootstrapping Language-Image Pre-training (BLIP) model is used for image caption generation.

[0063] OneFormer is a general semantic segmentation model that supports object segmentation on multiple datasets (COCO / ADE20K).

[0064] The Large Language-and-Vision Assistant (LLaVA) is used for multimodal interactive intent generation.

[0065] The Stable Diffusion XL Inpainting (SDXL-Inpaint) model is based on text-guided image inpainting and can repair, modify, or reconstruct images according to the images and relevant text prompts provided by users.

[0066] With the rapid development of generative AI technologies such as Diffusion models, the capabilities of image editing, synthesis, and tampering have been greatly improved. These models can achieve high-quality local tampering of images and generate visually realistic content, significantly increasing the difficulty of tampering detection. Moreover, since existing detection models rely on fixed feature templates, it is difficult to cope with the dynamic tampering characteristics of generative AI; the training dataset has low resolution and lacks high-quality images, making it impossible to fully simulate real scenarios; the tampering region selection strategy is simple and ignores semantic key regions, resulting in the Area Under Curve (AUC) values of current mainstream image tampering detection techniques (such as TruFor and PSCC-Net) being generally lower than 0.7 on new generative tampering data.

[0067] Currently, there is a GRE dataset, which is an image dataset with local tampering using AIGC (Artificial Intelligence Generated Content). However, the GRE dataset has a low resolution, and most of the images have a resolution less than 1024×768, which limits the generalization ability of the detection model on high-resolution images and cannot effectively train a high-resolution detection model; the GRE dataset has blindness in region selection. It selects regions based on a dense point grid of SAM and cannot preferentially select key semantic regions, resulting in blind selection of tampered regions and weakening the image realism; the GRE dataset only generates fixed template prompts through the LLaMA language model, lacking in-depth understanding of the semantics of the original image, and the generated editing intentions lack flexibility; post-processing is added after generating the tampered images, including compression and noise addition, which masks the original tampering traces; the GRE dataset lacks the true image - tampered image - mask Figure 3 triplet data, which is not conducive to classifying real and fake images, not conducive to subsequent model training, and cannot perform multi-task learning (detection, localization, tampered region recovery, etc.).

[0068] In view of this, as Figure 1 shown, the embodiments of the present invention provide a method for generating tampered images based on a multi-modal model, which may include but are not limited to steps S100 to S500:

[0069] Step S100, obtain the original image and the first mask image of the original image;

[0070] Step S200, screen the first object mask according to the original image and the first mask image to obtain the target region;

[0071] Step S300, perform semantic classification on the target region through a multi-model collaborative method to obtain the target candidate object category;

[0072] Step S400, generate a tampering prompt word according to the original image and the target candidate object category through the LLaVA model;

[0073] Step S500, tamper with the target region according to the tampering prompt word to obtain the target tampered image.

[0074] In some embodiments, step S100 may include but are not limited to steps S110 to S130:

[0075] Step S110, obtain the original image through a high-resolution image dataset;

[0076] Step S120, parse the original image and the corresponding JSON file of the original image to obtain object segmentation information;

[0077] Step S130: Generate the first mask image according to the object segmentation information.

[0078] In step S110 of some embodiments, a high-resolution original image is obtained through a high-resolution image dataset. Exemplarily, the SA-1B dataset is used to obtain the high-resolution original image. The SA-1B dataset contains a large number of high-resolution images with an average resolution of 1500×2250. The obtained high-resolution original image is in JPEG format with a resolution not lower than 1024×680.

[0079] In steps S120 to S130 of some embodiments, the original image is preprocessed, the original image and the corresponding JSON file are parsed. The JSON file contains the mask information of each object in the original picture. The object segmentation information recorded in the JSON file is read, and according to this object segmentation information, the first mask image can be generated.

[0080] In some embodiments, step S200 may include but is not limited to steps S210 to S240:

[0081] Step S210: Obtain the first mask area of the first object mask according to the first mask image;

[0082] Step S220: Perform hierarchical screening on the first object mask according to the ratio of the first mask area to the original image to obtain a number of initial regions;

[0083] Step S230: Perform non-overlapping processing on the number of initial regions to obtain a number of intermediate regions;

[0084] Step S240: Screen the second object masks of each intermediate region to obtain candidate object masks and the target region.

[0085] In steps S210 to S220 of some embodiments, the first mask area of the first object mask in the first mask image is obtained, and by stratifying according to the ratio of the first mask area to the original image, each initial region can be obtained. Exemplarily, the object masks are stratified according to the ratio of their areas to the whole image, and the object masks with area ratios of 0-10%, 10-20%, 20-30%, and 30-40% are respectively screened, so as to divide the object masks into different levels and retain the regions in each interval respectively.

[0086] In some embodiments, step S230 may include but is not limited to steps S231 to S234:

[0087] Step S231: Obtain the intersection over union of the first object mask in the initial region;

[0088] Step S232: Obtain the first object masks with an intersection over union greater than a preset threshold to obtain third object masks.

[0089] Step S233: Obtain the second mask area of the third object masks.

[0090] Step S234: Retain the third object mask corresponding to the largest second mask area to obtain the second object mask and the intermediate region.

[0091] In steps S231 to S234 of some embodiments, by calculating the IoU (Intersection over Union) of the first object masks, that is, calculating the intersection over union of the object masks, overlapping mask removal is achieved. Among the first object masks with an overlapping rate greater than 10%, only the object mask with the largest area is retained (that is, the object masks with non - largest areas among the first object masks with an overlapping rate greater than 10% are removed), then the third object masks can be obtained, thereby ensuring the uniqueness and rationality of region selection.

[0092] In step S240 of some embodiments, the object masks in the intermediate region after overlapping mask removal are screened. Exemplarily, during the screening process, at most 10 object masks are retained in each area interval as candidate object masks, and the target region containing the candidate object masks can be obtained after the intermediate region is screened. By restricting the number of object masks in each area region, it is prevented that too many object masks are selected, resulting in an increase in the complexity of subsequent tampering.

[0093] Through hierarchical filtering of the region area and overlapping elimination, the rationality of tampering can be guaranteed, focusing on small - object tampering, and intelligent screening of key regions can be achieved.

[0094] In step S300 of some embodiments, a multi - model collaborative method is used to provide the most likely candidate object categories. Among them, the OneFormer model provides candidate object categories based on the COCO and ADE20K datasets, the BLIP model provides open - domain candidate object categories, the categories are screened by the CLIP model, and pixel - level semantic segmentation is achieved by the CLIPSeg model, and the candidate object categories indicated by the candidate object masks are comprehensively obtained.

[0095] In some embodiments, step S300 may include but is not limited to steps S310 to S350:

[0096] Step S310: Obtain the first candidate category through the OneFormer model.

[0097] Step S320: Obtain the second candidate category through the BLIP model.

[0098] Step S330: Through the CLIP model, perform semantic feature matching on the target region according to the first candidate category and the second candidate category to obtain a third candidate category;

[0099] Step S340: Through the CLIPSeg model, perform semantic segmentation on the target region according to the third candidate category to obtain a pixel-level candidate object category;

[0100] Step S350: Obtain the pixel-level candidate object category with the largest proportion in the target region to obtain the target candidate object category.

[0101] In step S310 of some embodiments, the first candidate category is obtained through semantic segmentation by the OneFormer model. Exemplarily, the OneFormer provides candidate categories of candidate objects based on the COCO dataset and the ADE20K dataset respectively, and the accuracy is increased through different dataset models.

[0102] In step S320 of some embodiments, the BLIP model is used to generate a text description of the image, and the possible category names are used as the candidate categories of the candidate objects, further supplementing the candidate categories of the candidate objects.

[0103] In step S330 of some embodiments, in combination with the CLIP pre-trained model, the candidate categories of the candidate objects in the candidate object image after small-scale cropping are screened, the candidate object categories provided by the OneFormer and the BLIP are fused, global image features are matched with the semantic features of the candidate object image, and the top three categories are selected from the object categories with the highest correlation as the categories of the candidate objects. Among them, the candidate object image after small-scale cropping is a smaller image area cropped from the original image, which contains the candidate object. This image area is the input object for subsequent category screening using the CLIP pre-trained model.

[0104] In step S340 of some embodiments, based on the top three candidate object categories with the highest correlation given by the CLIP, semantic segmentation is performed on the candidate object image patch after large-scale cropping through the CLIPSeg model to obtain a pixel-level candidate object category. The candidate object image patch after large-scale cropping refers to a larger image area cropped from the original image, which contains the candidate object and its surrounding background and context information. This cropped image area (patch) will be used as the input of the semantic segmentation model to perform more detailed pixel-level classification and analysis of the candidate object.

[0105] In step S350 of some embodiments, in the target region, the proportion of these pixel-level candidate object categories is statistically analyzed, and finally the pixel-level candidate object category with the largest proportion is determined as the target candidate object category.

[0106] Through the multi-model collaborative method, combining the OneFormer model, BLIP model, CLIP model, and CLIPSeg model, a more accurate object classification description can be obtained.

[0107] In some embodiments, step S400 may include but is not limited to steps S410 to S430:

[0108] Step S410, label the first mask image according to the target candidate object category to obtain a second mask image;

[0109] Step S420, preset a question sentence according to the candidate object index of the second mask image and the target candidate object category;

[0110] Step S430, input the original image and the second mask image into the LLaVA model, and generate the tampering prompt word according to the question sentence.

[0111] In steps S410 to S430 of some embodiments, by labeling the first mask image according to the target candidate object category, a second mask image can be obtained. Taking the original image and the labeled second mask image as the input of the LLaVA model and asking corresponding questions, an adaptive tampering task description can be generated. Exemplarily, the question sentence is designed as follows:

[0112] Here are a pair of images for you. The first image is the original image, and the second image is a partially annotated semantic image of the first image (several masks and semantic categories have been annotated in the image). The image has these candidate regions: {obj_str}. You need to select a specific area from these regions that contributes the most to the semantic meaning of the image and replace it with specific misleading content, so that the image still appears natural and coherent after modification, but the semantic meaning of the image has significantly changed. You need to answer the tampering prompt of the specific misleading content. You only need to reply with one line of content, including the single specific area you have selected and corresponding tampering prompt, separated by commas.

[0113] Here is a pair of images for you. The first image is the original image, and the second image is a partially annotated semantic image of the first image (several masks and semantic classes have been annotated in the image). The image has the following candidate regions: {obj_str}. You need to select a specific region from these regions that contributes the most to the semantic meaning of the image and replace it with specific misleading content so that the modified image still looks natural and coherent, but the semantic meaning of the image has been significantly changed. You need to answer the tampering prompt for the specific misleading content. You only need to reply with one line of content, including the specific region you selected and the corresponding tampering prompt, separated by a comma.

[0114] Among them, obj_str refers to the indices and candidate object classes of the candidate objects in the picture, optionally in the format [(index1,class1),(index2,class2),...]. The LLaVA model selects a certain candidate region according to the semantic content of the candidate objects and generates a specific tampering prompt word (Prompt) in the format [(mask_index,mask_class), replace_prompt]. Exemplarily, the generated tampering prompt word is: (3,cup),a pen.

[0115] Through the LLaVA model, "image-semantic map-text" interactive intention generation is achieved. The most suitable mask is selected from the alternative objects, and a suitable replacement prompt word is generated, thereby improving the semantic misleadingness and making the image tampering more harmonious.

[0116] In step S500 of some embodiments, the original image and the first mask image are input into the Stable Diffusion XL inpainting model. According to the tampering prompt word, the target region is tampered with to obtain the target tampered image. Exemplarily, through the Stable Diffusion XL Inpaint model, combined with the prompt word (Prompt) generated by LLaVA, the original image and the first mask image of the original image are input to achieve natural tampering of the object mask region and generate a new image that meets the semantic misleading target. Optionally, the reconstructed image filled intelligently by the diffusion model (Stable Diffusion XL inpaint) is retained, and the splicing traces are eliminated, which can be used for subsequent multiple task learning.

[0117] In some embodiments, the tampered image is named with the original image name plus the suffix "_inpaint" and saved in JPG format; the binary mask image of the tampered area is saved, named with the original image name plus the suffix "_gt" and saved in PNG format; and the tampered area information (including the original prompt and the tampered prompt) is recorded for subsequent verification and tracking. Then, a one-to-one matching triple can be obtained: the real image (original image), the tampered image, and the mask image.

[0118] In summary, the overall processing framework of the tampered image generation method based on the multi-modal model in the embodiments of the present invention is as Figure 2 shown:

[0119] Step 1: Obtain the original image and the corresponding JSON file, where the JSON file contains the information of the masks of each object in the picture;

[0120] Step 2: According to the original image and the corresponding JSON file, select the object mask size and remove the overlap, and perform semantic classification through the multi-modal collaborative method (OneFormer+BLIP+CLIP+CLIPseg);

[0121] Step 3: Select a certain candidate area through LLaVA and generate a tampering prompt (Prompt);

[0122] Step 4: Input the replaced prompt, the original image, and the mask image into Stable Diffusion XL for intelligent filling;

[0123] Step 5: Output the tampered image.

[0124] The embodiments of the present invention also provide a tampered image generation device based on a multi-modal model, which can implement the above-mentioned tampered image generation method based on a multi-modal model. The device includes:

[0125] The first module is used to obtain the original image and the first mask image of the original image;

[0126] The second module is used to screen the first object mask according to the original image and the first mask image to obtain the target area;

[0127] The third module is used to perform semantic classification on the target area through a multi-model collaborative method to obtain the target candidate object category;

[0128] The fourth module is used to generate a tampering prompt according to the original image and the target candidate object category through the LLaVA model;

[0129] The fifth module is used to tamper with the target area according to the tampering prompt word to obtain a target tampered image.

[0130] It can be understood that the content in the above method embodiments is applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0131] An embodiment of the present invention further provides an electronic device, which includes a processor and a memory. The memory stores a computer program, and when the processor executes the computer program, it implements the above-mentioned method for generating a tampered image based on a multi-modal model. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0132] It can be understood that the content in the above method embodiments is applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0133] Reference Figure 3 , Figure 3 schematically shows the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0134] A processor 601, which can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention;

[0135] A memory 602, which can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 602 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 602 and are called by the processor 601 to execute a method for generating a tampered image based on a multi-modal model in the embodiments of the present invention;

[0136] An input / output interface 603, which is used to implement information input and output;

[0137] A communication interface 604, which is used to implement the communication interaction between this device and other devices. It can achieve communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0138] A bus 605, which transmits information between various components of the device (such as a processor 601, a memory 602, an input / output interface 603, and a communication interface 604);

[0139] Among them, the processor 601, the memory 602, the input / output interface 603, and the communication interface 604 are communicatively connected to each other inside the device through the bus 605.

[0140] An embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for generating tampered images based on a multimodal model.

[0141] It can be understood that the content in the above method embodiments is applicable to the embodiments of this storage medium. The functions specifically implemented by the embodiments of this storage medium are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0142] An embodiment of the present invention also provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned method for generating tampered images based on a multimodal model.

[0143] The embodiments of the present invention aim to improve the authenticity of image tampering and the difficulty of tampering detection. Through multimodal technology, tampered images with visual and semantic misleadingness are accurately generated, and at the same time, a paired dataset of high-quality real images, tampered images, and mask images is provided, so as to solve the problems of low resolution of the current dataset, blind editing areas, mechanized intention generation, lack of original tampering traces, etc. Optionally, the embodiments of the present invention can be applied to digital forensics tools to detect highly concealed tampered images; applied to image security detection systems to train robust tampering detection models; applied to content review platforms to identify maliciously tampered news or social media pictures; and applied to image editing software to provide intelligent tampering protection functions.

[0144] In summary, the method, device, and equipment for generating tampered images based on a multimodal model in the embodiments of the present invention have the following advantages:

[0145] 1. The embodiments of the present invention select images based on the SA-1B dataset (with an average resolution of 1500×2250 pixels) and perform high-precision region screening in combination with a mask (Mask) to ensure that the generated tampered images maintain a high resolution and meet the actual application requirements.

[0146] 2. The embodiments of the present invention perform hierarchical screening of the region area (0-10%, 10-20%, 20-30%, 30-40%) to ensure that objects of different sizes are representative; eliminate overlapping regions (IoU>10%) to ensure the uniqueness and rationality of region selection; retain at most 10 objects in each layer to reduce the complexity of candidate regions and improve the efficiency of subsequent tampering; use a multi-model collaborative method (OneFormer+BLIP+CLIP+CLIPseg) for object classification to obtain the categories of target candidate objects. By optimizing these tampered region screening strategies, the accuracy of object category recognition in key regions can be improved.

[0147] 3. The embodiments of the present invention analyze the original image and the semantic annotation image through the LLaVA model (a multi-modal large language model) and generate an adaptive tampering prompt (Prompt) based on the semantic information of the candidate object to ensure that the generated prompt not only conforms to the semantic background but also can accurately induce the tampered generated content, further enhancing the authenticity and misleading nature of the tampered image and improving the semantic misleadingness.

[0148] 4. The embodiments of the present invention use the Stable Diffusion XL Inpaint (SDXL-Inpaint) model for intelligent filling to achieve the generation of tampered images with consistent natural semantics. The generated pictures retain the original tampering traces, providing more challenging data for subsequent detection tasks.

[0149] 5. The tampered dataset output by the embodiments of the present invention retains the one-to-one correspondence between the real image, the tampered image, and the mask image (Mask), providing more accurate supervised data for the training of subsequent image tampering detection models, supporting multi-task learning (including detection, localization, tampered region restoration, etc.), and greatly improving the performance and robustness of the model.

[0150] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order mentioned in the operation diagrams. For example, depending on the functions / operations involved, two consecutive blocks shown may actually be executed substantially simultaneously or the blocks can sometimes be executed in the reverse order. Additionally, the embodiments presented and described in the flowcharts of the present invention are provided by way of example for a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logical flows presented herein. Alternative embodiments are foreseeable where the order of various operations is changed and where sub-operations described as part of a larger operation are executed independently.

[0151] In addition, although the present invention has been described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the described functions and / or features may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It should also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. Rather, considering the attributes, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the modules will be understood within the ordinary skills of an engineer. Therefore, those skilled in the art can implement the present invention as set forth in the claims without undue experimentation. It should also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0152] If the described functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0153] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a predefined sequence of executable instructions for implementing a logical function, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in conjunction with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0154] More specific examples (nonexhaustive list) of computer-readable media include the following: an electrical connection (electronic device) having one or more wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other suitable processing as necessary, and then stored in a computer memory.

[0155] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0156] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0157] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the claims and their equivalents.

[0158] The above has specifically described the preferred embodiments of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for generating tampered images based on a multimodal model, characterized in that, It includes the following steps: Obtain the original image and the first mask image of the original image; According to the original image and the first mask image, screen the first object mask to obtain the target area; Through a multi-model collaborative method, perform semantic classification on the target area to obtain the target candidate object category; Through the LLaVA model, generate a tampering prompt word according to the original image and the target candidate object category; According to the tampering prompt word, tamper with the target area to obtain the target tampered image.

2. The method for generating tampered images based on a multi-modal model according to claim 1, wherein, The obtaining of the original image and the first mask image of the original image includes the following steps: Obtain the original image through a high-resolution image dataset; Parse the original image and the corresponding JSON file of the original image to obtain object segmentation information; Generate the first mask image according to the object segmentation information.

3. A method for generating tampered images based on a multimodal model according to claim 1, characterized in that, The screening of the first object mask according to the original image and the first mask image to obtain the target area includes the following steps: According to the first mask image, obtain the first mask area of the first object mask; According to the ratio of the first mask area to the original image, perform hierarchical screening on the first object mask to obtain several initial areas; Perform de-overlapping processing on the several initial areas to obtain several intermediate areas; Screen the second object mask of each intermediate area to obtain the candidate object mask and the target area.

4. A method for generating tampered images based on a multi-modal model according to claim 3, characterized in that, The performing of de-overlapping processing on the several initial areas to obtain several intermediate areas includes the following steps: Obtain the intersection over union of the first object masks in the initial areas; Obtain the first object masks with the intersection over union greater than the preset threshold to obtain the third object mask; Obtain the second mask area of the third object mask; Retain the third object mask corresponding to the largest second mask area to obtain the second object mask and the intermediate area.

5. A method for generating tampered images based on a multi-modal model according to claim 1, characterized in that, The performing of semantic classification on the target area through a multi-model collaborative method to obtain the target candidate object category includes the following steps: Obtain the first candidate category through the OneFormer model; Obtain the second candidate category through the BLIP model; Through the CLIP model, perform semantic feature matching on the target area according to the first candidate category and the second candidate category to obtain the third candidate category; Through the CLIPSeg model, perform semantic segmentation on the target area according to the third candidate category to obtain the pixel-level candidate object category; Obtain the pixel-level candidate object category with the largest proportion in the target area to obtain the target candidate object category.

6. A method for generating tampered images based on a multimodal model according to claim 1, wherein, The generating of the tampering prompt word through the LLaVA model according to the original image and the target candidate object category includes the following steps: According to the target candidate object category, annotate the first mask image to obtain the second mask image; Preset a question sentence according to the candidate object index of the second mask image and the target candidate object category; Input the original image and the second masked image into the LLaVA model, and generate the tampering prompt word according to the interrogation question sentence.

7. A method for generating tampered images based on a multi-modal model according to claim 1, characterized in that, Tampering with the target area according to the tampering prompt word to obtain the target tampered image includes the following steps: Input the original image and the first masked image into the Stable Diffusion XL inpainting model, and tamper with the target area according to the tampering prompt word to obtain the target tampered image.

8. An image forgery generation device based on a multi-modal model, characterized in that, It includes: A first module for obtaining the original image and the first masked image of the original image; A second module for screening the first object mask according to the original image and the first masked image to obtain the target area; A third module for semantically classifying the target area through a multi-model collaboration method to obtain the target candidate object category; A fourth module for generating a tampering prompt word through the LLaVA model according to the original image and the target candidate object category; A fifth module for tampering with the target area according to the tampering prompt word to obtain the target tampered image.

9. An electronic device, characterized in that, It includes a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by the processor to implement the method according to any one of claims 1 to 7.

Images