Semantic-based AI edited image tampering data generation method and system

Abstract

The invention discloses a semantic-based AI edited image tampering data generation method and system, and the method comprises the steps: obtaining original image data, and recognizing a to-be-tampered region of the original image data; segmenting the original image data based on a semantic segmentation algorithm and the to-be-tampered region, and generating a candidate region mask for image editing; determining a collaborative language model, and inputting the original image data and the to-be-tampered area into the collaborative language model to obtain an output tampering prompt word; and editing the original image data based on the candidate region mask and the tampering prompt word, and generating AI edited image tampering data. By optimizing a semantic segmentation model and using a collaborative language model and a diffusion model, an AI editing image tampering data set with diversity and authenticity can be efficiently generated, and powerful support is provided for constructing a tampering detection and positioning model for the aspect of AI editing images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a semantic-based AI image tampering data generation method and system. Background Art

[0002] Images can carry more information than text and play a crucial role in social media, news reporting, and commercial applications. With the continuous advancement of generative artificial intelligence (AIGC) technology, image generation and manipulation have become increasingly accessible and efficient, significantly impacting the authenticity and credibility of images. Therefore, identifying whether an image has been tampered with, identifying tampered areas, and further developing tampering detection and localization models for AI-powered image editing have both significant academic significance and practical application value.

[0003] In the field of AIGC image forensics, data diversity, authenticity, and scale integrity directly determine the robustness and generalization ability of detection models in complex scenarios. However, existing image tampering datasets lack diversity and authenticity. Summary of the Invention

[0004] The main purpose of the present invention is to provide a semantic-based AI editing image tampering data generation method and system, aiming to solve the problem of lack of diversity and authenticity of image tampering data in the existing technology.

[0005] To achieve the above-mentioned object, the present invention provides a semantic-based AI-edited image tampering data generation method, the semantic-based AI-edited image tampering data generation method comprising the following steps:

[0006] Acquire original image data, and identify a region of the original image data to be tampered with;

[0007] Segmenting the original image data based on a semantic segmentation algorithm and the area to be tampered with, and generating a candidate area mask for image editing;

[0008] Determining a collaborative language model, inputting the original image data and the area to be tampered into the collaborative language model, and obtaining an output tampering prompt word;

[0009] The original image data is edited based on the candidate area mask and the tampering prompt word to generate AI-edited image tampering data.

[0010] Optionally, the original image data is image data containing identifiable semantic information, including semantic information of objects and backgrounds.

[0011] Optionally, identifying the area to be tampered with of the original image data includes:

[0012] identify the original image data using a target detection algorithm to obtain coordinate ranges and confidence levels of all detected targets in the original image data, the detected targets being regions in the original image data that have identifiable semantic information;

[0013] filter all the detected targets based on the confidence levels to obtain candidate tampered targets with complete semantic information;

[0014] define an intersection-over-union threshold and a coordinate range threshold, filter all the candidate tampered targets based on the intersection-over-union threshold and the coordinate range threshold, and obtain and output a region to be tampered with and its corresponding coordinate range.

[0015] Optionally, the method of segmenting the original image data based on the semantic segmentation algorithm and the region to be tampered with to generate a candidate region mask for image editing comprises:

[0016] obtaining the coordinate range of the region to be tampered with in the original image data to be segmented;

[0017] segmenting the original image data based on a semantic segmentation algorithm and the coordinate range of the region to be tampered with;

[0018] generating a candidate region mask for image editing corresponding to the coordinate range of the region to be tampered with;

[0019] optimizing the candidate region mask through a morphological processing method.

[0020] Optionally, the collaborative language model at least includes a multi-modal large language model and an AI large language model, and the method of inputting the original image data and the region to be tampered with into the collaborative language model to obtain an output tampering prompt word comprises:

[0021] inputting the original image data and the region to be tampered with into the collaborative language model;

[0022] generating a background description corresponding to the original image data and a foreground description corresponding to the region to be tampered with through the multi-modal large language model;

[0023] generating and outputting a tampering prompt word corresponding to the background description and the foreground description through the AI large language model.

[0024] Optionally, the method of editing the original image data based on the candidate region mask and the tampering prompt word to generate AI edited image tampering data comprises:

[0025] inputting the candidate region mask, the tampering prompt word, and the original image data into a diffusion model using an image tampering algorithm based on the diffusion model;

[0026] positioning a region to be tampered in the original image data through the candidate region mask;

[0027] tampering the region to be tampered based on the tampering prompt word and the diffusion model, to generate AI edited image tampering data.

[0028] Optionally, the AI edited image tampering data generation method based on semantics further comprises:

[0029] obtaining the original image data, the candidate region mask and the tampering prompt word corresponding to the AI edited image tampering data, and constructing a structured data set based on the AI edited image tampering data.

[0030] In addition, to achieve the above-mentioned purpose, the present application further provides an AI edited image tampering data generation system based on semantics, which comprises:

[0031] an image recognition module, configured to obtain original image data and identify a region to be tampered in the original image data;

[0032] a semantic segmentation module, configured to segment the original image data according to a semantic segmentation algorithm and the region to be tampered, to generate a candidate region mask for image editing;

[0033] a prompt word generation module, comprising a determined collaborative language model, configured to input the original image data and the region to be tampered into the collaborative language model, to obtain an output tampering prompt word;

[0034] an image tampering module, configured to edit the original image data according to the region mask and the tampering prompt word, to generate AI edited image tampering data.

[0035] In addition, to achieve the above-mentioned purpose, the present application further provides a terminal, wherein the terminal comprises a memory, a processor and an AI edited image tampering data generation program based on semantics stored on the memory and executable on the processor, and the AI edited image tampering data generation program based on semantics implements the steps of the AI edited image tampering data generation method based on semantics when executed by the processor.

[0036] In addition, to achieve the above-mentioned purpose, the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores an AI edited image tampering data generation program based on semantics, and the AI edited image tampering data generation program based on semantics implements the steps of the AI edited image tampering data generation method based on semantics when executed by a processor.

[0037] In the present application, the original image data is obtained, and the tampering region to be tampered with in the original image data is identified; the original image data is segmented based on a semantic segmentation algorithm and the tampering region to be tampered with, to generate a candidate region mask for image editing; a collaborative language model is determined, the original image data and the tampering region to be tampered with are input into the collaborative language model, and an output tampering prompt word is obtained; the original image data is edited based on the candidate region mask and the tampering prompt word, to generate AI edited image tampering data. By optimizing the semantic segmentation model, the specific region with semantic information in the input image is extracted, a high-precision candidate region mask can be generated, the collaborative language model containing a multi-modal large language model and a high-performance large language model is used, a tampering prompt word highly related to the tampering region to be tampered with can be generated; the diffusion model is used to perform high-quality and accurate editing on the tampering region to be tampered with in the original image data based on the tampering prompt word, the high authenticity of the generated image in terms of semantic consistency and visual consistency can be ensured, thereby the AI edited image tampering data set with diversity and authenticity can be efficiently generated, and strong support can be provided for constructing the tampering detection and positioning model for AI edited images. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a flowchart of a preferred embodiment of the AI edited image tampering data generation method based on semantics of the present application;

[0039] Figure 2 is a schematic diagram of the optimized candidate region obtained by using the target detection algorithm and the coordinate range optimization method of the present application;

[0040] Figure 3 is a schematic diagram of the final candidate region mask generated by using the semantic segmentation algorithm and the mask optimization method of the present application;

[0041] Figure 4 is an example diagram of the AI edited image tampering data generated by using the method of the present application;

[0042] Figure 5 is a result schematic diagram of the tampering detection and positioning of the AutoSplice data set by using the tampering detection and positioning model and training on the training data set generated by the present application;

[0043] Figure 6 is a structure diagram of a preferred embodiment of the AI edited image tampering data generation system based on semantics of the present application;

[0044] Figure 7 is a structure diagram of a preferred embodiment of the terminal of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present application clearer and more explicit, the present application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0046] The continuous progress of artificial intelligence generated content (AIGC) technology has resulted in more and more image editing software on the market that uses artificial intelligence editing methods as the core. Using these image editing software, realistic tampering of images can be achieved with simple operations, which brings people creative expression and more possibilities for commercial applications, but also has a significant impact on the authenticity and credibility of images. Therefore, identifying whether an image has been tampered with and identifying the tampered areas, and further constructing a tampering detection and positioning model for AI edited images, not only has important academic significance, but also has outstanding practical application value in maintaining information credibility and social stability.

[0047] A high-quality data set is the basic guarantee for deep learning model training, algorithm evaluation and applicability verification. In the field of AIGC image forensics, the diversity, authenticity and integrity of the data directly determine the robustness and generalization ability of the detection model in complex scenarios. However, existing image tampering data sets have the following significant shortcomings: (1) lack of diversity, most existing image tampering data sets cover a single scene and tampering type, mainly focusing on traditional image splicing, copy-paste and other static operations, lacking the depiction of complex and diverse image editing behaviors in real AIGC scenarios; (2) lack of authenticity, most existing data is generated by manual operation or programmed scripts, which has a large gap with real AI generated images in terms of texture fusion, color coordination, edge details and other aspects, making it difficult to reflect the visual features and tampering traces of current advanced AIGC models (such as diffusion models); (3) low efficiency in scale construction, due to the lack of efficient and automated AI edited image and data labeling methods, the current publicly available data set is limited in size, making it difficult to support the adaptive training and performance evaluation of deep models for multi-modal and high complexity scenarios.

[0048] Therefore, there is an urgent need to construct a method that can automatically and systematically generate AI edited image tampering data with diversity, authenticity and scalability, and generate corresponding data sets to meet the demand for comprehensive and representative data for AIGC image forensics tasks.

[0049] To solve one or more of the above problems, the application obtains original image data, identifies a tampering region to be tampered with in the original image data; segments the original image data based on a semantic segmentation algorithm and the tampering region to be tampered with, to generate a candidate region mask for image editing; determines a collaborative language model, inputs the original image data and the tampering region to be tampered with into the collaborative language model, and obtains an output tampering prompt word; edits the original image data based on the candidate region mask and the tampering prompt word, and generates AI edited image tampering data.

[0050] The AI edited image tampering data generation method based on semantics according to the preferred embodiment of the application, as shown in Figure 1 The method comprises the following steps:

[0051] In step S10, original image data is obtained, and a tampering region to be tampered with in the original image data is identified.

[0052] In step S11, original image data is obtained.

[0053] It should be noted that, in the present application, in order to make the AI edited image tampering data generation method meet the requirement of semantic authenticity, the original image data refers to image data containing identifiable semantic information, including object and background semantic information, therefore, original image data containing object and background semantic information is obtained. In addition, the object and background semantic information contained in the original image data is not unique, that is, a single image data can contain multiple background semantic information, and can also contain multiple object semantic information.

[0054] In this embodiment, the AI edited image tampering data generation method based on semantics is performed on a single original image data, and corresponding AI edited image tampering data is generated based on a single original image data set. However, in actual application, a single AI edited image tampering data cannot meet the demand, therefore, in order to generate an AI edited image tampering data set for an AIGC image forensics task, the number of original image data selected should be as large as possible, that is, an original image data set containing rich types of original image data is selected based on the original image data, and the original image data set containing rich types of original image data is selected, and all original image data in the original image data set is sequentially generated into corresponding AI edited image tampering data using the AI edited image tampering data generation method based on semantics.

[0055] In this embodiment, the COCO (Microsoft target detection dataset, Common Objects in Context) dataset is selected as the original image dataset, and the core features of the COCO dataset are high scene complexity and fine annotation quality. In this embodiment, the dataset is used mainly to take advantage of the large-scale data and complex scene coverage features of the dataset. The COCO dataset used is the image of the COCO2017 data validation set, and the original image data obtained is the image data in the COCO2017 data validation set.

[0056] Further, identifying the tamperable region of the original image data comprises:

[0057] In step S12, a target detection algorithm is used to identify the original image data, and the coordinate range and confidence of all detection targets in the original image data are obtained. The detection target is a region in the original image data that has identifiable semantic information.

[0058] It should be noted that the target detection algorithm is used to identify the coordinate range and confidence of all detection targets in the original image data, and a deep learning-based target detection algorithm is preferentially selected. This type of target detection algorithm includes but is not limited to: Faster R-CNN (Faster Region-CNN) and Mask R-CNN (Mask Region-CNN) based on a convolutional neural network (CNN, Convolutional Neural Networks), a real-time target detection algorithm based on deep learning such as YOLO (You Only Look Once) algorithm, a single-stage target detection algorithm based on deep learning such as SSD (Single Shot MultiBox Detector) algorithm, and a one-stage target detection algorithm based on FocalLoss (loss function for solving class imbalance problem) such as RetinaNet (Efficient Single Shot MultiBox Detector) algorithm.

[0059] The detection target is a region in the original image data that has identifiable semantic information. Specifically, when using a target detection algorithm, a candidate box (which can also be an anchor box or a grid) that may contain a target is generated based on the original image, and semantic feature extraction is performed, that is, the semantic information contained in the candidate box is identified, and the region that can identify the semantic information is taken as the detection target. In addition, when using a target detection algorithm to detect a target, a corresponding confidence is also generated for each candidate box, and the region coordinates corresponding to the candidate box are the coordinate range of the detection target.

[0060] Step S13, screening all the detection targets based on the confidence to obtain candidate tampering targets with complete semantic information.

[0061] It should be noted that the detection targets obtained in step S12 are not the final detection targets. Because the detection algorithm generally uses less types of detection box sizes, there are many different cases, including but not limited to: the size of the detection box is smaller than the actual size of the detection target, the size of the detection box is equal to the actual size of the detection target, the size of the detection box is larger than the actual size of the detection target, and the detection box contains multiple detection targets.

[0062] To ensure that the detection target is a candidate tampering target with complete semantic information, all detection targets also need to be screened based on the confidence. By setting a confidence threshold, the confidence of each candidate box is compared with the confidence threshold to determine whether the current candidate box contains a candidate tampering target with complete semantic information.

[0063] Step S14, define an intersection-over-union threshold and a coordinate range threshold, and screen all the candidate tampering targets based on the intersection-over-union threshold and the coordinate range threshold to obtain and output the tampering region and its corresponding coordinate range.

[0064] As shown in FIG. 6, a schematic diagram of obtaining an optimized candidate region using a target detection algorithm and a coordinate range optimization method. Figure 2 Figure 2 To utilize the target detection algorithm and the coordinate range optimization method to obtain an optimized candidate region, as shown in FIG. 6, when using the target detection algorithm, multiple recognition boxes for the same target may be generated, which is mainly due to the redundant design of the algorithm itself (generating a large number of candidate boxes to cover the target). Therefore, the candidate region in step S13 needs to be optimized.

[0065] Specifically, based on the coordinate range of the candidate region, an optimization strategy is developed. The intersection-over-union between the coordinate ranges of the candidate regions is calculated pair by pair, a threshold is set for the intersection-over-union, and the coordinate ranges of two candidate regions with an intersection-over-union higher than the threshold are merged into a new candidate region. After all the merging of the current candidate regions is completed, the optimized candidate region and its coordinate range are finally output.

[0066] It should be noted that when merging the coordinate ranges of two subsequent regions, the coordinate range of the final candidate region needs to be limited to not more than 70% of the overall size of the picture (only for reference, the limitation can be adjusted according to the actual situation), so as to prevent multiple targets from being merged to form a too large region, ensure the reasonableness of the candidate region, effectively reduce the interference of irrelevant regions, and improve the accuracy and efficiency of generating the candidate region mask in the subsequent step.

[0067] ​The candidate region optimized finally is output as the tampered region, and the coordinate range of the tampered region is output based on the coordinate range of the candidate region.

[0068] As shown in the figure, the semantic-based AI edited image tampering data generation method comprises the following steps: Figure 1

[0069] In step S20, the original image data is segmented based on a semantic segmentation algorithm and the tampered region, to generate a candidate region mask for image editing.

[0070] As shown in the figure, Figure 3 Figure 3 As shown in the figure, a schematic diagram for generating a final candidate region mask using a semantic segmentation algorithm and a mask optimization method, the semantic segmentation algorithm is mainly used for segmenting the original image data, and the original image data is segmented into different semantic regions, thereby generating a candidate region mask corresponding to the candidate region.

[0071] It should be noted that the semantic segmentation algorithm used in the present application is not unique, but can be selected according to actual needs. The optional semantic segmentation algorithm includes but is not limited to: FCN (image semantic segmentation algorithm, Fully Convolutional Networks) using encoder-decoder architecture, SegNet (improved semantic segmentation algorithm, Segmentation Network) and other algorithms, PSPNet (pyramid scene parsing semantic segmentation algorithm, Pyramid Scene Parsing Network) based on multi-scale context fusion model, DeepLab (image segmentation algorithm combining deep convolutional neural network and probability model), RefineNet (path optimization image segmentation algorithm), BiseNet (bilateral segmentation network algorithm, Bilateral Segmentation Network) based on real-time lightweight model, ICNet (real-time semantic segmentation algorithm, Image Cascade Network) and other algorithms, advanced models based on Transformer (improved neural network) driven, such as SETR (semantic segmentation algorithm, Semantic Segmentation Transformer), MaskFormer (mask-based semantic segmentation algorithm, Mask Transformer), Mask2Former and other algorithms. Semantic segmentation models such as SAM (Segment Anything Model, prompt-based zero-shot generalization image segmentation model) can also be used.

[0072] Step S21, obtaining the coordinate range of the tampered region in the original image data to be segmented.​​

[0073] Specifically, the original image data requiring to perform the semantic segmentation task is selected, i.e., the original image data to be segmented is acquired, a target to be tampered with in the original image data is selected, the target to be tampered with corresponds to a region to be tampered with, and the coordinate range of the region to be tampered with is further acquired.

[0074] In this embodiment, the region to be tampered with in step S14 and the coordinate range corresponding thereto are directly acquired, or the candidate region optimized in this step and the coordinate range thereof are acquired.

[0075] Step S22: performing segmentation on the original image data based on the semantic segmentation algorithm and the coordinate range of the region to be tampered with.

[0076] According to actual needs, original image data set types, and the like, the most suitable semantic segmentation algorithm is selected from the above semantic segmentation algorithms, and the original image data is segmented based on the semantic segmentation algorithm and the coordinate range of the region to be tampered with.

[0077] Alternatively, the screened candidate tampering target can also be directly acquired, and the candidate tampering target corresponds to complete semantic information and a region to be tampered with, so that the coordinate range of the region to be tampered with can be acquired, and the original image data containing the screened candidate tampering target is directly segmented based on the semantic segmentation algorithm.

[0078] Step S23: generating a candidate region mask for image editing corresponding to the coordinate range of the region to be tampered with.

[0079] The candidate tampering target corresponding to the region to be tampered with has complete semantic information, which can also be referred to as an accurate semantic structure. The candidate region mask for image editing corresponding to the coordinate range of the region to be tampered with is also a candidate region mask generated based on the accurate semantic structure, and the candidate region mask is used for subsequent image editing and can also realize the distinction between foreground objects and background information, thereby providing clear editing boundaries.

[0080] Step S24: optimizing the candidate region mask by a morphological processing method.

[0081] After the original image data is segmented based on the semantic segmentation algorithm and the candidate region, and the candidate region mask is generated, the candidate region mask needs to be optimized to further improve the accuracy and expression ability of the candidate region mask.

[0082] It should be noted that the morphological processing method in this application is specifically a morphological post-processing method, which refers to a morphological processing operation applied as a subsequent optimization step after the main processing flow (such as target detection, image segmentation, and the like in this embodiment) is completed.

[0083] In this embodiment, a mask optimization strategy including morphological processing and connected region analysis is designed. Specifically, a large-sized circular convolution kernel is used to perform morphological operations such as multiple dilations, erosions, and re-dilations on the generated initial candidate region mask, to remove small isolated regions, smooth edges, and enhance the coherence of the main region. Then, through connected region screening, white regions with an area ratio of no less than 50% (which can be set according to actual conditions) of the mask main region are retained, and black region holes with an area of less than 100 pixels are filled, to further optimize the semantic consistency and spatial integrity of the candidate region mask. These operations can compensate for the limitations of the segmentation model in boundary detail areas and region integrity, and also provide high-quality basis for subsequent image tampering, ultimately generating an optimized candidate region mask.

[0084] As shown in Figure 1 The semantic-based AI editing image tampering data generation method includes the following steps:

[0085] In step S30, a collaborative language model is determined, and the original image data and the region to be tampered with are input into the collaborative language model to obtain an output tampering prompt word.

[0086] It should be noted that the language model used in the present application is a collaborative language model, which at least includes a multi-modal large language model and an AI large language model. The multi-modal large language model is used to describe the feature information of the original image data and the region to be tampered with. The AI large language model, i.e., a high-performance large language model, is used to generate the tampering prompt word.

[0087] Specifically, the multi-modal large language model used in this embodiment is an Alpha_Blip2 (Alpha_Bootstrapping Language-Image Pretraining) model. The input of this model is: an entire picture img, a region of interest mask, and an instruction. The region of interest refers to the region that is intended to be highlighted, and the instruction refers to what the model is required to do. In this embodiment, the instruction is "describe the picture content". Using the Alpha_Blip2 model can output a description of the input, containing semantic information of img and mask, and focusing on the semantic information of the mask region.

[0088] The AI large language model used includes but is not limited to: ChatGPT (Chat Generative Pre-trained Transformer), DeepSeek (DeepSeek AI assistant), Wenxin Yiyang, Tencent Yuanbao, etc. In this embodiment, the high-performance large language model used can be ChatGPT3.5, which can generate high-quality tampering prompt words according to the description of the foreground object and the background information. Through the powerful language ability of ChatGPT3.5, tampering sentences that conform to logic and have imagination can be generated.

[0089] In the collaborative language model of this embodiment, the Alpha_Blip2 model can input pictures / text, but its pure text conversation ability is weak, while the ChatGPT3.5 input is text, the conversation is more natural, and the thinking mode is closer to human. Therefore, the Alpha_Blip2 is used to generate description text for the region, and the ChatGPT3.5 is used to ask for tampering prompt words that are appropriate and consistent with human thinking by using the global description of the image and the local description of the mask region.

[0090] Step S31, inputting the original image data and the region to be tampered into the collaborative language model.

[0091] In practical applications, the collaborative language model is a language model containing multiple types of large language models, and each type of large language model in the collaborative language model is based on a fixed use order. The use order of the multi-modal large language model of the present application is earlier than the use order of the AI large language model.

[0092] In this embodiment, when the original image data and the region to be tampered are input into the collaborative language model, the original image data is first input into the multi-modal large language model in the collaborative language model, and the output of the multi-modal large language model is used as the input of the AI large language model.

[0093] Step S32, generating a background description corresponding to the original image data and a foreground description corresponding to the region to be tampered through the multi-modal large language model.

[0094] In this embodiment, the multi-modal large language model Alpha_Blip2 model is used to generate the background description and the foreground description, specifically, the Alpha_Blip2 model is used to generate the background description corresponding to the original image data and the foreground description corresponding to the region to be tampered in turn, which includes:

[0095] The first input: the original image img, the region of interest mask, the instruction "describe the picture content", and the first output generated by the Alpha_Blip2 model is the background description, reflecting the context information of the entire image in the region to be tampered with.

[0096] The second input: the picture cropped according to the coordinate range of interest, the all-white mask consistent with the size of the cropped picture, the instruction "describe the picture content", and the second output generated by the Alpha_Blip2 model is the foreground description, used to depict the semantic content of the region itself. Among them, the all-white mask indicates that the region of interest is the entire image range, because the Alpha_Blip2 model requires a mask input to be displayed, i.e., the mask input cannot be empty, and when there is no region of interest that needs to be highlighted, a white mask of the full image size is used as the default mask input.

[0097] For example, the input image information is an office table with various items on it. After target detection and processing, the coordinate range of the region of interest is obtained, which is a mug, and the coordinate range is the range of the region to be tampered with where the mug is located. The coordinate range and the picture are input into the semantic segmentation model SAM to obtain the output mask of the model, and the mask is morphologically processed to obtain the tampering region mask. Based on the above, the background description and the foreground description generated by the multi-modal large language model Alpha_Blip2 model are as follows:

[0098] The first input image of the Alpha_Blip2 model is the entire picture, and the mask is the tampering region mask described above, generating the background description "a mug on the office table";

[0099] The second input image of the Alpha_Blip2 model is the cropped mug region image, and the mask is the all-white mask consistent with the size of the input image, generating the foreground description "a mug".

[0100] Through the Alpha_Blip2 model, the background description "a mug on the office table" and the foreground description "a mug" can be generated, rather than just "an office table" description, so that the multi-modal large language model can describe the feature information of the original image data and the region to be tampered with.

[0101] Step S33, generating and outputting a tampering prompt word corresponding to the background description and the foreground description through the AI large language model.

[0102] The background description corresponding to the original image data and the foreground description corresponding to the tampering region to be tampered with generated in the acquisition step S32 are input into the AI large language model, i.e. ChatGPT3.5 used in the present application, so as to generate and output tampering prompt words corresponding to the background description and the foreground description.

[0103] For example, under the premise of the background description "a mug on the office table" and the foreground description "a mug" output by the Alpha_Blip2 model described above, ChatGPT3.5 can output high-quality tampering prompt words based on this, specifically, by inputting the existing background description "a mug on the office table" and the foreground description "a mug" into ChatGPT3.5, the foreground description is the region that needs to be tampered with, and please generate tampering prompt words that conform to human thinking logic and have imagination according to the two descriptions. Thus, the high-quality tampering prompt words "a metal pen container filled with pens on the office table" output by ChatGPT3.5 can be obtained. It can be seen that, compared with the tampering prompt words used by other tampering models, which are usually relatively short prompt words such as single nouns, the tampering prompt words generated by ChatGPT3.5 are significantly different from the tampering prompt words generated by other tampering models.

[0104] As shown in Figure 1 The semantic-based AI edited image tampering data generation method includes the following steps:

[0105] In step S40, the original image data is edited based on the candidate region mask and the tampering prompt words to generate AI edited image tampering data.

[0106] Specifically, the image tampering algorithm based on the diffusion model is used in the present embodiment, the candidate region mask and the original image data are input into the diffusion model, and the output AI edited image tampering data is obtained. The diffusion model uses a generation mechanism of step-by-step noise adding and noise removing, which can ensure that the generated tampering region has high-quality visual consistency with the original image in terms of texture, color and edge, so as to generate tampering images with diverse content and strong realism.

[0107] Step S41, using an image tampering algorithm based on a diffusion model, inputting the candidate region mask, tampering prompt words and the original image data into the diffusion model.

[0108] As shown in Figure 4 , Figure 4 The AI edited image tampering data generated by the semantic-based AI edited image tampering data generation method is an example of an AI edited image tampering data generated by the semantic-based AI edited image tampering data generation method. The candidate region mask generated in step S20 and the tampering prompt words obtained in step S30 are input into the diffusion model.

[0109] Step S42, positioning the region to be tampered in the original image data through the candidate region mask.

[0110] By positioning the region to be tampered in the original image data through the candidate region mask, the diffusion model is facilitated to edit the original image data, thereby completing the image tampering of the target region and generating the AI edited image tampering data.

[0111] Step S43, tampering the region to be tampered based on the tampering prompt words and the diffusion model, and generating the AI edited image tampering data.

[0112] Based on the tampering prompt words and the diffusion model, the region to be tampered is tampered to generate the AI edited image tampering data. Based on the original image content in step S10, the candidate region mask obtained in step S20 is used to generate natural language descriptions highly related to the tampering context. Specifically, the AI large language model generates high-quality tampering prompt words based on the natural language descriptions of the foreground objects and background information of the candidate region mask coverage area. These prompt words can effectively guide the image editing process, making the generated tampered images more reasonable in semantics and more realistic in content. With the support of the multi-model collaborative generation mechanism, the prompt words not only adapt reasonably to the semantics of the background and accurately depict the tampering semantics, but also achieve high matching with the mask region, thereby effectively improving the semantic consistency of the tampering operation and enhancing the quality and accuracy of the generated results.

[0113] Step S44, obtaining the original image data, candidate region mask and tampering prompt words corresponding to the AI edited image tampering data, and constructing a structured data set based on the AI edited image tampering data.

[0114] By systematically recording the supporting information related to the tampered image, including the original image data, candidate region mask, tampering prompt words and AI edited image tampering data (tampered image), the various supporting information is corresponded with the AI edited image tampering data as the core, and is organized into a structured data set in a unified format, supporting subsequent training and evaluation of image tampering and positioning models.

[0115] In this embodiment, the data set formed is a BLD_train (mixed latent diffusion model training data set) data set.

[0116] Among them, the image tampering algorithms based on diffusion model include three kinds of Brushnet (a plug-and-play image inpainting model with decomposed dual-branch diffusion), BLD (blended latent diffusion) and PowerPaint (a task is worth one word: learning with task prompts for high-quality versatile image inpainting), using the diffusion model kernel as SD (stable diffusion) model and LDMs (latent diffusion models) model, version v1.5, v2 and XL (specific version number) three kinds, these algorithms only need to input the mask of the candidate tampering area and the tampering prompt word to realize high-quality tampering, finally generate 3000 tampered images, this paper selects the original images in PASCAL VOC 2007 and MS COCO 2017 datasets as input examples, these two datasets contain a large number of diversified real scene images and are widely used in computer vision research, the results are shown in Table 1, where voc_2007 corresponds to PASCAL VOC 2007 dataset and coco_2017 corresponds to MS COCO 2017 dataset.

[0117] Table 1 Composition of large-scale tampered image dataset.

[0118] Tampering algorithm Source Diffusion model Number Picture size Use BLD voc_2007 LDMs 2313 256×256 Training set BLD coco_2017 LDMs 283 256×256 Test set BLD coco_2017 SDv2 283 512×512 Test set Brushnet coco_2017 SDv1.5 5703 200×144-640×640 Training set Brushnet coco_2017 SDv1.5 273 496×280-640×640 Test set Brushnet coco_2017 SDXL 273 1024×1024-1848×1024 Test set PowerPaint coco_2017 SDv1.5 288 640×640-1408×640 Test set

[0119] In addition, in order to verify the performance of the generated AI edited image tampering dataset BLD_train in the AIGC image forensics task, we use the tampering detection positioning model MVSS (MVSS-Net: Multi-View Multi-Scale Supervised Networks for Image Manipulation Detection) to construct the training set using the traditional tampering dataset CASIA, joint CASIA and BLD_train respectively, and detect and locate the image dataset AutoSplice which is also tampered by AI editing method, part of the results are as follows Figure 5As shown in the table 2, the F1 value of the detection positioning result and the IoU index are shown, and the experimental results show that the model trained only using the traditional data set CASIA has poor performance in dealing with AI edited generated image tampering scene; and after introducing the BLD_train data generated by the method, the detection performance of the model under multiple JPEG (image file format) compression levels is significantly improved, which reflects the effectiveness of the application in improving the robustness and adaptability of the model, and in the table, F1 refers to F1-Score, also known as Balanced Score, which is defined as the harmonic mean of precision and recall; IoU (Intersection over Union) refers to the intersection over union, which is used to calculate the intersection and union ratio of the true value and the predicted value set.

[0120] Table 2 Comparison of detection positioning performance of tampering detection models using different training data on AutoSplice dataset.

[0121]

[0122] In the application, by optimizing the semantic segmentation model, the specific region with semantic information in the input image is extracted, a high-precision candidate region mask can be generated, a collaborative language model containing a multi-modal large language model and a high-performance large language model is used, a tampering prompt word highly related to the region to be tampered with can be generated; using the diffusion model, the region to be tampered with in the original image data is edited with high quality and precision based on the tampering prompt word, which can ensure that the generated image has high authenticity in terms of semantic consistency and visual consistency, thereby efficiently generating AI edited image tampering data set with diversity and authenticity, and providing strong support for constructing tampering detection and positioning model for AI edited image.

[0123] Further, as Figure 6 shown, based on the above method, the application also correspondingly provides an AI edited image tampering data generation system based on semantics, wherein the system comprises:

[0124] An image recognition module 61 is configured to acquire an original image and identify a region to be tampered with in the original image data.

[0125] A semantic segmentation module 62 is configured to segment the original image data according to a semantic segmentation algorithm and the region to be tampered with, and generate a candidate region mask for image editing.

[0126] A prompt word generation module 63 contains a determined collaborative language model, which is configured to input the original image data and the region to be tampered with into the collaborative language model to obtain an output tampering prompt word.

[0127] An image tampering module 64 is configured to edit the original image data according to the region mask and the tampering prompt word to generate AI edited image tampering data.

[0128] Further, as shown in Figure 7 Based on the above-mentioned AI edited image tampering data generation method and system, the application further provides an AI edited image tampering data generation terminal based on semantics, which comprises a processor 10, a memory 20 and a display 30. Figure 7 Only some components of the terminal are shown, but it should be understood that all the shown components are not required, and more or less components can be implemented instead.

[0129] The memory 20 can be an internal storage unit of the terminal in some embodiments, such as a hard disk or a memory of the terminal. The memory 20 can also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 20 can include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store application software and various data installed on the terminal, such as program codes of the terminal, etc. The memory 20 can also be used to temporarily store data that has been output or will be output. In an embodiment, the memory 20 stores a program 40 that can be executed by the processor 10, thereby implementing the method of the application.

[0130] The processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, which is used to run program codes or process data stored in the memory 20, such as executing the method, etc.

[0131] The display 30 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 30 is used to display information of the terminal and to display a visualized user interface.

[0132] In an embodiment, the processor 10 implements the steps of the above method when executing the program 40 in the memory 20.

[0133] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a semantic-based AI edited image tampering data generation program, and the semantic-based AI edited image tampering data generation program, when executed by a processor, implements the steps of the semantic-based AI edited image tampering data generation method.

[0134] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or terminal including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article, or terminal. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or terminal that includes the element.

[0135] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware (such as a processor, a controller, etc.) to complete, and the program can be stored in a computer readable storage medium, and the program can include the processes of the above-mentioned method embodiments when executed. The computer readable storage medium can be a memory, a magnetic disc, an optical disc, etc.

[0136] It should be understood that the application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all these improvements and changes should be within the protection scope of the appended claims of the application.

Claims

1. A semantic-based AI editing image tampering data generation method, characterized in that: The semantic-based AI-edited image tampering data generation method comprises the following steps: Acquire original image data, and identify a region of the original image data to be tampered with; Segmenting the original image data based on a semantic segmentation algorithm and the area to be tampered with, and generating a candidate area mask for image editing; Determining a collaborative language model, inputting the original image data and the area to be tampered into the collaborative language model, and obtaining an output tampering prompt word; The original image data is edited based on the candidate area mask and the tampering prompt word to generate AI-edited image tampering data.

2. The semantic-based AI editing image tampering data generation method according to claim 1 is characterized in that: The original image data is image data containing identifiable semantic information, including semantic information of objects and backgrounds.

3. The semantic-based AI editing image tampering data generation method according to claim 2 is characterized in that: The identifying the area to be tampered with of the original image data includes: Identify the original image data using a target detection algorithm, and obtain coordinate ranges and confidence levels of all detected targets in the original image data, where the detected targets are regions in the original image data containing identifiable semantic information; Screening all the detection targets based on the confidence level to obtain candidate tampering targets with complete semantic information; An intersection-in-union ratio threshold and a coordinate range threshold are defined, and all candidate tampering targets are screened based on the intersection-in-union ratio threshold and the coordinate range threshold to obtain and output the area to be tampered and its corresponding coordinate range.

4. The semantic-based AI editing image tampering data generation method according to claim 3 is characterized in that: The step of segmenting the original image data based on a semantic segmentation algorithm and the area to be tampered with to generate a candidate area mask for image editing includes: Obtaining a coordinate range of the area to be tampered in the original image data to be segmented; Segmenting the original image data based on a semantic segmentation algorithm and the coordinate range of the area to be tampered with; Generating a candidate region mask for image editing corresponding to the coordinate range of the region to be tampered; The candidate region mask is optimized by a morphological processing method.

5. The semantic-based AI editing image tampering data generation method according to claim 1 is characterized in that: The collaborative language model includes at least a multimodal large language model and an AI large language model. The original image data and the area to be tampered with are input into the collaborative language model, and the output tampering prompt words include: Inputting the original image data and the area to be tampered into the collaborative language model; Generating a background description corresponding to the original image data and a foreground description corresponding to the area to be tampered with by using the multimodal large language model; The AI ​​large language model generates and outputs tampering prompt words corresponding to the background description and the foreground description.

6. The semantic-based AI editing image tampering data generation method according to claim 1, characterized in that: The method of editing the original image data based on the candidate area mask and the tampering prompt word to generate AI-edited image tampering data includes: Using an image tampering algorithm based on a diffusion model, the candidate region mask, the tampering prompt word and the original image data are input into the diffusion model; Locating the area to be tampered in the original image data by using the candidate area mask; The area to be tampered is tampered with based on the tampering prompt word and the diffusion model to generate AI edited image tampering data.

7. The semantic-based AI editing image tampering data generation method according to claim 1, characterized in that: The semantic-based AI editing image tampering data generation method further includes: The original image data, the candidate region mask, and the tampering prompt word corresponding to the AI-edited image tampering data are obtained, and a structured data set based on the AI-edited image tampering data is constructed.

8. A semantic-based AI-edited image tampering data generation system, characterized by: The semantic-based AI editing image tampering data generation system includes: An image recognition module, configured to obtain original image data and identify a region of the original image data to be tampered with; A semantic segmentation module, configured to segment the original image data according to a semantic segmentation algorithm and the area to be tampered with, and generate a candidate area mask for image editing; a prompt word generation module, comprising a determined collaborative language model, configured to input the original image data and the area to be tampered into the collaborative language model to obtain an output tampering prompt word; The image tampering module is used to edit the original image data according to the area mask and the tampering prompt word to generate AI-edited image tampering data.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a semantic-based AI-edited image tampering data generation program stored in the memory and runnable on the processor. When the semantic-based AI-edited image tampering data generation program is executed by the processor, the steps of the semantic-based AI-edited image tampering data generation method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a semantic-based AI editing image tampering data generation program, which, when executed by a processor, implements the steps of the semantic-based AI editing image tampering data generation method as described in any one of claims 1 to 7.

Images