Image forgery detection method, device and storage medium

Abstract

The application discloses an image forgery detection method and device and a storage medium, relates to the technical field of artificial intelligence, and comprises the following steps: acquiring a to-be-detected image; performing forgery detection on the to-be-detected image by using a first-stage model to obtain a forgery detection result, wherein the first-stage model is obtained by training a sample data set with different label granularities in advance; if the forgery detection result is that the to-be-detected image is a forged image, performing forgery analysis on the to-be-detected image by using a second-stage model according to a preset forgery semantic library to obtain a forgery semantic text, wherein the second-stage model is obtained by training a plurality of forged images and semantic annotation texts associated with the forged images that are collected in advance. The first-stage model is trained by using the sample data set with different label granularities, so that the accuracy of image forgery detection is improved. In addition, the second-stage model is trained by using the forged images with rich semantic annotations, so that the explanation capability of the model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image forgery detection method, device and storage medium. Background Technology

[0002] Currently, methods for detecting face spoofing mainly fall into two categories, as detailed below:

[0003] The first category consists of detection methods based on traditional image classification algorithms. These methods extract features from input samples using deep neural networks and then output the probability and judgment result of whether the sample is real or fake using a classification head. Although the detection accuracy is relatively high, the basis for determining forgery lacks interpretability and is difficult to assist in tracing the source of forgery behavior.

[0004] The second category is detection methods based on multimodal large language models. In addition to outputting the forgery probability and judgment result of the sample, the large language model can also provide the decision basis of the model at the same time, realizing semantic-level interpretation of forged images. However, the detection accuracy of the large language model itself is low. Summary of the Invention

[0005] The main objective of this application is to provide an image forgery detection method, device, and storage medium, which aims to improve the accuracy of image forgery detection and enhance the interpretability of the detection results.

[0006] To achieve the above objectives, this application proposes an image forgery detection method, the method comprising: Acquire the image to be detected; Based on the image to be detected, a forgery detection is performed using a first-stage model to obtain a forgery detection result. The first-stage model is trained on a pre-constructed sample dataset with different label granularities. If the forgery detection result indicates that the image to be detected is a forgery image, then based on the image to be detected and a preset forgery semantic library, a second-stage model is used to perform forgery analysis to obtain forgery semantic text. The second-stage model is trained based on several pre-collected forgery images and semantic annotation text associated with the forgery images.

[0007] In one embodiment, the first-stage model is trained according to the following steps: Obtain sample datasets with different label granularities; Each image sample in the sample dataset is aligned to obtain an aligned image corresponding to each image sample, wherein the label associated with the aligned image is the same as the label associated with the corresponding image sample; Based on each of the image samples and each of the aligned images, multiple sub-training sample sets are obtained by dividing the data according to preset different scale input ratios; For each training sample in each of the sub-training sample sets, the training sample is input into the detection model to be trained, and the target detection result of the training sample is output. The detection model to be trained is trained based on the target detection result of the training sample to obtain the first stage model.

[0008] In one embodiment, the sample dataset includes a multi-label dataset and a binary classification dataset, wherein the image samples in the multi-label dataset are associated with at least one forgery attribute label, the forgery attribute label being used to characterize the forged regions in the image samples and the forgery type corresponding to the forgery regions; the image samples in the binary classification dataset are associated with binary classification labels; The step of training the detection model to be trained based on the target detection results of the training samples to obtain the first-stage model includes: If the training sample belongs to the aligned image corresponding to the multi-label dataset, then the model loss value is determined based on the target detection result and the fake attribute label associated with the training sample. If the training sample belongs to an image sample or an aligned image corresponding to an image sample in the binary classification dataset, then the model loss value is determined based on the target detection result and the binary classification label associated with the training sample. Based on the model loss value, the detection model to be trained is trained to obtain the first-stage model.

[0009] In one embodiment, the multi-label dataset is constructed according to the following steps: Acquire several real images, and a forged image corresponding to each real image; For each of the forged images, a differential analysis is performed on each local region in the forged image and each local region in the corresponding real image to obtain the relative differences; Based on the relative differences, the forgery area is determined, and combined with the preset forgery type, the forgery type corresponding to the forgery area is analyzed and obtained; The multi-label dataset is constructed based on each forged image, the forged region in the forged image, and the forgery type corresponding to the forged region.

[0010] In one embodiment, the forged semantic library is constructed according to the following steps: Based on the forged region in each forged image and the forged type corresponding to the forged region, construct guidance prompt information for each forged image; Each of the forged images, the corresponding real images, and the guidance prompts are input into a preset large model, and the semantic annotation text corresponding to each of the forged images is output. The forgery semantic library is constructed based on the semantic annotation text corresponding to each forged image.

[0011] In one embodiment, the second-stage model is trained according to the following steps: Obtain several forged images and the semantic annotation text associated with the forged images; Determine the text similarity between each of the semantically annotated texts, and group the semantically annotated texts whose text similarity is greater than a preset similarity threshold into a group; A mapping relationship is constructed based on the semantically annotated text of the same group and the forged images associated with each of the semantically annotated texts; Training batches are sampled according to the mapping relationship, and the second-stage model is trained using each sampled training batch. Each training batch includes semantically annotated text in different groups and forged images associated with the semantically annotated text.

[0012] In one embodiment, training the second-stage model using each sampled training batch includes: The semantically labeled text in the training batch and the forged images associated with the semantically labeled text are processed by inner product to form multiple image-text pairs; Each image-text pair is input into the analysis model to be trained, and the training image encoding vector and training text encoding vector corresponding to each image-text pair are output. Based on the training image encoding vector and training text encoding vector corresponding to each image-text pair, the analysis model to be trained is subjected to comparative learning training to obtain the second-stage model.

[0013] In one embodiment, the step of performing forgery analysis using a second-stage model based on the image to be detected and a preset forgery semantic library to obtain forged semantic text includes: Based on each semantically annotated text in the forged semantic library, feature encoding processing is performed using the text encoder in the second-stage model to obtain the target text encoding vector corresponding to each semantically annotated text. Based on the image to be detected, feature encoding processing is performed using the image encoder in the second stage model to obtain the target image encoding vector; Determine the feature similarity between the target image encoding vector and each of the target text encoding vectors; The forged semantic text is determined based on the feature similarity corresponding to each target text encoding vector.

[0014] Furthermore, to achieve the above objectives, this application also proposes an image forgery detection device, which includes: The acquisition module is used to acquire the image to be detected; The forgery detection module is used to perform forgery detection based on the image to be detected using a first-stage model to obtain forgery detection results. The first-stage model is trained based on a pre-constructed sample dataset with different label granularities. The semantic analysis module is used to perform forgery analysis based on the image to be detected and a preset forgery semantic library if the forgery detection result indicates that the image to be detected is a forgery image, and to obtain forgery semantic text by using a second-stage model. The second-stage model is trained based on a number of pre-collected forgery images and semantic annotation text associated with the forgery images.

[0015] In addition, to achieve the above objectives, this application also proposes an image forgery detection device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image forgery detection method as described above.

[0016] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the image forgery detection method described above.

[0017] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the image forgery detection method described above.

[0018] This application provides an image forgery detection method, device, and storage medium. The image forgery detection method includes acquiring an image to be detected; performing forgery detection using a first-stage model based on the image to be detected to obtain a forgery detection result, wherein the first-stage model is trained on a pre-constructed sample dataset with different label granularities; if the forgery detection result indicates that the image to be detected is a forgery, then performing forgery analysis using a second-stage model based on the image to be detected and a pre-defined forgery semantic library to obtain forgery semantic text, wherein the second-stage model is trained on a number of pre-collected forgery images and semantically labeled text associated with the forgery images. By training the first-stage model using sample datasets with different label granularities, the first-stage model's ability to perceive different types of forgery features is enhanced, improving the accuracy of image forgery detection. Furthermore, by training the second-stage model using forgery images with rich semantic annotations, the model is endowed with the ability to perceive and express forgery semantic information, effectively improving the model's interpretability and thus enhancing the interpretability of the detection results. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating an embodiment of the image forgery detection method of this application. Figure 2 A schematic diagram of the model inference stage provided in an embodiment of this application; Figure 3 This is a flowchart illustrating Embodiment 2 of the image forgery detection method of this application; Figure 4 This is a flowchart illustrating Embodiment 3 of the image forgery detection method of this application; Figure 5 This is a flowchart illustrating Embodiment 4 of the image forgery detection method of this application; Figure 6 This is a flowchart illustrating Embodiment 5 of the image forgery detection method of this application; Figure 7 This is a schematic diagram of the module structure of the image forgery detection device according to an embodiment of this application; Figure 8This is a schematic diagram of the device structure of the hardware operating environment involved in the image forgery detection method in this application embodiment.

[0022] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0025] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device, big data service platform, or image forgery detection system capable of the above functions. The following description uses an image forgery detection system as an example to illustrate this embodiment and the subsequent embodiments.

[0026] Based on this, embodiments of this application provide an image forgery detection method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the image forgery detection method of this application.

[0027] Step S11: Obtain the image to be detected; It should be noted that real-time frames can be captured by a camera as the image to be detected, or the image to be detected can be extracted from a live stream or video call stream at a fixed frame rate, or the user can upload the image to be detected to the detection system.

[0028] Step S12: Based on the image to be detected, perform forgery detection using the first-stage model to obtain the forgery detection result; It should be noted that the first-stage model is trained on pre-constructed sample datasets with different label granularities. These sample datasets include multi-label datasets and binary classification datasets. In the multi-label dataset, image samples are associated with at least one forgery attribute label. This forgery attribute label characterizes the forged regions within the image sample and the corresponding forgery type. For example, in face forgery detection, the forgery regions include the entire face, eyes, nose, and mouth. The forgery attribute labels include the location of the forgery region, whether it is blurred, whether there is color inconsistency, whether there is texture inconsistency, whether there is structural distortion, and whether there are traces of fusion forgery. In the binary classification dataset, image samples are associated with binary classification labels, which characterize whether the image sample is forged.

[0029] Understandably, the image samples include three types of images: fake images carrying fine-grained fake attribute labels, real images containing the "true" label, and fake images containing the "fake" label.

[0030] Here, a real image is a naturally captured, unaltered image that retains its original, authentic information, providing feature samples for the classifier. Real images contain objects; in the field of face detection, these objects are faces. Forged images, on the other hand, are images artificially modified or generated using various techniques. While visually similar to real images, forged images contain traces of forgery. Forged images include various types of forgery, such as face replacement and face attribute manipulation. Face replacement involves replacing a face in a target image with another person's face while retaining other factors such as actions, expressions, and background from the original image. Face attribute manipulation involves modifying specific attributes of a real face image, such as the eyes, nose, and mouth.

[0031] Here, in the first stage, forgery detection is modeled as a multi-label classification problem. The first-stage model is trained using both a multi-label dataset and a binary classification dataset.

[0032] In this embodiment, the image to be detected is input into the first-stage model to perform forgery detection and obtain the forgery detection result.

[0033] Step S13: If the forgery detection result indicates that the image to be detected is a forged image, then based on the image to be detected and a preset forgery semantic library, a second-stage model is used to perform forgery analysis to obtain forged semantic text.

[0034] It should be noted that the second-stage model is trained based on a number of pre-collected forged images and their associated semantic annotation text. The semantic annotation text is obtained by combining the forged image, the corresponding real image, and the forgery type corresponding to the forged region in the forged image, using a large model for prediction. During training, the second-stage model no longer predicts the authenticity of the image but focuses on the semantic interpretation of forgery features; that is, the second-stage model outputs specific semantic evidence for "judging the sample as forged." Optionally, the model uses image-text alignment loss during training to enhance its semantic understanding ability.

[0035] In this embodiment, refer to Figure 2 , Figure 2This is a flowchart illustrating the model inference stage according to an embodiment of this application. If the forgery detection result indicates that the image to be detected is a forged image, the specific semantic basis for determining that the image is forged is further analyzed. Then, the image to be detected is input into the second-stage model to combine with the semantically annotated text in a pre-set forgery semantic library to analyze and obtain the forged semantic text corresponding to the image to be detected. It should be noted that the second-stage model is a CLIP (Contrastive Language-Image Pre-training) model, which includes modules such as an image encoder and a text encoder.

[0036] Furthermore, if the forgery detection result indicates that the image to be detected is not a forged image, then step S13 does not need to be executed.

[0037] In one feasible implementation, step S13 includes: Step S131: Based on each semantically annotated text in the forged semantic library, feature encoding processing is performed using the text encoder in the second stage model to obtain the target text encoding vector corresponding to each semantically annotated text. Step S132: Based on the image to be detected, perform feature encoding processing using the image encoder in the second stage model to obtain the target image encoding vector; In this embodiment, for each semantically annotated text in the forged semantic library, the semantically annotated text is input into the second-stage model to perform feature encoding processing using a text encoder, and outputs the target text encoding vector corresponding to the semantically annotated text.

[0038] In addition, the image to be detected is input into the second stage model to perform feature encoding processing using an image encoder to obtain the target image encoding vector.

[0039] In this embodiment, the execution order of steps S131 and S132 is not limited. Optionally, steps S131 and S132 can also be executed in parallel.

[0040] Step S133: Determine the feature similarity between the target image encoding vector and each of the target text encoding vectors; Step S134: Determine the forged semantic text based on the feature similarity corresponding to each target text encoding vector.

[0041] In this embodiment, the feature similarity between the target image encoding vector and each target text encoding vector is calculated, and the semantically annotated text with the highest feature similarity is used as the final forgery interpretation output to obtain the forged semantic text. This embodiment uses a single visual encoder to complete feature extraction and semantic alignment during the inference phase, which simplifies system design and reduces resource overhead during the inference phase.

[0042] This embodiment trains the first-stage model using sample datasets with different label granularities, enhancing the first-stage model's ability to perceive different types of forgery features and improving the accuracy of image forgery detection. In addition, it trains the second-stage model using forged images with rich semantic annotations, endowing the model with the ability to perceive and express forged semantic information, effectively improving the model's interpretability.

[0043] In one feasible implementation, refer to Figure 3 , Figure 3 This is a flowchart illustrating Embodiment 2 of the image forgery detection method of this application; the multi-label dataset is constructed according to the following steps: Step S21: Obtain several real images and a fake image corresponding to each real image; Step S22: For each of the forged images, perform a differential analysis on each local region in the forged image and each local region in the corresponding real image to obtain the relative differences; Step S23: Based on the relative differences, determine the forgery area, and in conjunction with the preset forgery type, analyze and obtain the forgery type corresponding to the forgery area; Step S24: Construct the multi-label dataset based on each forged image, the forged region in the forged image, and the forgery type corresponding to the forged region.

[0044] It should be noted that training data for deepfake face detection tasks typically only contains real and fake labels. To enhance the interpretability of the model's output, this embodiment constructs interpretable annotation information for fake images and uses it for subsequent model training. Accurate identification of fake regions is a key step in interpretability modeling, effectively defining the spatial scope of interpretation. A fake image refers to a real image that has been cropped and aligned.

[0045] In this embodiment, specifically, several real images and a forged image corresponding to each real image are obtained. It should be noted that during the generation process of the forged image, several typical forgery traces are often accompanied by, such as inconsistent skin color, local blurring, structural distortion, abnormal texture, and fusion boundary.

[0046] For each of the forged images, the following steps are performed: Based on facial key points, local regions of the forged image and corresponding local regions in the real image are located. These local regions include the entire face, eyes, nose, and mouth. A difference analysis is then performed on each local region in the forged image and each local region in the real image to obtain the relative differences between each region. The local region with the largest relative difference is then selected as the forged region. In other embodiments, multiple local regions with significant relative differences can also be selected as the forged region.

[0047] Furthermore, after identifying the forged region, the forgery types contained within that region are analyzed. These forgery types include categories such as whether there is blurring, color inconsistency, texture inconsistency, structural distortion, and traces of fusion forgery. Then, based on each forged image, the forged region within the forged image, and the corresponding forgery type, a multi-label dataset is constructed. The forged image serves as an image sample in the multi-label dataset, and the forgery type corresponding to the forged region is used to construct the forged region and its forgery attribute labels within the image sample.

[0048] This embodiment analyzes the relative differences between fake and real images to locate fake regions in fake images; then it performs fine-grained identification of various fake types contained in fake regions, and finally constructs a multi-label dataset to improve the model's ability to understand fake features in a fine-grained manner.

[0049] In one feasible implementation, refer to Figure 4 , Figure 4 This is a flowchart illustrating Embodiment 3 of the image forgery detection method of this application; the forgery semantic library is constructed according to the following steps: Step S31: Based on the forged region in each forged image and the forged type corresponding to the forged region, construct guidance prompt information corresponding to each forged image; Step S32: Input each of the forged images, the corresponding real images, and the guidance prompts into the preset large model, and output the semantic annotation text corresponding to each of the forged images; Step S33: Construct the forgery semantic library based on the semantic annotation text corresponding to each forged image.

[0050] It should be noted that this embodiment utilizes the multi-label dataset constructed in the first stage, including forged images and the forged attribute labels corresponding to the forged images. In addition, it combines real images and uses multiple prompts to guide the multimodal large model to automatically generate richer and more detailed natural language descriptions, thereby achieving more detailed semantic-level annotation of forged samples.

[0051] Specifically, for each of the forged images, the following steps are performed: based on the forged regions in the forged image and the forged type corresponding to the forged regions, guide prompt information corresponding to the forged image is constructed. That is, using the forged attribute label as a priori, the model is guided to focus on the key forged regions and feature attributes.

[0052] In addition, task description prompts can be pre-set to provide clear task instructions that specify the task objective of the model to analyze visual evidence and generate a comprehensive description of forgery traces.

[0053] In addition, formatting prompts can be pre-set to ensure that the output format is a standardized JSON structure, thus ensuring that the annotation results have good structured properties and are easy to call and evaluate by subsequent downstream tasks.

[0054] Furthermore, the forged image, its corresponding real image, and guidance prompts are input into a pre-defined large model, which outputs semantically annotated text corresponding to the forged image. Introducing a real image as a reference effectively alleviates the model's illusionary behavior, thereby improving the ability to identify forgery traces.

[0055] Furthermore, to improve the quality of text annotation, a rule-based filtering mechanism is introduced after the model pre-annotation is completed, supplemented by manual review, to remove invalid semantics or noisy annotation text. Then, based on the semantic annotation text corresponding to each forged image, the forged semantic library is constructed.

[0056] This embodiment utilizes the multi-label dataset constructed in the first stage and employs a multi-prompt guidance strategy to drive a large model, enabling the large model to generate richer and more detailed natural language descriptions, thereby achieving semantic-level fine-grained annotation of forged samples.

[0057] In one feasible implementation, refer to Figure 5 , Figure 5 This is a flowchart illustrating Embodiment 4 of the image forgery detection method of this application; the first-stage model is trained according to the following steps: Step S41: Obtain sample datasets with different label granularities; It should be noted that the sample dataset includes a multi-label dataset and a binary classification dataset. In the multi-label dataset, the image samples are associated with at least one forgery attribute label, which is used to characterize the forged regions in the image samples and the forgery type corresponding to the forgery regions. In the binary classification dataset, the image samples are associated with binary classification labels.

[0058] Step S42: Align each image sample in the sample dataset to obtain an aligned image corresponding to each image sample; Step S43: Based on each of the image samples and each of the aligned images, multiple sub-training sample sets are obtained by dividing them according to preset different scale input ratios; It should be noted that when only undone original face images are used during training, the model training process is difficult to converge due to the large amount of background information unrelated to forgery and complex pose variations contained in the images, exhibiting strong instability. If only aligned and cropped face regions are used as input to the model, although the model tends to be stable during training, in actual forgery samples, forgery traces are often not limited to the face region but also appear at the edges of the face where they blend with the background. This makes it difficult for models trained using only aligned images to effectively capture key forgery features, leading to a decline in model recognition performance.

[0059] Therefore, in this embodiment, to enhance the robustness of the model in real-world application scenarios, each image sample in the binary classification dataset of the sample dataset is cropped and aligned to obtain an aligned image corresponding to each image sample, so that the original image and the aligned image can be used for training simultaneously. It should be noted that the label associated with the aligned image is the same as the label associated with the corresponding image sample.

[0060] Furthermore, based on each of the image samples and each of the aligned images, multiple sub-training sample sets are obtained according to preset different scale input ratios, so as to balance the stability and generalization ability of training.

[0061] Step S44: For each training sample in each of the sub-training sample sets, input the training sample into the detection model to be trained, output the target detection result of the training sample, and train the detection model to be trained based on the target detection result of the training sample to obtain the first stage model.

[0062] It should be noted that the backbone network in the detection model being trained is the next-generation ConvNeXt convolutional network architecture. Unlike the Transformer structure, which primarily focuses on global feature dependencies, forgery detection tasks rely more on capturing subtle and localized forgery traces in the face region. These subtle differences are highly localized in spatial distribution, and convolutional neural networks have a natural advantage in local feature modeling and spatial neighborhood information capture, thus better meeting the feature requirements of this task. Furthermore, compared to traditional convolutional networks, ConvNeXt significantly improves global information modeling and overall performance while maintaining the local feature modeling capabilities of the convolutional structure by optimizing the convolutional kernel design, deepening the network structure, and improving normalization and activation methods.

[0063] In this embodiment, the following operations are performed for each training sample in each of the sub-training sample sets: The training samples are input into the detection model to be trained, and the model outputs the target detection results of the training samples. It should be noted that the classification head of the detection model to be trained includes multiple branches to simultaneously predict the global judgment result and multi-dimensional local interpretation information. Among them, the main branch is used to output the overall true / false prediction (i.e., binary classification result), and the other branches respectively forge different forgery attributes corresponding to the forged regions in the image, including: the location of the forged region, whether there is blurring, whether there is color inconsistency, whether there is texture inconsistency, whether there is structural distortion, and whether there are traces of fusion forgery, etc.

[0064] Further, based on the target detection results and the labels associated with the training samples (fake attribute labels or binary classification labels), the model loss value is calculated, and then the detection model to be trained is trained and optimized based on the model loss value to obtain the first stage model.

[0065] In one feasible implementation, training the detection model to be trained based on the target detection results of the training samples to obtain the first-stage model includes: Step S441: If the training sample belongs to the aligned image corresponding to the multi-label dataset, then determine the model loss value based on the target detection result and the fake attribute label associated with the training sample. In this embodiment, if the training sample belongs to the aligned image corresponding to the multi-label dataset, the model loss value is calculated according to the target detection result and the fake attribute label associated with the training sample, based on a preset multi-class classification loss function. The formula for the multi-class classification loss function is as follows:

[0066] Where N represents the number of samples in the sub-training sample set, y i,k ∈{0,1} represents the label of the k-th fake attribute, P i,k It is the target detection result predicted by the model for this attribute.

[0067] Step S442: If the training sample belongs to an image sample or an aligned image corresponding to an image sample in the binary classification dataset, then the model loss value is determined based on the target detection result and the binary classification label associated with the training sample. In this embodiment, if the training sample belongs to an image sample in the binary classification dataset or an aligned image corresponding to an image sample in the binary classification dataset, that is, for a training sample containing a "true / false" label, the model loss value is calculated using the binary classification cross-entropy loss function based on the target detection result and the binary classification label associated with the training sample. The formula for the cross-entropy loss function is as follows:

[0068] Where N represents the number of samples in the sub-training sample set, y i ∈{0,1} represents the binary classification label of the i-th training sample, p i It is the target detection result of the i-th training sample.

[0069] Step S443: Based on the model loss value, train and optimize the detection model to be trained to obtain the first stage model.

[0070] In this embodiment, the model parameters of the detection model to be trained are optimized based on the model loss value until the model training termination condition is met, thus obtaining the first stage model. The model training termination condition includes the model having converged and the number of iterations reaching the preset maximum number of training iterations.

[0071] This embodiment trains the model by combining sample datasets corresponding to fine-grained fake labels and genuine / fake labels, modeling the fake detection task as a multi-label classification problem. This improves the model's ability to understand fake features in a fine-grained manner, thereby effectively improving the model's fake detection accuracy. Furthermore, considering the differences in feature representation of input images at different preprocessing scales, this embodiment introduces a hybrid training mechanism using the original image and the aligned image. This improves training stability while enhancing the model's generalization ability to different fake types.

[0072] In one feasible implementation, refer to Figure 6 , Figure 6 This is a flowchart illustrating Embodiment 5 of the image forgery detection method of this application; the second-stage model is trained according to the following steps: Step S51: Determine the text similarity between each semantically annotated text, so as to group the semantically annotated texts whose text similarity is greater than a preset similarity threshold. Step S52: Construct a mapping relationship based on the semantically annotated texts of the same group and the forged images associated with each of the semantically annotated texts; It should be noted that, considering the diversity of the goals and text types in image-text matching, this embodiment employs a contrastive learning framework for training. The training objective is to maximize the similarity between matched image-text pairs while minimizing the similarity between non-matching pairs. A problem that needs to be addressed during training is data sampling conflicts.

[0073] It should be noted that although richer textual semantics are introduced, the semantic scope is still mainly limited to the forged region and its detailed description, and redundant information unrelated to forgery is actively filtered out. Therefore, duplicate labeled text is still unavoidable. To make full use of all available data, this embodiment does not forcibly remove duplicates during the data construction stage. Instead, a sample selection mechanism is introduced during the model training stage. By rewriting the sampler, the same text is avoided from being sampled repeatedly in the same training batch, thereby ensuring the diversity of training.

[0074] It should be noted that during training, if the uniqueness of text within the same training batch is not constrained, different images may correspond to the same text, leading to conflicts in the objective function. For example, if the samples include three forged images and three semantically labeled texts associated with those forged images, performing inner product processing on the samples yields the following training batch:

[0075] Assuming Text1 = Text3, taking the first row as an example, the training objective is to make... Approaching (1 0 0), that is, It is a positive sample, however, since Text1 = Text3, In reality, these samples also constitute positive samples, meaning their labels should be positive, but the training objective treats them as negative examples, leading to optimization conflicts and affecting model learning performance. Therefore, to avoid semantic label confusion, it is necessary to ensure the uniqueness of text descriptions within the same training batch during the data sampling phase.

[0076] In this embodiment, specifically, several forged images and semantic annotation text associated with the forged images are obtained; the semantic annotation text associated with the forged images is generated according to the implementation schemes of steps S31 to S33 above. The text similarity between each semantic annotation text is determined, and semantic annotation texts with a text similarity greater than a preset similarity threshold are grouped together; that is, semantic annotation texts with high text similarity can be considered duplicate texts. To ensure the uniqueness of text descriptions in each training batch, a mapping relationship is constructed based on the semantic annotation texts in the same group and the forged images associated with each semantic annotation text. Here, any semantic annotation text in the group is used as the key, and the index corresponding to the forged image associated with all texts in the group is used as the value. For example, following the above example, Text1 = Text3. In this case, a mapping relationship is established between Text1 and the index corresponding to the forged image associated with Text1 and Text3, respectively.

[0077] Step S53: Perform training batch sampling according to the mapping relationship, so as to use the training batch of each sampling to train the second stage model; In this embodiment, during the sampling process of each training batch, the key in the mapping relationship is taken as the semantically labeled text according to the mapping relationship, and a corresponding forged image is extracted at the same time. This ensures that each training batch includes semantically labeled text from different groups and the forged images associated with the semantically labeled text, thereby effectively avoiding the situation of repeatedly sampling the same text within the same training batch. Furthermore, the second-stage model is trained using each training batch.

[0078] Furthermore, to fully utilize the computing resources of multi-GPU parallel training, all text in the training batch is evenly distributed across the GPUs, ensuring a balanced text distribution across devices. Simultaneously, to ensure a degree of randomness in data distribution across different training epochs, a fixed-seed random shuffling mechanism based on the epoch number is introduced for each training epoch, ensuring that each GPU obtains a consistent data arrangement order within the same epoch, facilitating parallel synchronization and result reproducibility.

[0079] This embodiment constructs a mapping relationship to effectively avoid repeatedly sampling the same text in the same training batch during the sampling process, thereby maximizing the use of existing data, ensuring the diversity of training samples, and thus effectively improving the interpretability of the model and enhancing the interpretability of the detection results.

[0080] In one feasible implementation, the second-stage model is trained using training batches from each sample, including: Step S61: Perform inner product processing on the semantically labeled text in the training batch and the forged images associated with the semantically labeled text to form multiple image-text pairs; Step S62: Input each image-text pair into the analysis model to be trained, and output the training image encoding vector and training text encoding vector corresponding to each image-text pair. Step S63: Based on the training image encoding vector and training text encoding vector corresponding to each image-text pair, perform comparative learning training on the analysis model to be trained to obtain the second-stage model.

[0081] It should be noted that the analysis model to be trained includes an image encoder and a text encoder, etc. Optionally, the image encoder reuses the ConvNeXt network of the first-stage model. By leveraging the fine-grained understanding of the forged features of the ConvNeXt network and aligning it with rich semantically annotated text, it ultimately achieves more interpretable representation learning for forged samples.

[0082] In this embodiment, the following operations are performed for each training batch sampled: The semantically labeled text in the training batch and the forged images associated with the semantically labeled text are subjected to inner product processing to form multiple image-text pairs. The inner product processing can refer to the example of inner product processing of three forged images and semantically labeled text in step S52. Each image-text pair is then input into the analysis model to be trained. An image encoder is used to encode the forged images in the image-text pair to obtain a training image encoding vector, and a text encoder is used to encode the semantically labeled text in the image-text pair to obtain a training text encoding vector.

[0083] Furthermore, based on the training image encoding vector and training text encoding vector corresponding to each image-text pair in the training batch, the target similarity for each image-text pair is calculated, and a similarity matrix is ​​constructed based on the target similarity for each image-text pair. In addition, a label matrix is ​​constructed based on the real labels associated with each image-text pair in the training batch. Then, based on the number of samples in the training batch, the similarity matrix, and the label matrix, the contrastive learning loss value is calculated. The formula for calculating the contrastive learning loss value is as follows:

[0084] Where v represents the training image encoding vector, Let s(v, l) represent the training text encoding vector, s(v, l) represent the similarity matrix, I represent the label matrix, and B represent the number of samples in the training batch.

[0085] Furthermore, based on the contrastive learning loss value, the parameters of the analysis model to be trained are optimized to obtain the second-stage model. It should be noted that the parameters of the analysis model to be trained include parameters from the image encoder and text encoder, etc.

[0086] This embodiment trains the analysis model to be trained by comparing and contrasting semantically annotated text and forged images to obtain the second-stage model. This model is endowed with the ability to perceive and express forged semantic information, effectively improving the model's interpretability and thus enhancing the interpretability of the detection results.

[0087] It should be noted that the examples in the figure are only for understanding this application and do not constitute a limitation on the image forgery detection method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0088] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0089] This application also provides an image forgery detection device; please refer to... Figure 7 , Figure 7 This is a schematic diagram of the module structure of an image forgery detection device according to an embodiment of this application; the image forgery detection device includes: Acquisition module 71 is used to acquire the image to be detected; The forgery detection module 72 is used to perform forgery detection based on the image to be detected using a first-stage model to obtain a forgery detection result, wherein the first-stage model is trained based on a pre-constructed sample dataset with different label granularities. The semantic analysis module 73 is used to perform forgery analysis based on the image to be detected and a preset forgery semantic library if the forgery detection result indicates that the image to be detected is a forgery image, and to obtain forgery semantic text by using a second-stage model. The second-stage model is trained based on a number of pre-collected forgery images and semantic annotation text associated with the forgery images.

[0090] The image forgery detection device provided in this application, employing the image forgery detection method described in the above embodiments, can solve the technical problems mentioned in the background section. Compared with the prior art, the beneficial effects of the image forgery detection device provided in this application are the same as those of the image forgery detection method provided in the above embodiments, and other technical features in the image forgery detection device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0091] This application provides an image forgery detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the image forgery detection method in the first embodiment described above.

[0092] The following is for reference. Figure 8 , Figure 8 This is a schematic diagram of the hardware operating environment involved in the image forgery detection method in this application embodiment. The image forgery detection device in this application embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), vehicle terminals (such as vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8The image forgery detection device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0093] like Figure 8 As shown, the image forgery detection device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the image forgery detection device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the image forgery detection device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows image forgery detection devices with various systems, it should be understood that implementing or possessing all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0094] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0095] The image forgery detection device provided in this application, employing the image forgery detection method described in the above embodiments, can solve the technical problems mentioned in the background section. Compared with the prior art, the beneficial effects of the image forgery detection device provided in this application are the same as those of the image forgery detection method provided in the above embodiments, and other technical features of this image forgery detection device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0096] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0097] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0098] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the image forgery detection method in the above embodiments.

[0099] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0100] The aforementioned computer-readable storage medium may be included in the image forgery detection device; or it may exist independently and not assembled into the image forgery detection device.

[0101] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the image forgery detection device, cause the image forgery detection device to: acquire an image to be detected; perform forgery detection using a first-stage model based on the image to be detected to obtain a forgery detection result, wherein the first-stage model is trained based on a pre-constructed sample dataset with different label granularities; if the forgery detection result indicates that the image to be detected is a forgery image, then perform forgery analysis using a second-stage model based on the image to be detected and a pre-defined forgery semantic library to obtain forgery semantic text, wherein the second-stage model is trained based on a number of pre-collected forgery images and semantically labeled text associated with the forgery images.

[0102] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0103] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0104] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0105] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described image forgery detection method, and is capable of solving the technical problems described in the background art. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the image forgery detection method provided in the above embodiments, and will not be repeated here.

[0106] This application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image forgery detection method described above.

[0107] The computer program product provided in this application can solve the technical problems described in the background section. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiments of this application are the same as the beneficial effects of the image forgery detection method provided in the above embodiments, and will not be repeated here.

[0108] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for detecting image forgery, characterized in that, include: Acquire the image to be detected; Based on the image to be detected, a forgery detection is performed using a first-stage model to obtain a forgery detection result. The first-stage model is trained on a pre-constructed sample dataset with different label granularities. If the forgery detection result indicates that the image to be detected is a forgery image, then based on the image to be detected and a preset forgery semantic library, a second-stage model is used to perform forgery analysis to obtain forgery semantic text. The second-stage model is trained based on several pre-collected forgery images and semantic annotation text associated with the forgery images.

2. The image forgery detection method as described in claim 1, characterized in that, The first-stage model was trained using the following steps: Obtain sample datasets with different label granularities; Each image sample in the sample dataset is aligned to obtain an aligned image corresponding to each image sample. Based on each of the image samples and each of the aligned images, multiple sub-training sample sets are obtained by dividing the data according to preset different scale input ratios; For each training sample in each of the sub-training sample sets, the training sample is input into the detection model to be trained, and the target detection result of the training sample is output. The detection model to be trained is trained based on the target detection result of the training sample to obtain the first stage model.

3. The image forgery detection method as described in claim 2, characterized in that, The sample dataset includes a multi-label dataset and a binary classification dataset. In the multi-label dataset, image samples are associated with at least one forgery attribute label, which is used to characterize the forgery region in the image sample and the forgery type corresponding to the forgery region. The image samples in the binary classification dataset are associated with binary classification labels. The step of training the detection model to be trained based on the target detection results of the training samples to obtain the first-stage model includes: If the training sample belongs to the aligned image corresponding to the multi-label dataset, then the model loss value is determined based on the target detection result and the fake attribute label associated with the training sample. If the training sample belongs to an image sample or an aligned image corresponding to an image sample in the binary classification dataset, then the model loss value is determined based on the target detection result and the binary classification label associated with the training sample. Based on the model loss value, the detection model to be trained is trained to obtain the first-stage model.

4. The image forgery detection method as described in claim 3, characterized in that, The multi-label dataset was constructed according to the following steps: Acquire several real images, and a forged image corresponding to each real image; For each of the forged images, a differential analysis is performed on each local region in the forged image and each local region in the corresponding real image to obtain the relative differences; Based on the relative differences, the forgery area is determined, and combined with the preset forgery type, the forgery type corresponding to the forgery area is analyzed and obtained; The multi-label dataset is constructed based on each forged image, the forged region in the forged image, and the forgery type corresponding to the forged region.

5. The image forgery detection method as described in claim 4, characterized in that, The forged semantic library is constructed according to the following steps: Based on the forged region in each forged image and the forged type corresponding to the forged region, construct guidance prompt information for each forged image; Each of the forged images, the corresponding real images, and the guidance prompts are input into a preset large model, and the semantic annotation text corresponding to each of the forged images is output. The forgery semantic library is constructed based on the semantic annotation text corresponding to each forged image.

6. The image forgery detection method as described in claim 1, characterized in that, The second-stage model is trained according to the following steps: Obtain several forged images and the semantic annotation text associated with the forged images; Determine the text similarity between each of the semantically annotated texts, and group the semantically annotated texts whose text similarity is greater than a preset similarity threshold into a group; A mapping relationship is constructed based on the semantically annotated text of the same group and the forged images associated with each of the semantically annotated texts; Training batches are sampled according to the mapping relationship, and the second-stage model is trained using each sampled training batch. Each training batch includes semantically annotated text in different groups and forged images associated with the semantically annotated text.

7. The image forgery detection method as described in claim 6, characterized in that, The process of training the second-stage model using each sampled training batch includes: The semantically labeled text in the training batch and the forged images associated with the semantically labeled text are processed by inner product to form multiple image-text pairs; Each image-text pair is input into the analysis model to be trained, and the training image encoding vector and training text encoding vector corresponding to each image-text pair are output. Based on the training image encoding vector and training text encoding vector corresponding to each image-text pair, the analysis model to be trained is subjected to comparative learning training to obtain the second-stage model.

8. The image forgery detection method as described in claim 1, characterized in that, The step involves performing forgery analysis using a second-stage model based on the image to be detected and a pre-defined forgery semantic library to obtain forged semantic text, including: Based on each semantically annotated text in the forged semantic library, feature encoding processing is performed using the text encoder in the second-stage model to obtain the target text encoding vector corresponding to each semantically annotated text. Based on the image to be detected, feature encoding processing is performed using the image encoder in the second stage model to obtain the target image encoding vector; Determine the feature similarity between the target image encoding vector and each of the target text encoding vectors; The forged semantic text is determined based on the feature similarity corresponding to each target text encoding vector.

9. An image forgery detection device, characterized in that, The image forgery detection device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image forgery detection method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the image forgery detection method as described in any one of claims 1 to 8.

Images