Image authenticity detection method and device

Abstract

The invention discloses an image authenticity detection method, and the method comprises the steps: obtaining a forged image sample generated by an image generation model, and intermediate state information generated by the image generation model in the process of generating the forged image sample; the real image sample and the forged image sample are used as training data to be input into the detection model for multi-task learning training; the multiple tasks comprise a classification task for detecting whether an input image sample is a forged image sample or not; and reconstructing a reconstruction task of the intermediate state information, calculating classification loss of the classification task and reconstruction loss of the reconstruction task; the reconstruction loss of the reconstruction task is the loss generated by learning training by taking intermediate state information generated by the image generation model as a self-supervised learning target; taking the classification loss and the reconstruction loss as model loss, and performing parameter adjustment on model parameters contained in the detection model; wherein the trained detection model is used for detecting whether the input image is a forged image or not.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of artificial intelligence, and more particularly, to a method and device for detecting the authenticity of an image. Background Art

[0002] In recent years, with the emergence of more and more artificial intelligence models with powerful image generation capabilities, significant technological breakthroughs have occurred in the field of image generation. For example, with the widespread application of GAN (Generative Adversarial Network) models in the image field, significant results have been achieved in application scenarios such as image synthesis, image editing, image style transfer, and image data enhancement.

[0003] With the continuous development of artificial intelligence technology, especially the emergence of the AIGC (AI Generated Content) model based on deep learning, its creative image generation capabilities have far surpassed the traditional GAN ​​model in terms of image detail processing and image authenticity.

[0004] However, technological breakthroughs will inevitably lead to some new attack methods, which may lead to huge security risks in some fields. For example, taking facial images as an example, some black industries can use the AIGC model to generate a fake facial image with facial movements, attributes, and style that are completely similar to the real facial image by inputting conditional prompts, which puts some face-based security verification systems at risk.

[0005] It can be seen that how to defend against this new type of attack method derived from technological breakthroughs is the direction that needs to be explored at present. Summary of the invention

[0006] This specification proposes a method for detecting the authenticity of an image, the method comprising:

[0007] Acquire a forged image sample generated by an image generation model and corresponding to a real image sample, and intermediate state information generated by the image generation model in a process of generating the forged image sample;

[0008] The real image samples and the forged image samples are used as training data and input into a preset detection model for multi-task learning training; wherein the multi-task includes a classification task of detecting whether the image samples input into the detection model are forged image samples generated by the image generation model; and a reconstruction task of reconstructing the intermediate state information based on the image information contained in the forged image samples;

[0009] Calculating the classification loss of the classification task and the reconstruction loss of the reconstruction task; wherein the reconstruction loss of the reconstruction task is the loss generated by taking the intermediate state information generated by the image generation model as a self-supervised learning target for learning and training;

[0010] The classification loss and the reconstruction loss are used as model losses, and the model parameters contained in the detection model are adjusted to complete multi-task learning and training for the detection model; wherein the trained detection model is used to detect whether an input image is a forged image.

[0011] Optionally, the detection model includes a model based on deep learning; and the image generation model includes an AIGC model based on deep learning.

[0012] Optionally, the image generation model includes a diffusion model.

[0013] Optionally, the intermediate state information includes: an image latent code obtained by encoding the real image sample into a latent space supported by the diffusion model; and a conditional information code obtained by encoding semantic information extracted from conditional information input to the image generation model; wherein the conditional information is generation conditional information corresponding to the forged image sample;

[0014] The reconstruction task includes: a first reconstruction task of reconstructing the image hidden code; and a second reconstruction task of reconstructing the conditional information code; accordingly, the reconstruction loss includes a first reconstruction loss corresponding to the first reconstruction task; and a second reconstruction loss corresponding to the second reconstruction task.

[0015] Optionally, the image hidden code includes: the image hidden code includes, based on the image hidden code obtained after the diffusion model performs forward noise addition on the image hidden code, continuing to perform a plurality of reverse denoising steps on the image hidden code to output the image hidden code; or,

[0016] Based on the image hidden code obtained after forward noise addition is performed on the image hidden code, the image hidden code outputted from the last step of the multiple reverse denoising steps performed on the image hidden code is continued.

[0017] Optionally, the first reconstruction loss includes the similarity between the image latent code reconstructed by the detection model and the image latent code generated by the image generation model; the second reconstruction loss includes the similarity between the conditional information code reconstructed by the detection model and the conditional information code generated by the image generation model.

[0018] Optionally, the image includes a face image; the condition information includes condition information in text form. Optionally, the image generation model includes a stable diffusion model.

[0019] This specification also proposes a method for detecting the authenticity of an image, the method comprising:

[0020] Acquire the target image to be detected;

[0021] The target image is input into a preset detection model; wherein the detection model is a model obtained by performing multi-task learning and training using real image samples and forged image samples generated by an image generation model and corresponding to the real image samples as training data; the multi-task includes a classification task of detecting whether the image input into the detection model is a forged image generated by the image generation model; and a reconstruction task of reconstructing the intermediate state information generated by the image generation model in the process of generating the forged image sample based on the image information contained in the forged image sample; the model loss of the detection model includes the classification loss of the classification task and the reconstruction loss of the reconstruction task; the reconstruction loss of the reconstruction task is the loss generated by performing learning and training using the intermediate state information generated by the image generation model in the process of generating the forged image sample as a self-supervised learning target;

[0022] A classification result corresponding to the classification task output by the detection model is obtained, and based on the classification result, it is confirmed whether the target image is a forged image generated by the image generation model.

[0023] This specification also proposes an image authenticity detection device, the device comprising:

[0024] A first acquisition module is used to acquire a forged image sample generated by an image generation model and corresponding to a real image sample, as well as intermediate state information generated by the image generation model in a process of generating the forged image sample;

[0025] A training module, which uses the real image samples and the forged image samples as training data and inputs them into a preset detection model for multi-task learning training; wherein the multi-tasks include a classification task of detecting whether the image samples input into the detection model are forged image samples generated by the image generation model; and a reconstruction task of reconstructing the intermediate state information based on the image information contained in the forged image samples;

[0026] A calculation module calculates the classification loss of the classification task and the reconstruction loss of the reconstruction task; wherein the reconstruction loss of the reconstruction task is the loss generated by taking the intermediate state information generated by the image generation model as a self-supervised learning target for learning and training;

[0027] The adjustment module uses the classification loss and the reconstruction loss as model losses and performs parameter adjustment on the model parameters contained in the detection model to complete multi-task learning and training for the detection model; wherein the trained detection model is used to detect whether an input image is a forged image.

[0028] This specification also proposes a device for detecting the authenticity of an image, the device comprising:

[0029] The second acquisition module acquires the target image to be detected;

[0030] An input module inputs the target image into a preset detection model; wherein the detection model is a model obtained by multi-task learning and training using real image samples and forged image samples generated by an image generation model and corresponding to the real image samples as training data; the multi-task includes a classification task of detecting whether the image input into the detection model is a forged image generated by the image generation model; and a reconstruction task of reconstructing the intermediate state information generated by the image generation model in the process of generating the forged image samples based on the image information contained in the forged image samples; the model loss of the detection model includes the classification loss of the classification task and the reconstruction loss of the reconstruction task; the reconstruction loss of the reconstruction task is the loss generated by learning and training using the intermediate state information generated by the image generation model in the process of generating the forged image samples as a self-supervised learning target;

[0031] A confirmation module obtains a classification result corresponding to the classification task output by the detection model, and confirms whether the target image is a forged image generated by the image generation model based on the classification result.

[0032] In the above embodiment, during the process of training the detection model, the intermediate state information generated by the image generation model in the process of generating a forged image corresponding to the real image is introduced as a self-supervised learning target, so that the trained detection model can mine more implicit tampering clues from the image to be detected, thereby improving the detection model's ability to detect forged images generated by the image generation model, and providing effective security protection against attack behaviors initiated by forged images generated by the image generation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0034] Figure 1 is a flow chart of a method for detecting the authenticity of an image shown in an embodiment of this specification;

[0035] Figure 2 is a schematic diagram of generating a forged image sample based on a Stable Diffusion model shown in an embodiment of this specification;

[0036] Figure 3 is a schematic diagram of performing multi-task learning and training on a detection model shown in an embodiment of this specification;

[0037] Figure 4 is a flow chart of another method for detecting authenticity of an image shown in an embodiment of this specification;

[0038] Figure 5 is a schematic structural diagram of an electronic device shown in an embodiment of this specification;

[0039] Figure 6 is a block diagram of an image authenticity detection device shown in an embodiment of this specification;

[0040] Figure 7 It is a block diagram of another image authenticity detection device shown in an embodiment of this specification. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0042] This specification aims to propose a technical solution that introduces the intermediate state information generated by the image generation model in the process of generating forged images as a self-supervised learning target during the training of the detection model, so as to improve the detection model's ability to detect forged images generated by the image generation model.

[0043] See also Figure 1 , Figure 1 The flowchart of a method for detecting the authenticity of an image shown in this specification includes the following execution process:

[0044] Step 102, obtaining a forged image sample generated by an image generation model and corresponding to a real image sample, and intermediate state information generated by the image generation model in a process of generating the forged image sample;

[0045] The above-mentioned image generation model may include any form of artificial intelligence model with image generation capabilities.

[0046] In one embodiment shown, the above-mentioned image generation model can specifically be an AIGC model based on deep learning.

[0047] It needs to be explained that AIGC models generally refer to artificial intelligence models that use artificial intelligence technology to generate various types of content. Among them, the content that the AIGC model can generate includes but is not limited to text, images, audio, video, etc. These contents can be generated completely automatically by the AIGC model, or they can be generated with a small amount of input or guidance provided by humans. The AIGC model usually contains a variety of different types of model variants. In practical applications, any one of these model variants can be selected as the above-mentioned image generation model.

[0048] For example, the Diffusion model is a specific AIGC model for generating images. In practical applications, the Diffusion model can be used as the above-mentioned image generation model.

[0049] It should be noted that the Diffusion model usually includes a variety of different types of model variants. For example, the common Diffusion model may include the Stable Diffusion model, the Standard Diffusion model, the Conditional Diffusion model, the Guided Diffusion model, etc. In practical applications, any one of the above model variants can be selected as the above image generation model. For example, in one example, the Stable Diffusion model, which is the most widely used in the Diffusion model series, can be selected as the above image generation model.

[0050] The above-mentioned image samples may specifically include images in any form; for example, in one example, the above-mentioned image samples may specifically include face images.

[0051] In this specification, in order to deal with attacks based on forged images generated by the above image generation model, a detection model for detecting whether an input image is a forged image generated by the above image generation model can be trained.

[0052] The specific type of the detection model is not particularly limited in this specification; for example, in one example, the detection model can also be an artificial intelligence model based on deep learning.

[0053] When training the detection model, a real image sample set can be prepared in advance as white samples.

[0054] In addition, the real image samples in the real image sample set can be input into the above-mentioned image generation model, and the image generation model can generate forged graphic samples corresponding to these real image samples, and the forged image samples corresponding to the above-mentioned real image samples generated by the above-mentioned image generation model can be obtained as black samples when training the detection model.

[0055] For example, see Figure 2 Taking the above-mentioned image generation model as a Stable Diffusion model as an example, the process of training the Stable Diffusion model usually includes a forward diffusion process and a reverse generation process. Among them, the forward diffusion process is mainly used to randomly add noise to the input image sample; and the reverse generation process is also called the reverse denoising process, which usually includes multiple denoising steps to gradually remove the noise on the image sample.

[0056] like Figure 2 As shown, when the Stable Diffusion model is trained, a real image sample X can be input into the Stable Diffusion model, and the encoder in the Stable Diffusion model will encode the real image sample X into the latent space supported by the Stable Diffusion model to obtain an image latent code Z; wherein, the specific details of encoding the real image sample X into the latent space supported by the Stable Diffusion model will not be described in detail in this specification.

[0057] After encoding the real image sample X into the latent space supported by the Stable Diffusion model to obtain an image latent code Z, the forward diffusion process of the Stable Diffusion model can be executed to add random noise to the image latent code to obtain the image latent code Z T .

[0058] In addition, the input generation condition information used to guide the Stable Diffusion model to generate forged image samples can also be obtained, and the semantic information extracted from these generation condition information is encoded to obtain the condition information encoding τ θ , and then encode the conditional information τ θ And the hidden code Z of the imageT Together they serve as input data for the subsequent reverse generation process.

[0059] In practical applications, these generated condition information may include condition information in the form of semantic map, text, feature representation, image, etc. For example, in one example, taking the above-mentioned generated condition information in the form of text as an example, the Stable Diffusion model can encode the semantic information extracted from the condition information in the form of text into a text information encoding τ based on the integrated language model. θ ; Among them, the language model integrated by the Stable Diffusion model can usually be a CLIP (Contrastive Language-Image Pre-training) model.

[0060] After completing the above forward diffusion process, you can continue to execute the multiple denoising steps included in the reverse generation process of the Stable Diffusion model to gradually remove the image hidden code Z T The noise contained in is obtained to obtain the image hidden code Z. For example, see Figure 2 In the reverse generation process of the Stable Diffusion model, multiple U-Nets are usually introduced to predict the denoising amount for each denoising step. Then, each denoising step can remove the noise contained in the above image latent code according to the denoising amount predicted by the corresponding U-Net.

[0061] Please continue to see Figure 2 , when the hidden code Z for the image is completed T After the denoising process, the decoder in the Stable Diffusion model decodes the hidden code Z of the image that has completed the denoising process back to the pixel space of the original image sample to generate the real image sample X T .

[0062] Finally, the real image sample X generated by the above denoising process can be obtained based on the real image sample X and the Stable Diffusion model. T , to calculate the training loss of the model, and adjust the model parameters contained in the Stable Diffusion model according to the training loss. Subsequently, the training of the Stable Diffusion model can be completed by iteratively executing the above process.

[0063] After the Stable Diffusion model is trained, when generating forged image samples based on the Stable Diffusion model, the pre-prepared noise image and the generation condition information for guiding the Stable Diffusion model to generate forged image samples can be input into the Stable Diffusion model, and the Stable Diffusion model will implicitly encode the noise image into the latent space supported by the Stable Diffusion model to obtain an image implicit code Z T , and the semantic information extracted from the generated conditional information is also encoded into the conditional information code.

[0064] Furthermore, by executing multiple denoising steps included in the reverse generation process of the Stable Diffusion model, under the guidance of the above-mentioned conditional information coding, the noise contained in the noisy image can be gradually removed to obtain the denoised image hidden code Z, and then the decoder in the Stable Diffusion model will decode the image hidden code that has completed the denoising process back to the pixel space of the original image sample to generate the final forged image sample. Then, when training the above-mentioned detection model, the forged image sample corresponding to the real image sample generated by the Stable Diffusion model can be obtained as a black sample when training the detection model.

[0065] In this specification, in addition to obtaining the forged image samples generated by the above-mentioned image generation model as black samples when training the detection model, the intermediate state information generated by the image generation model in the process of generating the forged image samples can also be obtained.

[0066] The intermediate state information may specifically include all forms of information generated by the image generation model in the intermediate process of generating the forged image sample.

[0067] In an illustrated embodiment, taking the image generation model as a diffusion model as an example, the intermediate state information may specifically include an image latent code obtained by encoding a real image sample into a latent space supported by the diffusion model.

[0068] Among them, since the reverse generation process in the diffusion model usually includes multiple denoising steps, and each denoising step will generate an image latent code that has been encoded in the latent space supported by the diffusion model; therefore, in practical applications, if the image latent code encoded in the latent space supported by the diffusion model is required to be used as the intermediate state information, the image latent code output by the appropriate step can be flexibly selected from the above-mentioned multiple denoising steps as the above-mentioned intermediate state information.

[0069] For example, see Figure 2 , still taking the above-mentioned image generation model as a Stable Diffusion model as an example, in one case, the encoder in the Stable Diffusion model encodes the input real image sample X into the latent space supported by the Stable Diffusion model to obtain an image latent code Z, and then the image latent code Z can be obtained after forward noise addition is performed on the image latent code Z. T Based on this, it is preferred to continue to encode the image hidden code Z T The last step of performing multiple reverse denoising steps outputs the image hidden code Z (see Figure 2 The intermediate state information is represented by .

[0070] In another case, the encoder in the Stable Diffusion model encodes the input real image sample X into the latent space supported by the Stable Diffusion model to obtain an image latent code Z, and then performs forward noise addition on the image latent code Z to obtain the image latent code Z. T Based on this, we choose to continue to encode the image Z T The image hidden code outputted by any of the multiple reverse denoising steps performed is used as the intermediate state information; for example, assuming that the image hidden code Z is selected T The image hidden code outputted in the T-1th step of the multiple reverse denoising steps performed is used as the intermediate state information, and the image hidden code Z outputted in the T-1th step can be obtained. T-1 As intermediate information.

[0071] It should be noted that, in practical applications, when the hidden code Z for the image is T When selecting the image latent code output by a certain step from the multiple reverse denoising steps as the intermediate state information, a random sampling algorithm can be specifically run to randomly select a step from the multiple reverse denoising steps.

[0072] In another embodiment shown, still taking the above-mentioned image generation model as a diffusion model as an example, the above-mentioned intermediate state information may further specifically include conditional information encoding obtained by encoding semantic information extracted from the conditional information input to the image generation model.

[0073] For example, see Figure 2 , still taking the above image generation model as the Stable Diffusion model as an example, you can choose Figure 2 The conditional information encoding τ generated in θ As the intermediate state information.

[0074] Step 104: input the real image sample and the forged image sample as training data into a preset detection model for multi-task learning training; wherein the multi-task includes a classification task of detecting whether the image sample input into the detection model is a forged image sample generated by the image generation model; and a reconstruction task of reconstructing the intermediate state information based on the image information contained in the forged image sample;

[0075] After obtaining the forged image samples generated by the above-mentioned image generation model and corresponding to the real image samples in the above-mentioned real image sample set, the pre-prepared real image sample set can be used as white samples for training the detection model, and the forged image samples generated by the above-mentioned image generation model and corresponding to the real image samples in the real image sample set can be used as black samples for training the detection model, and input into the above-mentioned detection model to train the detection model.

[0076] In order to improve the detection model's ability to detect forged images generated by the image generation model, so that the detection model can effectively protect against attacks initiated by forged images generated by the image generation model, the detection model can be a multi-task model. In this case, the training of the detection model can be multi-task training.

[0077] Among them, the multi-task may specifically include a classification task of detecting whether the image sample input to the detection model is a forged image sample generated by the image generation model; and a reconstruction task of reconstructing the above-mentioned intermediate state information generated by the image generation model in the process of generating the forged image sample based on the image information contained in the input forged image sample.

[0078] That is to say, in addition to having the model capability to detect whether the image sample input to the detection model is a forged image sample generated by the image generation model, the multi-task model can also have the model capability to reconstruct the above-mentioned intermediate state information generated by the image generation model in the process of generating the forged image sample based on the image information contained in the input forged image sample.

[0079] For example, taking the above-mentioned intermediate state information including the image latent code obtained by encoding the real image sample into the latent space supported by the diffusion model, and the conditional information code obtained by encoding the semantic information extracted from the conditional information input to the image generation model as an example, in this scenario, the above-mentioned reconstruction task may specifically include a first reconstruction task of reconstructing the image latent code; and a second reconstruction task of reconstructing the conditional information code.

[0080] It should be noted that the model type of the above-mentioned detection model and the model architecture adopted by the model are not specifically limited in this specification.

[0081] In one embodiment shown, the above detection model may specifically be an artificial intelligence model based on deep learning.

[0082] For example, in practical applications, the detection model can specifically adopt the architecture of CNN (Convolutional Neural Networks), which can specifically include an input layer, several convolutional layers, a pooling layer, a fully connected layer and three different output layers. Among them, the three different output layers can correspond to three different outputs, one of which can correspond to the classification result of the upper classification task output by the detection model, another can correspond to the reconstruction result of the above-mentioned first reconstruction task output by the detection model, and the third can correspond to the reconstruction result of the above-mentioned second reconstruction task output by the detection model.

[0083] Step 106, calculating the classification loss of the classification task and the reconstruction loss of the reconstruction task; wherein the reconstruction loss of the reconstruction task is the loss generated by taking the intermediate state information generated by the image generation model as a self-supervised learning target for learning and training;

[0084] In the process of inputting the above-mentioned white samples and black samples into the above-mentioned detection model and performing multi-task learning and training on the detection model, on the one hand, the classification results of the classification tasks output by the detection model and included in the above-mentioned multi-tasks can also be obtained, and the classification loss of the classification task can be further calculated based on the classification results.

[0085] For example, in practical applications, the above classification task may specifically be a binary classification task, and the classification loss may specifically be calculated using a cross entropy loss function (CEloss) commonly used in binary classification tasks.

[0086] On the other hand, the reconstruction result corresponding to the above-mentioned reconstruction task output by the detection model can also be obtained, and the reconstruction loss of the reconstruction task can be further calculated based on the reconstruction result.

[0087] In one embodiment shown, the intermediate state information includes the image latent code obtained by encoding the real image sample into the latent space supported by the diffusion model, and the conditional information code obtained by encoding the semantic information extracted from the conditional information input to the image generation model. In this scenario, the reconstruction task may specifically include a first reconstruction task for reconstructing the image latent code; and a second reconstruction task for reconstructing the conditional information code. The reconstruction loss may include a first reconstruction loss corresponding to the first reconstruction task; and a second reconstruction loss corresponding to the second reconstruction task.

[0088] For example, in practical applications, the first reconstruction loss can be specifically represented by the similarity between the image latent code reconstructed by the detection model and the image latent code generated by the image generation model; the second reconstruction loss can be specifically represented by the similarity between the conditional information code reconstructed by the detection model and the conditional information code generated by the image generation model.

[0089] The specific measurement method of the above similarity is not particularly limited in this specification.

[0090] For example, in one example, the dot product between two codes can be used to represent the similarity between the two codes. In another example, the vector distance between the vectors corresponding to the two codes can also be used to represent the similarity between the two codes. It should be noted that, unlike the above-mentioned classification task, in the process of learning and training for the above-mentioned reconstruction task, the intermediate state information obtained by the above-mentioned image generation model in the process of generating forged image samples can be used as a self-supervised learning target. In this case, the reconstruction loss can specifically be the model loss generated by self-supervised learning training using the intermediate state information obtained by the above-mentioned image generation model in the process of generating forged image samples as a self-supervised learning target. For example, please refer to Figure 3 , still taking the above image generation model as the Stable Diffusion model, the above intermediate state information includes the image latent code obtained by encoding the real image sample into the latent space supported by the diffusion model, and the conditional information code obtained by encoding the semantic information extracted from the conditional information input to the image generation model. In this scenario, the image latent code Z and the conditional information code τ generated by the Stable Diffusion model in the process of generating fake image samples can be taken as examples. θ As a self-supervised item, it continues to be input into the detection model.

[0091] In the process of multi-task learning and training of the detection model using the above-mentioned black samples and the above-mentioned white samples as training data, for the binary classification task included in the multi-task, the classification loss can be calculated using the cross entropy loss function commonly used in the binary classification task, and for the above-mentioned first reconstruction task included in the multi-task, the similarity between the image latent code reconstructed by the detection model and the image latent code generated by the above-mentioned image generation model as a self-supervisory item can be calculated as the reconstruction loss of the first reconstruction task; for the above-mentioned second reconstruction task included in the multi-task, the similarity between the conditional information code reconstructed by the detection model and the conditional information code generated by the above-mentioned image generation model as a self-supervisory item can be calculated as the reconstruction loss of the second reconstruction task.

[0092] Step 108, using the classification loss and the reconstruction loss as model losses, and adjusting the model parameters contained in the detection model to complete multi-task learning and training for the detection model; wherein the trained detection model is used to detect whether the input face image is a forged image.

[0093] After calculating the classification loss of the above classification task and the reconstruction loss of the above reconstruction task, these two parts of loss can be further used as model losses for multi-task learning training of the detection model to adjust the model parameters contained in the detection model.

[0094] For example, still taking the above intermediate state information including the image latent code obtained by encoding the real image sample into the latent space supported by the diffusion model, and the conditional information code obtained by encoding the semantic information extracted from the conditional information input to the image generation model as an example, in this scenario, the calculated model loss includes three parts: the classification loss of the above classification task, the reconstruction loss of the above first reconstruction task, and the reconstruction loss of the above second reconstruction task. Assuming that the classification loss of the above classification task is represented by L1, the reconstruction loss of the above first reconstruction task is represented by L2, and the reconstruction loss of the above second reconstruction task is represented by L3, these three parts of loss can be weighted and calculated to obtain the model loss of the detection model.

[0095] Assume that the model loss of the detection model is represented by L, then L can be expressed as follows:

[0096] L = w1*L1+w2*L2+w3*L3;

[0097] It should be noted that w1, w2, and w3 are weighted coefficients of the losses of each part, and can be a decimal between 0 and 1 respectively, and the sum of w1, w2, and w3 can be equal to 1.

[0098] It should be noted that a parameter adjustment is performed on the model parameters contained in the detection model based on the above model loss, which is usually called a round of training. Subsequently, the above process can be iteratively executed to complete the multi-task learning and training of the detection model.

[0099] After the above detection model is trained, the trained detection model can be used to detect whether the image input to the detection model is a forged image generated by the above image generation model.

[0100] See also Figure 4 , Figure 4 The flowchart of a method for detecting the authenticity of an image shown in this specification includes the following execution process:

[0101] Step 402, obtaining a target image to be detected;

[0102] Step 404, inputting the target image into a preset detection model; wherein the detection model is a model obtained by performing multi-task learning and training using real image samples and forged image samples generated by an image generation model and corresponding to the real image samples as training data; the multi-task includes a classification task of detecting whether the image input into the detection model is a forged image generated by the image generation model; and a reconstruction task of reconstructing intermediate state information generated by the image generation model in the process of generating the forged image samples based on image information contained in the forged image samples; the model loss of the detection model includes the classification loss of the classification task and the reconstruction loss of the reconstruction task; the reconstruction loss of the reconstruction task is the loss generated by performing learning and training using the intermediate state information generated by the image generation model in the process of generating the forged image samples as a self-supervised learning target;

[0103] When using the trained detection model, a target image to be detected can be obtained, and then the target image can be input into the trained detection model, and the detection model can be used to confirm whether the target image is a forged image generated by the above-mentioned image generation model. The training process of the above-mentioned detection model can be specifically referred to in the previous embodiment, and will not be repeated in this embodiment.

[0104] Step 406: Obtain a classification result corresponding to the classification task output by the detection model, and confirm whether the target image is a forged image generated by the image generation model based on the classification result.

[0105] Among them, although the detection model is a multi-task model, when using the detection model, we can only focus on the output results of the detection model and the classification task included in the multi-task, and ignore the output results of the reconstruction task other than the classification task included in the multi-task.

[0106] In this case, the classification result corresponding to the above classification task output by the detection model can be further obtained, and then based on the classification result, it can be confirmed whether the target image is a forged image generated by the image generation model.

[0107] For example, taking the above classification task as a binary classification task, the classification result 1 can be used to indicate that the target image is a forged image generated by the image generation model, and the classification result 0 can be used to indicate that the target image is not a forged image generated by the image generation model. At this time, if the classification result corresponding to the above classification task output by the detection model obtained is 1, it can be confirmed that the target image is a forged image generated by the image generation model; conversely, if the classification result corresponding to the above classification task output by the detection model obtained is 0, it can be confirmed that the target image is a real image, and not a forged image generated by the image generation model.

[0108] In the above technical scheme, in the process of training the detection model, the intermediate state information generated by the image generation model in the process of generating a forged image corresponding to the real image is introduced as a self-supervised learning target, so that the trained detection model can mine more implicit tampering clues from the image to be detected, thereby improving the detection model's detection ability for the forged image generated by the image generation model, and providing effective security protection against attack behaviors initiated by the forged image generated by the image generation model.

[0109] For example, although when using the detection model, we only need to pay attention to the output results of the detection model and the classification task included in the above-mentioned multi-tasks, and ignore the output results of other tasks included in the multi-tasks except the classification task, but in the process of multi-task learning of the detection model, the intermediate state information generated by the above-mentioned image generation model in the process of generating a forged image corresponding to the real image is introduced as a self-supervised learning target, which makes the final model loss can cover the reconstruction loss of the detection model trying to reconstruct these intermediate state information; therefore, after multi-task learning training of the detection model in this way, the detection model can learn from the training data the generation rules of the above-mentioned image generation model in the process of generating forged images, so that when the detection model is applied to the authenticity detection of the image, more implicit tampering clues can be mined from the image to be detected, thereby improving the detection model's detection and recognition capabilities for forged images generated by the image generation model.

[0110] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of an apparatus, an electronic device, and a storage medium.

[0111] Figure 5 is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Figure 5 At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and may also include other required hardware. One or more embodiments of this specification may be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 508 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or logic devices.

[0112] like Figure 6 As shown, Figure 6 is a block diagram of an image authenticity detection device according to an exemplary embodiment of the present specification. The device can be operated in Figure 5 In the electronic device shown in the figure, the technical solution of this specification is implemented. The device 600 includes:

[0113] A first acquisition module 601 is configured to acquire a forged image sample generated by an image generation model and corresponding to a real image sample, and intermediate state information generated by the image generation model in a process of generating the forged image sample;

[0114] A training module 602 uses the real image samples and the forged image samples as training data and inputs them into a preset detection model for multi-task learning training; wherein the multi-tasks include a classification task of detecting whether the image samples input into the detection model are forged image samples generated by the image generation model; and a reconstruction task of reconstructing the intermediate state information based on the image information contained in the forged image samples;

[0115] The calculation module 603 calculates the classification loss of the classification task and the reconstruction loss of the reconstruction task; wherein the reconstruction loss of the reconstruction task is the loss generated by taking the intermediate state information generated by the image generation model as a self-supervised learning target for learning and training;

[0116] The adjustment module 604 uses the classification loss and the reconstruction loss as model losses, and performs parameter adjustment on the model parameters included in the detection model to complete multi-task learning and training for the detection model; wherein the trained detection model is used to detect whether the input image is a forged image.

[0117] like Figure 7 As shown, Figure 6 is a block diagram of another image authenticity detection device according to an exemplary embodiment of the present specification. The device can also be operated in the following manner: Figure 5 In the electronic device shown in the figure, the technical solution of this specification is implemented. The device 700 includes:

[0118] The second acquisition module 701 acquires the target image to be detected;

[0119] The input module 702 inputs the target image into a preset detection model; wherein the detection model is a model obtained by multi-task learning and training using real image samples and forged image samples generated by an image generation model and corresponding to the real image samples as training data; the multi-task includes a classification task of detecting whether the image input into the detection model is a forged image generated by the image generation model; and a reconstruction task of reconstructing the intermediate state information generated by the image generation model in the process of generating the forged image sample based on the image information contained in the forged image sample; the model loss of the detection model includes the classification loss of the classification task and the reconstruction loss of the reconstruction task; the reconstruction loss of the reconstruction task is the loss generated by learning and training using the intermediate state information generated by the image generation model in the process of generating the forged image sample as a self-supervised learning target;

[0120] The confirmation module 703 obtains the classification result corresponding to the classification task output by the detection model, and confirms whether the target image is a forged image generated by the image generation model based on the classification result.

[0121] Accordingly, the present specification also provides an electronic device, which includes a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the steps in all the method flows described above.

[0122] Accordingly, the present specification also provides a computer-readable storage medium on which executable computer program instructions are stored; wherein, when the instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0123] Accordingly, the present specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0124] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0125] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0126] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, the present application does not exclude that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0127] Although one or more embodiments of the present specification provide method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0128] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0129] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0132] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0133] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0134] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0135] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0137] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0138] The above description is only an example of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included in the scope of the claims.

Claims

1. A method for detecting the authenticity of an image, the method comprising: Acquire a forged image sample generated by an image generation model and corresponding to a real image sample, and intermediate state information generated by the image generation model in a process of generating the forged image sample; The real image samples and the forged image samples are used as training data and input into a preset detection model for multi-task learning training; wherein the multi-task includes a classification task of detecting whether the image samples input into the detection model are forged image samples generated by the image generation model; and a reconstruction task of reconstructing the intermediate state information based on the image information contained in the forged image samples; Calculating the classification loss of the classification task and the reconstruction loss of the reconstruction task; wherein the reconstruction loss of the reconstruction task is the loss generated by taking the intermediate state information generated by the image generation model as a self-supervised learning target for learning and training; The classification loss and the reconstruction loss are used as model losses, and the model parameters contained in the detection model are adjusted to complete multi-task learning and training for the detection model; wherein the trained detection model is used to detect whether an input image is a forged image.

2. According to the method as claimed in claim 1, the detection model includes a model based on deep learning; the image generation model includes an AIGC model based on deep learning.

3. The method of claim 2, wherein the image generation model comprises a diffusion model; The intermediate state information includes: Encoding the real image sample into the latent space supported by the diffusion model to obtain an image latent code; and encoding the conditional information obtained by encoding the semantic information extracted from the conditional information input to the image generation model; wherein the conditional information is generation conditional information corresponding to the forged image sample; The reconstruction task includes: a first reconstruction task of reconstructing the image hidden code; and a second reconstruction task of reconstructing the conditional information code; accordingly, the reconstruction loss includes a first reconstruction loss corresponding to the first reconstruction task; and a second reconstruction loss corresponding to the second reconstruction task.

4. The method according to claim 3, wherein the image hidden code comprises: the diffusion model further performs a plurality of reverse denoising steps on the image hidden code to output the image hidden code based on the image hidden code obtained after forward denoising is performed on the image hidden code; or Based on the image hidden code obtained after forward noise addition is performed on the image hidden code, the image hidden code outputted from the last step of the multiple reverse denoising steps performed on the image hidden code is continued.

5. In the method as claimed in claim 3, the first reconstruction loss includes the similarity between the image latent code reconstructed by the detection model and the image latent code generated by the image generation model; the second reconstruction loss includes the similarity between the conditional information code reconstructed by the detection model and the conditional information code generated by the image generation model.

6. The method according to claim 5, wherein the image comprises a face image; and the condition information comprises condition information in text form.

7. A method for detecting the authenticity of an image, the method comprising: Acquire the target image to be detected; The target image is input into a preset detection model; wherein the detection model is a model obtained by performing multi-task learning and training using real image samples and forged image samples generated by an image generation model and corresponding to the real image samples as training data; the multi-task includes a classification task of detecting whether the image input into the detection model is a forged image generated by the image generation model; and a reconstruction task of reconstructing the intermediate state information generated by the image generation model in the process of generating the forged image sample based on the image information contained in the forged image sample; the model loss of the detection model includes the classification loss of the classification task and the reconstruction loss of the reconstruction task; the reconstruction loss of the reconstruction task is the loss generated by performing learning and training using the intermediate state information generated by the image generation model in the process of generating the forged image sample as a self-supervised learning target; A classification result corresponding to the classification task output by the detection model is obtained, and based on the classification result, it is confirmed whether the target image is a forged image generated by the image generation model.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images