General AI generated image detection method and system based on few-sample learning

Abstract

The invention discloses a general AI generated image detection method and system based on few sample learning, the method comprises a training stage and a testing stage, and in order to solve the problem that a current detection method depends on a large number of training samples, a counterfeit sensing module based on low-rank adaptation is designed for the training stage of the method; the feature extraction model can be suitable for AI generated image detection by using a small number of training samples, and meanwhile, overfitting is avoided. Besides, the invention also designs a comparative learning strategy based on semantic feature guidance, so that the counterfeit perception module based on low-rank adaptation learns the generalization between the real image and the AI generated image and also learns the difference between the real image and the AI generated image, thereby enabling the method to have better generalization ability.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and AI-generated image detection, and specifically to a general AI-generated image detection method and system based on few-sample learning. Background Art

[0002] In recent years, due to the rapid development of generative models such as generative adversarial networks (GAN) and diffusion models, the images generated by these AI models show extraordinary realism. However, the progress of generative AI has also raised concerns about the safety and privacy of human society. Therefore, there is a growing demand for the development of detection methods that can distinguish AI-generated images.

[0003] Existing AI-generated image detection algorithms rely on pre-trained deep learning models. Existing AI-generated image detection relies on a large number of training samples, which requires high training costs and training time. However, despite the use of a large number of training samples, its generalization is still insufficient, especially for images generated by generative models that are not visible during the training phase. At a time when AI image generation models are emerging in an endless stream, these methods cannot meet the requirements for effectively detecting AI-generated images. Summary of the invention

[0004] The purpose of the present invention is to provide a general AI generated image detection method and system based on few-sample learning to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] In a first aspect, the present application provides a general AI generated image detection method based on few-sample learning, the method comprising a training phase and a testing phase, the training phase comprising the following steps:

[0007] Step S11: extracting embedding features from the training images respectively through the guidance model and the feature extraction model;

[0008] Step S12: calculating a similarity score of an embedded feature pair consisting of embedded features extracted by the guide model and the feature extraction model;

[0009] Step S13: Calculate the label of the embedded feature pair;

[0010] Step S14: Calculate the loss function according to the similarity score of the embedded feature pair and the label of the embedded feature pair.

[0011] The testing phase includes the following steps:

[0012] Step S21: Extract the embedding features of the real image and the AI-generated image;

[0013] Step S22: Extract the embedding features of the image to be tested;

[0014] Step S23: Using the embedding features of the image to be tested, the real image, and the AI-generated image, calculate the cosine distance between the image to be tested and the real image, and the cosine distance between the image to be tested and the AI-generated image respectively;

[0015] Step S24: Obtain the prediction result based on the relationship of the cosine distances, that is, whether the image to be tested is an AI-generated image.

[0016] As a further solution of the present invention: The feature extraction model in Step S11 is obtained by designing a forgery perception module based on low-rank adaptation on the basis of the CLIP: ViT-L / 14 model.

[0017] As a further solution of the present invention: In the vision transformer module of the CLIP: ViT-L / 14 model, a low-rank adaptation technology is introduced to obtain the forgery perception module based on low-rank adaptation, which specifically includes:

[0018] In each multi-head attention module of the vision transformer module, a low-rank adaptation module with a rank of 2 is used to modify the query matrix, key matrix, value matrix, and output matrix of the multi-head attention module: Let represent any pre-trained matrix in the query matrix, key matrix, value matrix, and output matrix, and use low-rank decomposition to represent and constrain the update of W0, and the formula is as follows:

[0019] W0 + ΔW = W0 + BA,

[0020] where, Both A and B represent low-rank matrices, and the rank r of the low-rank matrices A and B < min(d, k); during the training process, the parameters of W0 are frozen and do not participate in the update, and the parameters of the low-rank matrices A and B are trainable.

[0021] As a further solution of the present invention: A contrast learning strategy based on semantic feature guidance is designed for the training stage.

[0022] As a further solution of the present invention: The contrast learning strategy based on semantic feature guidance specifically includes the following steps:

[0023] Step S31: Collect the labeled training data X, where X contains the real image x real and the AI-generated image x fake , the label of the training data X is Y, and the real image label contained in the label Y is y realand AI generates image labels as y fake ; where x fake Generated using a specific generative model;

[0024] Step S32: Two pre-trained CLIP:ViT models are trained, one of which is the guidance model G and the other is the feature extraction model T. They are enhanced using a forgery-aware module based on low-rank adaptation. In each training batch, N images {x i |1≤i≤N}∈X and the label corresponding to the image {y i |1≤i≤N}∈Y, N represents the training batch size, and then each image x in the current training batch i Send them to G and T respectively to get the embedded features extracted by the guided model G And the embedded features extracted by the feature extraction model T

[0025] Step S33: Based on and Generate N×N embedded feature pairs and assign a similarity score p to each embedded feature pair i,j , where p i,j pass Calculated, each p i,j The corresponding label l i,j =y i ⊙y j , ⊙ represents the XOR operation; where 1≤i≤N, 1≤j≤N;

[0026] Step S34: Using p i,j and l i,j , the loss function is calculated using the following formula

[0027]

[0028] In the formula, σ represents the sigmoid function, and τ is a learnable parameter used to adjust the similarity score p i,j to zoom in or out.

[0029] As a further solution of the present invention: in step S23 of the test phase, the embedding feature E of the image to be tested is extracted by the feature extractor t , E t Embedded features E of the real image r The cosine distance between them is d r , E t Embedded features E of AI-generated images f The cosine distance between them is d f , use the following formula to calculate dr and d f :

[0030]

[0031] As a further solution of the present invention: In the test phase step S24, the prediction result is calculated using the following formula:

[0032]

[0033] Prediction results , it means that the image to be tested is an AI-generated image; prediction result , it indicates that the image to be tested is a real image.

[0034] In a second aspect, the present invention further provides a general AI generated image detection system based on few-sample learning, comprising a training unit and a testing unit, wherein the training unit comprises:

[0035] The embedded feature extraction module extracts embedded features from the image through the guidance model and the feature extraction model respectively;

[0036] A similarity calculation module is used to calculate the similarity score of the embedded feature pair consisting of the embedded features extracted by the guidance model and the feature extraction model;

[0037] A label calculation module, which calculates the label of the embedded feature pair;

[0038] The loss calculation module calculates the model loss according to the similarity of the embedded feature pairs and the labels of the embedded feature pairs.

[0039] The test unit comprises:

[0040] Embedded feature extraction module 1, extracts embedded features of real images and AI-generated images;

[0041] Embedded feature extraction module 2, extracting embedded features of the image to be tested;

[0042] Cosine distance calculation module: This module uses the embedded features of the image to be tested, the real image and the AI ​​generated image to calculate the cosine distance between the image to be tested and the real image and the AI ​​generated image respectively;

[0043] Prediction result output module: obtains the prediction result according to the relationship of the cosine distance, that is, whether the image to be tested is an AI generated image.

[0044] The system of the present invention corresponds to the method, and the specific technical solutions applicable to the method are also applicable to the system.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention uses only a small number of generated images of a generative model for training, and can achieve effective detection of images generated by generative models that are invisible in multiple training stages. The present invention designs a forgery perception module based on low-rank adaptation, which can make the feature extraction model suitable for AI-generated image detection with a small number of training samples while avoiding overfitting. In addition, the present invention designs a contrastive learning strategy guided by semantic features, so that the forgery perception module based on low-rank adaptation can learn the commonalities between real images and AI-generated images while also learning the differences between real images and AI-generated images, thereby making the present invention have better generalization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The overall framework diagram of the training phase and the testing phase of the general AI generation image detection method of the present invention;

[0047] Figure 2 A flow chart of the training phase of the general AI image detection method of the present invention;

[0048] Figure 3 A flowchart of the testing phase of the general AI-generated image detection method of the present invention;

[0049] Figure 4 Flowchart of the forgery perception module based on low-rank adaptation designed for the present invention;

[0050] Figure 5 It is a feature space comparison diagram of the model CLIP:ViT-L / 14 and the feature extractor of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The embodiments of the present application provide a general AI generated image detection method and system based on few-sample learning, which realizes effective detection of images generated by generative models that are invisible in multiple training stages by using only a small number of generated images of one generative model for training.

[0053] Embodiment 1

[0054] This embodiment provides a general AI generated image detection method based on few-sample learning, such as Figure 1As shown, the method consists of a training phase and a testing phase. The fake images in the figure are AI-generated images.

[0055] The training phase is as Figure 2 shown and includes the following steps:

[0056] Step S11: Extract embedding features from the training images through a guiding model and a feature extraction model respectively;

[0057] Step S12: Calculate the similarity score of the embedding feature pairs composed of the embedding features extracted by the guiding model and the feature extraction model;

[0058] Step S13: Calculate the labels of the embedding feature pairs;

[0059] Step S14: Calculate the loss function according to the similarity score of the embedding feature pairs and the labels of the embedding feature pairs.

[0060] The feature extraction model in Step S11 is designed with a forgery perception module based on low-rank adaptation on the basis of the CLIP: ViT-L / 14 model. As Figure 4 shown, the low-rank adaptation algorithm is used for the multi-head attention modules of the last 12 ViT blocks in the CLIP: ViT model. In each multi-head attention module, a low-rank adaptation module with a rank of 2 is used to modify the query matrix, key matrix, value matrix, and output matrix of the multi-head attention module. Let represent any pre-trained matrix in the query matrix, key matrix, value matrix, and output matrix. Low-rank decomposition is used to represent and constrain the update of W0, and the formula is as follows:

[0061] W0 + ΔW = W0 + BA,

[0062] where both A and B represent low-rank matrices, and the rank r of the low-rank matrices A and B < min(d, k); during the training process, the parameters of W0 are frozen and do not participate in the update, and the parameters of the low-rank matrices A and B are trainable.

[0063] In addition, the embodiment of the present invention designs a contrastive learning strategy based on semantic feature guidance in the training phase. The contrastive learning strategy specifically includes the following steps:

[0064] Step S31: Collect the labeled training data X, where X contains the real image x real and the AI-generated image x fake , the label of the training data X is Y, and the real image label contained in the label Y is y real and the AI-generated image label is y fake ; where x fake is generated using a specific generation model;

[0065] Step S32: Two pre-trained CLIP:ViT models are trained, one of which is the guidance model G and the other is the feature extraction model T. They are enhanced using a forgery-aware module based on low-rank adaptation. In each training batch, N images {x i |1≤i≤N}∈X and the label corresponding to the image {y i |1≤i≤N}∈Y, N represents the training batch size, and then each image x in the current training batch i Send them to G and T respectively to get the embedded features extracted by the guided model G And the embedded features extracted by the feature extraction model T

[0066] Step S33: Based on and Generate N×N embedded feature pairs and assign a similarity score p to each embedded feature pair i,j , where p i,j pass Calculated, each p i,j The corresponding label l i,j =y i ⊙y j , ⊙ represents the XOR operation; where 1≤i≤N, 1≤j≤N;

[0067] Step S34: Using p i,j and l i,j , the loss function is calculated using the following formula

[0068]

[0069] in the formula It is the output result of converting the similarity of each positive and negative sample pair into probability. σ represents the sigmoid function, and the output range of this function is [0,1]. For sample pairs that are both positive or both negative, During the training process, it will gradually approach 1, and for a pair of samples where one is a positive sample and the other is a negative sample, It gradually approaches 0 during the training process; τ is a learnable parameter used to adjust the similarity p i,j Zoom in so that The output is smoothed, thus adjusting the model's sensitivity to variance.

[0070] The testing phase, such as Figure 3 As shown, the following steps are included:

[0071] Step S21: extracting embedded features of real images and AI-generated images;

[0072] Step S22: extracting embedded features of the image to be tested;

[0073] Step S23: using the embedded features of the image to be tested, the real image and the AI ​​generated image, respectively calculating the cosine distance between the image to be tested and the real image, and the cosine distance between the image to be tested and the AI ​​generated image;

[0074] Step S24: obtaining a prediction result according to the relationship of the cosine distance, that is, whether the image to be tested is an AI generated image.

[0075] In step S23 of the test phase, the embedding feature E of the image to be tested is extracted by the feature extractor. t , E t Embedded features E of the real image r The cosine distance between them is d r , E t Embedded features E of AI-generated images f The cosine distance between them is d f , use the following formula to calculate d r and d f :

[0076]

[0077] In step S24 of the test phase, the prediction result is calculated using the following formula:

[0078]

[0079] Prediction results , it means that the image to be tested is an AI-generated image; prediction result , it indicates that the image to be tested is a real image.

[0080] like Figure 5 As shown, the feature space of the CLIP:ViT-L / 14 model is compared with that of the feature extractor trained by the present invention. It can be seen from the figure that the feature extractor trained by the present method can better classify real images and AI generated images.

[0081] Based on the same inventive concept, the present application provides a system corresponding to the first embodiment, see the second embodiment for details.

[0082] Embodiment 2

[0083] This embodiment provides a general AI-generated image detection system based on few-sample learning, including a training unit and a testing unit. The training unit includes:

[0084] The embedded feature extraction module extracts embedded features from the image through the guidance model and the feature extraction model respectively;

[0085] A similarity calculation module is used to calculate the similarity score of the embedded feature pair consisting of the embedded features extracted by the guidance model and the feature extraction model;

[0086] A label calculation module, which calculates the label of the embedded feature pair;

[0087] The loss calculation module calculates the model loss according to the similarity of the embedded feature pairs and the labels of the embedded feature pairs.

[0088] The test unit comprises:

[0089] Embedded feature extraction module 1, extracts embedded features of real images and AI-generated images;

[0090] Embedded feature extraction module 2, extracting embedded features of the image to be tested;

[0091] Cosine distance calculation module: This module uses the embedded features of the image to be tested, the real image and the AI ​​generated image to calculate the cosine distance between the image to be tested and the real image and the AI ​​generated image respectively;

[0092] Prediction result output module: obtains the prediction result according to the relationship of the cosine distance, that is, whether the image to be tested is an AI generated image.

[0093] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0094] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A general AI generated image detection method based on few-shot learning, consisting of a training phase and a testing phase, characterized in that: The training phase includes the following steps: Step S11: extracting embedding features from the training images respectively through the guidance model and the feature extraction model; Step S12: calculating a similarity score of an embedded feature pair consisting of embedded features extracted by the guide model and the feature extraction model; Step S13: Calculate the label of the embedded feature pair; Step S14: Calculating a loss function according to the similarity score of the embedded feature pair and the label of the embedded feature pair; The testing phase includes the following steps: Step S21: extracting embedded features of real images and AI-generated images; Step S22: extracting embedded features of the image to be tested; Step S23: using the embedded features of the image to be tested, the real image and the AI ​​generated image, respectively calculating the cosine distance between the image to be tested and the real image, and the cosine distance between the image to be tested and the AI ​​generated image; Step S24: obtaining a prediction result according to the relationship of the cosine distance, that is, whether the image to be tested is an AI generated image.

2. The general AI generated image detection method based on few-sample learning according to claim 1, characterized in that: The feature extraction model in step S11 is obtained by designing a forgery perception module based on low-rank adaptation on the basis of the CLIP:ViT-L / 14 model.

3. The general AI generated image detection method based on few-sample learning according to claim 2, characterized in that: In the visual converter module of the CLIP:ViT-L / 14 model, low-rank adaptation technology is introduced to obtain the forgery perception module based on low-rank adaptation, which specifically includes: In each multi-head attention module of the visual converter module, a low-rank adaptation module with rank 2 is used to modify the query matrix, key matrix, value matrix, and output matrix of the multi-head attention module: Represents any pre-trained matrix among the query matrix, key matrix, value matrix, and output matrix, and uses low-rank decomposition to represent and constrain the update of W0. The formula is as follows: W0+ΔW=W0+BA, Among them, both A and B represent low-rank matrices, and the ranks r of the low-rank matrices A and B are less than min(d, k); during the training process, the parameters of W0 are frozen and do not participate in the update, and the parameters of the low-rank matrices A and B are trainable.

4. The general AI generated image detection method based on few-sample learning according to claim 1, characterized in that: A contrastive learning strategy guided by semantic features is designed for the training stage.

5. The general AI generated image detection method based on few-sample learning according to claim 4, characterized in that: The contrastive learning strategy based on semantic feature guidance specifically includes the following steps: Step S31: Collect labeled training data X, which contains real images x real and AI generated image x fake , the label of training data X is Y, the real image label contained in label Y is yreal and the AI ​​generated image label is yfake; where x fake Generated using a specific generative model; Step S32: Two pre-trained CLIP:ViT models are trained, one of which is the guidance model G and the other is the feature extraction model T. They are enhanced using a forgery-aware module based on low-rank adaptation. In each training batch, N images {x i |1≤i≤N}∈X and the label corresponding to the image {y i |1≤i≤N}∈Y, N represents the training batch size, and then each image x in the current training batch i Send them to G and T respectively to get the embedded features extracted by the guided model G And the embedded features extracted by the feature extraction model T Step S33: Based on and Generate NNN×NNN embedded feature pairs and assign a similarity score p to each embedded feature pair i，j , where p i，j pass Calculated, each p i，j The corresponding label l i，j =y i ⊙y j , ⊙ represents the XOR operation; where 1≤i≤NNN, 1≤j≤NNN; Step S34: Using p i，j and l i，j , the loss function is calculated using the following formula In the formula, σ represents the sigmoid function, and τ is a learnable parameter used to adjust the similarity score p i,j to zoom in or out.

6. The general AI generated image detection method based on few-sample learning according to claim 1, characterized in that: In step S23 of the test phase, the embedding feature E of the image to be tested is extracted by the feature extractor. t , E t Embedded features E of the real image r The cosine distance between them is d r , E t Embedded features E of AI-generated images f The cosine distance between them is d f , use the following formula to calculate d r and d f :

7. The general AI generated image detection method based on few-sample learning according to claim 6, characterized in that: In step S24 of the test phase, the prediction result is calculated using the following formula: Prediction results , it means that the image to be tested is an AI-generated image; prediction result , it indicates that the image to be tested is a real image.

8. A general AI generated image detection system based on few-shot learning, consisting of a training unit and a testing unit, characterized in that: The training unit comprises: The embedded feature extraction module extracts embedded features from the image through the guidance model and the feature extraction model respectively; A similarity calculation module is used to calculate the similarity score of the embedded feature pair consisting of the embedded features extracted by the guidance model and the feature extraction model; A label calculation module, which calculates the label of the embedded feature pair; A loss calculation module, which calculates the model loss according to the similarity of the embedded feature pairs and the labels of the embedded feature pairs; The test unit comprises: Embedded feature extraction module 1, extracts embedded features of real images and AI-generated images; Embedded feature extraction module 2, extracting embedded features of the image to be tested; Cosine distance calculation module: This module uses the embedded features of the image to be tested, the real image and the AI ​​generated image to calculate the cosine distance between the image to be tested and the real image and the AI ​​generated image respectively; Prediction result output module: obtains the prediction result according to the relationship of the cosine distance, that is, whether the image to be tested is an AI generated image.

Images