A generative image unsupervised detection method based on visual language model

Abstract

The present invention discloses a generative image unsupervised detection method based on a visual language model, comprising the following steps: obtaining an image data set; effectively assigning noise labels to unlabeled data through noise fingerprint extraction, frequency domain feature extraction and clustering algorithm, and establishing an efficient feature extractor by using a pre-trained visual language model and a contrastive learning strategy; using the trained feature extractor to extract high-dimensional discriminant features of the sample to be tested, and classifying the obtained discriminant features into two clusters through a clustering algorithm, and distinguishing the real image from the generated image by calculating the cosine similarity. This method does not require the use of labeled data from beginning to end through unsupervised learning, and overcomes the problem that supervised learning relies on large-scale data with correct labeling. The generalization and robustness of the detector are improved through a pre-trained visual language model. A certain generalization ability is demonstrated when detecting images generated by different generative models.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a generative image unsupervised detection method based on a visual language model. Background Art

[0002] In recent years, the rapid development of generative adversarial networks and diffusion models has attracted a lot of attention. Malicious attackers may use these technologies to generate realistic fake images, and even post these unrealistic images to social networking platforms, or fabricate fake news to undermine the public's credibility. Although some reliable forensic methods have emerged to detect generated fake images, current methods mainly rely on supervised learning, which requires a large number of correctly labeled samples. In fact, collecting enough labeled data in the real world is usually expensive and laborious, and the detection accuracy may drop sharply when faced with problems such as data labeling errors. When detecting forged images generated by different generative models, the performance of the detector often drops significantly. In addition, when the detected image is subjected to post-processing attacks, the performance of the detector will also drop to a certain extent. In view of the above limitations, the present invention proposes an unsupervised detection method that does not require the use of labeled data from beginning to end, and improves the generalization and robustness of the detector through a pre-trained visual language model.

[0003] The generative adversarial network consists of a generator and a discriminator. The generator generates images, while the discriminator identifies the authenticity of images. The principle is a zero-sum game process: the generator and the discriminator continuously optimize the parameters of the network to improve their own capabilities during the confrontation process, and the generator eventually generates images that the discriminator cannot recognize. In recent years, with the rapid development of generative adversarial networks, various methods have sprung up, such as StarGAN, StyleGAN, StyleGAN2, StyleGAN3, etc. Among them, StyleGAN, StyleGAN2, and StyleGAN3 can generate high-resolution (1024×1024) and high-quality face images, and can even deceive humans.

[0004] Recently, diffusion models such as Midjourney and Stablediffusion have attracted more and more attention. Diffusion models are also called diffusion probability models. Diffusion models mainly include two steps, the forward denoising process and the reverse denoising process. The forward denoising process gradually adds noise to the image until pure noise is finally obtained. The reverse denoising process trains a neural network, starting from pure noise and gradually denoising until an image is obtained. In addition, the diffusion model also adds a text encoder, which encodes the input prompt word Prompt into a text vector and guides the image to be generated in the direction of the text vector through the attention mechanism.

[0005] Existing generated image detection methods are divided into two categories: spatial domain-based methods and frequency domain-based methods. Spatial domain-based methods mainly rely on the obvious difference in imaging principles between generated images and real images. Due to the inherent limitations of the generative model, the generated false images have obvious traces of forgery in the spatial domain. For example, detection methods based on color component differences; detection methods based on eye highlight inconsistency; detection methods based on color co-occurrence matrix, etc. Frequency domain-based methods mainly rely on the fact that upsampling operations will leave periodic traces of forgery in the frequency domain of generated images. For example, through DCT transformation, frequency domain features are input into SVM to train the detector; generated images are detected by combining color channel spectrum and capsule network; generated images are detected by the distribution of the first digit of DCT coefficients, etc.

[0006] However, current generative image detectors all rely on supervised learning mechanisms. Supervised learning relies on large-scale data with correct labels. In the real world, collecting enough labeled data is usually expensive and laborious. In addition, supervised learning suffers from generalization errors, false correlations, and adversarial attacks. When faced with problems such as incorrect data labeling, detection accuracy may be greatly reduced. Although traditional self-supervised learning can significantly reduce the annotation cost of pre-training data and achieve good scalability, it still requires the use of correctly labeled samples to fine-tune the model in downstream tasks.

[0007] In addition, due to the continuous development of image generation technology, it is impractical to train a detection model that can fully fit the feature distribution of all generated images, which also leads to poor generalization ability of the detection model. In addition, when post-processing operations appear, the robustness of the detector tends to decrease to a certain extent. In addition, due to the limitations of the supervision mechanism, the generalization and robustness of the detector will be further reduced when the training samples are subjected to label flipping attacks. Summary of the invention

[0008] In view of the shortcomings of the prior art, the present invention proposes a generative image unsupervised detection method based on a visual language model. Through unsupervised learning, no labeled data is needed from beginning to end, which overcomes the problem that supervised learning relies on large-scale data with correct labels. And the generalization and robustness of the detector are improved through the pre-trained visual language model. A certain generalization ability is demonstrated when detecting images generated by different generative models. Good detection accuracy is maintained under image post-processing operations such as JPEG compression, cropping, and noise addition. When the training sample is subjected to a label flipping attack, this method can still maintain good detection performance.

[0009] In order to solve the above technical problems, the technical solution of the present invention is:

[0010] A generative image unsupervised detection method based on a visual language model comprises the following steps:

[0011] Step 1: Obtain image dataset;

[0012] Step 2: Feature extractor training

[0013] Step 2-1: Extract noise fingerprints. Input the image data set and use the denoising filter to extract noise fingerprints from the image. The noise fingerprint extraction method is as follows:

[0014] R=xF(x)

[0015] Where x is the input image and F(·) is the denoising filter.

[0016] Step 2-2, frequency domain feature extraction, first use discrete Fourier transform to convert the obtained noise fingerprint into the frequency domain, and then flatten the obtained frequency domain matrix as the frequency domain feature;

[0017] Step 2-3, noise label assignment, the extracted frequency domain features are divided into two clusters by clustering algorithm, and then noise labels are randomly assigned to the two clusters (for example, cluster 0 samples and cluster 1 samples, the above noise labels are used to distinguish different clusters);

[0018] Step 2-4: perform data augmentation on the clusters assigned with noise labels;

[0019] Step 2-5: Feature extractor training. Use the pre-trained visual language model CLIP as the feature extraction backbone network. Take the augmented image data as input and train it through the contrast loss function. The contrast loss function is defined as follows:

[0020]

[0021] Wherein, set I = {1, ..., 2N} represents all samples obtained after data augmentation, i is a sample in set I, set P(i) = {1, ..., M} represents samples in set I with the same noise label as i, set A(i) = {1, ..., K} represents the remaining samples after removing i from set I, h represents the features output by the feature extraction network, · represents the inner (dot) product, and τ represents the temperature parameter.

[0022] Step 3: Binary Classifier

[0023] Use the trained feature extractor to extract the high-dimensional discriminant features of the sample to be tested, and classify the obtained discriminant features into two clusters through the clustering algorithm. The real image and the generated image are distinguished by calculating the cosine similarity. The cosine similarity calculation method is:

[0024]

[0025] Where (x, x′) represents a pair of different images in the same cluster. The cosine similarity is calculated for the samples belonging to the two clusters respectively, and then the average cosine similarity of each cluster is calculated. The average cosine similarities of the two clusters are compared. The image cluster with a higher average cosine similarity is considered to be the generated image, and the image cluster with a lower average cosine similarity is considered to be the real image.

[0026] Preferably, in step 2-1, the method for extracting noise fingerprint is:

[0027] R=xF(x)

[0028] Where x is the input image and F(·) is the denoising filter.

[0029] Preferably, in step 2-2, the method for transforming the noise fingerprint into a frequency domain matrix is:

[0030]

[0031] Where R represents the noise fingerprint matrix of size M×N. u and v are the indices of the frequency domain matrix.

[0032] Preferably, the data augmentation methods include cropping, horizontal flipping, vertical flipping, 90-degree rotation, erasing, color jittering, histogram equalization, Gaussian blurring, and JPEG compression.

[0033] Preferably, in the steps 2-5, the training method of the feature extractor is: first, the augmented image is input into the CLIP network to obtain high-dimensional features, then normalized, and then projected into a low-dimensional space through a projection head, and the distance between samples with the same noise label is minimized by a contrast loss function, while maximizing the distance between samples with different noise labels. The contrast loss function is defined as follows:

[0034]

[0035] Wherein, set I = {1, ..., 2N} represents all samples obtained after data augmentation, i is a sample in set I, set P(i) = {1, ..., M} represents samples in set I with the same noise label as i, set A(i) = {1, ..., K} represents the remaining samples after removing i from set I, h represents the features output by the feature extraction network, · represents the inner (dot) product, and τ represents the temperature parameter.

[0036] The present invention has the following characteristics and beneficial effects:

[0037] (1) Without providing any label information of the sample, an unsupervised detector is designed to detect the generated image, which saves a lot of manpower and material resources required for sample annotation.

[0038] (2) When the training samples are subjected to label flipping attacks, this method can still maintain good detection performance, while the detection performance of the supervised method will decrease.

[0039] (3) This method shows a certain degree of generalization and robustness. It shows a certain degree of generalization when detecting images generated by different generative models. In addition, it can still maintain good detection performance when facing image post-processing operations such as JPEG compression, cropping, and noise addition. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0041] Figure 1 It is a schematic diagram of the process of the present invention;

[0042] Figure 2 for Figure 1 Schematic diagram of the basic process of feature extractor training;

[0043] Figure 3 for Figure 1 Schematic diagram of the basic process of binary classifier. DETAILED DESCRIPTION

[0044] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0045] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0046] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0047] The present invention provides a generative image unsupervised detection method based on a visual language model, and implements the generative image detection method of this embodiment, such as Figure 1 As shown, it consists of two parts: feature extractor training and binary classifier.

[0048] (1) Feature extractor training. The basic process of feature extractor training is as follows: Figure 2 shown.

[0049] (1.1) Noise fingerprint extraction: Although the generated images have high visual quality, they still bear unique traces of the generation process. Each forgery model will insert a digital fingerprint into all synthesized images, which depends on its architecture and training details. A denoising filter is used to extract the noise fingerprint from the image, where the noise fingerprint is extracted as follows:

[0050] R=xF(x)

[0051] Where x is the input image and F(·) is the denoising filter.

[0052] (1.2) Frequency domain feature extraction: The obtained noise fingerprint is converted into the frequency domain using discrete Fourier transform, and then the obtained frequency domain matrix is ​​flattened as the frequency domain feature:

[0053]

[0054] Where R represents the noise fingerprint matrix of size M×N. u and v are the indices of the frequency domain matrix.

[0055] (1.3) Noise label assignment: The extracted frequency domain features are divided into two clusters through a clustering algorithm, and then noise labels are randomly assigned to the two clusters (for example, cluster 0 samples and cluster 1 samples, and the above noise labels are used to distinguish different clusters).

[0056] (1.4) Data augmentation: For each input sample, two random augmentations are generated. Augmentation methods include: cropping, horizontal flipping, vertical flipping, 90 degree rotation, erasing, color jittering, histogram equalization, Gaussian blur, and JPEG compression.

[0057] (1.5) Feature extractor training: The pre-trained visual language model CLIP is used as the feature extraction backbone network. First, the augmented image is input into the CLIP network to obtain high-dimensional features, then normalized, and then projected into a low-dimensional space through the projection head. The contrast loss function is used to minimize the distance between samples with the same noise label and maximize the distance between samples with different noise labels. The contrast loss function is defined as follows:

[0058]

[0059] Wherein, set I = {1, ..., 2N} represents all samples obtained after data augmentation, i is a sample in set I, set P(i) = {1, ..., M} represents samples in set I with the same noise label as i, set A(i) = {1, ..., K} represents the remaining samples after removing i from set I, h represents the features output by the feature extraction network, · represents the inner (dot) product, and τ represents the temperature parameter.

[0060] (2) Binary classifier, the basic process is as follows Figure 3 As shown in the figure: Use the trained feature extractor to directly extract the high-dimensional discriminant features of the sample to be tested, and use the clustering algorithm to classify the obtained high-dimensional discriminant features into two clusters. In fact, after extracting effective discriminant features, the image to be tested can only be divided into two clusters by the clustering algorithm, but it is impossible to distinguish which cluster is the real image or the generated image. Since the generated images are synthesized through the standard pipeline, this leads to a high degree of similarity between the generated images. More specifically, the cosine similarity is calculated to judge:

[0061]

[0062] Where (x, x') represents a pair of different images in the same cluster. The real image and the generated image are identified by comparing the average similarity of each cluster. The cosine similarity is calculated for the samples belonging to the two clusters respectively, and then the average cosine similarity of each cluster is calculated. Finally, the average cosine similarity of the two clusters is compared. The image cluster with a higher average cosine similarity is considered to be the generated image, and the image cluster with a lower average cosine similarity is considered to be the real image.

[0063] In the above technical solution, unsupervised learning is used to overcome the problem that supervised learning relies on large-scale correctly labeled data, and the generalization and robustness of the detector are improved through the pre-trained visual language model.

[0064] In this embodiment, noise labels are effectively assigned to unlabeled data through noise fingerprint extraction, frequency domain feature extraction and clustering algorithm. In addition, an efficient feature extractor is established to continuously refine the discriminant features of the model through pre-trained visual language model and contrastive learning strategy.

[0065] The detection accuracy of the generated image detection method provided by this embodiment is comparable to that of the supervised method. In addition, when the training sample is subjected to a label flip attack, the method of this embodiment can still maintain good detection performance.

[0066] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A generative image unsupervised detection method based on a visual language model, characterized in that: The steps include: Step 1: Obtain image dataset; Step 2: Feature extractor training Step 2-1, noise fingerprint extraction, input image data set and use denoising filter to extract noise fingerprint from the image; Step 2-2, frequency domain feature extraction, first use discrete Fourier transform to convert the obtained noise fingerprint into the frequency domain, and then flatten the obtained frequency domain matrix as the frequency domain feature; Step 2-3: Noise label assignment: The extracted frequency domain features are divided into two clusters through a clustering algorithm, and then noise labels are randomly assigned to the two clusters; Step 2-4: perform data augmentation on the clusters assigned with noise labels; Step 2-5: Use the pre-trained visual language model CLIP as the feature extraction backbone network, take the augmented image data as input and train it through the contrast loss function; The training method of the feature extractor is as follows: first, the augmented image is input into the CLIP network to obtain high-dimensional features, then normalized, and then projected into a low-dimensional space through a projection head. The distance between samples with the same noise label is minimized by the contrast loss function, while the distance between samples with different noise labels is maximized. The contrast loss function is defined as follows: Wherein the set I = {1, ..., 2N} represents all samples obtained after data augmentation, i is a sample in the set I, the set P(i) = {1, ..., M} represents the samples in the set I with the same noise label as i, the set A(i) = {1, ..., K} represents the remaining samples after removing i from the set I, h represents the features output by the feature extraction network, · represents the inner product, and τ represents the temperature parameter Step 3: Binary Classifier Use the trained feature extractor to extract the high-dimensional discriminant features of the sample to be tested, and classify the obtained discriminant features into two clusters through the clustering algorithm. The real image and the generated image are distinguished by calculating the cosine similarity. The cosine similarity calculation method is: Where (x, x) represents a pair of different images in the same cluster. The cosine similarity is calculated for the samples belonging to the two clusters respectively, and then the average cosine similarity of each cluster is calculated. The average cosine similarities of the two clusters are compared. The image cluster with a higher average cosine similarity is considered to be the generated image, and the image cluster with a lower average cosine similarity is considered to be the real image.

2. The method of unsupervised detection of generative images based on a visual language model according to claim 1, characterized in that: In step 2-1, the method of extracting noise fingerprint is: R=xF(x) Where x is the input image and F(·) is the denoising filter.

3. The method for generating unsupervised image detection based on a visual language model according to claim 1, characterized in that: In step 2-2, the method of transforming the noise fingerprint into a frequency domain matrix is: Where R represents the noise fingerprint matrix of size M×N, and u and v are the indices of the frequency domain matrix.

4. The method of unsupervised detection of generative images based on a visual language model according to claim 1, characterized in that: The data augmentation methods include cropping, horizontal flipping, vertical flipping, 90-degree rotation, erasing, color jittering, histogram equalization, Gaussian blurring, and JPEG compression.

Images