A GAN Image Forensic Method Integrating Robust Features of Color Spaces

Abstract

The present invention provides a GAN image forensics method that fuses robust features of color spaces. By combining the advantages of handcrafted features with DNN, a reliable and efficient detector is designed. Under the guidance of the experience of selecting color channels, the features are input into the DNN for training, and good detection performance can be achieved, including the following steps: Step 1, convert the RGB image into the HSV color space and the YCbCr color space; Step 2, calculate the mean of the pixels in each color channel in Step 1, calculate its correlation coefficient, and calculate the chi-square distance for the correlation coefficients of each color channel; Step 3, select the four color channels with the largest chi-square distance; Step 4, extract the cross-color space co-occurrence matrix using the color channels selected in Step 3; Step 5, input the features in Step 4 into a shallow neural network for training to obtain a model and use it for final classification.

Description

Technical Field

[0001] The present invention belongs to the field of image machine learning, and particularly relates to a GAN image forensics method integrating robust features of color spaces. Background Art

[0002] With the continuous development and improvement of digital media technology, social networks have become the main platforms for people to obtain information and communicate. Due to the characteristics of easy storage, transmission, and processing, digital images have become one of the main types of information disseminated on social networks. At the same time, with the continuous development of image generation technology based on generative adversarial networks (GANs), fake images have become increasingly realistic. Therefore, it is of great practical significance to develop powerful image forensics technologies. In the process of competing with each other, image forgery technology and image forensics technology have also promoted each other.

[0003] Existing GAN image detection methods are divided into two categories: methods based on manually extracted features and methods based on deep learning features. Methods based on manually extracted features mainly rely on various manually extracted features and combine traditional support vector machine (SVM) mechanisms or advanced network structures to identify GAN-generated images. For example, the SRM feature in steganalysis and a linear SVM classifier are combined to identify CycleGAN images; head pose fingerprints composed of facial landmarks are used to identify GAN-generated facial images; the red-green bivariate histogram and a feature of the proportion of abnormally exposed pixels are combined with classifiers such as SVM for GAN image detection. Methods based on deep neural network (DNN) features mainly refer to the DNN algorithm adjusting the typical weights of neurons by calculating the gradient of the loss function to automatically learn features for identifying GAN-generated images. For example, by using a high-pass filter, residual noise is input into the DNN to train the detector; a customized DNN is designed, such as CGFace to verify fake images based on the PGGAN model; the input image is subjected to DCT transformation to obtain a normalized spectrum and then input into a convolutional neural network to obtain better detection results.

[0004] To facilitate the transmission of images in social networks, they are usually compressed and processed, so that while maintaining their semantics and quality, the traffic is saved and the transmission speed is increased. Existing image forensics methods are generally trained and detected on standard data sets, but social network images often lose some information after passing through compression algorithms, making it difficult to accurately detect them. The present invention uses the texture features and color features of images and combines a shallow neural network to train a classifier to accurately identify natural images and GAN-generated images disseminated in social networks. Summary of the Invention

[0005] In existing GAN image detection methods, methods based on manually extracting features often require in-depth understanding and analysis of fake images. These methods are designed for specific forgeries and are often vulnerable to image compression, resizing, transmission noise, etc. Classifiers designed based on DNN generally have problems such as long training model time, large computational overhead, and low model interpretability. Moreover, the interpretability of DNN has always been a difficult problem in research. The challenging problem of learning-based algorithms is the dependence on high-dimensional features. In addition, when the training set and the test set do not match, the detection accuracy cannot always be guaranteed.

[0006] Social networks are an important platform for image dissemination. The detection technology of GAN images should be robust to image post-processing operations (such as JPEG compression, cropping, Gaussian noise, contrast enhancement, etc.) during its dissemination process. In existing methods, in the scenario of simple forensics, when using standard data sets, they generally have good detection performance. However, in actual forensics scenarios, they are less robust to common post-processing operations or combinations of multiple operations.

[0007] This method combines the advantages of manual features with DNN to design a reliable and efficient detector. Under the guidance of the experience of selecting color channels, the features are successfully input into the designed DNN for training, and good detection performance can be achieved. In addition, these features can be smoothly input into other benchmark DNN models with high detection rates, and have good detectability even for images disseminated in social networks under post-processing image operations such as JPEG compression, cropping, Gaussian noise, and contrast enhancement.

[0008] The method provided by the present invention includes:

[0009] Step 1, obtain an RGB image and convert the RGB image into the HSV color space and the YCbCr color space;

[0010] Step 2, calculate the mean value of the pixels in each color channel in Step 1, calculate its correlation coefficient, and calculate the chi-square distance for the correlation coefficients of each color channel;

[0011] Step 3, the four color channels with the largest chi-square distance;

[0012] Step 4, perform cross-color space co-occurrence matrix extraction using the rotated color channels in Step 3;

[0013] Step 5, input the features in Step 4 into a shallow neural network for training to obtain a model and use it for final classification, and input the features in Step 4 into a benchmark DNN model for detection.

[0014] Although forged GAN images are visually indistinguishable from real images, there are still essential differences in their generation principles and procedures, which can be summarized as follows: 1) GAN images are synthesized in the RGB color space and it is difficult to simulate the data distribution of the original images in other color spaces; 2) Compared with the original images, the upsampling process of GAN images directly leads to higher correlation between adjacent pixels; 3) GAN images are difficult to model the global information of the original images, such as textures, structures or spectral features in the frequency domain. Therefore, by analyzing the correlation between pixels, the above differences can be effectively discovered.

[0015] Preferably, in the second step, the correlation coefficient is calculated by formulas (1) and (2);

[0016]

[0017]

[0018] Among them, for the color spaces RGB, HSV, and YCbCr, 9 candidate color channels are selected, denoted as β ∈ {R, G, B, H, S, V, Y, Cb, Cr}, Cor {β} represents the correlation between adjacent pixels in multiple color spaces; P i,j represents the pixel value, M×N represents the size of the image, and the larger the value of Cor {β} , the greater the correlation between adjacent pixels of the image;

[0019] For each color channel, calculate the chi-square distance between the histograms of adjacent pixels of the two types of images:

[0020]

[0021] where and respectively represent the frequency values of the two types of images in the histogram, and the larger the value of the chi-square distance, the greater the difference between these two types of images in this color channel.

[0022] Color and texture information are the main underlying visual features widely used. In practice, the naked eye usually discovers significant information by capturing color features and then further analyzes them based on texture features. In addition, since color and texture are complementary visual features and are relatively simple and efficient to calculate, fusing these two features becomes a crucial preliminary step before designing the following network. The gray-level co-occurrence matrix is an effective method for texture feature extraction. It is a second-order statistical feature that clearly depicts the change in image pixel intensity, mainly manifested as the frequency of the occurrence of pixel pairs with a specified value, that is, the positional distribution feature between pixels with the same or similar intensity. The rapid progress of GAN image technology makes it difficult for people to judge authenticity solely with the naked eye. Therefore, the relationship between different color channels is an effective basis for judgment. That is, the gray-level co-occurrence matrix is extended to the cross-color-space co-occurrence matrix to extract the co-occurrence matrix between multiple color channels, where the color channels refer to the four channels of R, B, Cb, and Cr with relatively large differences mentioned above.

[0023] Preferably, in step four, the extraction of the cross-color-space co-occurrence matrix includes:

[0024] Extend the gray-level co-occurrence matrix to the cross-color-space co-occurrence matrix to extract the co-occurrence matrix between multiple color channels;

[0025] For the co-occurrence matrix of a single color channel, it can be expressed as:

[0026]

[0027] For the co-occurrence matrix of cross-color channels, it can be expressed as:

[0028]

[0029] where i,j∈[0,255],(C x ,C x ) represents the same color channel; there are a total of four co-occurrence matrices of single color channels, namely T(C R ,C R ),T(C B ,C B ),T(C Cb ,C Cb ),T(C Cr ,C Cr ); there are a total of six co-occurrence matrices of cross-color-space channels, namely T(C R ,C B ),T(C R ,C Cb ),T(C R ,C Cr ),T(C B ,C Cb),T(C B ,C Cr ),T(C Cb ,C Cr ); Δm and Δn represent offsets. In a single channel, the offset only considers the influence of spatial position. In a cross-color space channel, the offset considers the combined influence of different color channels and spatial positions. Therefore, all co-occurrence matrices are fused to obtain a feature matrix with a dimension of 256×256×10.

[0030] Preferably, in step five, the training of the shallow neural network includes:

[0031] The first convolutional layer filters the input cross-color space co-occurrence matrix features using 32 kernels of size 3×3. By applying the rectified linear unit ReLU to truncate negative values to 0, the operation result of the CNN is prevented from approaching 0 or infinity. The output feature dimension is 254×254×32;

[0032] The second convolutional layer performs convolution using 32 kernels of size 5×5. By using a max-pooling layer with a kernel size of 3×3, features with a dimension of 126×126×32 are obtained;

[0033] The third convolutional layer performs convolution using 64 kernels of size 3×3, where the activation function is also ReLU, and features with a size of 126×126×64 are output;

[0034] The fourth convolutional layer performs convolution using 64 kernels of size 5×5. By using a max-pooling layer with a kernel size of 3×3, features with a size of 62×62×64 are output;

[0035] The fifth convolutional layer performs convolution using 128 kernels of size 3×3 and a ReLU activation function;

[0036] The sixth convolutional layer performs convolution using 128 kernels of size 5×5. By using a max-pooling layer with a kernel size of 3×3, features with a size of 30×30×128 are output and passed through a flattening layer to be output between the fully connected layers, converting the features into one dimension;

[0037] Two 256-node perceptron layers and a dropout layer are stacked on the output of the fully connected layer, and the output is sent to the two-node perceptron layer to judge the result.

[0038] The method of the present invention has the following substantial features:

[0039] 1) The correlation between adjacent pixels in the original image and the GAN image in multiple color channels is studied, and the most distinguishable channels are selected for feature extraction.

[0040] 2) The cross-color space co-occurrence matrix is used to characterize features, and the texture and color characteristics of the image are obtained simultaneously and input into the designed lightweight convolutional neural network to train a robust and reliable detector. Using a lightweight convolutional neural network can solve the problem of too long model training time.

[0041] 3) The cross-color space co-occurrence matrix features can be smoothly input into a benchmark DNN model with a high detection rate, adapt to other networks, and obtain good detection performance.

[0042] It not only considers multiple post-processing attacks in the simulated scenario but also considers actual social network platforms such as Weibo, WeChat, Facebook, and Twitter. After large-scale experimental verification, the designed detector has superiority. Description of the Drawings

[0043] Figure 1 Basic flowchart of the present invention;

[0044] Figure 2 Neural network structure diagram proposed by the present invention. Detailed Embodiments

[0045] The technical solutions of the present invention will be further specifically described below through specific embodiments and in conjunction with the drawings.

[0046] Embodiment 1

[0047] Before feature extraction, first, according to the differences in the generation methods of GAN-generated images and natural images, the differences between the two types of images on different color channels are analyzed. After calculating the mean value of the pixels in each color channel, its correlation coefficient is calculated, and the chi-square distance is calculated for the correlation coefficients of each color channel. The four color channels with the largest chi-square distance are selected: R, B, Cb, Cr. Then the selected color channels are used for cross-color space co-occurrence matrix feature extraction to obtain the texture features and color features of the image. Then the obtained 256×256×10-dimensional features are input into the designed shallow neural network for training to obtain a model and used for final classification, realizing high-efficiency and high-accuracy detection of GAN-generated images. In addition, the cross-color space co-occurrence matrix is input into other benchmark DNN models with high detection rates for detection, and the detection of images spread in social networks such as WeChat, Weibo, Twitter, and Facebook is also realized. It is a robust GAN image detection method. The basic process of the present invention is as Figure 1 shown.

[0048] For Figure 1 the above-mentioned color channel analysis, cross-color space co-occurrence matrix feature extraction, training of the shallow neural network, transfer to the benchmark neural network, and model training for social network images, the detailed processes are described as follows:

[0049] 1) Color channel analysis;

[0050] Although forged GAN images are visually indistinguishable from real images, there are still essential differences in their generation principles and procedures, which can be summarized as follows: 1) GAN images are synthesized in the RGB color space and it is difficult to simulate the data distribution of the original images in other color spaces; 2) Compared with the original images, the upsampling process of GAN images directly leads to higher correlation between adjacent pixels; 3) GAN images are difficult to model the global information of the original images, such as textures, structures or spectral features in the frequency domain. Therefore, by analyzing the correlation between pixels, the above differences can be effectively mined.

[0051] First, analyze the correlation between adjacent pixels of the images to find suitable color channels for feature extraction. Nine candidate color channels are selected in three representative color spaces RGB, HSV, and YCbCr, denoted as β ∈ {R, G, B, H, S, V, Y, Cb, Cr}. The correlation between adjacent pixels in multiple color spaces is measured by the coefficient Cor {β} as follows:

[0052]

[0053]

[0054] where P i,j represents the pixel value, M×N represents the size of the image, and the larger the value of Cor {β} , the greater the correlation between adjacent pixels of the image. For each color channel, calculate the chi-square distance between the histograms of adjacent pixels of the two types of images:

[0055]

[0056] where and represent the frequency values of the two types of images in the histogram respectively. The larger the value of the chi-square distance, the greater the difference between these two types of images in this color channel. Through experimental verification, the correlation between adjacent pixels of GAN synthetic images is generally larger than that of natural images, and the chi-square distance is the largest in the four channels of R, B, Cb, and Cr. Therefore, it can be considered that when extracting features in the four channels of R, B, Cb, and Cr, the differences in the images are more obvious.

[0057] 2) Cross-color space co-occurrence matrix feature extraction;

[0058] Color and texture information are the main underlying visual features widely used. In practice, the human eye usually discovers significant information by capturing color features and then further analyzes them based on texture features. In addition, since color and texture are complementary visual features and are relatively simple and efficient to calculate, fusing these two features becomes a crucial preliminary step before designing the following network. The gray-level co-occurrence matrix is an effective method for texture feature extraction. It is a second-order statistical feature that clearly depicts the change in image pixel intensity, mainly manifested as the frequency of the occurrence of pixel pairs with a specified value, that is, the positional distribution feature between pixels with the same or similar intensity. The rapid development of GAN image technology makes it difficult for people to judge authenticity solely by the naked eye. Therefore, the relationship between different color channels is an effective basis for judgment. That is, the gray-level co-occurrence matrix is extended to the cross-color space co-occurrence matrix to extract the co-occurrence matrix between multiple color channels, where the color channels refer to the four channels of R, B, Cb, and Cr with relatively large differences mentioned above.

[0059] For the co-occurrence matrix of a single color channel, it can be expressed as:

[0060]

[0061] For the co-occurrence matrix of cross-color channels, it can be expressed as:

[0062]

[0063] where i, j ∈ [0, 255], (C x , C x ) represents the same color channel. There are a total of four co-occurrence matrices of single color channels, namely T(C R , C R ), T(C B , C B ), T(C Cb , C Cb ), T(C Cr , C Cr ). There are a total of six co-occurrence matrices of cross-color space channels, namely T(C R , C B ), T(C R , C Cb ), T(C R , C Cr ), T(C B , C Cb ), T(C B , C Cr ), T(C Cb , C Cr)。Δm and Δn represent offsets. In a single-channel, the offset only considers the influence of spatial position. In a cross-color space channel, the offset considers the combined influence of different color channels and spatial positions. Therefore, all co-occurrence matrices are fused to obtain a feature matrix with a dimension of 256×256×10.

[0064] 3) Train a shallow neural network;

[0065] This shallow neural network consists of six convolutional layers and three pooling layers. The neural network architecture is as Figure 2 shown, clearly describing the size of each filter layer, the size of the corresponding feature map, and the relationship between the positions of the convolutional layer, pooling layer, and fully connected layer. This DNN network can further significantly reveal the separability of channel features between the same and different color space channels of an image. The input layer is a tensor with a size of 256×256×10. The first convolutional layer filters the input cross-color space co-occurrence matrix features using 32 kernels of size 3×3. By applying the rectified linear unit (ReLU) to truncate negative values to 0 to avoid the operation results of the CNN approaching 0 or infinity, the output feature dimension is 254×254×32. The second convolutional layer takes this feature as input and convolves it with 32 kernels of size 5×5. Then, through a max pooling (Maxpooling) layer with a kernel size of 3×3, the resulting feature with a dimension of 126×126×32 is fed into the third convolutional layer with 64 kernels of size 3×3, where the activation function is also ReLU, to obtain a feature with a size of 126×126×64. The fourth convolutional layer has 64 kernels of size 5×5, and after passing through the Maxpooling layer, the output feature with a size of 62×62×64 is obtained. The 5th convolutional layer has 128 kernels of size 3×3 and a ReLU activation function, followed by the 6th convolutional layer with 128 kernels of size 5×5. Through the Maxpooling layer, a feature with a size of 30×30×128 is obtained. The Flatten layer is used between the convolutional layer and the fully connected layer to convert the feature into one-dimensional, and then there are two 256-node perceptron layers stacked with a dropout layer. The last two-node perceptron layer maps the output to 0 or 1 representing two classes, namely GAN-generated images and natural images.

[0066] Experiments prove that the features and neural network proposed by the present invention have excellent detection performance for the detection rate of GAN images on a standard dataset, and the accuracy rate can reach more than 99%.

[0067] 4) Transfer to the benchmark neural network;

[0068] The features proposed in the present invention can not only be combined with the proposed neural network to obtain excellent detection performance through training the model, but also achieve comparable performance when the benchmark network structure is combined with the features proposed in the present invention.

[0069] Therefore, the present invention designs a model. For the benchmark network, only the data of its input layer is modified without changing its original structure, and the above cross-color-space co-occurrence matrix is input into the benchmark network training model, where the benchmark network can be a network with high detection rate such as CapsuleNet, XceptionNet, EfficientNet, etc. Experiments prove that the design proposed in the present invention can detect more than 99% of the GAN-generated images.

[0070] 5) For social network images;

[0071] Using the model trained in 3), it is possible to identify GAN-generated images that have undergone post-processing operations such as JPEG compression (with different quality factors), median filtering, Gaussian noise, gamma correction, average pooling, resizing, cropping, contrast enhancement, etc. Even GAN-generated images that have undergone multiple post-processing operations can be well detected.

[0072] To improve the accuracy of the present invention, a model for social network images is additionally designed. The cross-color-space co-occurrence matrices of the original image and images with equal JPEG compression quality factors of 90, 80, 70, 60, and 50 are input into the neural network for training to obtain a detection model. This model can not only achieve high accuracy on standard datasets and post-processed images, but also achieve very good results in blindly forensically analyzing social network images spread on WeChat, Weibo, Twitter, and Facebook.

Claims

1. A GAN image forensics method integrating robust features of color spaces, characterized in that, Including: Step 1: Obtain an RGB image and convert the RGB image into the HSV color space and the YCbCr color space; Step 2: Calculate the mean value of the pixels in each color channel in Step 1, calculate its correlation coefficient, and calculate the chi-square distance for the correlation coefficients of each color channel; Step 3: Select the four color channels with the largest chi-square distance, where the color channels refer to the four channels of R, B, Cb, and Cr with relatively large differences mentioned above; Step 4: Use the color channels selected in Step 3 to extract the cross-color-space co-occurrence matrix; The extraction of the cross-color-space co-occurrence matrix includes: expanding the gray-level co-occurrence matrix to the cross-color-space co-occurrence matrix to extract the co-occurrence matrix between multiple color channels; For the co-occurrence matrix of a single color channel, it can be expressed as: For the co-occurrence matrix of cross-color channels, it can be expressed as: where A and B represent the dimensions of the channels, i, j ∈ [0, 255], (C x , C x ) represent the same color channel; there are four co-occurrence matrices of single color channels, namely T(C R , C R ), T(C B , C B ), T(C Cb , C Cb ), T(C Cr , C Cr ); (C x , C y ), there are a total of six co-occurrence matrices for cross-color-space channels, namely T(C R , C B ), T(C R , C Cb ), T(C R , C Cr ), T(C B , C Cb ), T(C B , C Cr ), T(C Cb , C Cr ); Δm and Δn represent the offsets. In a single channel, the offset only considers the influence of spatial position. In cross-color-space channels, the offset considers the combined influence of different color channels and spatial positions; Therefore, fuse all the co-occurrence matrices to obtain a feature matrix with a dimension of 256×256×10; Step 5: Input the features in Step 4 into a shallow neural network for training to obtain a model and use it for final classification, and input the features in Step 4 into the baseline DNN model for detection.

2. The GAN image forensics method integrating robust features of a fusion color space according to claim 1, wherein In the above-mentioned Step 2, the correlation coefficient is calculated by Formulas (1) and (2); Among them, 9 candidate color channels are selected from the color spaces RGB, HSV, and YCbCr, denoted as β ∈ {R, G, B, H, S, V, Y, Cb, Cr}, Cor {β} represents the correlation between adjacent pixels on multiple color spaces; P i,j represents the pixel value, M×N represents the size of the image, Cor {β The larger the value of}, the greater the correlation between adjacent pixels of the image; For each color channel, calculate the chi-square distance of the adjacent pixel histograms of two types of images: where N {β} and G {β} respectively represent the frequency values of two types of images in the histogram. The larger the value of the chi-square distance, the greater the difference between these two types of images in this color channel.

3. The GAN image forensics method for fusing robust features of a color space, according to claim 1, is characterized in that In the above-mentioned Step 5, the training of the shallow neural network includes: The first convolutional layer uses 32 kernels of size 3×3 to filter the input cross-color-space co-occurrence matrix features, and truncates negative values to 0 by applying the rectified linear unit ReLU to avoid the operation results of the CNN approaching 0 or infinity. The output feature dimension is 254×254×32; The second convolutional layer uses 32 kernels of size 5×5 for convolution, and through the max-pooling layer with a kernel size of 3×3, obtains features with a dimension of 126×126×32; The third convolutional layer uses 64 kernels of size 3×3 for convolution, where the activation function is also ReLU, and outputs features with a size of 126×126×64; The fourth convolutional layer uses 64 kernels of size 5×5 for convolution, and through the max-pooling layer with a kernel size of 3×3, outputs features with a size of 62×62×64; The fifth convolutional layer uses 128 kernels of size 3×3 for convolution and a ReLU activation function; The sixth convolutional layer uses 128 kernels of size 5×5 for convolution, and through the max-pooling layer with a kernel size of 3×3, outputs features with a size of 30×30×128, and outputs to between the fully connected layers through the flattening layer to convert the features into one-dimensional; Stack two 256-node perceptron layers and a dropout layer on the output of the fully connected layer, and output to the two-node perceptron layer to judge the result.

Images