An AI-generated image detection method based on graph topology learning

Abstract

The present invention provides an AI-generated image detection method based on graph topology learning, and relates to the field of image processing technology. The AI-generated image detection method based on graph topology learning includes the following steps: extracting global visual features of multiple image samples based on frozen CLIP; constructing a graph topology structure based on the multiple image samples, and obtaining the relationship features between nodes in the graph topology structure based on a graph convolutional network; performing splicing and fusion to obtain node feature vectors, and transmitting them to a linear layer for true and false classification prediction to obtain detection results; updating the parameters of the graph convolutional network based on a metric learning loss function, and updating the parameters of the linear layer based on a binary cross entropy loss. The present invention designs a graph topology structure that is difficult to distinguish samples based on the visual features of CLIP, and inputs the features of the graph topology into a graph convolutional network for feature learning, which further enriches and enhances the discriminative power of feature representation, and improves the accuracy and generalization during detection.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an AI generated image detection method based on graph topology learning. Background Art

[0002] Nowadays, with the remarkable success of generative adversarial networks and diffusion models in visual synthesis, the ways of forging images have become more widespread, resulting in an increasing risk of misuse of false information. In the existing technology, CNNspot and PatchCraft are mainly used to detect forged images of generative adversarial networks. However, when dealing with complex generated images, there are still problems with insufficient feature extraction and low detection accuracy. In terms of research algorithms, the detection of generated images mainly relies on global regional artifacts, which will lead to special local forgery traces. For example, delicate artifacts such as eyes and hair are difficult to detect. At the same time, the current detection method ignores the correspondence between image samples, which is very important for natural image detection tasks. Therefore, it is urgent to provide a solution to improve the above problems. Summary of the invention

[0003] The purpose of the present invention is to provide an AI-generated image detection method based on graph topology learning, which improves the detection capability of the existing technology in complex scenes.

[0004] The present invention provides an AI-generated image detection method based on graph topology learning, which adopts the following technical solutions:

[0005] Extracting global visual features of multiple image samples based on the frozen CLIP and adding labels to the multiple image samples respectively;

[0006] The multiple image samples are regarded as nodes, and the similarity between two pairs of nodes is obtained based on the dot product operation, and a similarity matrix between the node pairs is constructed, and the node pairs with the same label and similarity lower than the threshold in the similarity matrix are constituted as difficult sample pairs, and a graph topology structure is constructed based on the difficult sample pairs, and the relationship characteristics between the nodes in the graph topology structure are obtained based on the graph convolution network;

[0007] The global visual features of the image sample and the relationship features between nodes are spliced ​​and fused to obtain a node feature vector, and the node feature vector is transmitted to the linear layer for true and false classification prediction to obtain a detection result;

[0008] The parameters of the graph convolutional network are updated based on the metric learning loss function, and the parameters of the linear layer are updated based on the binary cross entropy loss.

[0009] Optionally, in the process of forming difficult sample pairs from the node pairs with the same label and similarity lower than a threshold, the threshold is calculated as follows:

[0010] The threshold is obtained by calculating the mean similarity between pairs of nodes with different labels. The threshold calculation formula is:

[0011] ;

[0012] in, is the similarity threshold, represents the set of all nodes, and Node With Node Tags, For Node With Node The similarity.

[0013] Optionally, the process of constructing a graph topology structure based on the difficult sample pairs includes: when the node pair is a difficult sample pair, there is a connecting edge between the two nodes, otherwise there is no connecting edge between the two nodes, thereby constructing an adjacency matrix of the nodes, and the adjacency matrix is ​​the graph topology structure of the image samples.

[0014] Optionally, the relationship features between nodes in the graph topology structure are obtained based on the graph convolutional network, including:

[0015] The global visual features and graph topological structure of the multiple image samples are input into a graph convolutional network, and the feature information of adjacent nodes of each node in the graph topological structure is aggregated based on the graph convolutional network to obtain the relationship features between the nodes in the graph topological structure; the graph convolutional network includes two graph convolutional layers and one ReLU activation layer, and the feature information of the node is the global visual feature of the image sample.

[0016] Optionally, in the process of aggregating feature information of each node in the graph topology structure based on a graph convolutional network, the convolution formula is as follows:

[0017] ;

[0018] in, For the The node feature matrix of the layer, For the The node feature matrix of the layer, is the layer index in the graph convolutional network, is a nonlinear activation function of the ReLU activation layer, for The degree matrix of , is the adjacency matrix, is the identity matrix, is the global visual feature of the multiple image samples, is the trainable parameter matrix of the current convolutional layer.

[0019] Optionally, the global visual features of the image sample and the relationship features between nodes are concatenated and fused to obtain a node feature vector, using the following formula:

[0020] ;

[0021] in, is the node feature vector after concatenation and fusion, For feature fusion, is the global visual feature of the image sample, It is the relationship feature between nodes in the graph topology.

[0022] Optionally, the test result is obtained by sending it to the linear layer for true and false classification prediction using the following formula:

[0023] ;

[0024] in, For the test results, is the trainable parameter matrix of the linear layer, is the node feature vector after concatenation and fusion, and b is the linear layer parameter.

[0025] Optionally, updating the parameters of the graph convolutional network based on the metric learning loss function includes: when obtaining the relationship features between nodes in the graph topology structure based on the graph convolutional network, simultaneously using the back propagation algorithm to calculate the gradient of the metric learning loss function with respect to the graph convolutional network parameters, and updating the graph convolutional network parameters based on the gradient using an optimization algorithm.

[0026] Optionally, the metric learning loss function is:

[0027] ;

[0028] in, To measure the learning loss function, Represents the set of all connected edges in the graph topology, and They are the node feature vectors after splicing and fusion and node feature vector .

[0029] The beneficial effects of the AI ​​generated image detection method based on graph topology learning proposed in the present invention are:

[0030] 1. The present invention proposes a graph topology structure that is difficult to distinguish between samples, and aggregates the feature information of adjacent nodes of each node in the graph topology structure based on a graph convolutional network, thereby shortening the distance between image samples with the same labels and better distinguishing image samples with different labels.

[0031] 2. The present invention designs a metric learning based on graph topology, and tunes the parameters of the graph convolutional network during the training process until the optimal detection effect is obtained, thereby improving the accuracy and precision of detecting forged images. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flow chart of an AI-generated image detection method based on graph topology learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0034] The embodiment of the present invention provides an AI generated image detection method based on graph topology learning, see Figure 1 ,include:

[0035] S1. extracting global visual features of multiple image samples based on frozen CLIP and adding labels to the multiple image samples respectively;

[0036] S2, treating the multiple image samples as nodes, obtaining the similarity between any two node pairs based on a dot product operation and constructing a similarity matrix between the node pairs, constituting the node pairs with the same label and a similarity lower than a threshold in the similarity matrix as difficult sample pairs, constructing a graph topology structure based on the difficult sample pairs, and obtaining the relationship features between the nodes in the graph topology structure based on a graph convolutional network;

[0037] S3, concatenating and fusing the global visual features of the image sample and the relationship features between nodes to obtain a node feature vector, and transmitting it to the linear layer for true and false classification prediction to obtain a detection result;

[0038] S4. Update the parameters of the graph convolutional network based on the metric learning loss function, and update the parameters of the linear layer based on the binary cross entropy loss.

[0039] In some embodiments, the process of executing step S1 includes:

[0040] S1.1, extracting global visual features of multiple image samples based on frozen CLIP;

[0041] S1.2. Add labels to the multiple image samples respectively.

[0042] Specifically, in the process of executing step S1.1, extracting global visual features of multiple image samples based on frozen CLIP, it can be expressed as: , that is, given an input image , and then output a feature vector based on the frozen CLIP model .

[0043] In some embodiments, the process of executing step S2 includes:

[0044] S2.1, constructing a graph topology structure based on the labels and global visual features of the multiple image samples;

[0045] S2.2. Obtain the relationship features between nodes in the graph topology structure based on graph convolutional network.

[0046] Specifically, when executing step S2.1, it includes:

[0047] S2.1.1, construct a similarity matrix, and form difficult sample pairs with the same label and similarity below the threshold;

[0048] S2.1.2. Construct a graph topology structure based on the difficult sample pairs.

[0049] Specifically, when executing step S2.1.1, it includes: regarding the multiple image samples as nodes, obtaining the similarity between any two node pairs based on the dot product operation and constructing a similarity matrix between the node pairs, and constituting the node pairs with the same label and similarity lower than the threshold into difficult sample pairs.

[0050] Furthermore, the threshold is obtained by calculating the mean similarity between pairs of nodes with different labels. The threshold calculation formula is as follows:

[0051] ;

[0052] in, is the similarity threshold, represents the set of all nodes, and Node With Node Tags, For Node With Node The similarity.

[0053] Specifically, when executing step S2.1.2, it includes: when the node pair is a difficult sample pair, there is a connecting edge between the two nodes, otherwise there is no connecting edge between the two nodes, thereby constructing an adjacency matrix of the nodes, and the adjacency matrix is ​​a graph topological structure of the image sample.

[0054] In fact, the adjacency matrix of the graph is constructed based on the similarity matrix and the label information of the sample. That is, if the sample pairs in the similarity matrix have the same label and the similarity of the sample pairs is less than or equal to the threshold, there is an edge between the pair of nodes. In this way, a graph topology that is difficult to distinguish samples is constructed, which is defined as:

[0055] ;

[0056] in, There is an edge between the sample pair representing node i and node j. It means no edge connection. and is the label of the image sample, is the similarity of the node pair, Through this connection method, the image features output by CLIP are converted into graph structure data. This graph topology not only considers the global visual features of the sample, but also further learns the relationship features between samples, making the same type of images closer in the sample space.

[0057] Specifically, when executing step S2.2, it includes: inputting the global visual features and graph topological structure of the multiple image samples into the graph convolutional network, aggregating the feature information of adjacent nodes of each node in the graph topological structure based on the graph convolutional network, and modeling the dependency relationship between the nodes to obtain the relationship features between the nodes in the graph topological structure; the graph convolutional network includes two graph convolutional layers and one ReLU activation layer, and the feature information of the node is the global visual feature of the image sample.

[0058] Furthermore, in the process of aggregating the feature information of the adjacent nodes of each node in the graph topology structure based on the graph convolution network, the convolution formula is as follows:

[0059] ;

[0060] in, For the The node feature matrix of the layer, For the The node feature matrix of the layer, is the layer index in the graph convolutional network, is a nonlinear activation function, for The degree matrix of , is the adjacency matrix, is the identity matrix, is the global visual feature, is the trainable parameter matrix of the current convolutional layer.

[0061] In fact, the input of the graph convolutional network is the global visual features of the image extracted by CLIP and the graph topology structure constructed for difficult sample distinction. The graph can be defined as , V is the set of nodes in the graph topology, and E is the set of edges.

[0062] Furthermore, the graph convolutional network includes two graph convolutional layers and one ReLU activation layer. The reason for choosing a two-layer architecture is that as the number of graph convolutional network layers increases, the embedding of each node will be closer, which will affect the classification performance.

[0063] In some embodiments, the process of executing step S3 includes:

[0064] S3.1, concatenating and fusing the global visual features of the image sample and the relationship features between nodes to obtain a node feature vector;

[0065] S3.2, send it to the linear layer for true and false classification prediction to obtain the detection result.

[0066] Specifically, when executing step S3.1, the global visual features of the image sample and the relationship features between nodes are spliced ​​and fused to obtain a node feature vector, using the following formula:

[0067] ;

[0068] in, is the node feature vector after concatenation and fusion, For feature fusion, is the global visual feature of the image sample, It is the relationship feature between nodes in the graph topology.

[0069] Specifically, in the process of executing step S3.2 and transmitting to the linear layer for true and false classification prediction to obtain the detection result, the following formula is used:

[0070] ;

[0071] in, For the test results, is the trainable parameter matrix of the linear layer, is the node feature vector after concatenation and fusion, and b is the linear layer parameter.

[0072] In some embodiments, the process of executing step S4 includes:

[0073] S4.1. Update the parameters of the graph convolutional network based on the metric learning loss function.

[0074] S4.2. Update the parameters of the linear layer based on the binary cross entropy loss.

[0075] Specifically, when executing step S4.1, when updating the parameters of the graph convolution network based on the metric learning loss function, it includes: when obtaining the relationship features between nodes in the graph topology structure based on the graph convolution network, while using the back propagation algorithm to calculate the gradient of the metric learning loss function to the graph convolution network parameters, and updating the graph convolution network parameters based on the gradient using an optimization algorithm.

[0076] Furthermore, the metric learning loss function is:

[0077] ;

[0078] in, To measure the learning loss function, Represents the set of all connected edges in the graph topology, and They are the node feature vectors after splicing and fusion and node feature vector .

[0079] Furthermore, using binary cross entropy loss not only updates the parameters of the linear layer, but also updates the parameters of the graph convolutional network.

[0080] In fact, after updating the parameters of the graph convolutional network and the linear layer, the global visual features of multiple image samples extracted based on the frozen CLIP will be re-input, and the parameters will be tuned through repeated training until the optimal detection effect is obtained.

[0081] Performance Testing

[0082] For the test data set, images generated by various types of generative models are used for in-domain and cross-domain testing, including ProGau, CycleGan, StarGan, StyleGan, GauGan, CRN, IMLE, VQDM, and Midjourney data sets. The present invention is used to detect the data set, and the comparison results are shown in Tables 1 and 2 compared with the two detection methods of CNNSpot and PatchCraft.

[0083] Based on the average value 1 of Table 1 and the average value 2 of Table 2, the average accuracy and average precision of different detection methods after detecting nine data sets are obtained. Among them, the average accuracy of the CNNSpot detection method with the variable Blur+JPEG(0.1) is 78.1% and the average precision is 89.1%, the average accuracy of the CNNSpot detection method with the variable Blur+JPEG(0.5) is 75.4% and the average precision is 87.7%, the average accuracy of the PatchCraft detection method is 85.4% and the average precision is 91.9%, and the average accuracy of the present invention is 89.2% and the average precision is 92.9%.

[0084] Experiments show that the method provided by the embodiment of the present invention has been significantly improved in average accuracy and precision compared with the CNNSpot and PatchCraft detection methods. Among them, compared with the CNNSpot detection method with the variable Blur+JPEG(0.1), the average accuracy is improved by 11.1%, and the average precision is improved by 3.8%. Compared with the CNNSpot detection method with the variable Blur+JPEG(0.5), the average accuracy is improved by 13.8%, and the average precision is improved by 5.1%. Compared with the PatchCraft detection method, the average accuracy is improved by 3.8%, and the average precision is improved by 1%.

[0085] Table 1

[0086]

[0087] Table 2

[0088]

[0089] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.

Claims

1. An AI generated image detection method based on graph topology learning, characterized in that: The following steps are involved: Extracting global visual features of multiple image samples based on the frozen CLIP and adding labels to the multiple image samples respectively; The multiple image samples are regarded as nodes, and the similarity between the two node pairs is obtained based on the dot product operation, and a similarity matrix between the node pairs is constructed, and the node pairs with the same label and similarity lower than the threshold in the similarity matrix are constituted as difficult sample pairs. When the node pair is a difficult sample pair, there is a connecting edge between the two nodes, otherwise there is no connecting edge between the two nodes, so as to construct an adjacency matrix of the nodes, and the adjacency matrix is ​​a graph topological structure of the image samples; The global visual features and graph topological structure of the multiple image samples are input into a graph convolutional network, and feature information of adjacent nodes of each node in the graph topological structure is aggregated based on the graph convolutional network to obtain relationship features between nodes in the graph topological structure; the graph convolutional network includes two graph convolutional layers and one ReLU activation layer, and the feature information of the node is the global visual feature of the image sample; The global visual features of the image sample and the relationship features between nodes are spliced ​​and fused to obtain a node feature vector, and the node feature vector is transmitted to the linear layer for true and false classification prediction to obtain a detection result; When obtaining the relationship features between nodes in the graph topology structure based on the graph convolutional network, the back propagation algorithm is used to calculate the gradient of the metric learning loss function to the graph convolutional network parameters, and the graph convolutional network parameters are updated based on the gradient using an optimization algorithm, and the parameters of the linear layer are updated based on the binary cross entropy loss.

2. According to claim 1, the AI ​​generated image detection method based on graph topology learning is characterized in that: In the process of forming difficult sample pairs from the node pairs with the same label and similarity lower than the threshold, the threshold is calculated as follows: The threshold is obtained by calculating the mean similarity between pairs of nodes with different labels. The threshold calculation formula is: ； in, is the similarity threshold, represents the set of all nodes, and Node With Node Tags, For Node With Node The similarity.

3. The AI-generated image detection method based on graph topology learning according to claim 1 is characterized in that: In the process of aggregating the feature information of each node in the graph topology structure based on the graph convolutional network, the convolution formula is as follows: ； in, For the The node feature matrix of the layer, For the The node feature matrix of the layer, is the layer index in the graph convolutional network, is a nonlinear activation function of the ReLU activation layer, for The degree matrix of , is the adjacency matrix, is the identity matrix, is the global visual feature of the multiple image samples, is the trainable parameter matrix of the current convolutional layer.

4. The AI ​​generated image detection method based on graph topology learning according to claim 1 is characterized in that: The global visual features of the image sample and the relationship features between nodes are spliced ​​and fused to obtain the node feature vector using the following formula: ； in, is the node feature vector after concatenation and fusion, For feature fusion, is the global visual feature of the image sample, It is the relationship feature between nodes in the graph topology.

5. The AI-generated image detection method based on graph topology learning according to claim 4 is characterized in that: The test result is sent to the linear layer for true and false classification prediction, using the following formula: ； in, For the test results, is the trainable parameter matrix of the linear layer, is the node feature vector after concatenation and fusion, and b is the linear layer parameter.

6. The AI ​​generated image detection method based on graph topology learning according to claim 1 is characterized in that: The metric learning loss function is: ； in, To measure the learning loss function, Represents the set of all connected edges in the graph topology, and They are the node feature vectors after splicing and fusion and node feature vector .

Images