Multi-image forgery detection method and system based on cross-modal visual large language model

Abstract

The invention provides a multi-image forgery detection method and system based on a cross-modal vision large language model, and relates to the technical field of image processing and computer vision, and the method comprises the steps: constructing a data set, and carrying out the preprocessing of the data set; according to the preprocessed data, respectively extracting visual features and language features through a pre-trained visual Transform and a language model so as to obtain cross-modal features; according to the cross-modal features, the visual features and the language features are clustered, cross-modal similarity is calculated, a matching relation is established, fusion is carried out, and fused multi-modal features are obtained; and according to the fused multi-modal features, performing adversarial training through a generator and a discriminator to generate an adversarial network. According to the method, high-precision detection and effective detection of multiple image counterfeiting types such as positioning splicing, copying and pasting, AIGC generation and the like are realized.

Description

Technical Field

[0001] The present invention relates to the technical fields of image processing and computer vision, and particularly to a method and system for detecting various types of image forgeries based on a cross-modal vision large language model. Background Art

[0002] With the rapid development of digital image processing technology and artificial intelligence technology, image forgery has become increasingly easy, and the forgery means have become increasingly complex and diverse, such as splicing, copy-pasting, AIGC generation, etc. These forged images may be used for malicious purposes such as the spread of false news, fraud, and privacy infringement, posing a serious threat to social order, commercial activities, and personal rights and interests.

[0003] Existing image forgery detection methods have many limitations. On the one hand, most methods only detect based on features of a single modality (such as visual features), ignoring the important role of text semantic information in forgery detection, and it is difficult to comprehensively and accurately identify complex forged images. On the other hand, different types of image forgeries have their own unique features, but existing methods often fail to fully exploit these feature differences, resulting in low detection accuracy for multiple forgery types. In addition, existing detection methods are less robust when facing new forgery techniques and attacks and are prone to misjudgment.

[0004] In applications of image forgery detection related to generative adversarial networks (GANs), the existing methods combining a generator and a discriminator mainly include the following several types:

[0005] Traditional image generation adversarial: When traditional GANs are applied to the image field, the generator usually takes random noise as input and generates forged images through a series of convolutional layers and upsampling operations. The discriminator is a binary classifier that judges whether the input image is a real image or a generated forged image. This method is simple and direct, but lacks pertinence to the features of different forgery types and is difficult to effectively detect various complex image forgery situations.

[0006] Feature-based adversarial: Some methods attempt to perform adversarial training at the feature level. The generator generates not only images but also image feature vectors with specific features, and the discriminator discriminates these feature vectors. However, this method often does not fully utilize multi-modal information and has insufficient semantic understanding of forged images.

[0007] Conditional generative adversarial: Some conditional GAN methods introduce additional conditional information, such as labels, etc., to enable the generator to generate specific types of forged images. However, these conditional information are often relatively single, without comprehensively considering the visual features and text semantic features of images, and have limited adaptability to complex multiple types of image forgeries. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a method and system for detecting various types of image forgery based on a cross-modal vision large language model, which can effectively detect and locate various types of image forgery such as splicing, copy-pasting, and AIGC generation with high precision.

[0009] To solve the above technical problems, the technical solution of the present invention is as follows:

[0010] In the first aspect, a method for detecting various types of image forgery based on a cross-modal vision large language model, the method includes:

[0011] Construct a dataset and preprocess the dataset;

[0012] According to the preprocessed data, extract visual features and language features respectively through a pre-trained vision Transformer and a language model to obtain cross-modal features;

[0013] According to the cross-modal features, cluster the visual and language features respectively, calculate the cross-modal similarity to establish a matching relationship, and perform fusion to obtain the fused multi-modal features;

[0014] According to the fused multi-modal features, perform adversarial training through a generator and a discriminator to generate an adversarial network;

[0015] According to the adversarial network, extract coarse-grained global features and fine-grained local features from the fused features, and perform authenticity identification and type and region detection respectively to obtain multi-granularity features;

[0016] According to the multi-granularity features, design a classification loss, a segmentation loss, and an adversarial loss, and perform weighted summation to obtain a total loss function;

[0017] According to the total loss function, adopt an end-to-end training method, use an Adam optimizer and a curriculum learning strategy, first perform coarse-grained identification during inference, and if it is forged, then perform fine-grained detection to obtain the output result.

[0018] Further, according to the preprocessed data, extracting visual features and language features respectively through a pre-trained vision Transformer and a language model to obtain cross-modal features includes:

[0019] According to the preprocessed data, extract the global and local features of the image through a pre-trained vision Transformer, and for different forgery types, dynamically adjust the feature weights through an attention mechanism to strengthen the feature response related to the forgery type;

[0020] Use a pre-trained language model to encode the text description to generate an embedding vector that is semantically aligned with the forgery area, ensuring the compatibility of text features and visual features in the semantic space to obtain cross-modal features.

[0021] Further, according to the cross-modal features, cluster the visual and language features respectively, calculate the cross-modal similarity to establish a matching relationship, and perform fusion to obtain the fused multi-modal features, including:

[0022] Perform clustering analysis on the visual features and language features respectively according to the extracted cross-modal features to generate visual clusters and language clusters;

[0023] Calculate the similarity between cross-modal clusters according to the visual clusters and language clusters, and establish a matching relationship between visual-language clusters;

[0024] Fuse the matched visual clusters and language clusters through an attention mechanism to generate multi-modal features.

[0025] Further, according to the fused multi-modal features, perform adversarial training through a generator and a discriminator to generate an adversarial network, including:

[0026] Construct a generator according to the fused multi-modal features. The generator takes a noise vector and semantic features as inputs and outputs a forged image consistent with the semantics;

[0027] Input the generated forged image and the real image into a cross-modal discriminator together. The discriminator adopts a visual branch and a language branch, and is fused through an attention mechanism or a gating mechanism to output the authenticity probability of the image and the semantic matching score discriminator to generate an adversarial network.

[0028] Further, according to the adversarial network, extract coarse-grained global features and fine-grained local features from the fused features, and perform authenticity recognition and type and region detection respectively to obtain multi-grained features, including:

[0029] Extract global features from the multi-modal fused features according to the adversarial network, and train a classifier to judge the authenticity of the image to obtain coarse-grained global features;

[0030] For the images determined to be forged at the coarse-grained level, further extract local features to obtain fine-grained local features;

[0031] Fuse the coarse-grained global features and the fine-grained local features through a graph neural network to generate a joint feature representation, and train a multi-task model based on the joint features to output the authenticity label, the forged type, and the region mask simultaneously to obtain multi-grained features.

[0032] Further, according to the multi-grained features, design a classification loss, a segmentation loss, and an adversarial loss, and perform weighted summation to obtain a total loss function, including:

[0033] Design a classification loss according to the multi-grained features, adopt a cross-entropy loss function to measure the difference between the predicted probability of the forged type and the real label, for the classification loss of the discriminator Let the true label be y and the predicted probability be Then where C is the number of categories of forgery types;

[0034] The similarity between the predicted forgery region mask and the true mask is measured by the Dice loss function as: where M pred is the predicted mask, M true is the true mask, and L seg is the similarity between the predicted forgery region mask and the true mask;

[0035] Through the generative adversarial network, the loss of the discriminator includes the classification loss and the adversarial loss. The adversarial loss of the discriminator aims to maximize the ability to distinguish between real images and forged images, and is defined as where D(·) represents the output probability of the discriminator, x real is the real image, and x fake is the forged image, is to maximize the probability that the discriminator misclassifies the forged image it generates as a real image;

[0036] The classification loss, segmentation loss, and adversarial loss are weighted and summed to obtain the total loss function. The total loss of the discriminator is The total loss of the generator is The total loss of the cross-modal vision large language model is L = ∈L cls + ζL seg + η(L D + L G ), where α, β, γ, δ, ∈, ζ, η are weight coefficients.

[0037] Furthermore, according to the total loss function, an end-to-end training method is adopted. Using the Adam optimizer and the curriculum learning strategy, at the time of inference, coarse-grained recognition is first performed, and if it is forged, then fine-grained detection is carried out to obtain the output results, including:

[0038] According to the total loss function, an end-to-end training method is adopted. The Adam optimizer is used to train the model, calculate the gradients, and update the parameters of the generator, discriminator, and cross-modal vision large language model;

[0039] During the training process, the curriculum learning strategy is adopted. Initially, the focus is mainly on the coarse-grained image forgery recognition task. As the training progresses, the weights of the fine-grained image forgery detection task and adversarial training are gradually increased;

[0040] According to model training, the input is the image to be detected and its corresponding text description. First, coarse-grained image forgery recognition is performed to determine whether the image is forged. If it is determined to be a forged image, then fine-grained image forgery detection is further carried out, and the forged type and forged area mask are output to obtain the output result.

[0041] In a second aspect, a multi-image forgery detection system based on a cross-modal vision large language model includes:

[0042] A data preprocessing module for constructing a data set and preprocessing the data set;

[0043] A feature extraction and fusion module for extracting visual features and language features respectively through a pre-trained vision Transformer and language model according to the preprocessed data to obtain cross-modal features; clustering the visual and language features respectively according to the cross-modal features, calculating the cross-modal similarity to establish a matching relationship, and performing fusion to obtain the fused multi-modal features;

[0044] A generative adversarial module for performing adversarial training through a generator and a discriminator according to the fused multi-modal features to generate an adversarial network;

[0045] A multi-granularity processing module for extracting coarse-grained global features and fine-grained local features from the fused features according to the adversarial network, respectively performing authenticity identification and type and area detection to obtain multi-granularity features;

[0046] A loss and calculation module for designing a classification loss, a segmentation loss, and an adversarial loss according to the multi-granularity features, and weighted summing to obtain a total loss function;

[0047] A training and inference module for adopting an end-to-end training method according to the total loss function, using an Adam optimizer and a curriculum learning strategy, performing coarse-grained recognition first during inference, and performing fine-grained detection if it is forged to obtain the output result.

[0048] In a third aspect, a computing device includes:

[0049] One or more processors;

[0050] A storage device for storing one or more programs, which when executed by the one or more processors cause the one or more processors to implement the method.

[0051] In a fourth aspect, a computer-readable storage medium stores a program that, when executed by a processor, implements the method.

[0052] The above solution of the present invention has at least the following beneficial effects:

[0053] The present invention integrates the visual features of images and the semantic features of texts, makes full use of multi-modal information, enables the model to understand images from different perspectives, and improves the detection ability of complex forged images. By deeply exploring the feature differences of different image forgery types and effectively fusing visual features with semantic information through intra-modal clustering and cross-modal matching, the pertinence and recognition accuracy of the model for various forgery types are enhanced.

[0054] The generative adversarial method is introduced to enable the model to continuously learn and adapt to new forgery patterns during adversarial training, improve the robustness of the model against various forgery techniques and attacks, and reduce misjudgments. A method combining coarse-grained and fine-grained features is adopted, corresponding to different detection tasks, which not only improves the detection efficiency by quickly judging authenticity through coarse-grained recognition but also ensures the detection accuracy by determining the specific type and area through fine-grained detection, realizing a comprehensive detection from the whole to the local. Brief Description of the Drawings

[0055] Figure 1 is a schematic flowchart of a method for detecting various image forgeries based on a cross-modal vision large language model provided by an embodiment of the present invention.

[0056] Figure 2 is a schematic diagram of a system for detecting various image forgeries based on a cross-modal vision large language model provided by an embodiment of the present invention.

[0057] Figure 3 is a data processing flowchart of a method for detecting various image forgeries based on a cross-modal vision large language model provided by an embodiment of the present invention. Detailed Description of the Embodiments

[0058] Hereinafter, exemplary embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in a form and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be completely conveyed to those skilled in the art.

[0059] As Figure 1 shown, an embodiment of the present invention proposes a method for detecting various image forgeries based on a cross-modal vision large language model, and the method includes the following steps:

[0060] Step 11, constructing a data set and preprocessing the data set;

[0061] Step 12, according to the preprocessed data, respectively extracting visual features and language features through a pre-trained vision Transformer and a language model to obtain cross-modal features;

[0062] Step 13: According to the cross-modal features, cluster the visual and language features respectively, calculate the cross-modal similarity to establish a matching relationship, and perform fusion to obtain the fused multi-modal features;

[0063] Step 14: According to the fused multi-modal features, perform adversarial training through a generator and a discriminator to generate an adversarial network;

[0064] Step 15: According to the adversarial network, extract coarse-grained global features and fine-grained local features from the fused features, respectively perform authenticity recognition and type and region detection to obtain multi-granularity features;

[0065] Step 16: According to the multi-granularity features, design classification loss, segmentation loss, and adversarial loss, and perform weighted summation to obtain the total loss function;

[0066] Step 17: According to the total loss function, adopt an end-to-end training method, use the Adam optimizer and the curriculum learning strategy, perform coarse-grained recognition first during inference, and if it is forged, then perform fine-grained detection to obtain the output result.

[0067] In the embodiment of the present invention, the visual features of the image and the semantic features of the text are fused, making full use of multi-modal information, enabling the model to understand the image from different perspectives, and improving the detection ability for complex forged images. By deeply mining the feature differences of different image forgery types, through intra-modal clustering and cross-modal matching, the visual features and semantic information are effectively fused, enhancing the pertinence and recognition accuracy of the model for various forgery types. The generative adversarial method is introduced, enabling the model to continuously learn and adapt to new forgery patterns during adversarial training, improving the robustness of the model to various forgery techniques and attacks, and reducing misjudgments. The method of combining coarse-grained and fine-grained features is adopted, corresponding to different detection tasks, which not only improves the detection efficiency (quickly judging authenticity through coarse-grained recognition), but also ensures the detection accuracy (determining the specific type and region through fine-grained detection), realizing a comprehensive detection from the whole to the local.

[0068] In a preferred embodiment of the present invention, the above step 11 may include:

[0069] Step 111: Collect normal images and images of various forgery types such as splicing, copy-pasting, and AIGC generation, construct a large-scale dataset, label detailed information for each forgery type of image, including the forgery area, forgery type, etc., and generate corresponding text descriptions;

[0070] Step 112: Preprocess the images according to the collected dataset.

[0071] In the embodiments of the present invention, by collecting normal images and images of various forgery types, a large-scale dataset containing rich and diverse samples can be constructed, which helps the model learn the features of different forgery types and improves the generalization ability of the model. Labeling detailed information for images of each forgery type, including the forgery area, forgery type, etc., provides a clear supervision signal for the model. The text description provides additional semantic information for the image, which helps the model understand the image content. Image preprocessing can remove noise, resize the image, normalize pixel values, etc., thereby improving the data quality. Processing steps such as data augmentation can increase the diversity of the data, making the model more robust to different image transformations, which helps improve the detection performance of the model for forged images and reduce the false detection rate and missed detection rate.

[0072] In the embodiments of the present invention, the specific steps include:

[0073] Step 111, collect normal images and images of various forgery types such as splicing, copy-pasting, and AIGC generation, construct a large-scale dataset, label detailed information for images of each forgery type, including the forgery area, forgery type, etc., and generate corresponding text descriptions. For spliced forged images, the text description can be "The image has splicing of regions from different sources, and the color transition at the splicing boundary is unnatural." For copy-pasted forged images, the description is "There are repeated textures and structures in some regions of the image." For images generated by AIGC, the description is "There are unnatural generation traces in the image details."

[0074] Step 112, according to the collected dataset, perform operations such as unified size adjustment and normalization on the images to meet the model input requirements, perform word segmentation, word embedding, etc. on the text description, and convert it into a vector representation that can be input into the language model. The processed data flows to the subsequent visual encoder and language encoder respectively.

[0075] In a preferred embodiment of the present invention, the above step 12 may include:

[0076] Step 121, according to the preprocessed data, extract the global and local features of the image through a pre-trained Vision Transformer. For different forgery types, dynamically adjust the feature weights through the attention mechanism to strengthen the feature response related to the forgery type;

[0077] Step 122, use a pre-trained language model to encode the text description to generate an embedding vector that is semantically aligned with the forgery area, ensuring the compatibility of text features and visual features in the semantic space to obtain cross-modal features.

[0078] In the embodiments of the present invention, the Vision Transformer can capture both the global structure and local details of an image, improving the accuracy of detection. Through the attention mechanism, the model can dynamically adjust the feature weights, enabling it to focus on the features most relevant to the forgery type, making the model adaptable to different forgery methods and enhancing the generalization ability and robustness of the model. The pre-trained language model can generate embedding vectors that are semantically aligned with the forged regions, ensuring the compatibility of text features and visual features in the semantic space. By combining visual features and text features, the model can generate richer cross-modal feature representations, which not only contain the visual information of the image but also incorporate the semantic information of the text, helping the model to more comprehensively understand the image content. The use of cross-modal features can improve the performance of the model. Especially when dealing with complex forgery methods, the text description provides additional supervision information, which helps the model converge faster during training and improves the training efficiency.

[0079] In the embodiments of the present invention, the specific steps include:

[0080] Step 121, use a pre-trained Vision Transformer as the visual encoder. The input image is segmented into multiple image patches, which are input into the multi-layer Transformer encoder after linear projection and position encoding to extract the local and global visual features of the image. For different forgery types, special attention is paid to their unique visual features, especially the color and texture discontinuities at the image edges and region boundaries, such as brightness mutations and color deviations; through the feature matching algorithm, similar texture and structure patterns in the image are found to detect duplicate regions; the high-frequency details, noise distribution, and consistency of local features of the image are analyzed to identify unnatural generation traces.

[0081] Step 122, use a pre-trained language model (such as BERT) as the language encoder. The text description is tokenized, token-embedded, position-encoded, and segment-encoded and then input into the multi-layer Transformer encoder of BERT to extract the semantic features of the text. The text descriptions of different forgery types endow the language features with specific semantic information, guiding the model to focus on the corresponding visual features, and the extracted language features are also passed to the multi-modal fusion module.

[0082] In a preferred embodiment of the present invention, the above step 13 may include:

[0083] Step 131, perform clustering analysis on the visual features and language features respectively according to the extracted cross-modal features to generate visual clusters and language clusters;

[0084] Step 132, calculate the similarity between cross-modal clusters according to the visual clusters and language clusters, and establish the matching relationship of visual-language clusters;

[0085] Step 133: Fuse the matched visual clusters and language clusters through an attention mechanism to generate multimodal features.

[0086] In the embodiment of the present invention, through clustering, high-dimensional cross-modal features can be mapped into a low-dimensional cluster space, thereby reducing the dimensionality of the data. Cluster analysis helps to group similar visual features and language features into the same cluster respectively, making the features more orderly and structured, which is helpful for more efficient processing and analysis of these features in subsequent steps. Cluster analysis helps to discover potential patterns and structures in visual features and language features. According to the visual clusters and language clusters, by calculating the similarity between cross-modal clusters, a matching relationship between the visual clusters and language clusters can be established. The establishment of the matching relationship helps the model to more accurately identify visual and language features related to forgery types, improve the accuracy of forgery detection, and reduce the false detection rate and missed detection rate. The matching relationship of visual-language clusters provides a basis for the fusion of multimodal features. In step 133, the attention mechanism helps to strengthen the visual and language information most relevant to the forgery type during the fusion process, which can make the generated multimodal features more focused on key information and improve the discriminability of the features. By fusing visual and language information, more expressive multimodal features can be generated, and the use of multimodal features can improve the performance of the model.

[0087] In the embodiment of the present invention, the specific steps include:

[0088] Step 131: Perform K-Means clustering on visual features and language features respectively. For visual features, set appropriate cluster numbers according to the feature differences of different forgery types, and divide them into multiple visual semantic pattern clusters. For example, cluster the edge features of splicing forgery, the repeated features of copy-paste forgery, and the high-frequency detail features of AIGC-generated images respectively. For language features, perform clustering in the same way to obtain multiple language semantic pattern clusters.

[0089] Step 132: Calculate the cosine similarity between the visual feature clusters and the language feature clusters, find the language feature cluster with the highest similarity to each visual feature cluster, and establish a cross-modal matching relationship. Through this matching, the visual features of different forgery types are associated with the corresponding semantic descriptions.

[0090] Step 133: According to the cross-modal matching relationship, splice the matched visual features and language features, and then perform a non-linear transformation through a multi-layer perceptron to obtain the fused multimodal features. During the fusion process, according to the feature weights of different forgery types, the features are weighted and combined to highlight the feature differences of different types. The fused features will flow to the generative adversarial module, classification head, and segmentation head simultaneously.

[0091] In a preferred embodiment of the present invention, the above step 14 may include:

[0092] Step 141: According to the fused multi-modal features, construct a generator. The generator takes a noise vector and semantic features as inputs and outputs a forged image consistent with the semantics.

[0093] Step 142: Input the generated forged image and the real image into a cross-modal discriminator. The discriminator adopts a visual branch and a language branch, which are fused through an attention mechanism or a gating mechanism, and outputs the authenticity probability of the image and the semantic matching score discriminator to form a generative adversarial network.

[0094] In the embodiment of the present invention, according to the fused multi-modal features and by inputting a noise vector, the generator can generate diverse forged images, improving the generalization ability of the model. The introduction of the noise vector makes the generated forged images more visually diverse and difficult to be detected by simple pattern recognition methods. The generator combines semantic features to generate forged images, and the generated images are consistent with the input features semantically, which helps to more accurately simulate the forgery means in the real world in the forgery detection task and improve the robustness of the model. By inputting the generated forged image and the real image into the discriminator together, the cross-modal discriminator can more comprehensively evaluate the authenticity and semantic matching degree of the image through the fusion of the visual branch and the language branch. The introduction of the attention mechanism or the gating mechanism enables the discriminator to more flexibly focus on key information, improving the discrimination accuracy and the stability of the generative adversarial network. The authenticity probability and semantic matching score output by the discriminator provide clear feedback signals for the generator, helping the generator to generate more realistic forged images. The design of the cross-modal discriminator enables the generative adversarial network to process multi-modal information and supports forgery detection in various forms such as images and texts.

[0095] In the embodiment of the present invention, the specific steps include:

[0096] Step 141: Concatenate a random noise vector and a semantic information vector to obtain an input vector, and perform upsampling operations through a series of transposed convolutional layers (also known as deconvolutional layers) and batch normalization layers, gradually increasing the size of the feature map while reducing the number of channels; after each transposed convolutional layer, use the ReLU activation function to introduce non-linearity, and use the Tanh activation function in the last layer to map the pixel values of the output image to the range [-1, 1]; the goal of the generator is to learn the feature distribution of different forgery types and generate a deceptive forged image G(x), and the generated forged image and the real image are input into the discriminator together.

[0097] Step 142: Input the fused multi-modal features into the subsequent network of the discriminator, which consists of a series of convolutional layers, pooling layers, and fully connected layers. The convolutional layers are used to extract local features of the image, the pooling layers are used to reduce the size of the feature map, and the fully connected layers are used to map the feature vectors to the classification space. The output of the discriminator is a probability value indicating the probability that the input image is a real image, and at the same time, it also outputs the classification result of the forgery type of the image. The output result of the discriminator is used to calculate the adversarial loss to guide the adversarial training of the generator and the discriminator.

[0098] In a preferred embodiment of the present invention, the above step 15 may include:

[0099] Step 151: Extract global features from the multi-modal fusion features according to the adversarial network, and train a classifier to judge the authenticity of the image to obtain coarse-grained global features.

[0100] Step 152: For the images determined to be forged at the coarse-grained level, further extract local features to obtain fine-grained local features.

[0101] Step 153: Fuse the coarse-grained global features and the fine-grained local features through a graph neural network to generate a joint feature representation. Train a multi-task model based on the joint features to simultaneously output authenticity labels, forgery types, and region masks to obtain multi-grained features.

[0102] In the embodiments of the present invention, by extracting global features from multi-modal fusion features according to the adversarial network, the overall information and main features of the image can be captured, which helps to provide a comprehensive perspective when initially judging the authenticity of the image and reduces the possibility of misjudgment. Training a classifier to judge the authenticity of the image can obtain a coarse-grained global feature representation, which is very useful in rapid screening and preliminary judgment and can improve the processing efficiency. Further analyzing the images determined to be forged at the coarse-grained level and extracting local features can more accurately locate the forgery areas and details, which helps to more accurately identify the forgery types and means in subsequent processing. The extraction of fine-grained local features can capture the subtle differences in forged images, thereby improving the detection accuracy. Fusing the coarse-grained global features and the fine-grained local features through a graph neural network can generate a more expressive joint feature representation. Such multi-grained features contain both the overall information of the image and capture local details, which helps to more comprehensively understand the image content. Training a multi-task model based on the joint features can simultaneously output authenticity labels, forgery types, and region masks, improving the practicality of the model, helping to share knowledge on multiple related tasks, and improving the overall performance.

[0103] In the embodiments of the present invention, the specific steps include:

[0104] Step 151: Extract global features from the multi-modal fusion features according to the adversarial network. These global features should contain the overall semantic information and structural features of the image. Construct a binary classifier (such as a fully connected layer + Softmax activation function) to judge the authenticity of the image. Use the extracted global features as input and train the classifier to minimize the prediction error of the authenticity label. The output of the middle layer (or the output of the last hidden layer) of the classifier can be used as the coarse-grained global features.

[0105] Step 152: Use the trained classifier in Step 151 to make a coarse-grained authenticity judgment on the input image. Screen out the images determined to be forged. Adopt the local feature extraction method of the convolutional neural network, such as using deeper convolutional layers or specific local feature extraction modules, to extract local features related to forgery, such as the texture, edges, colors, etc. of the forged area. Encode and pool the extracted local features to obtain fine-grained local features.

[0106] Step 153: Construct a graph neural network. Use the coarse-grained global features obtained in Step 151 and the fine-grained local features obtained in Step 152 as node inputs. Through the node update and edge propagation mechanisms of the GNN, fuse the global and local features to generate a joint feature representation. Based on the joint feature representation, construct a multi-task model that simultaneously includes three sub-tasks: authenticity label prediction, forgery type classification, and region mask generation. Use the annotated dataset (including authenticity labels, forgery types, and region masks) to train the multi-task model. Adopt a multi-task loss function to balance the loss contributions between different sub-tasks. Use an optimization algorithm to update the parameters of the model until the model achieves the best performance on the validation set. After training, the output of the middle layer of the multi-task model or the feature representation of a specific layer can be used as the multi-grained features.

[0107] In a preferred embodiment of the present invention, the above Step 16 may include:

[0108] Step 161: Design a classification loss according to the multi-grained features. Adopt the cross-entropy loss function to measure the difference between the predicted forgery type probability and the true label. For the classification loss of the discriminator Let the true label be y and the predicted probability be Then where C is the number of categories of forgery types;

[0109] Step 162: Measure the similarity between the predicted forgery region mask and the true mask through the Dice loss function as: where M pred is the predicted mask, M true is the true mask, and L seg is the measure of the similarity between the predicted forgery region mask and the true mask;

[0110] Step 163, through the generative adversarial network, the loss of the discriminator includes classification loss and adversarial loss, and the adversarial loss of the discriminator is aimed at maximizing the ability to distinguish real images from forged images, which is defined as where D(·) represents the output probability of the discriminator, and x real is a real image, and x fake is a forged image, is to maximize the probability that the discriminator misclassifies the forged image it generates as a real image;

[0111] Step 164, perform a weighted sum of the classification loss, segmentation loss, and adversarial loss to obtain the total loss function. The total loss of the discriminator is The total loss of the generator is The total loss of the cross-modal visual large language model is L = ∈L cls + ζL seg + η(L D + L G ), where α, β, γ, δ, ∈, ζ, η are weight coefficients.

[0112] In the embodiments of the present invention, the classification loss is designed according to multi-granularity features, and the cross-entropy loss function is used to measure the difference between the predicted probability of the forged type and the true label, which can make full use of the feature information of different granularities. This enables the model to analyze the image features from multiple angles and levels when judging the forged type, so as to more accurately capture the subtle differences of the forged type and effectively improve the classification accuracy of the forged type. Since the number of categories of the forged type is considered, this classification loss function has wide applicability and can handle various different types of forgery situations. Using the Dice loss function to measure the similarity between the predicted forged region mask and the true mask can more directly focus on the segmentation accuracy of the forged region. The loss of the discriminator includes classification loss and adversarial loss, where the adversarial loss is aimed at maximizing the ability to distinguish real images from forged images. Through this design, the discriminator can continuously learn and optimize its own discrimination strategy to improve the discrimination accuracy between real images and forged images. Performing a weighted sum of the classification loss, segmentation loss, and adversarial loss to obtain the total loss function can comprehensively consider the performance of the model in multiple aspects such as classification, segmentation, and adversarial. By adjusting the weight coefficients, the importance of different loss terms can be flexibly balanced according to the actual application requirements, so that the model can achieve better performance in all aspects. The loss calculation module receives the classification result of the classification head, the segmentation result of the segmentation head, and the discrimination result of the discriminator for loss calculation. This modular design makes the loss calculation process clearer and more efficient, facilitating code writing and maintenance, and improving the training efficiency and final performance of the model.

[0113] In an embodiment of the present invention, the specific steps include:

[0114] Step 161: Input the multi-granularity features into the classification head to obtain the predicted probability of the forgery type. The predicted probability is a C-dimensional vector, where C is the number of categories of the forgery type. Let the true label be y. The cross-entropy loss function is used to calculate the classification loss through the above formula.

[0115] Step 162: Input the multi-granularity features into the segmentation head to obtain the predicted mask of the forgery area. The predicted mask is a two-dimensional matrix with the same size as the input image, and the value of each element represents the probability that the pixel belongs to the forgery area. The Dice loss function is used to calculate the segmentation loss through the above formula.

[0116] Step 163: Input the real image and the forged image into the discriminator respectively to obtain the output probability of the discriminator. The adversarial loss of the discriminator aims to maximize the ability to distinguish between real images and forged images, and the adversarial loss of the discriminator is calculated through the above formula; the goal of the generator is to generate more realistic forged images to deceive the discriminator. The adversarial loss of the generator aims to maximize the probability that the discriminator misclassifies the forged images generated by it as real images, and the adversarial loss of the generator is calculated through the above formula.

[0117] Step 164: Determine the weight coefficients of the classification loss, the segmentation loss, and the adversarial loss according to the actual situation. These weight coefficients can be adjusted through experiments or experience to balance the influence of different loss terms on the total loss. The classification loss, the segmentation loss, and the adversarial loss of the discriminator are weighted and summed to calculate the total loss of the discriminator through the above formula; the total loss of the generator mainly considers the adversarial loss, and in some cases, other regularization terms can also be combined to calculate the total loss of the generator through the above formula; implement the loss calculation module, which receives the classification result of the classification head, the segmentation result of the segmentation head, and the discrimination of the discriminator, and calculates the loss according to the above formula to obtain the total loss of the discriminator and the total loss of the generator.

[0118] In a preferred embodiment of the present invention, the above step 17 may include:

[0119] Step 171: According to the total loss function, adopt an end-to-end training method, use the Adam optimizer to train the model, calculate the gradient, and update the parameters of the generator, the discriminator, and the cross-modal vision-language model.

[0120] Step 172: During the training process, adopt the curriculum learning strategy. Initially, mainly focus on the coarse-grained image forgery recognition task. As the training progresses, gradually increase the weights of the fine-grained image forgery detection task and the adversarial training.

[0121] Step 173: According to model training, input the image to be detected and its corresponding text description. First, perform coarse-grained image forgery recognition to determine whether the image is forged. If it is determined to be a forged image, then further perform fine-grained image forgery detection to output the forgery type and forgery area mask, obtaining the output result.

[0122] In the embodiment of the present invention, an end-to-end training method is adopted, and the Adam optimizer is used to train the model, which can directly update the parameters of the entire cross-modal vision large language model based on the total loss function. By accurately calculating the gradient, the direction and amplitude of model parameter adjustment can be determined, so that the parameters of the generator, discriminator, and cross-modal vision large language model can be reasonably updated in each iteration, further accelerating the convergence speed of the model. During the training process, a curriculum learning strategy is adopted. Initially, mainly focus on the coarse-grained image forgery recognition task. As the training progresses, gradually increase the weights of the fine-grained image forgery detection task and adversarial training, enabling the model to gradually delve into the learning of more complex forgery features. In the model application stage, after inputting the image to be detected and its corresponding text description, first perform coarse-grained image forgery recognition to determine whether the image is forged. This coarse-grained recognition method can quickly screen out obvious forged images, greatly improving the detection efficiency. If it is determined to be a forged image, then further perform fine-grained image forgery detection to output the forgery type and forgery area mask. The fine-grained detection can more accurately identify the forgery type and specific forgery area in the image. The cross-modal vision large language model can simultaneously process image and text description information. During the training and application processes, the text description can provide additional semantic information for image forgery detection. By fusing text and image information, the model can be applied to a wider range of scenarios. This cross-modal fusion method enables the model to handle more complex and realistic application scenarios, improving the practicality and adaptability of the model.

[0123] In the embodiment of the present invention, the specific steps include:

[0124] Step 171: Input the image and its corresponding text description into the cross-modal vision large language model, and perform forward propagation to obtain the output of the model, including the forged image of the generator, the discrimination result of the discriminator, and the outputs of the classification head and segmentation head. Calculate the loss value according to the total loss function, use the backpropagation algorithm to calculate the gradients of the parameters of the generator, discriminator, and cross-modal vision large language model according to the loss value, adopt the Adam optimizer, and update the parameters of the generator, discriminator, and cross-modal vision large language model according to the calculated gradients. Perform multiple training iterations until the model converges or reaches the preset number of training epochs.

[0125] Step 172: Divide the image forgery detection task into two levels: coarse-grained and fine-grained. The coarse-grained task mainly focuses on whether the image is forged, while the fine-grained task further focuses on the forgery type and forgery area. At the initial stage of training, set a larger weight for the coarse-grained task, and smaller weights for the fine-grained task and adversarial training. As the training progresses, gradually increase the weights of the fine-grained task and adversarial training. A weight adjustment strategy can be set, such as increasing the weights of the fine-grained and adversarial training by a certain proportion every certain number of training epochs.

[0126] Step 173: Input the image to be detected and its corresponding text description into the trained cross-modal vision-language model. The model first performs coarse-grained image forgery recognition, and uses the output of the classification head to determine whether the image is forged. If the classification result indicates that the image is a real image, directly output the result of "real image". If it is determined to be a forged image, further perform fine-grained image forgery detection, use the output of the segmentation head to obtain the forgery area mask, and at the same time determine the forgery type according to the output of the classification head; output the forgery type and forgery area mask to obtain the final detection result. For example, output "This image is a forged image, the forgery type is [specific type], and the forgery area is [mask image]".

[0127] As Figure 2 shown, an embodiment of the present invention also provides a multi-image forgery detection system 20 based on a cross-modal vision-language model, including:

[0128] A data preprocessing module 21 for constructing a data set and preprocessing the data set;

[0129] A feature extraction and fusion module 22 for extracting visual features and language features respectively through a pre-trained vision Transformer and language model according to the preprocessed data to obtain cross-modal features; according to the cross-modal features, clustering the visual and language features respectively, calculating the cross-modal similarity to establish a matching relationship, and performing fusion to obtain the fused multi-modal features;

[0130] A generative adversarial module 23 for performing adversarial training through a generator and a discriminator according to the fused multi-modal features to generate an adversarial network;

[0131] A multi-granularity processing module 24 for extracting coarse-grained global features and fine-grained local features from the fused features according to the adversarial network, respectively performing authenticity recognition and type and area detection to obtain multi-granularity features;

[0132] A loss and calculation module 25 for designing a classification loss, a segmentation loss, and an adversarial loss according to the multi-granularity features, and weighted summing to obtain a total loss function;

[0133] The training and inference module 26 is used to adopt an end-to-end training method according to the total loss function, use the Adam optimizer and the curriculum learning strategy, perform coarse-grained recognition first during inference, and then perform fine-grained detection if it is forged to obtain the output result.

[0134] In the present invention, as Figure 3 shown, a method for detecting various image forgeries based on a cross-modal vision large language model includes the following steps:

[0135] S1. Data preparation:

[0136] S11. Dataset construction: Collect normal images and images of various forgery types such as splicing, copy-pasting, and AIGC generation to construct a large-scale dataset. Label detailed information for images of each forgery type, including the forgery area, forgery type, etc., and generate corresponding text descriptions. For spliced forged images, the text description can be "The image has splicing of regions from different sources, and the color transition at the splicing boundary is unnatural"; for copy-pasted forged images, the description is "There are repeated texture and structure patterns in some regions of the image"; for images generated by AIGC, the description is "There are unnatural generation traces in the image details."

[0137] S12. Data preprocessing: Perform operations such as unified size adjustment and normalization on the images to meet the model input requirements. Perform word segmentation, word embedding, etc. on the text descriptions to convert them into vector representations that can be input into the language model. The processed data flows to the subsequent visual encoder and language encoder respectively.

[0138] S2. Cross-modal feature extraction:

[0139] S21. Visual feature extraction: Adopt a pre-trained vision Transformer (such as ViT) as the visual encoder. Divide the input image into multiple image patches, and after linear projection and position encoding, input them into a multi-layer Transformer encoder to extract local and global visual features of the image. For different forgery types, focus on their unique visual features.

[0140] Splicing forgery: Focus on the color and texture discontinuities at the edges and intersections of regions in the image, such as sudden brightness changes, color deviations, etc.

[0141] Copy-pasting forgery: Through a feature matching algorithm, find similar texture and structure patterns in the image to detect repeated regions.

[0142] AIGC-generated images: Analyze the high-frequency details, noise distribution, and consistency of local features of the image to identify unnatural generation traces. The extracted visual features will be passed to the multi-modal fusion module.

[0143] S22. Language Feature Extraction: Use a pre-trained language model (such as BERT) as the language encoder. After tokenizing, token embedding, position encoding, and segment encoding the text description, input it into the multi-layer Transformer encoder of BERT to extract the semantic features of the text. The text descriptions of different forgery types endow specific semantic information to the language features, guiding the model to focus on the corresponding visual features. The extracted language features are also passed to the multi-modal fusion module.

[0144] S3. Multi-modal Feature Fusion:

[0145] S31. Intra-modal Feature Clustering: Perform K-Means clustering on the visual features and language features respectively. For visual features, set an appropriate number of clusters according to the feature differences of different forgery types, and divide them into multiple visual semantic pattern clusters. For example, cluster the edge features of splicing forgery, the repeated features of copy-paste forgery, and the high-frequency detail features of AIGC-generated images respectively. For language features, perform clustering in the same way to obtain multiple language semantic pattern clusters.

[0146] S32. Cross-modal Feature Matching: Calculate the cosine similarity between the visual feature clusters and the language feature clusters, find the language feature cluster with the highest similarity for each visual feature cluster, and establish a cross-modal matching relationship. Through this matching, associate the visual features of different forgery types with the corresponding semantic descriptions.

[0147] S33. Feature Fusion: According to the cross-modal matching relationship, splice the matched visual features and language features, and then perform a non-linear transformation through a multi-layer perceptron (MLP) to obtain the fused multi-modal features. During the fusion process, according to the feature weights of different forgery types, perform weighted combination on the features to highlight the feature differences of different types. This fused feature will flow to the generative adversarial module, classification head, and segmentation head simultaneously.

[0148] S4. Generative Adversarial Method:

[0149] S41. Generator:

[0150] Structure Design: The generator adopts a variant architecture of the deep convolutional generative adversarial network (DCGAN) to meet the requirements of multi-modal information fusion in the present invention. Its input includes a random noise vector and the semantic information vector of the forgery type, where n and m are the dimensions of the noise vector and the semantic information vector respectively. The semantic information vector is extracted from the text description through the language encoder.

[0151] Workflow: First, concatenate the random noise vector and the semantic information vector to obtain the input vector x = [z; s]. Then, perform upsampling operations through a series of transposed convolutional layers (also known as deconvolutional layers) and batch normalization layers, gradually increasing the size of the feature map while reducing the number of channels. After each transposed convolutional layer, use the ReLU activation function to introduce non-linearity, except for the last layer which uses the Tanh activation function to map the pixel values of the output image to the range [-1, 1]. The goal of the generator is to learn the feature distributions of different forgery types and generate deceptive forged images G(x). The generated forged images are input into the discriminator together with the real images.

[0152] S42. Discriminator:

[0153] Structure design: The discriminator uses the architecture of a cross-modal vision-language large model, which can process both the visual features of images and the semantic features of text simultaneously. It receives the input images (including real images, generated forged images, and actual forged images) and the corresponding text descriptions, extracts visual features and language features through a visual encoder and a language encoder respectively, and then obtains the fused multi-modal features through a multi-modal fusion module.

[0154] Workflow: Input the fused multi-modal features into the subsequent network of the discriminator, which consists of a series of convolutional layers, pooling layers, and fully connected layers. The convolutional layers are used to extract the local features of the image, the pooling layers are used to reduce the size of the feature map, and the fully connected layers are used to map the feature vector to the classification space. The output of the discriminator is a probability value indicating the probability that the input image is a real image, and it also outputs the classification result of the forgery type of the image. The output result of the discriminator is used to calculate the adversarial loss to guide the adversarial training of the generator and the discriminator.

[0155] Adversarial training process: During the training process, the generator and the discriminator perform adversarial training. The generator continuously optimizes the generated forged images to make them more difficult to be recognized by the discriminator; the discriminator continuously improves its discrimination ability to distinguish real images and forged images. Specifically, the training goal of the discriminator is to maximize the probability of correctly classifying real images and forged images, while the training goal of the generator is to maximize the probability that the discriminator misclassifies the forged images it generates as real images. Through this adversarial training, the robustness of the model to various forgery types is enhanced.

[0156] S5. Multi-granularity feature combination:

[0157] S51. Coarse-grained Feature and Image Forgery Recognition: Extract the global features of the image from the fused multi-modal features as coarse-grained features, and use a simple classifier (such as a fully connected layer) to perform preliminary image forgery recognition based on the coarse-grained features. This task mainly judges whether the image is forged, without involving specific forgery types and region localization.

[0158] S52. Fine-grained Feature and Image Forgery Detection: On the basis of coarse-grained recognition, further extract the local features of the image from the fused features as fine-grained features, combine the features after multi-modal fusion, and use a more complex classifier and semantic segmentation head to perform image forgery detection to determine the specific forgery type and forgery region.

[0159] S6. Loss Function Design:

[0160] S61. Classification Loss: Adopt the cross-entropy loss function to measure the difference between the predicted forgery type probability and the true label.

[0161] S62. Segmentation Loss: Use the Dice loss function to measure the similarity between the predicted forgery region mask and the true mask.

[0162] S63. Adversarial Loss: For the generative adversarial network, the loss of the discriminator includes classification loss and adversarial loss. The adversarial loss of the discriminator aims to maximize the ability to distinguish between real images and forged images, and the adversarial loss of the generator aims to maximize the probability that the discriminator misclassifies the forged images generated by it as real images.

[0163] S64. Total Loss: Weightedly sum the classification loss, segmentation loss, and adversarial loss to obtain the total loss function. The total loss of the discriminator, the total loss of the generator, and the total loss of the cross-modal vision-large language model, where the weight coefficients can be adjusted according to the actual situation. The loss calculation module receives the classification results of the classification head, the segmentation results of the segmentation head, and the discrimination results of the discriminator to perform loss calculation.

[0164] S7. Model Training and Inference:

[0165] S71. Model Training: Adopt an end-to-end training method and use the Adam optimizer to train the model. During the training process, alternately update the parameters of the generator, discriminator, and cross-modal vision-large language model. Adopt the curriculum learning strategy. At the beginning of training, mainly focus on the coarse-grained image forgery recognition task. As the training progresses, gradually increase the weight of the fine-grained image forgery detection task and adversarial training to improve the comprehensive performance of the model. For each training batch, the specific training steps are as follows:

[0166] Training the discriminator: Fix the parameters of the generator. Sample a batch of real images from the real image dataset. At the same time, the generator generates a batch of forged images based on random noise and semantic information. Input these images and their corresponding text descriptions into the discriminator, calculate the total loss of the discriminator, and update the parameters of the discriminator through backpropagation.

[0167] Training the generator: Fix the parameters of the discriminator. The generator generates a batch of forged images and inputs them into the discriminator to calculate the total loss \(L\) of the generator G , and update the parameters of the generator through backpropagation.

[0168] Training the cross-modal vision-language model: Input the input image and its corresponding text description into the cross-modal vision-language model, calculate its total loss \(L\), and update the parameters of the model through backpropagation.

[0169] S72. Model inference: Input the image to be detected and its corresponding text description (hint information can be provided according to requirements, such as "detect whether the image is generated by AIGC"). The model first performs coarse-grained image forgery recognition to determine whether the image is forged. If it is determined to be a forged image, further fine-grained image forgery detection is performed to output the specific forgery type and the mask of the forged area.

[0170] The above is the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for detecting various image forgeries based on a cross-modal vision-language model, characterized in that, The method includes: Construct a dataset and preprocess the dataset; According to the preprocessed data, extract visual features and language features respectively through a pre-trained vision Transformer and a language model to obtain cross-modal features; According to the cross-modal features, cluster the visual and language features respectively, calculate the cross-modal similarity to establish a matching relationship, and perform fusion to obtain the fused multi-modal features; According to the fused multi-modal features, perform adversarial training through a generator and a discriminator to generate an adversarial network; According to the adversarial network, extract coarse-grained global features and fine-grained local features from the fused features, and perform authenticity recognition and type and region detection respectively to obtain multi-granularity features; According to the multi-granularity features, design a classification loss, a segmentation loss, and an adversarial loss, and perform weighted summation to obtain a total loss function; According to the total loss function, adopt an end-to-end training method, use the Adam optimizer and the curriculum learning strategy, first perform coarse-grained recognition during inference, and if it is forged, then perform fine-grained detection to obtain the output result.

2. The method for detecting various image forgeries based on a cross-modal vision-language model according to claim 1, wherein According to the preprocessed data, extract visual features and language features respectively through a pre-trained vision Transformer and a language model to obtain cross-modal features, including: According to the preprocessed data, extract the global and local features of the image through a pre-trained vision Transformer, and for different forgery types, dynamically adjust the feature weights through an attention mechanism to strengthen the feature response related to the forgery type; Use a pre-trained language model to encode the text description to generate an embedding vector aligned with the semantics of the forged area, ensuring the compatibility of the text features and the visual features in the semantic space to obtain cross-modal features.

3. The method for detecting various image forgeries based on a cross-modal vision-large language model according to claim 1, wherein According to the cross-modal features, cluster the visual and language features respectively, calculate the cross-modal similarity to establish a matching relationship, and perform fusion to obtain the fused multi-modal features, including: According to the extracted cross-modal features, perform clustering analysis on the visual features and the language features respectively to generate visual clusters and language clusters; According to the visual clusters and the language clusters, calculate the similarity between the cross-modal clusters to establish a matching relationship between the visual-language clusters; Fuse the matched visual clusters and language clusters through an attention mechanism to generate multi-modal features.

4. The method for detecting various image forgeries based on a cross-modal vision-language model according to claim 1, wherein According to the fused multi-modal features, perform adversarial training through a generator and a discriminator to generate an adversarial network, including: According to the fused multi-modal features, construct a generator, where the generator inputs a noise vector and semantic features and outputs a forged image consistent with the semantics; Input the generated forged image and the real image into a cross-modal discriminator together. The discriminator adopts a visual branch and a language branch, and is fused through an attention mechanism or a gating mechanism to output the authenticity probability of the image and the semantic matching score discriminator to generate an adversarial network.

5. The method for detecting various image forgeries based on a cross-modal vision-large language model according to claim 1, characterized in that, According to the adversarial network, extract coarse-grained global features and fine-grained local features from the fused features, and perform authenticity recognition and type and region detection respectively to obtain multi-granularity features, including: According to the adversarial network, extract global features from the multi-modal fused features, and train a classifier to judge the authenticity of the image to obtain coarse-grained global features; For the images that are coarsely determined to be forged, local features are further extracted to obtain fine-grained local features; The coarse-grained global features and the fine-grained local features are fused through a graph neural network to generate a joint feature representation. Based on the joint features, a multi-task model is trained to simultaneously output authenticity labels, forgery types, and region masks to obtain multi-grained features.

6. The method for detecting various image forgeries based on a cross-modal vision-large language model according to claim 1, characterized in that, According to the multi-grained features, a classification loss, a segmentation loss, and an adversarial loss are designed, and weighted summation is used to obtain a total loss function, including: According to multi-granularity features, a classification loss is designed. The cross-entropy loss function is adopted to measure the difference between the predicted probability of the forgery type and the true label. For the classification loss of the discriminator Let the true label be y and the predicted probability be Then where C is the number of categories of forgery types; The similarity between the predicted forged region mask and the ground truth mask is measured by the Dice loss function as follows: where M pred is the predicted mask, M true is the ground truth mask, and K seg is the similarity between the predicted forged region mask and the ground truth mask; Through the generative adversarial network, the loss of the discriminator includes the classification loss and the adversarial loss. The adversarial loss of the discriminator aims to maximize the ability to distinguish between real images and forged images, which is defined as where D(·) represents the output probability of the discriminator, x real is a real image, and x fake is a forged image, is to maximize the probability that the discriminator misclassifies the forged image it generates as a real image; The classification loss, segmentation loss and adversarial loss are weighted summed to obtain the total loss function. The total loss of the discriminator is: The total loss of the generator is The total loss of the cross-modal visual language model is L = ∈ L cls +ζL seg +η(L D +L G ), where α, β, γ, δ, ∈, ζ, η are weight coefficients.

7. The method for detecting various image forgeries based on a cross-modal vision-large language model according to claim 1, wherein According to the total loss function, in an end-to-end training manner, the Adam optimizer and the curriculum learning strategy are used. During inference, coarse-grained recognition is performed first. If it is determined to be forged, then fine-grained detection is carried out to obtain the output results, including: According to the total loss function, in an end-to-end training manner, the Adam optimizer is used to train the model, calculate the gradients, and update the parameters of the generator, discriminator, and cross-modal vision-language model; During the training process, the curriculum learning strategy is adopted. Initially, the focus is mainly on the coarse-grained image forgery recognition task. As the training progresses, the weights of the fine-grained image forgery detection task and adversarial training are gradually increased; Based on model training, the image to be detected and its corresponding text description are input. First, coarse-grained image forgery recognition is performed to determine whether the image is forged. If it is determined to be a forged image, then fine-grained image forgery detection is further carried out to output the forgery type and the forgery region mask to obtain the output results.

8. A multi-image forgery detection system based on a cross-modal vision-large language model, which implements the method described in any one of claims 1 to 7, characterized in that, Including: A data preprocessing module for constructing a data set and preprocessing the data set; A feature extraction and fusion module for extracting visual features and language features respectively through a pre-trained vision Transformer and language model according to the preprocessed data to obtain cross-modal features; According to the cross-modal features, the visual and language features are respectively clustered, the cross-modal similarity is calculated to establish a matching relationship, and fusion is performed to obtain the fused multi-modal features; A generative adversarial module for performing adversarial training through a generator and a discriminator according to the fused multi-modal features to generate an adversarial network; A multi-grained processing module for extracting coarse-grained global features and fine-grained local features from the fused features according to the adversarial network, and respectively performing authenticity recognition and type and region detection to obtain multi-grained features; A loss and calculation module for designing a classification loss, a segmentation loss, and an adversarial loss according to the multi-grained features, and weighted summation to obtain a total loss function; A training and inference module for adopting an end-to-end training manner according to the total loss function, using the Adam optimizer and the curriculum learning strategy, and performing coarse-grained recognition first during inference. If it is forged, then fine-grained detection is carried out to obtain the output results.

9. A computing device, characterized in that, Including: One or more processors; A storage device for storing one or more programs, which when executed by the one or more processors cause the one or more processors to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A program is stored in the computer-readable storage medium, and when the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images