Large model training method for forged image detection and forged image detection method

Abstract

The invention provides a large model training method for forged image detection and a forged image detection method, and belongs to the technical field of image recognition. According to the method, the understanding of the preset large model on the description information of the forged image is enhanced through the comparative learning of the image and the image description; furthermore, by calculating the second loss value and the third loss value, the accuracy of the preset large model on a forged image result is improved, and an accurate reasoning reason of the forged image is given.

Description

Technical Field

[0001] The present disclosure relates to the field of image recognition technology, and particularly to a large model training for forged image detection and a forged image detection method. Background Art

[0002] Deep Generative Technology (DGT) is a branch in the field of deep learning, mainly involving generating new data samples similar to the training data.

[0003] The forged images generated by using deep generative technology have high resolution and lack obvious artifacts. It is difficult for humans to identify the authenticity of the generated forged images. The spread of these forged images poses a threat to the credibility of the media and the security of face privacy. Summary of the Invention

[0004] The present disclosure proposes a large model training for forged image detection and a forged image detection method.

[0005] The first aspect embodiment of the present disclosure proposes a large model training method for forged image detection, including:

[0006] Generating description information corresponding to each image in a pre-acquired image training set;

[0007] Performing contrastive learning based on the images in the image training set and the description information, and calculating a first loss value;

[0008] For any image in the image training set, inputting the image and the description information corresponding to the image into a preset large model, and determining a second loss value based on the output content of the preset large model;

[0009] Calculating the probability that the preset large model outputs the description information corresponding to the image based on the image and the preset output rule of the preset large model, and taking the probability as the third loss value of the image;

[0010] Generating a target loss value corresponding to the image based on the first loss value, the second loss value, and the third loss value;

[0011] Optimizing the preset large model based on the relationship between the target loss value and a preset loss threshold, and outputting a large model for forged image detection.

[0012] In the embodiments of the present disclosure, the image training set includes a forged image training subset and a real image training subset, and the description information includes forged attribute descriptions corresponding to each forged image in the forged image training subset and image content descriptions corresponding to each real image in the real image training subset.

[0013] Generating the description information corresponding to each image in the pre-acquired image training set includes:

[0014] Generating the forgery attribute description of each forged image in the forged image training subset based on a preset large model; and generating the content description of each real image in the real image training subset based on the preset large model; wherein, the forgery attribute description is the reason for forgery at a preset angle, and the preset angle includes at least one of illumination, texture, semantics, edge, and quality.

[0015] In the embodiments of the present disclosure, calculating the first loss value by performing contrastive learning based on the images in the image training set and the description information includes:

[0016] For any object in the image training set, matching a matching object that matches the object in the image training set; wherein, the any object is any image and / or any description information in the image training set;

[0017] Generating the first loss value based on the object, the matching object, and the unmatched objects in the image training set.

[0018] In the embodiments of the present disclosure, for any object in the image training set, matching a matching object that matches the object in the image training set includes:

[0019] For any image in the image training set, matching the first matching description information and the first matching image that match the image in the image training set; and for any description information in the image training set, matching the second matching image and the second matching description information that match the description information.

[0020] In the embodiments of the present disclosure, generating the first loss value based on the object, the matching object, and the unmatched objects in the image training set includes:

[0021] Calculating the first similarity between the object and the matching object; calculating the first ratio of the first similarity to a preset temperature parameter;

[0022] Calculating the second similarity between the object and any unmatched object in the image training set; calculating the second ratio of the second similarity to the preset temperature parameter; calculating the sum of the second ratios corresponding to all unmatched objects in the image training set to obtain a first value;

[0023] Taking the logarithm of the ratio of the first value to the second value as the first loss value; wherein the second value is the exponent of the first ratio.

[0024] In an embodiment of the present disclosure, generating the first loss value based on the object, the matching object, and the unmatched objects in the image training set includes:

[0025] Generating a first sub-loss value according to the image, the first matching description information, and the unmatched description information in the image training set;

[0026] Generating a second sub-loss value according to the description information, the second matching image, and the unmatched images in the image training set;

[0027] Generating a third sub-loss value according to the image, the first matching image, and the unmatched images in the image training set;

[0028] Generating a fourth sub-loss value according to the description information, the second matching description information, and the unmatched description information in the image training set;

[0029] Taking the mean of the first sub-loss value, the second sub-loss value, the third sub-loss value, and the fourth sub-loss value as the first loss value.

[0030] In an embodiment of the present disclosure, inputting the image and the description information corresponding to the image into a preset large model and determining a second loss value based on the output content of the preset large model includes:

[0031] Inputting the image and the description information corresponding to the image into a preset large model to obtain the output content of the preset large model;

[0032] Performing an activation operation based on the output content to obtain an output probability;

[0033] Calculating the second loss value based on the output probability.

[0034] In an embodiment of the present disclosure, calculating the probability that the preset large model outputs the description information corresponding to the image based on the image and the preset output rule of the preset large model, and taking the probability as the third loss value of the image includes:

[0035] For any vocabulary in the description information corresponding to the image, predicting the probability of the vocabulary based on a first vocabulary set and a first image set, where the first vocabulary set is the vocabulary before the vocabulary, and the first image set is the images in the image training set before the image corresponding to the description information to which the vocabulary belongs;

[0036] Calculating the mean of the probability logarithms of the respective vocabularies in the description information corresponding to the image as the probability of the description information corresponding to the image.

[0037] In an embodiment of the present disclosure, generating the target loss value corresponding to the image based on the first loss value, the second loss value, and the third loss value includes:

[0038] Calculating the product of the first loss value and a first preset parameter to obtain a first initial value;

[0039] Calculating the product of the second loss value and a second preset parameter to obtain a second initial value;

[0040] Calculating a third initial value based on the third loss value, the first preset parameter, and the second preset parameter;

[0041] Taking the sum of the first initial value, the second initial value, and the third initial value as the target loss value.

[0042] An embodiment of the second aspect of the present disclosure provides a forged image detection method, and the method includes:

[0043] Obtaining an image to be detected;

[0044] Inputting the image to be detected and a preset output rule into a large model for forged image detection obtained by the large model training method for forged image detection as described in the first aspect or any optional implementation manner of the first aspect above, to obtain a detection result of the image to be detected and the reason corresponding to the detection result; wherein, if the detection result is a real image, the reason corresponding to the detection result is the description information of the image to be detected; if the detection result is a forged image, the reason corresponding to the detection result is the reason for the forged image of the image to be detected.

[0045] An embodiment of the third aspect of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor runs the computer program to implement the methods described in the first aspect or any optional implementation manner of the first aspect, the second aspect, and any optional implementation manner of the second aspect above.

[0046] An embodiment of the fourth aspect of the present disclosure provides a computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to implement the methods described in the first aspect and any optional implementation manner of the first aspect, the second aspect, and any optional implementation manner of the second aspect above.

[0047] The technical solution provided in the embodiment of the present disclosure has at least the following technical effects or advantages:

[0048] Generate description information corresponding to each image in the pre-acquired image training set; perform contrastive learning based on the images and description information in the image training set, and calculate the first loss value; further, input the image and the description information corresponding to the image into a preset large model, and determine the second loss value based on the output content of the preset large model; calculate the probability that the preset large model outputs the description information corresponding to the image based on the image and the preset output rule of the preset large model, and use the probability as the third loss value; generate the target loss value corresponding to the image based on the first loss value, the second loss value, and the third loss value; optimize the preset large model based on the relationship between the target loss value and the preset loss threshold, and output a large model for forged image detection.

[0049] Through the above implementation method, by performing contrastive learning on the image and the image description, the understanding of the description information of the forged image by the preset large model is enhanced; further, by calculating the second loss value and the third loss value, the accuracy of the result of the preset large model for the forged image is improved, and the reasoning reason for the accurate forged image is given.

[0050] Additional aspects and advantages of the present disclosure will be given in part in the following description, will become apparent in part from the following description, or will be understood through the practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present disclosure. Moreover, throughout the drawings, the same reference numerals are used to represent the same components.

[0052] In the drawings:

[0053] Figure 1 Shows a flowchart of a method for training a large model for forged image detection provided by an embodiment of the present disclosure;

[0054] Figure 2 Shows a schematic diagram of generating description information in a method for training a large model for forged image detection provided by an embodiment of the present disclosure;

[0055] Figure 3 Shows a schematic diagram of generating each loss value in a method for training a large model for forged image detection provided by an embodiment of the present disclosure;

[0056] Figure 4 Shows a flowchart of a method for forged image detection provided by an embodiment of the present disclosure;

[0057] Figure 5Shows an application schematic diagram of a forged image detection method provided by an embodiment of the present disclosure;

[0058] Figure 6 Shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure;

[0059] Figure 7 Shows a schematic diagram of a storage medium provided by an embodiment of the present disclosure. Detailed implementation manners

[0060] Hereinafter, the exemplary embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0061] It should be noted that unless otherwise specified, the technical terms or scientific terms used in the present disclosure should have the ordinary meanings understood by those skilled in the art to which the present disclosure belongs.

[0062] The embodiment of the present disclosure proposes a large model training method for forged image detection, as Figure 1 Shown is a large model training method for forged image detection provided by an embodiment of the present disclosure, including the following steps:

[0063] In step S11, generate description information corresponding to each image in the pre-acquired image training set.

[0064] Exemplarily, the image training set is an image for training a large model for forged image detection. Specifically, it can be obtained through an open-source data set. Generally, in order to improve the training accuracy of the preset large model, the image training set needs to include real images and forged images so that the preset large model can better learn the difference between real images and forged images.

[0065] The description information can be the content description corresponding to each image in the image training set, specifically, it can be description information in text form. In some embodiments, the images in the image training set can be input into models such as semantic recognition models and GPT-4o respectively to obtain the description information corresponding to the images.

[0066] In some embodiments, the image training set includes a forged image training subset and a real image training subset, and the description information includes the forged attribute description corresponding to each forged image in the forged image training subset and the image content description corresponding to each real image in the real image training subset.

[0067] Based on this, the above-mentioned step S11 is implemented in the following manner: generating forged attribute descriptions for each forged image in the forged image training subset based on a preset large model; and generating content descriptions for each real image in the real image training subset based on the preset large model; wherein, the forged attribute description is the reason for forgery at a preset angle, and the preset angle includes at least one of illumination, texture, semantics, edge, and quality.

[0068] Exemplarily, for the forged images and real images in the image training set, the corresponding description information can be different, so that the preset large model can learn the differences between real images and forged images according to the description information. Specifically, the description information for the forged images can be forged attribute descriptions. Usually, the forged attribute description can be the content description of the forged image at a preset angle. To improve the accuracy of the forged attribute description, the preset angle can be multiple. For example, the forged attribute description can be generated from aspects such as illumination, texture, and semantics.

[0069] For the description information of the real images, it only needs to describe the content included in the real images. For example, the real images can be input into BLIP-3 to obtain the content descriptions corresponding to the real images. Similarly, the content descriptions of the real images can also be described from multiple angles. For example, they can be described from angles including objects, the forms of objects, etc. The embodiments of the present disclosure do not limit the description information corresponding to each image in the image training set, and those skilled in the art can determine it according to the actual situation.

[0070] As Figure 2 shown, it is a schematic diagram of generating description information for real images and forged images respectively. For the real image, the corresponding description information is "In the center of the photo, a professional wrestler stands upright in the wrestling ring. He is shirtless, revealing a bright red and white checkered pair of shorts. The black center of a gold belt, symbolizing his championship status, is shown around his waist. The wrestler's pose in the ring indicates that he has just won a match, and he raises his arms in celebration." For the forged image, the corresponding description information is "Semantic unreasonableness: The positions and interactions between the wrestlers appear unnatural. Abnormal image texture: The textures of the skin and clothes are inconsistent, especially on the arms and back. Inconsistent lighting and shadows: Their shadows and the shadow of the wrestler do not match the direction of the light source."

[0071] In step S12, contrastive learning is performed based on the images and description information in the image training set, and a first loss value is calculated.

[0072] Exemplarily, before performing contrastive learning on the images in the image training set and the corresponding description information of the images, it is first necessary to extract the features of the images and the description information. Specifically, a pre-trained convolutional neural network can be used, such as ResNet, ViT, and obtained based on the CLIP visual encoder and linear mapping layer, etc., to extract the image feature F v . Use a pre-trained text encoder, such as BERT, GPT, etc., to extract the image description feature F of the description information a .

[0073] To facilitate contrastive learning of the image features and the image description features, the image features and the image description features can also be encoded in advance, and the encoded image features and the image description features are used for contrastive learning. Further, dimensionality reduction, normalization, etc. can also be used to reduce the complexity of the encoded image features and the image description features

[0074] The real images and the forged images in the image training set are positive samples and negative samples respectively. Among them, the real images and the corresponding content descriptions are used as positive samples; the forged images and the corresponding forged attribute descriptions are used as negative samples. When performing contrastive learning, the cosine similarity or other metric methods can be used to calculate the similarity between the image features and the image description features. The loss function for calculating the similarity using InfoNCE and triplet loss is used as the first loss value

[0075] In some embodiments, the above step S12 can be implemented in the following manner: for any object in the image training set, a matching object that matches the object is matched in the image training set; wherein, any object is any image and / or any description information in the image training set; based on the object, the matching object, and the unmatched objects in the image training set, a first loss value is generated

[0076] Exemplarily, for any object (image or description information), a matching object that matches it is found in the image training set. The matching can be performed based on the similarity of features (such as cosine similarity). The selection of the matching object can be to sort the similarity scores and select the object with the highest score, or to determine whether to match based on a certain threshold. The matching objects that match any object are excluded from the training set to obtain a set of unmatched objects

[0077] Design a loss function based on objects, matching objects, and unmatched objects. The loss function should be able to reflect the similarity between an object and a matching object, as well as the difference from an unmatched object. Commonly used loss functions include Contrastive Loss, Triplet Loss, etc., which can encourage the similarity between matching objects to be greater than that between unmatched objects. According to the loss function, calculate the loss values between the object, the matching object, and the unmatched object, that is, the first loss value.

[0078] By optimizing the first loss value, the model can learn the matching relationship between images and description information, thereby improving performance in related tasks (such as image retrieval, image caption generation, etc.). Since the design of the loss function takes into account unmatched objects, the model can learn how to distinguish between matching and non-matching objects during training, thus enhancing the generalization ability. By introducing unmatched objects, the model can be exposed to more diverse data during training, which helps to improve the model's robustness to noise and abnormal data.

[0079] In some embodiments, the first loss value can be calculated in the following way: Generate the first loss value based on the object, the matching object, and the unmatched objects in the image training set, including:

[0080] Calculate the first similarity between the object and the matching object; calculate the first ratio of the first similarity to the preset temperature parameter;

[0081] Calculate the second similarity between the object and any unmatched object in the image training set; calculate the second ratio of the second similarity to the preset temperature parameter; calculate the sum of the second ratios corresponding to all unmatched objects in the image training set to obtain the first value;

[0082] Take the logarithm of the ratio of the first value to the second value as the first loss value; where the second value is the exponent of the first ratio.

[0083] For any object in the image training set, match the matching object that matches the object in the image training set, including:

[0084] For any image in the image training set, match the first matching description information and the first matching image that match the image in the image training set; and for any description information in the image training set, match the second matching image and the second matching description information that match the description information.

[0085] Based on this, generate the first loss value based on the object, the matching object, and the unmatched objects in the image training set, including:

[0086] Generate a first sub-loss value based on the image, the first matching description information, and the unmatched description information in the image training set;

[0087] Generate a second sub-loss value based on the description information, the second matching image, and the unmatched images in the image training set;

[0088] Generate a third sub-loss value based on the image, the first matching image, and the unmatched images in the image training set;

[0089] Generate a fourth sub-loss value based on the description information, the second matching description information, and the unmatched description information in the image training set;

[0090] Take the mean of the first sub-loss value, the second sub-loss value, the third sub-loss value, and the fourth sub-loss value as the first loss value.

[0091] Exemplarily, the first sub-loss value is calculated by the following formula:

[0092]

[0093] where, represents the first sub-loss value; F v represents the image; represents the first matching description information; represents the unmatched description information; n represents the number of images in the image training set; τ represents a trainable temperature hyperparameter, and the initial value is set to 0.07.

[0094] The second sub-loss value The third sub-loss value and the fourth sub-loss value are calculated in the same way as above and will not be elaborated here.

[0095] Among them, the first loss value is calculated by the following formula:

[0096]

[0097] In step S13, for any image in the image training set, input the image and the description information corresponding to the image into the preset large model, and determine the second loss value based on the output content of the preset large model.

[0098] Exemplarily, input the image, the description content, and the corresponding output content specified by the preset large model in the image training set into the preset large model. Here, the image, the description content, and the output content can be converted into the format adapted to the preset large model. For example, the image, the description content, and the output content can also be encoded and then input into the preset large model.

[0099] After the pre-set large model processes the input data, it generates output content. This output content can be a probability distribution (such as the output of a classification task) or an embedding vector (such as the output of a retrieval task). If the task is to determine whether an image matches the description information, the output content can be compared with a label indicating match / mismatch, and the binary cross-entropy loss function can be used. Using the defined loss function, a second loss value is calculated based on the output content of the model and the true label (or the expected output).

[0100] In some embodiments, inputting an image and the description information corresponding to the image into the pre-set large model, and determining the second loss value based on the output content of the pre-set large model includes:

[0101] Inputting an image and the description information corresponding to the image into the pre-set large model to obtain the output content of the pre-set large model; performing an activation operation based on the output content to obtain an output probability; calculating the second loss value based on the output probability.

[0102] Exemplarily, encoding the image features and encoding the image description as inputs and passing them to the pre-set large model (which may be a multi-modal model capable of processing image and text inputs).

[0103] The pre-set large model outputs a result, which may be a vector representing the probability distribution of whether the image is real or fake, or a sequence of words containing words describing the authenticity of the image.

[0104] Use the output template to locate the authenticity classification result. The output template is a judgment on whether the image is a real image or a forged image, and the reason for the judgment. The output template represents a pre-set string or sequence of words used to locate the key position determining the authenticity classification result in the sequence of words output by the model. The output template should match the format of the model output and be able to uniquely identify the words representing the authenticity classification result.

[0105] Search for a subsequence in the sequence of words output by the model that matches the output template. Find the starting position index i of the matching subsequence. According to the length l of the output template, determine the specific position of the word determining the authenticity classification result in the output sequence.

[0106] For example, obtain the position index in the output of the pre-set large model that determines the authenticity classification result:

[0107] k = {i + l | y i:i+l = p, i ∈ [0, N - l]},

[0108] N represents the length of the text of the output result; i is the text position, that is, the index; l is the length of the output template (search for the text of length l in y, where y represents the vocabulary of the preset large model), and the vocabulary probability at this position determines the result of the image authenticity classification.

[0109] The vocabulary probability determines the authenticity classification: Extract the probability of the vocabulary that determines the authenticity classification result. If this vocabulary is "fake", then use its probability as the probability that the binary classification result is the positive class (that is, the image is a forged image).

[0110] Since the probability output by the model may be very large or close to 1, directly using it may lead to numerical instability or be unfavorable for subsequent processing. Therefore, use an activation function (such as the sigmoid function) to compress the probability value to between 0 and 1 for better subsequent processing of the binary classification task.

[0111]

[0112] Among them, represents the output result, id fake represents the position index of the forged image output structure in the vocabulary; k is the vocabulary at the id fake position.

[0113] Optimize the binary classification task of image authenticity based on the binary cross-entropy loss, which is used to measure the difference between the model prediction result and the true label. The formula is as follows:

[0114]

[0115] Among them, represents the second loss value; is the true value of the binary classification of image authenticity, and y' represents the value between 0 and 1 (1 for fake, 0 for true) obtained by the activation function.

[0116] In step S14, based on the image and the preset output rule of the preset large model, calculate the probability of the description information corresponding to the image output by the preset large model, and use the probability as the third loss value of the image.

[0117] Exemplarily, the third loss value is the probability of obtaining the description information corresponding to the image on the premise of the known image and the specified output rule of the preset large model.

[0118]

[0119] Among them, represents the third loss value; L represents the number of words in the description information; X at represents the description information; X v represents the image; X q represents the preset output rule.

[0120] In some embodiments, in the above step S14, it can be determined in the following manner: Based on the image and the preset output rules of the preset large model, calculate the probability of the description information corresponding to the output image of the preset large model, and use the probability as the third loss value of the image, including:

[0121] For any vocabulary in the description information corresponding to the image, based on the first vocabulary set and the first image set, predict the probability of the vocabulary. The first vocabulary set is the vocabulary before the vocabulary, and the first image set is the image in the image training set before the image corresponding to the vocabulary to which the vocabulary belongs;

[0122] Calculate the mean of the probability logarithms of each vocabulary in the description information corresponding to the image as the probability of the description information corresponding to the image.

[0123] Exemplarily, among them, the probability of the target vocabulary p(X at |X v ,X q ) is calculated as follows:

[0124]

[0125] Among them, θ represents the trainable parameters of the large model, and i represents the position of the vocabulary in the description information.

[0126] Use the predicted large language model to predict the probability of the current vocabulary based on the previous vocabulary (the vocabulary before the current vocabulary in the description information).

[0127] For any vocabulary in the description information, use the preset large model to predict the probability of the vocabulary based on its previous vocabulary sequence (the first vocabulary set).

[0128] For all the vocabulary in the description information, calculate the logarithm of their predicted probabilities. Calculate the mean of these logarithmic probabilities as the probability estimate of the entire description information. Here, logarithmic probabilities are used because they are easier to handle mathematically. For example, it can avoid the underflow problem caused by too small probability values.

[0129] As Figure 3 shown, it is a schematic diagram for calculating each loss value. Generate the preset output rule X q : Is this photo real or fake? Please describe the age of the photo and answer "This photo is real" or "The age of this photo is fake". Encode the preset output rule to get F q . The description information X a of the forged image is: The positions and interactions between the wrestlers look unnatural. The textures of the skin and clothes are inconsistent, especially on the arms and back. The shadows on the mat and the wrestling do not match the direction of the light source. Encode the description information to get F a。For F q Perform average pooling (operations such as dimensionality reduction or normalization) to obtain F q '; Input F a into a preset large model (large language model).

[0131] Input the forged image into the visual encoder to obtain the encoding F of the forged image v 。For F v Perform average pooling (operations such as dimensionality reduction or normalization) to obtain F v '. Compare and learn between F v ' and F q ' to obtain the first loss value. Input F a , F q and F v into the large language model (preset large model), and obtain the second loss value according to the output content of the large language model; obtain the third loss value according to the answer of the large language model to the question.

[0132] In step S15, based on the first loss value, the second loss value, and the third loss value, generate the target loss value corresponding to the image.

[0133] Exemplarily, a weight can be assigned to each loss value. These weights can be fixed hyperparameters or can be dynamically adjusted through a certain strategy (such as learning rate scheduling). Multiply each loss value by its corresponding weight, and then add these weighted loss values to obtain the target loss value.

[0134] In some optional embodiments, generating the target loss value corresponding to the image based on the first loss value, the second loss value, and the third loss value includes:

[0135] Calculate the product of the first loss value and the first preset parameter to obtain the first initial value;

[0136] Calculate the product of the second loss value and the second preset parameter to obtain the second initial value;

[0137] Based on the third loss value, the first preset parameter, and the second preset parameter, calculate to obtain the third initial value;

[0138] Take the sum of the first initial value, the second initial value, and the third initial value as the target loss value.

[0139] Exemplarily,

[0140] Among them, λ1 represents the second preset parameter; λ2 represents the first preset parameter. Through experimental tuning of parameters, it can be determined that λ1 = 0.1 and λ2 = 0.4; represents the second loss value; represents the first loss value; Represents the third loss value. The embodiments of the present disclosure do not limit the specific values of the second preset parameter and the first preset parameter, and those skilled in the art can determine them according to actual situations.

[0141] In step S16, based on the relationship between the target loss value and the preset loss threshold, optimize the preset large model and output a large model for forged image detection.

[0142] Exemplarily, compare the calculated target loss value with the preset loss threshold. If the loss value is higher than the preset loss threshold, it indicates that there is still room for improvement in the performance of the preset large model and optimization is required.

[0143] The ADAM optimizer can be initialized, and the learning rate can be set to 2×10^-4 (or other appropriate values). The ADAM optimizer requires several key parameters: learning rate, β1 (exponential decay rate of the first moment estimate), β2 (exponential decay rate of the second moment estimate), and ε (a small constant for numerical stability).

[0144] Perform backpropagation to calculate the gradients of the target loss value with respect to all the parameters in the large model. Use the ADAM optimizer to update the parameters of the model according to the gradients. Repeat this process until the loss value converges to an acceptable range or reaches the preset number of iterations.

[0145] Model verification: Evaluate the performance of the optimized model on the validation dataset. If the validation performance meets the expectations, output the optimized large model as the final forged image detection model. Save the optimized model for future use.

[0146] Through the large model training method for forged image detection according to the embodiments of the present application, generate description information corresponding to each image in the pre-acquired image training set; perform contrastive learning based on the images and description information in the image training set, and calculate the first loss value; further, input the images and the description information corresponding to the images into the preset large model, and determine the second loss value based on the output content of the preset large model; calculate the probability that the preset large model outputs the description information corresponding to the image based on the image and the preset output rule of the preset large model, and use the probability as the third loss value; generate the target loss value corresponding to the image based on the first loss value, the second loss value, and the third loss value; based on the relationship between the target loss value and the preset loss threshold, optimize the preset large model and output a large model for forged image detection. Through the above implementation, by performing contrastive learning on the images and the image descriptions, the understanding of the description information of the forged images by the preset large model is enhanced; further, by calculating the second loss value and the third loss value, the accuracy of the results of the forged images by the preset large model is improved, and the reasoning reasons for the accurate forged images are given.

[0147] Corresponding to the above embodiments, an embodiment of the present disclosure further provides a forged image detection method, as Figure 4 shown, the method includes:

[0148] In step S41, obtain the image to be detected;

[0149] In step S42, input the image to be detected and a preset output rule into the large model for forged image detection obtained by the large model training method for forged image detection in the above embodiment, to obtain the detection result of the image to be detected and the reason corresponding to the detection result; wherein, if the detection result is a real image, the reason corresponding to the detection result is the description information of the image to be detected; if the detection result is a forged image, the reason corresponding to the detection result is the reason why the image to be detected is a forged image.

[0150] Exemplarily, as Figure 5 shown, it is a schematic diagram of image detection based on the forged image detection method of the embodiment of the present disclosure. Input the authenticity classification prompt and the image to be detected into the trained large model for forged image detection. Obtain the forged image detection result and the reason for the forged image detection inference.

[0151] Among them, the authenticity classification prompt: Is this photo real or fake? Please describe the image and answer "This image is real" or "This image is fake".

[0152] Forged image detection result: This image is fake.

[0153] Forged image detection inference: The shadows on the house and the ground are inconsistent with the light source direction. The texture of the house and the surrounding vegetation look unnatural and lack realistic details. The structure of the house and the surrounding environment appear distorted.

[0154] The embodiment of the present disclosure is an embodiment for detecting forged images by using the large model for forged image detection trained in the above embodiment. The specific implementation manner will not be elaborated.

[0155] Corresponding to the implementation manner of the above signal acquisition method, an embodiment of the present disclosure further provides a large model training device for forged image detection, which is used to execute the above Figure 1 any one of the embodiments of the large model training method for forged image detection shown, and the large model training device for forged image detection includes:

[0156] A description information generation module, configured to generate description information corresponding to each image in the pre-obtained image training set;

[0157] A first loss value calculation module, configured to perform contrastive learning based on the images in the image training set and the description information, and calculate a first loss value;

[0158] A second loss value calculation module, configured to input, for any image in the image training set, the image and the description information corresponding to the image into a preset large model, and determine a second loss value based on the output content of the preset large model;

[0159] A third loss value calculation module, configured to calculate the probability that the preset large model outputs the description information corresponding to the image based on the image and a preset output rule of the preset large model, and use the probability as the third loss value of the image;

[0160] A target loss value calculation module, configured to generate a target loss value corresponding to the image based on the first loss value, the second loss value, and the third loss value;

[0161] An optimization module, configured to optimize the preset large model based on the relationship between the target loss value and a preset loss threshold, and output a large model for forged image detection.

[0162] The large model training device for forged image detection provided in the above embodiments of the present disclosure and the large model training method for forged image detection provided in the embodiments of the present disclosure are based on the same inventive concept, and have the same beneficial effects as the methods adopted, run, or implemented by the application programs stored therein.

[0163] Corresponding to the implementation manners of the above signal acquisition method, the embodiments of the present disclosure further provide a forged image detection device for executing the forged image detection method in any of the above Figure 4 schematic embodiments. The forged image detection device includes:

[0164] An image acquisition module, configured to acquire an image to be detected;

[0165] A detection module, configured to input the image to be detected and a preset output rule into a large model for forged image detection obtained by the large model training method for forged image detection as described in the above embodiments, to obtain a detection result of the image to be detected and the reason corresponding to the detection result; wherein, if the detection result is a real image, the reason corresponding to the detection result is the description information of the image to be detected; if the detection result is a forged image, the reason corresponding to the detection result is the reason for the forged image of the image to be detected.

[0166] The forged image detection device provided in the above embodiments of the present disclosure and the forged image detection method provided in the embodiments of the present disclosure are based on the same inventive concept, and have the same beneficial effects as the methods adopted, run, or implemented by the application programs stored therein.

[0167] The embodiments of the present disclosure also provide an electronic device for executing the above method. Please refer toFigure 6 , which shows a schematic diagram of an electronic device provided by some embodiments of the present disclosure. As Figure 6 shown, the electronic device includes: a processor 600, a memory 601, a bus 602, and a communication interface 603. The processor 600, the communication interface 603, and the memory 601 are connected through the bus 602; a computer program that can run on the processor 600 is stored in the memory 601, and when the processor 600 runs the computer program, it executes the method provided by any of the foregoing Figure 1 or Figure 2 embodiments schematically illustrated.

[0168] Among them, the memory 601 may include a high-speed random access memory (Random Access Memory, RAM), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface 603 (which can be wired or wireless), a communication connection is realized between this system network element and at least one other network element, and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0169] The bus 602 can be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Among them, the memory 601 is used to store a program. After receiving an execution instruction, the processor 600 executes the program, and the method disclosed in any of the foregoing Figure 1 embodiments schematically illustrated can be applied to the processor 600 or implemented by the processor 600.

[0170] The processor 600 may be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 600 or the instructions in the form of software. The above-mentioned processor 600 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present disclosure. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present disclosure can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 601, and the processor 600 reads the information in the memory 601 and combines its hardware to complete the steps of the above method.

[0171] The electronic device provided in the embodiments of the present disclosure and the method provided in the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the method adopted, run, or implemented by it.

[0172] The embodiments of the present disclosure also provide a computer-readable storage medium corresponding to the method provided in the foregoing embodiments. Please refer to Figure 7 Figure, in which the computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the method provided in any of the foregoing embodiments.

[0173] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other optical or magnetic storage media, which will not be elaborated here one by one.

[0174] The computer-readable storage medium provided in the above embodiments of the present disclosure and the method provided in the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the method adopted, run, or implemented by the application program stored in it.

[0175] It should be noted that:

[0176] In the specification provided herein, a large number of specific details are set forth. However, it will be understood that embodiments of the present disclosure may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0177] Similarly, it should be understood that, in order to streamline the present disclosure and assist in understanding one or more of the various inventive aspects, in the foregoing description of the exemplary embodiments of the present disclosure, various features of the present disclosure are sometimes grouped together in a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting an intention that: the claimed present disclosure requires more features than are expressly recited in each embodiment. The inventive aspects lie in less than all the features of the single foregoing disclosed embodiment. Thus, the implementation of the following specific embodiments is hereby expressly incorporated into that specific embodiment, where each embodiment stands on its own as a separate embodiment of the present disclosure.

[0178] Furthermore, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not others, the combination of features of different embodiments is meant to be within the scope of the present disclosure and forms different embodiments.

[0179] The foregoing is only a preferred specific implementation manner of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any variation or replacement that can be easily thought of by those skilled in the art within the technical scope disclosed by the present disclosure should be covered by the protection scope of the present disclosure.

Claims

1. A large model training method for forged image detection, characterized in that: The method comprises: Generate description information corresponding to each image in the pre-acquired image training set; Perform comparative learning based on the images in the image training set and the description information, and calculate a first loss value; For any image in the image training set, input the image and the description information corresponding to the image into a preset large model, and determine a second loss value based on the output content of the preset large model; Based on the image and a preset output rule of the preset large model, calculate the probability that the preset large model outputs the description information corresponding to the image, and use the probability as a third loss value of the image; Based on the first loss value, the second loss value, and the third loss value, generating a target loss value corresponding to the image; Based on the relationship between the target loss value and the preset loss threshold, the preset large model is optimized and a large model for forged image detection is output.

2. The method according to claim 1, characterized in that The image training set includes a forged image training subset and a real image training subset, the description information includes a forged attribute description corresponding to each forged image in the forged image training subset, and an image content description corresponding to each real image in the real image training subset, The generating of description information corresponding to each image in the pre-acquired image training set includes: A forgery attribute description of each forged image in the forged image training subset is generated based on a preset large model; and a content description of each real image in the real image training subset is generated based on the preset large model; wherein the forgery attribute description is the cause of forgery at a preset angle, and the preset angle includes at least one of lighting, texture, semantics, edge, and quality.

3. The method according to claim 2, characterized in that The performing comparative learning based on the images in the image training set and the description information to calculate a first loss value includes: For any object in the image training set, matching an object in the image training set that matches the object; wherein the any object is any image and / or any description information in the image training set; The first loss value is generated based on the object, the matching object, and the unmatched object in the image training set.

4. The method according to claim 3, characterized in that The step of matching any object in the image training set with a matching object that matches the object in the image training set includes: For any image in the image training set, first matching description information and a first matching image matching the image are matched in the image training set; and for any description information in the image training set, a second matching image and a second matching description information matching the description information are matched in the image training set.

5. The method according to claim 3 or 4, characterized in that: The generating the first loss value based on the object, the matching object, and the unmatched object in the image training set comprises: Calculating a first similarity between the object and the matching object; calculating a first ratio of the first similarity to a preset temperature parameter; Calculating a second similarity between the object and any unmatched object in the image training set; calculating a second ratio between the second similarity and the preset temperature parameter; calculating the sum of the second ratios corresponding to all unmatched objects in the image training set to obtain a first value; The logarithm of the ratio of the first value to the second value is taken as the first loss value; wherein the second value is an exponent of the first ratio.

6. The method according to claim 4, characterized in that The generating the first loss value based on the object, the matching object, and the unmatched object in the image training set comprises: generating a first sub-loss value according to the image, the first matching description information, and the unmatched description information in the image training set; generating a second sub-loss value according to the description information, the second matching image, and an unmatched image in the image training set; generating a third sub-loss value according to the image, the first matching image, and an unmatched image in the image training set; generating a fourth sub-loss value according to the description information, the second matching description information, and the unmatched description information in the image training set; The average of the first sub-loss value, the second sub-loss value, the third sub-loss value and the fourth sub-loss value is taken as the first loss value.

7. The method according to claim 1, characterized in that The step of inputting the image and the description information corresponding to the image into a preset large model, and determining the second loss value based on the output content of the preset large model, comprises: Inputting the image and the description information corresponding to the image into a preset large model, and obtaining the output content of the preset large model based on the preset output rule of the preset large model; Perform an activation operation based on the output content to obtain an output probability; The second loss value is calculated based on the output probability.

8. The method according to claim 1, characterized in that The calculating, based on the image and the preset output rule of the preset large model, the probability that the preset large model outputs the description information corresponding to the image, and taking the probability as the third loss value of the image, comprises: For any word in the description information corresponding to the image, predict the probability of the word based on a first vocabulary set and a first image set, wherein the first vocabulary set is words located before the word, and the first image set is images in the image training set that are located before the image corresponding to the description information to which the word belongs; The mean of the probability logarithms of the words in the description information corresponding to the image is calculated as the probability of the description information corresponding to the image.

9. The method according to claim 1, characterized in that: The generating a target loss value corresponding to the image based on the first loss value, the second loss value, and the third loss value includes: Calculate the product of the first loss value and a first preset parameter to obtain a first initial value; Calculate the product of the second loss value and a second preset parameter to obtain a second initial value; Calculate a third initial value based on the third loss value, the first preset parameter and the second preset parameter; The sum of the first initial value, the second initial value and the third initial value is used as the target loss value.

10. A method for detecting forged images, characterized in that: The method comprises: Acquire the image to be detected; The image to be detected and the preset output rules are input into the large model for forged image detection obtained by the large model training method for forged image detection as described in any one of claims 1 to 9 above, to obtain the detection result of the image to be detected and the reason corresponding to the detection result; wherein, if the detection result is a real image, the reason corresponding to the detection result is the description information of the image to be detected; if the detection result is a forged image, the reason corresponding to the detection result is the reason why the image to be detected is a forged image.

Images