Image detection method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of image processing, and provides an image detection method and device, electronic equipment and a storage medium. The method comprises the following steps: inputting a to-be-identified image into a content feature extraction network and a clue feature extraction network to obtain an image content feature and a forged clue feature; inputting the forged clue features into a classifier to obtain a detection result; inputting the forged clue features into a generator to obtain the credibility of a clue reconstruction image and a detection result; inputting the image content features into a generator to obtain a content reconstruction image, a forgery error and a forgery area; and displaying the clue reconstruction image, the credibility, the content reconstruction image, the forgery error and the forgery area. According to the method, after the detection result is obtained, the credibility of the detection result is calculated, the authentic identification error is determined, and the authentic identification error is visually displayed, so that a user can know the possibility of image forgery and what the image is determined to be forgery, and the persuasion and interpretability of the image detection result are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image detection method and device, electronic equipment, and storage medium. Background Art

[0002] With the rapid development of digital technology, it has become easy to edit and tamper with images, which has, to a certain extent, led to serious challenges to the authenticity and credibility of image information. Generated image detection (also known as image authentication) refers to the use of technical means to identify the authenticity of images. This is of great significance for maintaining the authenticity of information, protecting the public's right to know, and maintaining social fairness, justice and integrity.

[0003] The current mainstream image authentication technology mainly relies on deep learning algorithms to identify the authenticity of images by training neural network models. Neural networks generally consist of two parts: a feature extractor and a classifier. The feature extractor is responsible for extracting high-dimensional features from the input image, while the classifier performs authenticity authentication based on the extracted features.

[0004] However, existing image detection methods can only authenticate images and mark forged areas, allowing users to understand whether the image is forged and which specific part is forged. They cannot give the specific credibility of the judgment result and the basis for the judgment, nor can they explain why the part is judged to be forged and how likely it is to be forged. Therefore, they are not convincing enough and have poor explainability. Summary of the invention

[0005] In view of this, the purpose of this application is to provide an image detection method and device, an electronic device, and a storage medium. After obtaining the detection results, by calculating the credibility of the detection results, determining the authentication error, and visually displaying them, the user can understand how likely the image is to be forged and why the image is determined to be forged, thereby improving the persuasiveness and explainability of the image detection results.

[0006] The embodiment of the present application provides an image detection method, which is applied to an image detection model, wherein the image detection model includes a content feature extraction network, a clue feature extraction network, a classifier, and a generator, and the method includes:

[0007] Acquire an image to be identified, and input the image to be identified into a content feature extraction network and a clue feature extraction network respectively, to obtain image content features and forgery clue features, wherein the image content features include content information of the image to be identified, and the forgery clue features include clue information for identifying the authenticity of the image;

[0008] Input the forgery clue feature into the classifier, and output and display the detection result corresponding to the image to be identified;

[0009] Inputting the forged clue feature into a generator to obtain a clue reconstructed image based on the clue information, and obtaining the credibility of the detection result according to the clue reconstructed image;

[0010] Inputting the image content feature into the generator, obtaining a reconstructed image based on the content information, and determining a forgery error and a forgery area according to a difference between the content reconstructed image and a reference image, wherein the reference image is the image to be identified, or the reference image is generated by the image content feature and the forgery clue feature; the forgery error is the error between the image to be identified and the corresponding real image;

[0011] At least one of the clue reconstructed image, the credibility, the content reconstructed image, the forgery error, and the forgery area is displayed.

[0012] Furthermore, obtaining the credibility of the detection result according to the first reconstructed image includes:

[0013] Determining the credibility of the detection result according to the similarity between the clue reconstructed image and the real clue reference image, wherein the real clue reference image is a reconstructed image obtained by inputting the forged clue features of the real reference sample into the generator;

[0014] and / or,

[0015] Determine a first distance between the clue reconstructed image and a preset cluster center, and determine the credibility of the detection result based on a size relationship between the first distance and a preset distance threshold, wherein the preset cluster center is obtained by clustering the real clue reference image.

[0016] Furthermore, before determining the first distance between the clue reconstructed image and the preset cluster center, the method further includes:

[0017] respectively calculating a second distance between each real clue reference image and the forged clue reference image and the preset cluster center, and determining the preset distance threshold according to the second distance;

[0018] The forged clue reference image is a reconstructed image obtained by inputting the forged clue features of the forged reference sample into the generator.

[0019] Further, the generator is used to generate a reconstructed image using the image content feature and the specific forgery clue feature, and to generate a reconstructed image using the forgery clue feature and the specific image content feature;

[0020] The specific content feature is one of the following: a random vector, an all-0 vector, an all-1 vector, a mean of image content features of a preset specific sample, or a specific vector learned by the image detection model during training;

[0021] The specific clue feature is one of the following: a forged clue feature of a specific real sample, an average of forged clue features of multiple real samples, and an anchor point feature learned by the image detection model during training.

[0022] Furthermore, before inputting the image to be identified into the content feature extraction network and the clue feature extraction network respectively, the method further includes:

[0023] Designing a loss function, and using training samples to train the image detection model based on the loss function, wherein the training samples include real training samples and forged training samples;

[0024] Wherein, the loss function includes classification loss, contrast loss and reconstruction loss;

[0025] The contrast loss is determined according to the anchor clue feature in the anchor feature, the forgery clue feature of the training sample, and the image content feature of the training sample;

[0026] The reconstruction loss includes self-reconstruction loss, cross-reconstruction loss and reconstruction contrast loss;

[0027] The self-reconstruction loss is determined according to the content features and clue features of the training samples; the cross-reconstruction loss is determined according to the real training samples and the corresponding clue training samples; and the reconstruction contrast loss is determined according to the reconstructed images of each pair of training samples.

[0028] Furthermore, the design loss function includes:

[0029] Obtaining a training sample pair consisting of two training samples, and inputting the forged clue features of each training sample in the training sample pair into the generator to obtain clue reconstructed images of the two training samples;

[0030] The reconstruction contrast loss of the training sample pair is determined according to the contrast loss between the cue-reconstructed images of the two training samples.

[0031] Furthermore, the method further comprises:

[0032] Initialize the self-learning feature vector;

[0033] Freezing the self-learning feature vector, and adjusting the parameters of the image detection model based on the loss function;

[0034] Freezing parameters of the image detection model and adjusting the self-learning feature vector based on the loss function;

[0035] Returning to the step of freezing the self-learning feature vector until the loss function converges or is less than a preset loss threshold;

[0036] The self-learning feature vector is determined to be the specific vector and / or the anchor point feature.

[0037] The embodiment of the present application further provides an image detection device, which is applied to an image detection model, wherein the image detection model includes a content feature extraction network, a clue feature extraction network, a classifier, and a generator, and the device includes:

[0038] A feature extraction module is used to obtain an image to be identified, and input the image to be identified into a content feature extraction network and a clue feature extraction network respectively, to obtain image content features and forgery clue features, wherein the image content features include content information of the image to be identified, and the forgery clue features include clue information for identifying the authenticity of the image;

[0039] An image detection module, used to input the forgery clue features into a classifier and output a detection result corresponding to the image to be identified;

[0040] a forgery feature determination module, used for inputting the forgery clue feature into a generator, obtaining a clue reconstruction image based on the clue information, and obtaining the credibility of the detection result according to the clue reconstruction image;

[0041] a forgery clue determination module, configured to input the image content features into the generator, obtain a content reconstructed image based on the content information, and determine a forgery error and a forgery area according to a difference between the content reconstructed image and a reference image, wherein the reference image is the image to be identified, or the reference image is generated by the image content features and the forgery clue features; and the forgery error is an error between the image to be identified and a corresponding real image;

[0042] A visualization module is used to display the detection result; and to display at least one of the clue reconstruction image, the credibility, the content reconstruction image, the forgery error and the forgery area.

[0043] Furthermore, the forgery feature determination module is used to:

[0044] Determining the credibility of the detection result according to the similarity between the clue reconstructed image and the real clue reference image, wherein the real clue reference image is a reconstructed image obtained by inputting the forged clue features of the real reference sample into the generator;

[0045] and / or,

[0046] Determine a first distance between the clue reconstructed image and a preset cluster center, and determine the credibility of the detection result based on a size relationship between the first distance and a preset distance threshold, wherein the preset cluster center is obtained by clustering the real clue reference image.

[0047] Furthermore, the forgery clue determination module is used to:

[0048] respectively calculating a second distance between each real clue reference image and the forged clue reference image and the preset cluster center, and determining the preset distance threshold according to the second distance;

[0049] The forged clue reference image is a reconstructed image obtained by inputting the forged clue features of the forged reference sample into the generator.

[0050] Further, the generator is used to generate a reconstructed image using the image content feature and the specific forgery clue feature, and to generate a reconstructed image using the forgery clue feature and the specific image content feature;

[0051] The specific content feature is one of the following: a random vector, an all-0 vector, an all-1 vector, a mean of image content features of a preset specific sample, or a specific vector learned by the image detection model during training;

[0052] The specific clue feature is one of the following: a forged clue feature of a specific real sample, an average of forged clue features of multiple real samples, and an anchor point feature learned by the image detection model during training.

[0053] Furthermore, the device also includes a training module, which is used to:

[0054] Designing a loss function, and using training samples to train the image detection model based on the loss function, wherein the training samples include real training samples and forged training samples;

[0055] Wherein, the loss function includes classification loss, contrast loss and reconstruction loss;

[0056] The contrast loss is determined according to the anchor clue feature in the anchor feature, the forgery clue feature of the training sample, and the image content feature of the training sample;

[0057] The reconstruction loss includes self-reconstruction loss, cross-reconstruction loss and reconstruction contrast loss;

[0058] The self-reconstruction loss is determined according to the content features and clue features of the training samples; the cross-reconstruction loss is determined according to the real training samples and the corresponding clue training samples; and the reconstruction contrast loss is determined according to the reconstructed images of each pair of training samples.

[0059] Furthermore, the training module is used to:

[0060] Obtaining a training sample pair consisting of two training samples, and inputting the forged clue features of each training sample in the training sample pair into the generator to obtain clue reconstructed images of the two training samples;

[0061] The reconstruction contrast loss of the training sample pair is determined according to the contrast loss between the reconstructed images of the two training samples.

[0062] Furthermore, the device also includes a specific feature training module, which is used to:

[0063] Initialize the self-learning feature vector;

[0064] Freezing the self-learning feature vector, and adjusting the parameters of the image detection model based on the loss function;

[0065] Freezing parameters of the image detection model and adjusting the self-learning feature vector based on the loss function;

[0066] Returning to the step of freezing the self-learning feature vector until the loss function converges or is less than a preset loss threshold;

[0067] The self-learning feature vector is determined to be the specific vector and / or the anchor point feature.

[0068] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the image detection method as described above are performed.

[0069] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image detection method are executed.

[0070] The image detection method and device, electronic device, and storage medium provided by the embodiment of the present application can, on the basis of identifying the authenticity of the image, also display the credibility of the identification result, the clue features on which the counterfeit identification model relies on to make a judgment, and the estimated counterfeit error in a visual form, thereby improving the persuasiveness and interpretability of the algorithm. Specifically, the image content features and counterfeit clue features of the picture are extracted respectively by the decoupling learning strategy, and the extracted counterfeit clue features are combined with specific content features to use the generator network to generate clue reconstruction images, that is, the visualization results of the clue features. Then, the credibility of the detection result of the image detection model is evaluated by the visualization result. In addition, the image detection method provided by the embodiment of the present application also uses the extracted image content features combined with specific clue features to use the generator network to generate content reconstruction images, that is, to estimate the real image. By comparing the image with the reference image, the counterfeit area and the counterfeit error can be obtained, and the visualization result of the counterfeit error can be realized. Through such a design, the embodiment of the present application can not only enable the user to know the image detection result, but also enable the user to understand the credibility of the detection result, the counterfeit area in the image, and why the area is judged to be counterfeit. This improves the persuasiveness and interpretability of image authentication and avoids users from being confused about the detection results. On this basis, this application designs a loss function that includes classification loss, contrast loss, and reconstruction loss. The reconstruction contrast loss function calculates the visualization results of clue features based on the clue features, which can effectively improve the distinguishability of the visualization results of clue features corresponding to real samples and forged samples, so that users can intuitively and clearly see the difference between real images and forged images.

[0071] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 A flow chart of an image detection method provided by an embodiment of the present application is shown;

[0074] Figure 2 A schematic diagram of the structure of an image detection model provided in an embodiment of the present application is shown;

[0075] Figure 3A schematic diagram of the structure of an image detection device provided in an embodiment of the present application is shown;

[0076] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown.

[0077] In the figure:

[0078] 300 image detection device, 310 feature extraction module, 320 image detection module, 330 forgery feature determination module, 340 forgery clue determination module, 350 visualization module, 400 electronic device, 410 processor, 420 memory, 430 bus. DETAILED DESCRIPTION

[0079] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0080] First, the application scenarios to which this application can be applied are introduced. This application is used to identify the authenticity of images and can be applied to a variety of different scenarios, such as forensic evidence collection to identify the authenticity of evidence; in media monitoring, by identifying forged images, ensuring the authenticity of news reports and preventing the spread of false news; in e-commerce, preventing fraud in online transactions, etc.

[0081] Research has found that existing image detection methods mainly identify the authenticity of images by training neural network models. They can also use forgery location technology to mark the forged areas of the input images, so that users can understand which part of the image the algorithm model thinks is forged. However, this method can only display the forged area, and cannot give the credibility of the model's judgment results, nor can it further explain why the model thinks that the part is forged. In other words, this method cannot provide features or clues used to determine forgery. As the realism of forged images becomes higher and higher, it is difficult for users to find forgery clues in the area with the naked eye by only marking the forged area. Therefore, although these image detection methods can determine whether an image is forged, they are not convincing enough and have poor interpretability.

[0082] Based on this, an embodiment of the present application provides an image detection method to improve the persuasiveness and interpretability of image detection results.

[0083] See also Figure 1 , Figure 1 A flow chart of an image detection method provided in an embodiment of the present application. The method is used for an image detection model, and the image detection model includes at least two feature extraction networks, a classifier network, and a generator network. Among them, the feature extraction network includes a content feature extraction network and a clue feature extraction network, which are independent network structures, and classic network structures can be selected, including but not limited to ResNet, EfficientNet, Xception, ViT and its variants. The classifier network can adopt a simple network structure, such as a multi-layer fully connected network or a convolutional network, or a relatively complex classic network structure, which is not limited here. The generator network can adopt a classic image generation network, such as a Conditional UNet network.

[0084] Among them, the feature extraction network is used to extract high-dimensional embedded features from the input image, that is, to convert the input image to be identified into a feature vector. The classifier network is used to output the authenticity detection result of the image to be identified based on the extracted feature vector. The generator network is used to reconstruct the image to be identified through the extracted feature vector, that is, to complete the conversion of high-dimensional features to images, which can be considered as the reverse process of feature extraction.

[0085] like Figure 1 As shown in, the image detection method provided by the embodiment of the present application includes:

[0086] S101. Obtain an image to be identified, and input the image to be identified into a content feature extraction network and a clue feature extraction network respectively to obtain image content features and forgery clue features, wherein the image content features contain content information of the image to be identified, and the forgery clue features contain clue information for identifying the authenticity of the image.

[0087] In this step, the image to be identified is obtained, and the image content features in the image are extracted using the content feature extraction network, and the forgery clue features in the image are extracted using the clue feature extraction network. This step is used to decouple the features, and the features that were originally fused together are separated according to the information they contain. The image content features contain information related to the image content, and the forgery clue features contain clues required to determine whether the image is forged. Among them, the image to be identified can be a whole picture, a single frame in a video, or a part of a picture. For example, the area to be identified in the picture is marked by manual marking, target detection, forgery positioning algorithm, etc., and the area is used as the image to be identified. The area can have a fixed size, and if the image to be identified is a partial area of ​​the whole picture, the part can be normalized before feature extraction to meet the input requirements of the classifier.

[0088] In the feature extraction process, this step uses two independent feature extraction networks. The two networks are generally the same in structure, and their functional differences come from the training results. Specifically, the output of the clue feature extraction network will be connected to the classifier and the generator, while the output of the content feature extraction network is only connected to the generator. During the training process, the final content feature extraction network and clue feature extraction network can be automatically obtained through back propagation algorithm optimization.

[0089] In addition, forgery clue features refer to the features extracted from the input samples by the feature extraction network used to extract forgery clue related features, and are not the features corresponding to the forged images. Both real images and forged images have forgery clue features.

[0090] S102, inputting the forgery clue features into the classifier, outputting and displaying the detection results corresponding to the image to be identified.

[0091] In this step, the forgery clue features extracted by the aforementioned clue feature extraction network are input into the classifier, and the classifier can be used to obtain the probabilities of the image being real and forged, that is, the final detection result.

[0092] S103, inputting the forged clue features into the generator to obtain a clue reconstructed image based on the clue information, and obtaining the credibility of the detection result according to the clue reconstructed image.

[0093] In this step, the extracted forgery clue feature generator uses the forgery clue feature and specific content features to generate an image to obtain a clue reconstructed image based on the clue information, and then evaluates the clue reconstructed image to obtain the credibility of the detection result.

[0094] Optionally, the specific content feature is one of the following: a random vector, an all-0 vector, an all-1 vector, a mean of image content features of a preset specific sample, or a specific vector learned by the image detection model during training.

[0095] In addition, after the generator outputs clues to reconstruct the image, it can also perform image processing such as smoothing, normalization, and Gaussian filtering to improve its display effect, which is beneficial for users to observe the details of the image.

[0096] Optionally, in step 103, reconstructing the image according to the clues to obtain the credibility of the detection result includes the following steps:

[0097] S103-a, determining the credibility of the detection result according to the similarity between the clue reconstructed image and the real clue reference image.

[0098] Specifically, in step S103-a, considering that the contrast loss between features and the contrast loss between visualization results (i.e., reconstructed images) will be optimized during the training of the image detection model, the clue reconstructed images of different real images should present similar patterns, while the clue reconstructed images of fake images and the clue reconstructed images of real images have large differences. Based on this, the clue reconstructed images obtained in the above steps and the clue reconstructed images of real images (i.e., real clue reference images) can be compared, and the authenticity and significance (i.e., possibility) of the image to be identified can be judged according to the similarity between the two. If the significance is low or the judgment result is opposite to the detection result output by the classifier, the detection result of the classifier is considered to have low credibility. For example, if the similarity between the clue reconstructed images of the image to be identified obtained in the above steps and the clue reconstructed images of the real image is higher, the possibility that the image to be identified is true is greater. At this time, if the detection result is true, the credibility of the detection result can be considered to be high; on the contrary, the lower the similarity between the two, the smaller the possibility that the image to be identified is true. At this time, if the detection result is true, the credibility of the detection result can be considered to be low.

[0099] Similarly, the credibility of the detection result can also be obtained based on the similarity between the clue-reconstructed image of the image to be identified and the clue-reconstructed image of the forged image (i.e., the forged clue reference image). Alternatively, the similarity between the clue-reconstructed image of the image to be identified and the clue-reconstructed image of the real image and the clue-reconstructed image of the forged image can be calculated separately, and then analyzed comprehensively to obtain the credibility of the detection result.

[0100] The real clue reference image in this step refers to the reconstructed image obtained by inputting the clue features of the real reference sample into the generator; the forged clue reference image refers to the reconstructed image obtained by inputting the clue features of the forged reference sample into the generator.

[0101] and / or,

[0102] S103-b1, respectively calculating the second distance between each real clue reference image and the forged clue reference image and the preset cluster center, and determining a preset distance threshold according to the second distance;

[0103] S103-b2, determining a first distance between the clue reconstructed image and a preset cluster center, and determining the credibility of the detection result according to a magnitude relationship between the first distance and a preset distance threshold.

[0104] Specifically, in steps S303-b1 to S303-b2, the real clue reference images of the real reference samples are first clustered to obtain a preset cluster center, and then a preset distance threshold is set according to the distance between each real clue reference image and the forged clue reference image and the preset cluster center. Specifically, since the real clue reference image is closer to the preset cluster center, and the forged clue reference image is farther from the preset cluster center, the second distance between each real clue reference image and the forged clue reference image and the preset cluster center can be calculated respectively, and the preset distance threshold is obtained by comprehensive calculation. Finally, the credibility of the detection result is calculated according to the first distance between the clue reconstructed image corresponding to the image to be identified and the preset cluster center. For example, if the first distance between the clue reconstructed image and the preset cluster center is much greater than the preset distance threshold, it is considered that the possibility that the image to be identified is a forged image is relatively high, and if the detection result at this time is that the image is true, it can be considered that the credibility of the detection result is low.

[0105] S104: Input the image content features into the generator to obtain a content reconstructed image based on the content information, and determine the forgery error and forgery area according to the difference between the content reconstructed image and the reference image.

[0106] In this step, the extracted image content features are sent to the generator, which uses the image content features and specific forgery clue features to generate pictures, and obtains a content reconstructed image based on the image content information. At this time, the content reconstructed image is the real image corresponding to the image to be identified as considered by the image detection model. Then the content reconstructed image is evaluated to obtain the forged area and forgery error of the image to be identified. The forged area can be considered as the different area between the forged image and the original real image, and the forgery error can be considered as the numerical representation of the difference between the forged image and the original real image.

[0107] Specifically, for some forged images, since the forged errors of the forged area and the real area are significantly different, they can be used as the basis for image authentication and forged area judgment. Based on this, the difference between the content reconstructed image and the reference image can be calculated, and the difference can be used as the forged error, and the area where the forged error reaches a certain threshold is the forged area. Among them, the reference image is the image to be identified, or the image generated by the generator using content features and clue features.

[0108] This step can identify the forged area by forging errors, and also allow users to understand why the image detection model determines that the area is a forged area. For samples with high forgery quality, the influence of image content can be removed to more clearly show the difference in error characteristics between the real area and the forged area, further improving the persuasiveness of the detection results.

[0109] In addition, after the generator outputs the reconstructed image, it can also be subjected to image processing such as smoothing, normalization, and Gaussian filtering to improve its display effect, which is beneficial for users to observe the details of the image.

[0110] Optionally, the specific clue feature is one of the following: a forged clue feature of a specific real sample, an average of forged clue features of multiple real samples, and an anchor point feature learned by the image detection model during training.

[0111] S105 , display at least one of a clue-reconstructed image, credibility, a content-reconstructed image, a forged error, and a forged area.

[0112] In this step, the clue reconstructed image, detection result credibility, content reconstructed image, forged area and forged error obtained in the previous steps are displayed to the user. According to actual needs, only a part of it can be displayed to the user. For example, if the user only cares about why the image is determined to be forged, then only the content reconstructed image, forged error and forged area can be displayed.

[0113] This embodiment is applied to image detection models, such as Figure 2 As shown in the figure, the image detection model includes two independent feature extraction networks (i.e., content feature extraction network and clue feature extraction network), a classifier and a generator. In the specific application process, the image to be detected is input into two independent feature extraction networks respectively, and the image content features and forgery clue features are extracted from the image; the forgery clue features are input into the classifier network to obtain the classification result (image detection result), i.e., real or forged; then the image content features and forgery clue features are combined with specific clue features and specific content features respectively to obtain the content reconstructed image and clue reconstructed image.

[0114] In practical applications, the clue-reconstructed images of different real images usually have a high degree of similarity. Therefore, the clue-reconstructed image of the image to be identified and the clue-reconstructed image of the real image can be compared. If the similarity between the two is high, the image to be identified is more likely to be true; otherwise, the image to be identified is more likely to be false.

[0115] In addition, for the content reconstructed image of the image to be identified, since it is an image in which the forgery clue features of the image to be identified are eliminated, it can be considered that the content reconstructed image of the image to be identified has nothing to do with whether the image to be identified is forged or not, and it is considered to be a real image. Based on this, the image to be identified can be compared with its content reconstructed image. If the similarity between the two is high, it can be considered that the image to be identified is likely to be true. Otherwise, it can be considered that the image to be identified is likely to be false, and the area with a large difference between the image to be identified and its content reconstructed image is considered to be a forged area. Optionally, before step S101, the method further includes:

[0116] A loss function is designed, and an image detection model is trained based on the loss function using training samples, wherein the training samples include real training samples and forged training samples.

[0117] Specifically, the training samples can be constructed through open source data sets, self-made data sets, or a combination of the two, including real samples and forged samples. Ideally, for any forged sample in the training set, there is a unique real sample corresponding to it in the training set. During the model training process, optimizers such as ADAM (Adaptive Moment Estimation) and SGD (Stochastic Gradient Descent) can be used, using fixed learning rate or exponential decay, cosine annealing and other learning rate strategies. Through model training, the reconstructed images of any real image are relatively close, while the reconstructed images of forged images and real images are significantly different.

[0118] In addition, if the data set used for training has detailed annotations on the category of forgery methods used for each sample, content-related features, forgery method-related features and general clue features can be extracted separately during feature extraction, that is, the forgery clue features can be split into forgery method features and general clue features. When judging the authenticity of the image to be identified later, in addition to the forgery clues, the forgery method used for the image can also be output.

[0119] Among them, the loss function includes classification loss, contrast loss and reconstruction loss, and the final loss function is the weighted sum of these three parts of loss.

[0120] First, classification loss is the main loss function of the image detection model, which is used to optimize the classification results of the classifier, that is, the authenticity identification results of the image detection model for the forged image. In the specific design, the classification loss can use a simple cross entropy loss function, or an improved loss function for classification tasks such as the enhanced marginal cosine loss function (LMCL).

[0121] Secondly, the contrast loss is determined based on the forged clue features in the anchor features (i.e., anchor clue features), the forged clue features of the training samples, and the image content features of the training samples. Specifically, the contrast loss is an auxiliary loss function of the image detection model, which is used to increase the distance between the image content features and the forged clue features, and to reduce the distance between the forged clue features of different real samples, so as to achieve the decoupling between the image content features and the forged clue features. The formula of the contrast loss is as follows:

[0122]

[0123] Where N is the number of samples, i is the sample number, and y is the i represents the category of the i-th sample, the real sample category is 0, the generated sample category is 1, f a represents the forgery clue feature of the anchor sample, f i represents the forgery clue feature of the i-th sample, c i represents the image content feature of the i-th sample, ||f a ,f i ||2 means calculating two vectors f a and f i The distance between them can be measured by different distance metrics such as Euclidean distance or cosine distance. i ,f i ) represents the similarity between the content features and clue features, which can be achieved by dot product, Euclidean distance, cosine similarity or mutual similarity. The higher the similarity, the higher the D(c i ,f i ) value is smaller.

[0124] In the above contrast loss formula, when the input is a real sample, y i is 0, the first term (1-y i )||f a ,f i ||2 takes effect, the second term -y i ||f a ,f i||2 is invalid. At this time, the contrast loss function will make the forged clue features extracted by the optimized model close to the anchor sample; when the input is a forged sample, it will be just the opposite, which will make the forged clue features extracted by the optimized model far away from the anchor sample. The third term of the formula is used to increase the distance between the image content features and the forged clue features to separate the image content features and the forged clue features. In addition, the forged clue features of the anchor sample can be obtained by selecting specific training samples in the training set, or by dynamic updating during the training process.

[0125] Finally, the reconstruction loss includes self-reconstruction loss, cross-reconstruction loss and reconstruction contrast loss. The reconstruction loss is used to ensure the completeness of the image content features and forgery clue features obtained by decoupling learning, and also to ensure that the generator network can reasonably respond to the extracted image content features and forgery clue features to generate better reconstructed images and achieve better visualization effects.

[0126] In the specific design process, self-reconstruction loss is the basic form of reconstruction loss, which is determined according to the content features and clue features of the training samples. Specifically, the image content features and forged clue features extracted from the input training samples are input into the generator, and the generator is used to generate a reconstructed image. Then, the distance loss between the reconstructed image and the input training sample is calculated. The specific formula is as follows:

[0127]

[0128] Among them, x i represents the input sample, G(c i ,f i ) represents the generator network according to the content feature c i and clue feature f i The generated reconstructed sample, ||x i ,G(c i ,f i )||1 means calculating two vector elements x i and G(c i ,f i ) is the mean absolute error between .

[0129] The cross reconstruction loss is determined based on the real training samples and the corresponding forged training samples. Specifically, the cross reconstruction algorithm is similar to the self-reconstruction loss in that it extracts the content features of a certain category of input samples, combines them with the forged clue features of the opposite category, uses the generator network to reconstruct the image, and then calculates the distance loss with the corresponding samples of the opposite category. For example, for a forged sample x1 generated from a real sample x0, extract the image content feature c1 of x1 and combine it with the forged clue feature f0 of x0, and the generator generates a reconstructed sample x ′0=G(c1,f0), and calculate x ′ The distance loss between 0 and x0.

[0130] The reconstruction contrast loss is determined based on the reconstructed image of each pair of training samples. Specifically, the reconstruction contrast loss refers to selecting specific content features and using the generator network to generate visualization results of forged clue features. The visualization results of the features of the real samples are similar, while the visualization results of the forged samples and the real samples are quite different, thereby effectively improving the effect of the visualization display. The specific formula of the reconstruction contrast loss is as follows:

[0131]

[0132] Among them, c a is the selected anchor content feature, which can be obtained by extracting specific samples, averaging multiple samples, or setting special values ​​(such as all zeros, all ones, random values), or by model training; a is the anchor cue feature, which is the same as the anchor cue feature used in the aforementioned contrast loss; f i is the forgery clue feature of the sample; i is the sample label, y i 0 indicates a true sample, y i 1 indicates a fake sample; G(·,·) indicates the generator network, G(c a ,f a ) indicates that the generator network is used based on the feature c a and f generate an image; ||·,·||2 means calculating the distance between two vector elements, ||G(c a ,f a ),G(c a ,f i )||2 means G(c a ,f a ) and G(c a ,f i ), which can be the Euclidean distance, cosine distance or other distance metric functions used for error calculation.

[0133] Optionally, the reconstruction loss function can be obtained using the following steps:

[0134] A training sample pair consisting of two training samples is obtained, and the forgery clue features of each training sample in the training sample pair are respectively input into the generator to obtain reconstructed images of the two training samples; and the reconstruction contrast loss of the training sample pair is determined according to the contrast loss between the reconstructed images of the two training samples.

[0135] Optionally, in step S103 and step S104, the specific vector and anchor point features can be obtained by training an image detection model, and are specifically determined by the following steps:

[0136] Initialize the self-learning feature vector; freeze the self-learning feature vector, and adjust the parameters of the image detection model based on the loss function; freeze the parameters of the image detection model, and adjust the self-learning feature vector based on the loss function; return to the step of freezing the self-learning feature vector until the loss function converges or is less than a preset loss threshold; determine the self-learning feature vector as a specific vector and / or anchor feature.

[0137] Specifically, as mentioned above, the specific vector can be a feature of a specific sample, or a certain special value vector, or it can be obtained by autonomous model learning. The specific process for obtaining the specific vector by autonomous model learning is as follows:

[0138] First, initialize the specific vector, then freeze the value of the specific vector, and optimize the image detection model based on the entire loss function. During the optimization process, the parameters of the generator network will be continuously adjusted; then freeze the parameters of the image detection model, and optimize based on the contrast loss function of the clue feature reconstruction result. During the optimization process, the value of the specific vector will be continuously adjusted. Repeat the above steps until the loss function converges or the loss meets the threshold requirement.

[0139] The anchor point features can also be determined in the same way. Unlike traditional decoupled learning, in which the features of fixed anchor point samples are selected as anchor point features, the present application can determine the anchor point features through model autonomous learning. The specific determination method is similar to the determination method of the aforementioned specific vector, which will not be repeated here.

[0140] The image detection method provided by the embodiment of the present application, on the basis of identifying the authenticity of the image, can also display the credibility of the identification result, the clue features on which the counterfeit identification model relies on to make a judgment, and the estimated forgery error in a visual form, thereby improving the persuasiveness and interpretability of the algorithm. Specifically, the image content features and forgery clue features of the picture are extracted respectively by the decoupling learning strategy, and the extracted clue features are combined with specific content features to use the generator network to generate clue reconstruction images, that is, the visualization results of the forged clue features. Then the credibility of the detection result of the image detection model is evaluated by the visualization result. In addition, the image detection method provided by the embodiment of the present application, also uses the extracted image content features combined with specific clue features, and uses the generator network to generate content reconstruction images, that is, to estimate the real image. By comparing the image with the reference image, the forgery error and forgery area can be obtained, and the visualization result of the forgery error can be realized. Through such a design, the embodiment of the present application can not only enable the user to know the image detection result, but also enable the user to understand the credibility of the detection result, the forged area in the image, and why the area is determined to be forged. This improves the persuasiveness and interpretability of image authentication and avoids users from being confused about the detection results. On this basis, this application designs a loss function that includes classification loss, contrast loss, and reconstruction loss. The reconstruction contrast loss function is calculated based on the visualization results of the forged clue features, which can effectively improve the distinguishability of the visualization results of the forged clue features corresponding to the real samples and the forged samples, so that users can intuitively and clearly see the difference between the real image and the forged image.

[0141] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of an image detection device provided in an embodiment of the present application. Figure 3 As shown in , the image detection device 300 includes:

[0142] The feature extraction module 310 is used to obtain the image to be identified, and input the image to be identified into the content feature extraction network and the clue feature extraction network respectively, to obtain the image content feature and the forgery clue feature, wherein the image content feature contains the content information of the image to be identified, and the forgery clue feature contains the clue information for identifying the authenticity of the image;

[0143] Image detection module 320, used to input the forgery clue features into the classifier and output the detection result corresponding to the image to be identified;

[0144] The forgery feature determination module 330 is used to input the forgery clue feature into the generator, obtain a clue reconstruction image based on the clue information, and obtain the credibility of the detection result according to the clue reconstruction image;

[0145] a forgery clue determination module 340, configured to input the image content features into the generator, obtain a content reconstructed image based on the content information, and determine the forged area and the forgery error corresponding to the forged area according to the difference between the content reconstructed image and the reference image, wherein the reference image is the image to be identified, or the reference image is generated by the image content features and the forgery clue features;

[0146] The visualization module 350 is configured to display the detection result; and display at least one of the first reconstructed image, the credibility, the second reconstructed image, and the forgery error.

[0147] Furthermore, the counterfeit feature determination module 330 is used to:

[0148] Determine the credibility of the detection result according to the similarity between the clue reconstructed image and the real clue reference image, wherein the real clue reference image is a reconstructed image obtained by inputting the forged clue features of the real reference sample into the generator;

[0149] and / or,

[0150] A first distance between the clue reconstructed image and a preset cluster center is determined, and the credibility of the detection result is determined according to the magnitude relationship between the first distance and the preset distance threshold, wherein the preset cluster center is obtained by clustering the real clue reference image.

[0151] Furthermore, the forged clue determination module 340 is used to:

[0152] Calculating the second distance between each real clue reference image and the forged clue reference image and the preset cluster center respectively, and determining the preset distance threshold according to the second distance;

[0153] The forged clue reference image is a reconstructed image obtained by inputting the forged clue features of the forged reference sample into the generator.

[0154] Further, the generator is used to generate a reconstructed image using the image content feature and the specific forgery clue feature, and to generate a reconstructed image using the forgery clue feature and the specific image content feature;

[0155] The specific content feature is one of the following: a random vector, an all-0 vector, an all-1 vector, a mean of image content features of a preset specific sample, or a specific vector learned by the image detection model during training;

[0156] The specific clue feature is one of the following: a forged clue feature of a specific real sample, an average of forged clue features of multiple real samples, and an anchor feature learned by the image detection model during training.

[0157] Furthermore, the image detection device 300 also includes a training module, which is used to:

[0158] Designing a loss function, and using training samples to train an image detection model based on the loss function, wherein the training samples include real training samples and forged training samples;

[0159] Among them, the loss functions include classification loss, contrast loss and reconstruction loss;

[0160] The contrast loss is determined based on the anchor clue features in the anchor features, the forgery clue features of the training samples, and the image content features of the training samples;

[0161] Reconstruction loss includes self-reconstruction loss, cross-reconstruction loss and reconstruction contrast loss;

[0162] The self-reconstruction loss is determined based on the content features and clue features of the training samples; the cross-reconstruction loss is determined based on the real training samples and the corresponding clue training samples; and the reconstruction contrast loss is determined based on the reconstructed images of each pair of training samples.

[0163] Furthermore, the training module is used to:

[0164] A training sample pair consisting of two training samples is obtained, and the forged clue features of each training sample in the training sample pair are respectively input into the generator to obtain clue-reconstructed images of the two training samples;

[0165] According to the contrast loss between the cue-reconstructed images of the two training samples, the reconstruction contrast loss of the training sample pair is determined.

[0166] Furthermore, the image detection device 300 also includes a specific feature training module, which is used to:

[0167] Initialize the self-learning feature vector;

[0168] Freeze the self-learning feature vector and adjust the parameters of the image detection model based on the loss function;

[0169] Freeze the parameters of the image detection model and adjust the self-learning feature vector based on the loss function;

[0170] Return to the step of freezing the self-learning feature vector until the loss function converges or is less than a preset loss threshold;

[0171] The self-learning feature vector is determined to be a specific vector and / or an anchor feature.

[0172] It should be noted that, for other corresponding descriptions of the functional modules involved in the image detection device provided in the embodiment of the present application, reference can be made to the corresponding descriptions in the above method, which will not be repeated here.

[0173] See also Figure 4 , Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in , the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0174] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 as well as Figure 2 The specific implementation of the steps of the image detection method in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0175] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 as well as Figure 2 The specific implementation of the steps of the image detection method in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0176] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0177] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0178] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0180] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0181] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An image detection method, characterized in that: The image detection method is applied to an image detection model, the image detection model includes a content feature extraction network, a clue feature extraction network, a classifier and a generator, and the method includes: Acquire an image to be identified, and input the image to be identified into a content feature extraction network and a clue feature extraction network respectively, to obtain image content features and forgery clue features, wherein the image content features include content information of the image to be identified, and the forgery clue features include clue information for identifying the authenticity of the image; Input the forgery clue feature into the classifier, and output and display the detection result corresponding to the image to be identified; Inputting the forged clue feature into a generator to obtain a clue reconstructed image based on the clue information, and obtaining the credibility of the detection result according to the clue reconstructed image; Input the image content feature into the generator, obtain a content reconstructed image based on the content information, and forge errors and forge areas according to the difference between the content reconstructed image and a reference image, wherein the reference image is the image to be identified, or the reference image is generated by the image content feature and the forgery clue feature; the forgery error is the error between the image to be identified and the corresponding real image; At least one of the clue reconstructed image, the credibility, the content reconstructed image, the forgery error, and the forgery area is displayed.

2. The method according to claim 1, characterized in that: The reconstructing the image according to the clue to obtain the credibility of the detection result includes: Determining the credibility of the detection result according to the similarity between the clue reconstructed image and the real clue reference image, wherein the real clue reference image is a reconstructed image obtained by inputting the forged clue features of the real reference sample into the generator; and / or, Determine a first distance between the clue reconstructed image and a preset cluster center, and determine the credibility of the detection result based on a size relationship between the first distance and a preset distance threshold, wherein the preset cluster center is obtained by clustering the real clue reference image.

3. The method according to claim 2, characterized in that Before determining the first distance between the clue reconstructed image and the preset cluster center, the method further includes: respectively calculating a second distance between each real clue reference image and the forged clue reference image and the preset cluster center, and determining the preset distance threshold according to the second distance; The forgery clue reference image is a reconstructed image obtained by inputting the forgery clue features of the forged sample into the generator.

4. The method according to claim 1, characterized in that The generator is used to generate a reconstructed image using the image content feature and the specific forgery clue feature, and to generate a reconstructed image using the forgery clue feature and the specific image content feature; The specific content feature is one of the following: a random vector, an all-0 vector, an all-1 vector, a mean of image content features of a preset specific sample, or a specific vector learned by the image detection model during training; The specific clue feature is one of the following: a forged clue feature of a specific real sample, an average of forged clue features of multiple real samples, and an anchor point feature learned by the image detection model during training.

5. The method according to claim 4, characterized in that Before inputting the image to be identified into the content feature extraction network and the clue feature extraction network respectively, the method further includes: Designing a loss function, and using training samples to train the image detection model based on the loss function, wherein the training samples include real training samples and forged training samples; Wherein, the loss function includes classification loss, contrast loss and reconstruction loss; The contrast loss is determined according to the anchor clue feature in the anchor feature, the forgery clue feature of the training sample, and the image content feature of the training sample; The reconstruction loss includes self-reconstruction loss, cross-reconstruction loss and reconstruction contrast loss; The self-reconstruction loss is determined according to the content features and clue features of the training samples; the cross-reconstruction loss is determined according to the real training samples and the corresponding clue training samples; and the reconstruction contrast loss is determined according to the reconstructed images of each pair of training samples.

6. The method according to claim 5, characterized in that The design loss function includes: Obtaining a training sample pair consisting of two training samples, and inputting the forged clue features of each training sample in the training sample pair into the generator to obtain clue reconstructed images of the two training samples; The reconstruction contrast loss of the training sample pair is determined according to the contrast loss between the cue-reconstructed images of the two training samples.

7. The method according to claim 6, characterized in that The method further comprises: Initialize the self-learning feature vector; Freezing the self-learning feature vector, and adjusting the parameters of the image detection model based on the loss function; Freezing parameters of the image detection model and adjusting the self-learning feature vector based on the loss function; Returning to the step of freezing the self-learning feature vector until the loss function converges or is less than a preset loss threshold; The self-learning feature vector is determined to be the specific vector and / or the anchor point feature.

8. An image detection device, characterized in that: The image detection device is applied to an image detection model, the image detection model includes a content feature extraction network, a clue feature extraction network, a classifier and a generator, and the device includes: A feature extraction module is used to obtain an image to be identified, and input the image to be identified into a content feature extraction network and a clue feature extraction network respectively, to obtain image content features and forgery clue features, wherein the image content features include content information of the image to be identified, and the forgery clue features include clue information for identifying the authenticity of the image; An image detection module, used to input the forgery clue features into a classifier, and output and display the detection results corresponding to the image to be identified; a forgery feature determination module, configured to input the forgery clue feature into a generator, obtain a clue reconstruction image based on the clue information, and obtain the credibility of the detection result according to the clue reconstruction image; a forgery clue determination module, configured to input the image content features into the generator, obtain a content reconstructed image based on the content information, and determine a forgery error and a forgery area according to a difference between the content reconstructed image and a reference image, wherein the reference image is the image to be identified, or the reference image is generated by the image content features and the forgery clue features; and the forgery error is an error between the image to be identified and a corresponding real image; A visualization module is used to display the clue reconstruction image, the credibility, the forgery error of the content reconstruction image and at least one of the forgery errors.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and the machine-readable instructions are executed by the processor to execute the steps of the image detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image detection method according to any one of claims 1 to 7 are executed.

Images