Counterfeit detection method

Abstract

The application discloses a forgery detection method, and relates to the technical field of forgery detection. The method is based on an image encoder to perform feature extraction on a to-be-detected image, obtain a first feature vector, a second feature vector and a visual prompt vector set, splice a randomly generated context vector, a category vector and the visual prompt vector to generate an input vector, perform feature extraction on the input vector based on a text encoder, obtain a third feature vector, determine a fusion feature vector and a first similarity based on the second feature vector and the third feature vector, determine a second similarity based on the second feature vector and the fusion feature vector, generate a heat map according to the first similarity and the second similarity, and determine a forgery detection result according to the heat map. Thus, the method can generate a heat map based on a continuous input vector to provide a detection result and mark a forgery position, thereby improving the explainability of forgery detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of forgery detection technology, and in particular to a forgery detection method. BACKGROUND

[0002] In the current era of information explosion, the information on the network is massive and complex, and the emergence of deepfake technology has brought great challenges to the authenticity and reliability of information.

[0003] In related technologies, deepfake detection is performed by using a traditional neural network model. Due to the limitation of the number of parameters and training data, the method has poor adaptability, large model training difficulty, and it is difficult to provide clear and understandable explanations in the detection process, resulting in poor interpretability and low credibility of the method. SUMMARY

[0004] The present application provides a forgery detection method to at least solve the problem of poor interpretability of forgery detection in related technologies.

[0005] The present application provides a forgery detection method, which comprises: performing feature extraction on a to-be-detected image based on a preset image encoder to obtain a first feature vector and a second feature vector set, and determining a plurality of visual prompt vectors based on the first feature vector; performing dimension splicing on a plurality of randomly generated learnable context vectors, class vectors and the plurality of visual prompt vectors to generate an input vector, and performing feature extraction on the input vector based on a preset text encoder to obtain a third feature vector; determining a fusion feature vector set based on the second feature vector set and the third feature vector, determining a first similarity based on the second feature vector set and the third feature vector, and determining a second similarity based on the second feature vector set and the fusion feature vector set; generating a heat map according to the first similarity and the second similarity, and determining a forgery detection result according to the heat map.

[0006] According to the present application, the first feature vector, the second feature vector and the visual prompt vector set are obtained by using the preset image encoder to perform feature extraction on the to-be-detected image, then the learnable context vector, the class vector and the visual prompt vector are dimensionally spliced to generate a continuous input vector, the third feature vector is obtained by using the preset text encoder to perform feature extraction on the input vector, and the first similarity and the second similarity are determined based on the second feature vector, the third feature vector and their fusion feature vector. Finally, the heat map is generated according to the first similarity and the second similarity to determine the forgery detection result and mark the forgery position, which can solve the problem of poor interpretability of forgery detection in related technologies, and achieve the technical effects of improving the efficiency, accuracy and adaptability of forgery detection. BRIEF DESCRIPTION OF DRAWINGS

[0007] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 A flowchart illustrating a counterfeit detection method provided in this application embodiment;

[0009] Figure 2 This is a schematic diagram of the system framework for counterfeit detection according to a specific embodiment of this application;

[0010] Figure 3 This is a flowchart of a counterfeit detection method according to a specific embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0012] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0013] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] The counterfeit detection method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0015] like Figure 1 As shown, the counterfeit detection method of this application embodiment may include:

[0016] S1, based on a preset image encoder, perform feature extraction on the image to be detected to obtain a first feature vector and a set of second feature vectors, and determine multiple visual cue vectors based on the first feature vector.

[0017] Specifically, an image encoder transforms raw image data into a more compact and representative coded form, typically a vector or feature map, providing a foundation for subsequent image analysis. The core function of an image encoder is to extract effective features from the image. Pre-trained image encoders can be selected to reduce computational and data costs during training. For example, a pre-trained CLIP (Contrastive Language-Image Pre-training) model can be chosen. The key advantage of the CLIP model image encoder is its strong generalization ability. After pre-training, it does not require fine-tuning for specific tasks; using only task-related text, the model can complete tasks such as image classification and cross-modal retrieval, breaking the limitation of traditional single-modal models requiring training for a single task.

[0018] In today's information-saturated era, coupled with the development of artificial intelligence, the authenticity of online images is increasingly difficult to discern, and deepfake images are rampant, especially facial forgeries, which have severely impacted individuals and society in multiple ways. Therefore, forgery detection of images is necessary. A schematic diagram of a forgery detection system framework is shown below. Figure 2 As shown, by Figure 2 As can be seen, the image that needs to be deepfake detected is taken as the image to be detected, and subsequent feature extraction and further analysis are performed to determine whether the image to be detected is a fake image.

[0019] Depend on Figure 2 As shown, a preset image encoder extracts features from the image to be detected to obtain a first feature vector and a second feature vector set. The first and second feature vector sets are vector sets formed from the outputs of different levels of the image encoder, enabling layered capture of multi-scale feature information of the image and providing structured and differentiated feature support for subsequent image analysis. The role of the image encoder is to extract features from the image layer by layer. Its output has obvious hierarchical differences; for example, the output includes shallow output, mid-level output, and deep output. Shallow output corresponds to low-order features of the image, while high-level output corresponds to high-order features.

[0020] Visual cue vectors are machine-understandable numerical vectors derived from visual information, such as objects, textures, and spatial locations in an image. Figure 2 Z1, Z2…Zk in the model are used to assist the model in completing tasks such as recognition, localization, and generation. Multiple visual cue vectors are determined based on the first feature vector. The first feature vector can be input into the neural network, and after processing by the neural network, multiple visual cue vectors are generated and output.

[0021] S2, performing dimension splicing based on the plurality of learnable context vectors, the category vector and the plurality of visual prompt vectors to generate an input vector, and performing feature extraction on the input vector based on a preset text encoder to obtain a third feature vector.

[0022] Specifically, the traditional model training data input set generally uses discrete text, such as "a photo of a face", and is trained in combination with the corresponding label classification, such as "real / fake". However, such discrete natural language prompt words usually require a large amount of manual trial and error, which is high in labor cost. In the present application, the training data input is divided into three parts, as shown in the figure. Figure 2 As shown in the figure, the first part is a plurality of learnable context vectors V1, V2…Vm. The learnable context vector can encode context information, and each value in the vector is not fixed but can be updated through model training. For example, in a face forgery image, there are often details such as unnatural eye contact, blurred skin texture, and edge synthesis traces. These details are the context information that the model needs to capture. V1 may focus on encoding the logic of facial organ movement, such as the linkage of eye muscles when a real face blinks, and V2 may focus on encoding the consistency of skin pixel texture, such as abnormal local pixel distribution of a fake face. In the early stage of training, the initial values of the context vectors V1, V2, etc. are randomly generated and cannot accurately identify the details. As the training progresses, the model will continuously correct the values of the context vectors V1, V2, etc. based on the misjudgment of a fake face as a real face, for example, by allowing V1 to more accurately remember the muscle linkage rules of a real blink and allowing V2 to more sensitively capture pixel abnormalities of a fake texture, so as to more accurately distinguish whether a face is fake.

[0023] The initial values of the plurality of learnable context vectors are randomly generated, for example, by random initialization based on a Gaussian distribution. The Gaussian distribution can keep the initial values in a moderate range to ensure stable start of model training. The length m of the learnable context vector can be set according to actual conditions. Generally, a longer m can achieve a higher accuracy on the training set because a longer context can provide more information. However, an excessively long m may cause overfitting, which makes the result fit to a specific distribution of the training set. Therefore, a more accurate m value needs to be set according to actual application conditions, for example, after model training experiment verification, m=8, i.e., the length of the learnable context vector is 8.

[0024] The second part of the training data input is the category vector. The category vector is a vector that describes the category of the image. The conversion of the category name string into a vector is the embedding process of the category word. The category vector is a dense and continuous numerical representation that encapsulates the semantic information of the word, such as "real / fake", and provides category label information for model training.

[0025] The third part of the training data input consists of multiple visual cue vectors, such as... Figure 2 In this context, Z1, Z2…Zk, the visual cue vectors, are numerical vectors that convert specific visual content of an image into machine-understandable values ​​to guide the model in forgery detection. For example, in a face photo, Z1 might focus on the person's shape, Z2 on their facial expression, and so on. The length k of the visual cue vector can be set according to the specific situation. For instance, after model training experiments, k=2 can be set, meaning the length of the visual cue vector is 8.

[0026] The input vector is generated by concatenating multiple randomly generated learnable context vectors, category vectors, and visual cue vectors. This involves combining the three component vectors of the training data input according to their dimensions. For example, if the category vectors are A1 and A2, then the concatenation of multiple learnable context vectors, category vectors, and visual cue vectors can form a multi-dimensional input vector [V1, V2…Vm, A1, A2, Z1, Z2…Zk]. This replaces the discrete text input of the training data with continuous vectors. Discrete natural language prompts typically require extensive manual trial and error, while learnable vectors allow for the automatic search of optimal continuous vectors through gradient descent. This saves manpower and automates the prompt generation process. Furthermore, using continuous input vectors can represent an abstract contextual concept that may not be precisely described by one or a few words in human language, but is the most effective instruction for the model. The training data input includes both category text and visual contextual cues, making it a hybrid input. This dynamically encodes specific visual content of the input image into the text embedding space, thereby providing the subsequent text encoder with contextual information that is highly relevant to the image content. This eliminates the model's sole dependence on the text input and improves the model's detection accuracy.

[0027] like Figure 2 As shown, a third feature vector is obtained by extracting features from the input vector based on a pre-defined text encoder. The text encoder mines the hidden correlation information in the input vector, such as the matching relationship between contextual semantics and visual cues, and outputs the refined third feature vector. The third feature vector has integrated contextual, category, and visual information and can be directly used for subsequent similarity analysis. The pre-defined text encoder can be a pre-trained text encoder to reduce the computational and data costs during training. For example, a pre-trained CLIP model text encoder can be selected. The key advantage of the CLIP model text encoder is that it achieves effective alignment of text and image features in the same space through contrastive learning, which can more accurately understand the relationship between text description and visual content. Moreover, it has strong generalization ability, and after pre-training, it does not require fine-tuning for specific tasks and can be quickly integrated into different model frameworks.

[0028] S3, determine a fusion feature vector set based on the second feature vector set and the third feature vector, determine a first similarity based on the second feature vector set and the third feature vector, and determine a second similarity based on the second feature vector set and the fusion feature vector set.

[0029] Specifically, the fusion feature vector is a vector set obtained by integrating feature vectors of different sources or different types. A single feature can only reflect partial information of data, and through reasonable fusion, the integrated vector can contain more comprehensive and complementary information, thereby helping the model to more accurately understand the nature of the data. The fusion feature vector set is determined based on the second feature vector output by the preset image encoder and the third feature vector output by the preset text encoder. The second feature vector and the third feature vector can be interactively learned by a neural network, automatically mining the correlation between the features, and merging the second feature vector and the third feature vector into a new feature vector according to the correlation between the features. The purpose is to make the fused feature vector more representative than any single original vector.

[0030] Similarity calculation is a technology for measuring the similarity of two or more objects, such as text, image, data vector, etc. at the feature level. Similarity calculation converts abstract similarity into quantifiable numerical values. Common similarity calculation methods include Euclidean distance method, cosine similarity, etc. The first similarity is determined based on the second feature vector set and the third feature vector, and reflects the similarity between the second feature vector and the third feature vector. The second similarity is determined based on the second feature vector set and the fusion feature vector set, and reflects the similarity between the second feature vector and the fusion feature vector. The two different similarity calculation paths represent two different bases for the model to make a fake abnormality judgment. Combining the similarity calculation data of the two paths can obtain a more accurate and more robust fake detection result.

[0031] S4, generate a heat map according to the first similarity and the second similarity, and determine a fake detection result according to the heat map.

[0032] Specifically, the heat map is a visual chart that intuitively presents the value size in the data matrix through color depth changes. In the forgery detection scene, it can convert the distribution relationship of the two key indicators, the first similarity and the second similarity, into an understandable color heat, helping to quickly locate the abnormal area. The heat map is generated according to the first similarity and the second similarity, for example, different weights are assigned to the first similarity and the second similarity, and the similarity matrix is generated, and the similarity matrix is further processed to generate the heat map. According to the heat map, the forgery detection result is determined. The normal area and the abnormal area in the heat map are different in color, and the specific position of the image forgery can be determined, or the heat map of the forged image and the heat map of the real image are different in color, so as to determine whether the image is a forged image. The forgery detection result output can adopt a binary classification result, such as a forged image / non-forged image, etc.

[0033] Therefore, the embodiment generates an input vector based on a plurality of learnable context vectors, category vectors and a plurality of visual prompt vectors generated at random, breaks away from the discrete natural language input of the traditional model, can effectively avoid the interference caused by inaccurate natural language description, improves the precision of the model forgery detection, improves the accuracy of the forgery detection by determining the first similarity and the second similarity, and determines the forgery detection result according to the heat map. The heat map can clearly locate the forgery area, provides an intuitive and reliable basis for forgery detection, and greatly enhances the explainability of the forgery detection result.

[0034] In some embodiments of the present application, feature extraction is performed on the to-be-detected image based on a preset image encoder to obtain a first feature vector and a second feature vector set, including: determining the first feature vector based on the deep layer output feature of the preset image encoder; and determining the second feature vector set based on the shallow layer output feature, the middle layer output feature and the deep layer output feature of the preset image encoder.

[0035] Specifically, the first feature vector and the second feature vector set are vector sets formed for the output of different levels of the preset image encoder. The output of the preset image encoder can be divided into shallow layer output, middle layer output and deep layer output. The shallow layer feature contains more image details, but is insufficient for semantic understanding. The middle layer feature is helpful for understanding the medium scale image abnormal area and context relationship. The deep layer feature encodes high-level semantic concepts, but is slightly missing in the details of the image. Different levels of features provide multiple explainable perspectives for model decision-making. Assuming that the output of the preset image encoder is 24 layers, the output of the 8th layer, the 16th layer and the 24th layer can be taken as the shallow layer output, the middle layer output and the deep layer output respectively. At this time, the deep layer output feature of the 24th layer of the preset image encoder is extracted as the first feature vector, and the output features of the 8th layer, the 16th layer and the 24th layer of the preset image encoder are extracted to form the second feature vector set.

[0036] The embodiment determines a first feature vector based on deep output features of a preset image encoder, determines a second feature vector set based on shallow output features, medium output features and deep output features of the preset image encoder, and extracts multi-layer image features. The advantage is to allow the model to access information at different abstraction levels and spatial precision at the same time to realize cross-scale feature complementation and integration. If only a single level of features is relied on, the model is vulnerable to the specific representation bias or noise of the level. Multi-level features effectively smooth the uncertainty of single-level features, reduce the error probability of the model, enhance the robustness of the model, and provide multiple explainable perspectives for model decision-making, providing more explainable results for forgery detection.

[0037] In some embodiments of the present application, the first feature vector is transformed and expanded based on a preset neural network adapter to generate a plurality of visual prompt vectors, wherein the preset neural network adapter includes a one-dimensional convolution layer.

[0038] Specifically, as shown in Figure 2 The first feature vector is input into the preset neural network adapter, and the preset neural network adapter transforms and expands the first feature vector to generate a plurality of visual prompt vectors Z1, Z2…Zk as part of the input vector. The preset neural network adapter includes a one-dimensional convolution layer, which is good at processing sequence vectors and can extract more fine-grained correlation information in the vector through a convolution kernel (which can be understood as a feature detector), so that the expression of the plurality of visual prompt vectors is more accurate and more consistent with the actual image features. The purpose of generating a plurality of visual prompt vectors based on the preset neural network adapter is to dynamically encode specific visual content of the image to be detected into the text embedding space, thereby providing the subsequent text encoder with highly relevant context information about the image content, eliminating the model's single dependence on the text input, and realizing real visual context prompts.

[0039] The embodiment transforms and expands the first feature vector based on the preset neural network adapter including a one-dimensional convolution layer to generate a plurality of visual prompt vectors as part of the input vector, which can provide more fine-grained and multi-perspective image visual information to help the downstream model more accurately capture the key features of the image, reduce detection errors caused by insufficient original feature information, and provide the subsequent text encoder with highly relevant context information about the image content, eliminating the model's single dependence on the text input, and ultimately improving the precision, adaptability and stability of the forgery detection.

[0040] In some embodiments of the present application, the second feature vector set includes a shallow layer output feature vector, a middle layer output feature vector and a deep layer output feature vector, the fusion feature vector set includes a first fusion feature vector, a second fusion feature vector and a third fusion feature vector, and the fusion feature vector set is determined based on the second feature vector set and the third feature vector, including: taking the third feature vector as a text feature of a cross-modal multi-head attention mechanism, taking the shallow layer output feature vector as an image feature of the cross-modal multi-head attention mechanism, performing multi-head parallel attention calculation, and linearly projecting the outputs of the multi-heads to obtain the first fusion feature vector; taking the third feature vector as a text feature of a cross-modal multi-head attention mechanism, taking the middle layer output feature vector as an image feature of the cross-modal multi-head attention mechanism, performing multi-head parallel attention calculation, and linearly projecting the outputs of the multi-heads to obtain the second fusion feature vector; and taking the third feature vector as a text feature of a cross-modal multi-head attention mechanism, taking the deep layer output feature vector as an image feature of the cross-modal multi-head attention mechanism, performing multi-head parallel attention calculation, and linearly projecting the outputs of the multi-heads to obtain the third fusion feature vector, the fusion feature vector being a mixed feature modulated by image context information.

[0041] Specifically, the cross-modal multi-head attention mechanism is a core component for processing multi-source data in deep learning, and the multi-source data can be text, image or audio content, etc. On the basis of multi-head attention, the association and information fusion problem of different modal data is solved, which can make the model understand the semantics of multiple data and capture the complementary relationship between them. Taking the third feature vector as a text feature of a cross-modal multi-head attention mechanism, taking the shallow layer output feature vector of the second feature vector as an image feature of the cross-modal multi-head attention mechanism, using the query (Query) of the text feature to match the key (Key) and value (Value) of the image feature, calculating the attention weight of the text to the image area, and simultaneously designing the attention head of the text feature and the image feature, multi-head parallel attention calculation is performed, and finally the output results of the multi-heads are spliced and linearly projected to form a feature of a unified dimension. At this time, the feature has included the text semantic, image visual and the association information of the two, and the feature is the first fusion feature vector.

[0042] Similarly, the third feature vector can be taken as a text feature of a cross-modal multi-head attention mechanism, the middle layer output feature vector of the second feature vector can be taken as an image feature of the cross-modal multi-head attention mechanism to obtain the second fusion feature vector, and the third feature vector can be taken as a text feature of a cross-modal multi-head attention mechanism, the deep layer output feature vector of the second feature vector can be taken as an image feature of the cross-modal multi-head attention mechanism to obtain the third fusion feature vector. The first fusion feature vector, the second fusion feature vector and the third fusion feature vector constitute the fusion feature vector set.

[0043] The embodiment takes the third feature vector as the text feature of the cross-modal multi-head attention mechanism, takes the shallow, middle and deep output feature vectors of the second feature vector as the image features of the cross-modal multi-head attention mechanism, respectively calculates the attention to obtain a fusion feature vector set, realizes the accurate matching of the text and image information at different levels, makes the fusion feature more consistent with the internal correlation of the cross-modal data, and improves the cross-modal fusion accuracy. The fusion feature is a mixed feature modulated by the image context information, which strengthens the richness of feature expression.

[0044] In some embodiments of the present application, the second feature vector set includes a shallow output feature vector, a middle output feature vector and a deep output feature vector, and the first similarity value is determined based on the second feature vector set and the third feature vector, including: calculating a first similarity matrix of the shallow output feature vector and the third feature vector, and calculating a second similarity matrix of the middle output feature vector and the third feature vector, and calculating a third similarity matrix of the deep output feature vector and the third feature vector; determining the first similarity based on the average of the first similarity matrix, the second similarity matrix and the third similarity matrix.

[0045] Specifically, the similarity between vectors is usually calculated by cosine similarity, Euclidean distance, etc. Taking the calculation of the first similarity matrix of the shallow output feature vector and the third feature vector as an example, if the cosine similarity method is used, the angle between the two vectors of the shallow output feature vector and the third feature vector needs to be measured. First, calculate the dot product of the two vectors, then calculate the modulus of the two vectors respectively, and finally divide the result of the vector dot product by the product of the modulus of the two vectors. The resulting value is the cosine similarity. The value of the cosine similarity ranges between [-1, 1], and the closer the value is to 1, the more consistent the direction of the two vectors, so the similarity is higher. Since the shallow output feature vector and the third feature vector are both multi-dimensional vectors, a first similarity matrix is formed after all similarity values are calculated. Using the cosine similarity method to calculate the similarity between vectors has simple calculation logic, does not require complex operations, has low consumption of computing resources, is suitable for rapid comparison of large-scale vectors, and has strong adaptability to high-dimensional data. Even if the vector dimension is extremely high, it can still stably output the similarity result, avoiding the problem of failure of some distance algorithms under the dimension disaster. By analogy, the second similarity matrix of the middle output feature vector and the third feature vector, and the third similarity matrix of the deep output feature vector and the third feature vector can be calculated. Then the average of the first similarity matrix, the second similarity matrix and the third similarity matrix is calculated to determine the first similarity.

[0046] The embodiment determines the first similarity based on the average values of the first, second and third similarity matrices by respectively calculating the first, second and third similarity matrices of the shallow, middle and deep output feature vectors and the third feature vector, and the third feature vector is the output vector of the preset text encoder, and the shallow, middle and deep output feature vectors reflect the fine-grained image features, so the first similarity reflects the feature matching of the text and the image, and judges the content correlation degree of the two, which provides a judgment basis for subsequent forgery detection. If it is a real image, the text features and the image features at each level usually have a high similarity, and if it is a fake image, the similarity is usually abnormally low.

[0047] In some embodiments of the present application, the second feature vector set includes shallow output feature vectors, middle output feature vectors and deep output feature vectors, the fused feature vector set includes first fused feature vectors, second fused feature vectors and third fused feature vectors, the second similarity is determined based on the second feature vector set and the fused feature vector set, and the determination includes: calculating a fourth similarity matrix of the shallow output feature vectors and the first fused feature vectors, and calculating a fifth similarity matrix of the middle output feature vectors and the second fused feature vectors, and calculating a sixth similarity matrix of the deep output feature vectors and the third fused feature vectors; and determining the second similarity based on the average values of the fourth similarity matrix, the fifth similarity matrix and the sixth similarity matrix.

[0048] Specifically, the fourth similarity matrix of the shallow output feature vectors and the first fused feature vectors, the fifth similarity matrix of the middle output feature vectors and the second fused feature vectors, and the sixth similarity matrix of the deep output feature vectors and the third fused feature vectors can be calculated by using similarity calculation methods such as cosine similarity and Euclidean distance, and then the average values of the fourth similarity matrix, the fifth similarity matrix and the sixth similarity matrix are calculated to determine the second similarity. The fused feature vector is determined based on the second feature vector set output by the preset image encoder and the third feature vector output by the preset text encoder, and the second feature vector set includes shallow, middle and deep output feature vectors, so the second similarity reflects the similarity of the global text and image fused features and the image fine-grained features, and the correlation between global rationality and local authenticity is established. If the similarity of the fine-grained features of a local part of the image and the global fused features is low, the local part is likely to be an abnormal area. In the judgment of the authenticity of the image, the global features may cover the local abnormalities. For example, the global features of a fake image may seem reasonable, but there are vulnerabilities in the local details. By focusing on the similarity difference between the image fine-grained features and the global features, the local abnormal signal can be actively amplified, and the small tampered area that is not easy to detect can be more prominent in the similarity calculation result, which helps the subsequent module to accurately locate the abnormal position.

[0049] The embodiment determines the second similarity based on the image fine-grained feature set and the text and image fusion feature vector set, covers the authenticity of the image itself and the logical consistency of the image and text, provides more accurate and comprehensive judgment basis for the forgery detection, improves the reliability of the forgery judgment, can capture possible subtle tampering traces of the forged image, helps the subsequent module to accurately locate the abnormal position, and improves the explainability of the forgery detection.

[0050] In some embodiments of the present application, the heat map is generated according to the first similarity and the second similarity, including: obtaining a first product of the first similarity and a first weight coefficient, and obtaining a second product of the second similarity and a second weight coefficient, wherein the sum of the first weight coefficient and the second weight coefficient is 1; determining a target similarity matrix according to the sum value between the first product and the second product; and performing up-sampling processing on the target similarity matrix to obtain the heat map, wherein the heat map has the same size as the to-be-detected image.

[0051] Specifically, the first similarity matrix is denoted as M1, the second similarity matrix is denoted as M2, and the target similarity matrix is denoted as M final The formula for calculating the target similarity matrix is:

[0052] M final =a×M1+b×M2,

[0053] wherein a is the first weight coefficient, b is the second weight coefficient, a+b=1, a×M1 is the first product, b×M2 is the second product, and the values of the weight coefficients a and b can be adjusted according to different detection tasks, such as a=0.2 and b=0.8.

[0054] After obtaining the target similarity matrix, up-sampling processing is performed on the target similarity matrix. Up-sampling is a process of increasing the dimension or resolution of data by a specific method, so that the data is promoted from a lower dimension / resolution to a higher dimension / resolution. For example, the up-sampling method uses a bilinear interpolation method. By up-sampling the target similarity matrix, the detailed information is supplemented, and a high-definition heat map with the same size as the to-be-detected image can be obtained.

[0055] The embodiment assigns different weight coefficients to the first similarity and the second similarity to determine the target similarity matrix, performs up-sampling processing on the target similarity matrix to generate a high-definition heat map with the same size as the to-be-detected image, can intuitively map the forgery suspicion degree of each region of the image, improve the explainability of the forgery detection result, and different similarity weights can make the target similarity matrix more suitable for the needs of the forgery detection. Up-sampling processing can avoid losing details in a low-resolution matrix, accurately locate minor forgeries, and reduce missed judgments.

[0056] In some embodiments of the present application, the method for determining the forgery detection result according to the heat map comprises: performing analysis and processing on the heat map based on a global aggregator to obtain a target vector; performing analysis and processing on the target vector based on a fully connected neural network layer to output the forgery detection result; wherein the global aggregator is a two-way aggregator, and the target vector is obtained by performing analysis and processing on the heat map based on the global aggregator, which comprises: performing global maximum pooling processing on the heat map based on one way of the two-way aggregator to obtain a first result; performing global average pooling processing on the heat map based on the other way of the two-way aggregator to obtain a second result; and splicing the first result and the second result to obtain the target vector.

[0057] Specifically, the global aggregator is a component that aggregates and integrates the entire input feature or multiple local features, and can integrate the local spatial features of the heat map into a global one-dimensional vector. Common methods include global average pooling and global maximum pooling. The target vector is obtained by performing analysis and processing on the heat map based on the global aggregator, which is a condensed feature of the heat map and no longer contains the spatial position information of the pixels, but retains the global abnormality degree of the heat map. The core feature of the fully connected neural network layer is that each neuron in the layer is fully connected to all neurons in the previous layer, which is the basic structure for realizing linear transformation and nonlinear mapping of features. The fully connected neural network layer associates the abnormal features of the target vector, and gradually maps the high-dimensional target vector to a low-dimensional result vector through the calculation of multiple neurons, to output the image forgery detection result.

[0058] The global aggregator is a two-way aggregator, one way of which performs global maximum pooling processing on the heat map to output the highest abnormal score in the heat map as the first result, which reflects the abnormality degree of the most suspicious point in the image, and the global maximum pooling is very sensitive to small but obvious defects. The other way performs global average pooling processing on the heat map to calculate the average value of all pixel points in the heat map as the second result, which reflects the average level of the overall forgery region in the image. The first result and the second result are spliced to splice the outputs of the two ways into a feature vector to obtain the target vector. The target vector fuses the abnormality degree of the most suspicious single point in the image and the average level of the overall forgery region, neither missing small but clear key defects nor ignoring broad but slight overall abnormalities, avoiding the one-sidedness of a single pooling method, providing a two-dimensional basis for forgery detection judgment, and improving the accuracy and robustness of forgery detection.

[0059] The target vector is input into a fully connected neural network layer, each neuron in the layer is connected with all dimensions of the input vector one by one, and by assigning different weights to different dimensional features, the key features are strengthened and the irrelevant features are weakened, and a bias term is added to offset the interference of feature scale difference. Through the calculation of weighted summation and bias, the input target vector is converted into intermediate feature values, and the extraction of the counterfeit related features is completed. However, the intermediate feature values are continuous values and cannot be directly mapped to the binary classification results of counterfeit / non-counterfeit. Therefore, based on the sigmoid activation function, the intermediate feature values are mapped to the range of 0-1. When used, a threshold value such as 0.5 is set. When the output is greater than or equal to 0.5, it is determined to be counterfeit, and when the output is less than 0.5, it is determined to be non-counterfeit. Finally, a simple and intuitive binary classification counterfeit detection result is obtained.

[0060] This embodiment analyzes and processes the heat map based on the global aggregator to obtain the target vector, and analyzes and processes the target vector based on the fully connected neural network layer to output the counterfeit detection result. By analyzing the heat map through the global aggregator, global features related to counterfeiting can be more completely extracted, thereby improving the comprehensiveness of counterfeit feature capture. The fully connected neural network layer can accurately identify hidden counterfeit feature patterns in the target vector, reduce misjudgment / omission caused by feature ambiguity and noise interference, and enhance the accuracy and robustness of the detection result. The combination of global aggregation and fully connected network can not only extract common features of different types of counterfeits, but also adapt to heat map inputs of different data distributions through parameter learning of the network layer, thereby improving the universality and practicality of the counterfeit detection scheme in various counterfeit scenarios.

[0061] In some embodiments of the present application, the counterfeit detection method further comprises: comparing the first similarity and the first reference similarity based on a first preset loss function to calculate a first loss value; comparing the second similarity and the second reference similarity based on a second preset loss function to calculate a second loss value; determining a total loss value based on the sum of the first loss value and the second loss value; and calculating the parameter gradient by back propagation based on the total loss value, so as to adjust the corresponding parameters according to the parameter gradient.

[0062] Specifically, the reference similarity is the similarity corresponding to the real image sample, the preset loss function is used to quantify the difference between the first similarity and the first reference similarity, the first similarity and the first reference similarity are substituted into the first preset loss function for calculation to obtain a first loss value, the smaller the loss value, the closer the model's similarity calculation of the image to the real situation, the better the model effect, otherwise, the model deviation is large, and the parameters need to be adjusted and optimized. Similarly, the second similarity and the second reference similarity can be compared based on the second preset loss function to calculate a second loss value, and the first loss value and the second loss value are added to obtain a total loss value, the total loss value is used to comprehensively measure the performance of the model in learning real consistency and distinguishing fake difference, and the physical meaning of the total loss value is the overall performance error of the model under the current parameters, and the smaller the value, the better the model's consistency learning of real samples and difference distinguishing effect of fake samples.

[0063] The total loss value is a function of the model parameters, and the gradient of each parameter on the loss value, i.e. the influence degree of parameter change on the loss value, is calculated by the chain rule. According to the calculated gradient, the optimizer is used to update the parameters, and the core principle is to adjust the parameters along the gradient descent direction to reduce the total loss value. This process is performed by iteration, and each round is based on new image samples to calculate the loss, gradient and update the parameters, until the total loss value converges to a stable minimum value, at which time the model reaches the optimal state.

[0064] Here, the parameters of the preset image encoder and the preset text encoder are not included in the adjusted parameters, and the parameters of the preset image encoder and the preset text encoder are frozen, and other parameters in the system are fine-tuned, to realize a training strategy of partial freezing and partial fine-tuning, fully utilize the pre-training knowledge of the pre-trained image and text encoders, and efficiently introduce new learning parameters, so as to finally achieve the purposes of improving training efficiency, reducing training cost and improving model training accuracy.

[0065] The embodiment calculates the total loss value by the similarity and the reference similarity, calculates the parameter gradient by the chain rule based on the total loss value, adjusts the corresponding parameters according to the parameter gradient, and freezes the parameters of the preset image encoder and the preset text encoder, to realize a training strategy of partial freezing and partial fine-tuning, not only reserve the pre-training knowledge of the pre-trained image and text encoders, but also update other parameters, so as to achieve the technical effects of improving training efficiency, reducing training cost and improving model training accuracy.

[0066] In some embodiments of the present application, the first preset loss function is:

[0067]

[0068] The second preset loss function is:

[0069]

[0070] wherein, denotes the first loss value, denotes the second loss value, denotes a hyperparameter, denotes the number of feature vectors of the second feature vector set, denotes the layer index from which the image feature comes, denotes a hyperparameter of the weight, denotes the height of the feature vector matrix, denotes the width of the feature vector matrix, is a hyperparameter, denotes the probability predicted by the model of the i-th layer in the calculation of the first similarity, denotes the probability predicted by the model of the i-th layer in the calculation of the second similarity, denotes the label of the image to be detected, denotes the probability predicted by the model of the i-th layer in the calculation of the first similarity, denotes the probability predicted by the model of the i-th layer in the calculation of the second similarity, denotes the label of the image to be detected, denotes the similarity matrix calculated by the i-th layer in the calculation of the first similarity, denotes the similarity matrix calculated by the i-th layer in the calculation of the second similarity, denotes the pixel-level label of the image to be detected, is a hyperparameter, denotes the smoothing coefficient. Specifically,

[0071] denotes the weight coefficient of each branch of the dual-similarity branch, which can be adjusted according to actual use, for example, set . denotes the number of feature vectors of the second feature vector set, denotes the layer index from which the image feature comes, for example, the feature vectors of the second feature vector set come from the 8th, 16th and 24th layers of the preset image encoder, , . . denotes a hyperparameter of the weight, which can be adjusted according to actual use, for example, set . is a hyperparameter, used to adjust the weight of the real sample and the fake sample, in order to increase the weight of the fake sample, can be used . denotes the label of the image to be detected, the value of is 0 or 1. is a binary matrix, whose element value is 0 or 1. is a hyperparameter, The greater, the more the model will focus on difficult image samples, which can be set . The smooth coefficient represents the numerical stability, which can be set .

[0072] This embodiment defines the specific form of the first preset loss function and the second preset loss function, and explains the adjustable parameters in the function, to ensure the accuracy of the loss value calculation. The loss function calculates the difference between the final output of the model and the label of the real image, to guide the model learning and update the parameters, so as to improve the training accuracy of the model.

[0073] As a specific embodiment of the present application, the flow of the forgery detection method is as shown in Figure 3 , which can include the following steps:

[0074] S101, determining a first feature vector based on the deep output feature of the preset image encoder.

[0075] S102, determining a second feature vector set based on the shallow output feature, the middle output feature and the deep output feature of the preset image encoder.

[0076] S103, transforming and expanding the first feature vector based on the preset neural network adapter to generate a plurality of visual prompt vectors.

[0077] S104, performing dimension splicing based on the randomly generated plurality of learnable context vectors, category vectors and a plurality of visual prompt vectors to generate an input vector.

[0078] S105, performing feature extraction on the input vector based on the preset text encoder to determine a third feature vector.

[0079] S106, taking the second feature vector set and the third feature vector as the input of the cross-modal multi-head attention mechanism to generate a fusion feature vector set.

[0080] S107, calculating the first similarity of the second feature vector set and the third feature vector.

[0081] S108, calculating the second similarity of the second feature vector set and the fusion feature vector set.

[0082] S109, determining a target similarity matrix according to the first similarity and the second similarity, and performing upsampling processing on the target similarity matrix to obtain a heat map.

[0083] S110, analyzing and processing the heat map based on a global aggregator to obtain a target vector, and analyzing and processing the target vector based on a fully connected neural network layer to output a forgery detection result.

[0084] Therefore, the forgery detection method proposed in this embodiment generates text representation related to image content through visual context prompt dynamically, completely gets rid of the natural language description input required by the traditional model, can effectively avoid the interference caused by inaccurate natural language description, improves the forgery detection precision and the generalization ability of the model; injects visual information from the global and local two levels into the text prompt, so that the model can adaptively adjust the forgery judgment standard according to the input image content, significantly reduces the false positive rate in complex background, improves the accuracy of forgery part segmentation, enhances the adaptability and accuracy of the model; the model outputs a high-resolution abnormal heat map, which clearly locates the forgery area, provides an intuitive and reliable basis for forgery detection, and greatly enhances the explainability of the forgery detection result. This method can be used in social media supervision, intellectual property protection, news and advertisement supervision and other fields, and can play an important role in helping enterprises and individuals to protect intellectual property rights and prevent the spread of false information.

[0085] In this embodiment, a preset image encoder is used to extract features of the to-be-detected image to obtain a first feature vector, a second feature vector and a visual prompt vector set. Then, a randomly generated learnable context vector, a category vector and a visual prompt vector are dimensionally spliced to generate a continuous input vector. A preset text encoder is used to extract features of the input vector to obtain a third feature vector. The first similarity and the second similarity are determined based on the second feature vector, the third feature vector and their fusion feature vector. Finally, a heat map is generated according to the first similarity and the second similarity to determine the forgery detection result and mark the forgery position, which can solve the problem of poor explainability of forgery detection in related technologies, and achieve the technical effects of improving the efficiency, accuracy and adaptability of forgery detection.

[0086] Those skilled in the art will further appreciate that the elements and algorithms described in the examples illustrated by the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the elements and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0087] The above describes in detail the counterfeit detection method provided by the present application. The principles and implementation manners of the present application are described by using specific examples, and the above description of the examples is only applicable to helping understand the method of the present application and the core idea thereof. It should be pointed out that, for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A forgery detection method characterized by comprising: The method comprises: feature extraction on the to-be-detected image based on a preset image encoder to obtain a first feature vector and a second feature vector set, and determination of a plurality of visual prompt vectors based on the first feature vector; dimensional splicing based on a plurality of randomly generated learnable context vectors, category vectors and the plurality of visual prompt vectors to generate an input vector, and feature extraction on the input vector based on a preset text encoder to obtain a third feature vector; determination of a fusion feature vector set based on the second feature vector set and the third feature vector, determination of a first similarity based on the second feature vector set and the third feature vector, and determination of a second similarity based on the second feature vector set and the fusion feature vector set; generation of a heat map based on the first similarity and the second similarity, and determination of a forgery detection result based on the heat map.

2. The forgery detection method according to claim 1, characterized in that, The method comprises: feature extraction on the to-be-detected image based on a preset image encoder to obtain a first feature vector and a second feature vector set, and determination of a plurality of visual prompt vectors based on the first feature vector; determination of the first feature vector based on deep output features of the preset image encoder; 3. The forgery detection method according to claim 1 or 2, characterized in that, determination of the second feature vector set based on shallow output features, middle output features and deep output features of the preset image encoder. The method comprises:

4. The forgery detection method of claim 1, wherein transformation and expansion of the first feature vector based on a preset neural network adapter to generate the plurality of visual prompt vectors, wherein the preset neural network adapter comprises a one-dimensional convolution layer. The second feature vector set comprises shallow output feature vectors, middle output feature vectors and deep output feature vectors, the fusion feature vector set comprises first fusion feature vectors, second fusion feature vectors and third fusion feature vectors, and the method comprises: taking the third feature vector as a text feature of a cross-modal multi-head attention mechanism, taking the shallow output feature vector as an image feature of the cross-modal multi-head attention mechanism, performing multi-head parallel attention calculation, and concatenating and linearly projecting outputs of the plurality of heads to obtain the first fusion feature vector; taking the third feature vector as a text feature of a cross-modal multi-head attention mechanism, taking the middle output feature vector as an image feature of the cross-modal multi-head attention mechanism, performing multi-head parallel attention calculation, and concatenating and linearly projecting outputs of the plurality of heads to obtain the second fusion feature vector; 5. The forgery detection method of claim 1, wherein taking the third feature vector as a text feature of a cross-modal multi-head attention mechanism, taking the deep output feature vector as an image feature of the cross-modal multi-head attention mechanism, performing multi-head parallel attention calculation, and concatenating and linearly projecting outputs of the plurality of heads to obtain the third fusion feature vector. The second feature vector set comprises shallow output feature vectors, middle output feature vectors and deep output feature vectors, and the method comprises: compute a first similarity matrix of the shallow output feature vectors and the third feature vectors, a second similarity matrix of the middle output feature vectors and the third feature vectors, and a third similarity matrix of the deep output feature vectors and the third feature vectors; determine the first similarity based on an average value of the first similarity matrix, the second similarity matrix and the third similarity matrix.

6. The forgery detection method of claim 1, wherein, The second feature vector set includes shallow output feature vectors, middle output feature vectors and deep output feature vectors, the fusion feature vector set includes first fusion feature vectors, second fusion feature vectors and third fusion feature vectors, and the second similarity is determined based on the second feature vector set and the fusion feature vector set, including: compute a fourth similarity matrix of the shallow output feature vectors and the first fusion feature vectors, a fifth similarity matrix of the middle output feature vectors and the second fusion feature vectors, and a sixth similarity matrix of the deep output feature vectors and the third fusion feature vectors; determine the second similarity based on an average value of the fourth similarity matrix, the fifth similarity matrix and the sixth similarity matrix.

7. The forgery detection method of claim 1, wherein generate a heat map based on the first similarity and the second similarity, including: obtain a first product of the first similarity and a first weight coefficient, and obtain a second product of the second similarity and a second weight coefficient, wherein the sum of the first weight coefficient and the second weight coefficient is 1; determine a target similarity matrix based on a sum value between the first product and the second product; perform up-sampling processing on the target similarity matrix to obtain the heat map, wherein the heat map has the same size as the to-be-detected image.

8. The forgery detection method of claim 1, wherein, determine a forgery detection result based on the heat map, including: analyze and process the heat map based on a global aggregator to obtain a target vector; analyze and process the target vector based on a fully connected neural network layer to output the forgery detection result; wherein the global aggregator is a two-way aggregator, and analyzing and processing the heat map based on the global aggregator to obtain a target vector includes: performing global maximum pooling processing on the heat map based on one way of the two-way aggregator to obtain a first result; performing global average pooling processing on the heat map based on the other way of the two-way aggregator to obtain a second result; splicing the first result and the second result to obtain the target vector.

9. The forgery detection method of claim 1, wherein, The method further includes: comparing the first similarity with a first reference similarity based on a first preset loss function to calculate a first loss value; comparing the second similarity with a second reference similarity based on a second preset loss function to calculate a second loss value; determining a total loss value based on a sum value between the first loss value and the second loss value; calculating a parameter gradient by back propagation based on the total loss value, so as to adjust the corresponding parameters according to the parameter gradient.

10. The forgery detection method according to claim 9, characterized in that, The first preset loss function is: The second preset loss function is: in, This represents the first loss value. This represents the second loss value. Indicates hyperparameters, This indicates the number of eigenvectors in the second eigenvector set. This indicates the layer index from which the image features originate. Hyperparameters representing weights, The height of the eigenvector matrix is ​​represented by the height of the eigenvector matrix. This represents the width of the eigenvector matrix. For hyperparameters, This indicates that in calculating the first similarity, the... The probability predicted by the model of the layer. This indicates that in calculating the second similarity, the first... The probability predicted by the layer model. The annotation represents the image to be detected. Indicates the first The first similarity matrix is ​​calculated at each layer. Indicates the first The second similarity matrix is ​​calculated at each layer. This represents the pixel-level annotation of the image to be detected. It's a hyperparameter. This represents the smoothing coefficient.

Images