A visual encoder training method and a face forgery detection and explanation method

Abstract

The application provides a visual encoder training method and a face forgery detection and explanation method. The training method comprises the following steps: obtaining a face image, a corresponding authenticity label and an explanation text, and sequentially obtaining an image feature sequence by dividing the face image into blocks; initializing a learnable prompt vector for classification, general forgery and explanation perception, and splicing the image feature sequence into a composite input sequence; inputting the composite input sequence into a visual encoder to extract classification features, general pattern features and explanation features; using a frozen text encoder to extract semantic embeddings corresponding to the authenticity label and the explanation text; constructing a comprehensive loss function, updating the learnable prompt vector based on the comprehensive loss function, and obtaining a trained visual encoder. The application breaks the modal gap between the bottom-level visual discrimination features and the high-level language reasoning, and realizes the pre-alignment of the explanation semantics in the early stage of visual feature extraction.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a visual encoder training method and a face forgery detection and interpretation method. Background Technology

[0002] Facial forgery plays a crucial role in identity verification, financial risk control, and public safety systems. In recent years, with the rapid development of deep generative models (such as generative adversarial networks and diffusion models), highly realistic forged facial images and videos have become virtually indistinguishable from genuine ones. The misuse of these technologies poses serious threats to personal privacy, information security, and even social trust. Therefore, developing robust and reliable facial forgery detection algorithms is urgently needed.

[0003] Traditional face forgery detection methods focus on a binary classification task, aiming to uncover spatial artifacts or frequency domain anomalies in local images. These models typically only output a probability score regarding whether a face is real or fake, failing to provide specific forgery locations, forgery types, or intuitive visual evidence. This lack of interpretability severely limits their application in high-trust scenarios such as forensic identification and finance, making it difficult for human reviewers to verify and trust their judgments.

[0004] To address the aforementioned interpretability challenges, the development of multimodal large language models has provided a new opportunity for deep visual understanding and natural language reasoning. Currently, some research has attempted to apply multimodal large models to face forgery detection tasks, aiming to generate natural language-level detection reports. However, most existing interpretable detection methods are limited by a training paradigm that relies on post-interpretation injection. This involves using a simple binary classification loss to train a visual feature extractor, and then using these extracted features to fine-tune the large language model to generate interpretations. This paradigm suffers from a severe modal gap. When these semantically unaligned features are input into the large language model, it easily leads to model illusion, resulting in a mismatch between the generated interpretive text and the visual semantics of the image.

[0005] A search revealed a Chinese patent application (application number: 202511031987.8) disclosing a training method for a deepfake detection model and a deepfake detection method. This method constructs a training dataset containing real images and images of various forgery types, and assigns real / fake labels and forgery reason annotations to the images. This allows the model to learn diverse forgery patterns during training and understand the essential features of forged content through semantic annotation. However, this technique primarily focuses on utilizing semantic information in the later classification or interpretation stages, failing to achieve explicit alignment between visual features and textual semantics in the early stages of visual feature extraction. Due to the lack of early cross-modal semantic guidance, a modal gap exists between the extracted low-level visual features and high-level semantic concepts, resulting in a lack of reliable semantic evidence to support the downstream large language model when generating interpretations, easily leading to illusion phenomena.

[0006] In summary, bridging the modal gap between low-level visual discrimination features and high-level language reasoning, and achieving pre-alignment with interpretive semantics in the early stages of visual feature extraction, thereby constructing a face forgery detection and interpretation framework that possesses both high discrimination accuracy and generates reliable, accurate, and hallucination-free evidence interpretation, is a key technical challenge that urgently needs to be addressed in this field. Summary of the Invention

[0007] In view of the deficiencies in the prior art, the purpose of this application is to provide a visual encoder training method and a face forgery detection and interpretation method.

[0008] A first aspect of this application provides a visual encoder training method based on early semantic injection, comprising: Obtain a face image and its corresponding true / false labels and explanatory text, and serialize the face image into blocks to obtain an image feature sequence; Initialize a learnable cue vector that includes classification, general forgery, and interpretation-aware cue vectors, and concatenate it with the image feature sequence to form a composite input sequence; The composite input sequence is input into a visual encoder to extract classification features, general pattern features, and interpretive features; The semantic embeddings corresponding to the true and false tags and the explanatory text are extracted using a frozen text encoder; Construct a comprehensive loss function, which includes a classification loss based on the classification features and a contrastive loss that constrains the similarity between the general pattern features, the interpretive features, and the semantic embedding; The learnable cue vector is updated based on the comprehensive loss function to obtain a visual encoder configured with the trained learnable cue vector.

[0009] Optionally, the step of extracting the semantic embeddings corresponding to the true / false tags and the explanatory text using a frozen text encoder includes: Construct corresponding prompt text based on the true / false labels, and input it into the text encoder to obtain the classification semantic embedding; The explanatory text is input into the text encoder to obtain the explanatory semantic embedding.

[0010] Optionally, the constraint on the contrastive loss of the similarity between the general pattern features, the interpretive features, and the semantic embedding includes: Calculate the first cosine similarity between the general pattern features and the classification semantic embedding; Calculate the second cosine similarity between the explanatory features and the explanatory semantic embedding; Calculate the first difference between 1 and the first cosine similarity; Calculate the second difference between 1 and the second cosine similarity; The contrast loss is obtained by summing the product of the first difference and the preset first weight coefficient, and the product of the second difference and the preset second weight coefficient.

[0011] Optionally, the classification loss based on the classification features includes: The classification features, general pattern features, and explanatory features are input into the classifier to obtain the true / false prediction results; The difference between the true / false prediction result and the true / false label is calculated to obtain the classification loss.

[0012] A second aspect of this application provides an interpretable face forgery detection method based on early semantic injection, comprising: Using a trained visual encoder based on an early semantic injection method and a trained learnable cue vector, the face image to be detected is segmented and serialized, and then concatenated with the trained learnable cue vector before being input into the visual encoder to extract pre-aligned visual features. The pre-aligned visual features are mapped by a pre-trained projector and concatenated with the query text, then input into a large language model that integrates a pre-optimized low-rank adaptive module. The large language model is used to generate a response sequence to obtain the output results; wherein, the output results include the face authenticity determination results and the corresponding natural language interpretation report.

[0013] Optionally, the large language model integrating the pre-optimized low-rank adaptive module specifically involves inserting a trainable low-rank decomposition matrix only in the query and key-value projection layers of the self-attention module of the large language model.

[0014] Optionally, the projector is used to connect visual features with the language space, mapping the dimensions of the pre-aligned visual features to the language embedding space dimension of the large language model.

[0015] Optionally, the projector and the low-rank adaptive module are jointly optimized, including: Based on a deepfake visual question answering dataset, we calculate the standard language modeling loss between the generated response sequence and the real natural language interpretation label; Based on the standard language modeling loss, the parameters of the projector and the low-rank adaptive module are jointly updated using gradients.

[0016] A third aspect of this application provides a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform the visual encoder training method and the face forgery detection and interpretation method described in any one of the claims.

[0017] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the visual encoder training method and the face forgery detection and interpretation method described in any one of the claims.

[0018] The visual encoder training method provided in this application combines and deeply improves two techniques: multidimensional feature cue learning and cross-modal contrastive learning. On the one hand, by introducing multidimensional learnable cues, the visual encoder is guided to decouple complex facial manipulation traces, enabling the encoder to accurately capture different dimensions of forgery features, such as classification features for basic discrimination, general pattern features for improving generalization, and explanatory features for focusing on local visual defects. On the other hand, the designed forgery text-image contrastive loss provides the ability to inject semantics early, forcing the extracted low-level visual forgery features to be pre-aligned with the semantic embeddings of the corresponding real and fake labels and explanatory texts before entering the downstream large language model inference stage.

[0019] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0020] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a visual encoder training method based on early semantic injection according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating the framework of an interpretable face forgery detection method based on early semantic injection, according to an exemplary embodiment. Figure 3 This is an example of the output of an interpretable face forgery detection method based on early semantic injection, according to an exemplary embodiment. Detailed Implementation

[0021] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0022] In related technologies, visual feature extractors are typically trained using only binary classification loss, and then the extracted visual features are used to fine-tune a large language model to generate explanations. However, this paradigm suffers from a significant visual-language modality gap. Because visual features lack explicit alignment with textual semantics during the extraction stage, the downstream large language model suffers from a lack of reliable semantic evidence, leading to misinterpretations. To address these issues, this application provides a visual encoder training method based on early semantic injection to resolve the aforementioned problems.

[0023] Reference Figure 1 and Figure 2 As shown in one embodiment of this application, a visual encoder training method based on early semantic injection is provided, including the following steps: S100: Obtain the face image and its corresponding true / false labels and explanatory text, and serialize the face image into blocks to obtain the image feature sequence; S200, Initialize the learnable cue vector including classification, general forgery and interpretation perception cue vectors and concatenate it with the image feature sequence to form a composite input sequence; S300 inputs the composite input sequence into the visual encoder to extract classification features, general pattern features, and interpretive features; S400 uses a frozen text encoder to extract true and false tags and interpret the semantic embeddings corresponding to the text. S500 constructs a comprehensive loss function, which includes a classification loss based on classification features, and a contrastive loss that constrains the similarity between general pattern features, explanatory features, and semantic embeddings. S600 updates the learnable cue vector based on the comprehensive loss function, resulting in a visual encoder configured with the trained learnable cue vector.

[0024] The above embodiments construct an early cross-modal alignment mechanism between visual perception and language reasoning. This method introduces multi-dimensional learnable cues during the feature extraction stage of the visual detector and utilizes forged text-image contrastive learning to align low-level visual features with high-level semantic interpretations early on. This guides visual features to focus on genuine forgery traces and provides robust semantic evidence support for downstream large language models. This method significantly improves the model's cross-domain detection accuracy and robustness against unknown forgery techniques while also endowing the detection system with the ability to generate accurate and reliable explanations for forgery causes.

[0025] To achieve fine-grained feature representation of facial images and provide a foundation for subsequent semantic prompt injection, in some specific embodiments of this application, S100, the facial image and its corresponding true / false labels and explanatory text are obtained, and the facial image is segmented and serialized to obtain an image feature sequence, including the following steps: S101, Obtain the face images in the training set and their corresponding true / false labels and explanatory text.

[0026] Among them, the authenticity label is a binary label that represents the authenticity or forgery attribute of the image, and the natural language interpretation text is text data that describes forgery traces, abnormal areas or judgment criteria in the image. S102, Divide the face image into N blocks of size using a block-splitting operation. The image patch (p is an integer greater than 2) is flattened and projected into a feature sequence. .

[0027] The above embodiments convert two-dimensional image data into a one-dimensional sequence format, which is suitable for the input requirements of visual encoders based on the Transformer architecture; by preserving the local spatial information of the image through block operations, it is beneficial for the model to capture subtle forgery traces.

[0028] To achieve multi-dimensional decoupling and early semantic guidance of face forgery features, in some specific embodiments of this application, S200, the learnable cue vectors for classification, general forgery, and interpretation perception are initialized and concatenated with the image feature sequence to form a composite input sequence, including the following steps: S201, Initialize three types of learnable spoofing cue vectors using a normal distribution. and .in, These are classification cue vectors, used to guide the model to focus on basic classification features. It is a generic forgery hint vector used to focus on cross-domain generic forgery patterns. It is an interpretive cue vector used to associate interpretive semantics.

[0029] For example, three types of learnable cue vectors and All are initialized to a dimension of D A floating-point vector (e.g., 1024-dimensional) following a normal distribution N(0, 0.02) with a mean of 0 and a standard deviation of 0.02 can be represented in an example numerical form as a sequence containing 1024 floating-point numbers, such as [0.012, ..., ...]. [0.045, 0.003, ..., 0.021]. Classification cue vectors. Used to guide the model to focus on basic classification features, after training it will be embedded in the semantic space near the real and fake labels output by the text encoder, corresponding to the text examples "This is a real image" or "This is a fake image"; general fake hint vector Used to focus on cross-domain general forgery patterns, its semantic alignment target is the semantic embedding of "forgery traces exist", "frequency domain anomaly" or "generative artifact"; interpreting perceptual cue vectors Used to associate fine-grained interpretive semantics, with semantic alignment targets being semantic embeddings such as "blending marks at the eye edges", "inconsistent lighting", or "texture anomalies".

[0030] S202, the feature sequence The classifier [CLS] and the three types of forgery hint vectors mentioned above are concatenated to form the composite input sequence Z of the visual feature extractor. The classifier [CLS] is used to aggregate global sequence information.

[0031] Specifically, its splicing formula is expressed as follows:

[0032] In the above embodiments, the three types of cue vectors serve as dedicated anchors for classification, general pattern capture, and semantic association interpretation, respectively. This forces the visual encoder to distinguish information from different task dimensions early in feature extraction, avoiding feature confusion. This design not only enhances the model's sensitivity to subtle forgery traces and improves cross-domain generalization ability, but more importantly, it injects high-level semantic prior knowledge into low-level visual features through forward propagation. This provides solid semantic evidence to support the generation of accurate and interpretable identification reports by the subsequent large language model, effectively mitigating the illusion problem in multimodal tasks.

[0033] In some specific embodiments of this application, in S300, the composite input sequence is input into the visual encoder to extract classification features, general pattern features, and interpretive features, which can be achieved by the following steps: S301, Constructing a visual facial feature extractor (Visual encoder), which is used for further visual feature extraction of facial images.

[0034] S302, input the composite input sequence Z into the face feature visual extractor. In this process, feature interaction and transformation are performed through a multi-layer self-attention mechanism to obtain an output feature sequence with the same length as the input sequence. S303, based on three types of learnable forgery hint vectors and The position index in the composite input sequence is used to extract the feature vector at the corresponding position from the output feature sequence, resulting in the following: basic forgery classification features ( ), General forgery characteristics ( ) and explanatory forgery features ( ), and simultaneously output the visual basic features corresponding to the image patch ( ).

[0035] The above embodiments, through the deep fusion of preset cue vectors and self-attention mechanisms, achieve multi-dimensional and refined stripping of forgery features within a single backbone network. This not only improves the accuracy and generalization of forgery detection, but also, at the technical level, compensates for the shortcomings of existing forgery detection technologies in terms of transparency and understandability through the independent extraction of interpretable features.

[0036] To construct a stable cross-modal semantic alignment target, in some specific embodiments of this application, S400, the semantic embeddings corresponding to true and false tags and explanatory text are extracted using a frozen text encoder, which can be achieved by the following steps: S401, for a single face image I in the training set, obtain its corresponding true / false binary classification label. And detailed natural language forgery explanation labels .

[0037] S402, using a pre-trained and parameter-frozen text feature extractor, extracts explanatory labels. Corresponding Explanatory Semantic Embedding At the same time, the binary classification labels will be used. After being converted into prompt text, its corresponding categorical semantic embedding is extracted. .

[0038] The above embodiments transform the binary classification task of pure visual detection into a semantic alignment task, ensuring that the extracted forgery features are consistent with their corresponding natural language descriptions in spatial distribution, thus laying a solid semantic foundation for subsequent interpretable generation.

[0039] To optimize the parameters of the visual encoder and achieve semantic alignment, in some specific embodiments of this application, S500, a comprehensive loss function is constructed, including a classification loss based on classification features and a contrastive loss constraining the similarity between general pattern features, explanatory features, and semantic embeddings. This can be achieved by the following steps: S501, Classification loss calculation.

[0040] The basic forgery classification features output by the visual extractor Common counterfeiting characteristics and explanatory forgery features The data is then concatenated and input into a specialized forgery classifier. The algorithm then processes the data and outputs a scalar prediction value representing the probability of face image forgery after passing through an activation function. Then, the predicted value was calculated. Binary classification labels for images The binary cross-entropy loss between them is expressed by the formula: Comparative loss calculation.

[0041]

[0042]

[0043] S502, Comparison loss calculation.

[0044] Cosine similarity is used as the distance metric to calculate the directional consistency between general forgery features and categorical semantic embeddings, and between explanatory forgery features and explanatory semantic embeddings. Text-image contrast loss is also employed. The calculation formula is:

[0045] S503, parameter updates and optimizations.

[0046] Classification loss Compared with loss The weighted summation yields the comprehensive loss function, which is then used in the backpropagation algorithm to extract the learnable cue vectors from the facial feature visual extractor. Perform gradient backpropagation and updates.

[0047] The above embodiments optimize the model's image feature extraction process by constructing two different functions of loss: classification loss and contrast loss, enabling the model to understand the semantic features of forgery.

[0048] Based on the same technical concept, other embodiments of this application provide an interpretable face forgery detection method based on early semantic injection, such as... Figure 2 As shown, it includes the following steps: S1, using the trained visual encoder based on the early semantic injection method and the trained learnable cue vector, the face image to be detected is segmented and serialized, and then concatenated with the trained learnable cue vector and input into the visual encoder to extract pre-aligned visual features. S2, after mapping the pre-aligned visual features through a pre-trained projector, concatenates them with the query text and inputs them into a large language model that integrates a pre-optimized low-rank adaptive module; S3 uses a large language model to generate a response sequence and obtain the output results, which include the face authenticity determination results and the corresponding natural language interpretation report.

[0049] The above embodiments enhance model credibility and meet the application requirements of high-reputation scenarios. Traditional black-box detectors only output true / false probability scores, lacking traceable visual evidence. The above embodiments, by combining a multimodal model, construct an interpretable face forgery explanation model, ultimately outputting a detailed detection report containing specific forgery locations and natural language descriptions. This natural language-level interpretability allows this application to directly assist human reviewers in cross-validation, greatly expanding its application prospects in high-reputation security systems such as forensic identification, news verification, and financial risk control.

[0050] The above embodiments overcome the limitations of late-stage interpretive injection and eliminate the modal gap between forgery detection and language models. This application proposes an early semantic injection mechanism based on forged text-image contrast loss, which forces interpretable forgery features to be pre-aligned with text truth embeddings during the visual encoding stage. This design aligns semantic information with visual features before they enter the large language model, greatly reducing the reasoning burden of the language model, effectively avoiding the illusion that artifacts do not exist, and ensuring the authenticity and reliability of the detected evidence.

[0051] The above embodiments achieve multi-dimensional feature decoupling, ensuring detection accuracy while providing interpretation. Traditional methods often confuse classification signals with interpretation signals. This application successfully achieves decoupled feature extraction in a frozen visual encoder by introducing three dedicated learnable cue vectors for classification, general, and interpretation perception. This method guarantees the accuracy of the model's binary classification detection of forgeries.

[0052] To achieve efficient parameter fine-tuning while preserving the original common-sense knowledge of the large language model, in some specific embodiments of this application, S2, the pre-aligned visual features are mapped by a pre-trained projector and concatenated with the query text, then input into the large language model integrated with a pre-optimized low-rank adaptive module. This can be achieved by the following steps: S21, Keep the original pre-trained weight matrix of the large language model fixed and prohibit gradient updates; S22 bypasses only the query projection layer and key-value projection layer of the self-attention module in the large language model. Add trainable dimensionality reduction matrix and the increasing dimension matrix Set the rank parameter r and the scaling factor α.

[0053] S23, during forward propagation, the output of the projected layer after injecting the low-rank matrix is ​​represented as the sum of the original weight output and the low-rank branch output, expressed by the formula:

[0054] Where x represents the projection layer input and h represents the projection layer output.

[0055] The above embodiments require training only a very small number of low-rank matrix parameters, significantly reducing memory usage and training costs, and achieving rapid adaptation. Freezing the backbone parameters effectively prevents the model from forgetting during fine-tuning, preserving the general language common sense and reasoning ability of the pre-trained model. Through the low-rank adaptation layer, the general large model can quickly adapt to specific multimodal tasks of face forgery detection and interpretation, improving the professionalism and accuracy of the generated interpretation reports.

[0056] To achieve efficient detection and interpretable output of unknown face images, in some specific embodiments of this application, S3, a response sequence is generated using a large language model to obtain the output result; wherein, the output result includes the face authenticity determination result and the corresponding natural language interpretation report, including the following steps: S31, input multimodal sequence The input is fed into a large language model that integrates a low-rank adaptive module, and the model generates a word-by-word response sequence in an autoregressive manner. S32, the face authenticity determination conclusion and the corresponding natural language explanation text are parsed from the response sequence as the final detection result.

[0057] The above embodiments employ a clear inference chain and modular design, and rely on a large model to achieve human-understandable natural language explanations to address the problem of face forgery detection.

[0058] To achieve accurate understanding of multimodal input sequences and generate natural language interpretations that conform to semantic logic, some specific embodiments of this application employ the following steps to jointly optimize the projector and low-rank adaptive module parameters of a large language model through language modeling loss: First, obtain the deepfake visual question answering dataset.

[0059] The specific dataset includes face images, query text, and corresponding real natural language interpretation labels; Secondly, based on a deepfake visual question-answering dataset, the standard language modeling loss between the generated response sequence and the real natural language interpretation labels is calculated. :

[0060] Where T is the total length of the target interpretation text sequence. To explain the standard target word unit for truth value at time step i, This represents all historical context word sequences generated by the model up to the i-th time step. This represents the conditional probability distribution of the large language model's output of a specific lexical unit in the current step. This represents a multimodal input sequence.

[0061] Finally, based on the standard language modeling loss, the parameters of the projector and the low-rank adaptive module are jointly updated using gradients.

[0062] The above embodiments utilize a large language model as the core inference engine, transforming language interpretations into probability sequences to train the model's language output. Simultaneously, by optimizing the LoRA adapter instead of all parameters, the model can efficiently learn the descriptive specifications for the specific vertical domain of face forgery interpretation while retaining the powerful general-purpose language capabilities of the LLM.

[0063] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0064] This embodiment is an interpretable face forgery detection method based on early semantic injection. It implements the above steps S1-S3 and jointly optimizes the projector and low-rank adaptive module parameters of the large language model through language modeling loss. Step S1 adopts the above steps S100-S600.

[0065] Specifically, in step S100, the face image and its corresponding true / false labels and explanatory text are obtained as the training set. The specific process is as follows: High-quality face forgery detection datasets were collected and prepared. The datasets used in this embodiment for training are FaceForenisics++ (see A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner, “Faceforensics++: Learning to detect manipulated facial images,” in Proceedings of the IEEE / CVF international conference on computer vision, 2019, pp. 1–11) and DD-VQA (see Y. Zhang, B. Colman, X. Guo, A. Shahriyari, and G. Bharaj, “Common sense reasoning for deepfake detection,” in European conference on computer vision. Springer, 2024, pp. 399–415.). The FaceForenisics++ dataset contains images generated using various forgery methods, such as DeepFakes, Face2Face, FaceSwap, and NeuralTexture. The DD-VQA dataset is based on FF++ image data, with manually annotated question-answer pairs related to face forgery.

[0066] Specifically, the S300's visual encoder is constructed in the following manner: Following the structure of CLIP (see A. Radford, JW Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763.), a Transformer model for feature extraction is constructed. The model's parameters and design are consistent with the CLIP-large model.

[0067] Specifically, in S2, the Vicuna-7b model is selected as the base language model. To achieve efficient parameter fine-tuning while preserving the original common-sense knowledge of the large language model, all pre-trained parameters of the large language model are frozen. A trainable low-rank parameter matrix is ​​injected as a bypass into the query projection layer and key-value projection layer of the self-attention module, and the set rank parameter and scaling factor are configured. This embodiment sets the rank parameter... Scaling factor

[0068] Specifically, in the joint optimization of the projector and low-rank adaptive module parameters of the large language model, the standard language modeling loss between the generated response sequence and the real natural language interpretation label is calculated on the constructed deep fake visual question answering dataset DD-VQA. .

[0069] After performing all steps S1-S3, proceed to S4 to test and evaluate the model's ability to interpret and identify face forgeries. The specific steps are as follows: S41, Evaluation Test Set Setup. To further evaluate the model's generalization ability, in addition to the DD-VQA test set, three large benchmark datasets were used: Celeb-DF (CDF), Deepfake Detection Challenge Public (DFDC), and Deepfake Detection Challenge Preview (DFDCP) as test sets. These datasets contain a variety of fake images based on Generative Adversarial Networks (GANs), and these images come from different sources, exhibiting broad diversity.

[0070] S42: Comprehensive Evaluation of Face Forgery Reasoning System. For a complete face forgery detection system incorporating a large language model, a comprehensive evaluation is conducted primarily from two dimensions: the reasonableness of the interpretation and the accuracy of the judgment.

[0071] Reasonableness assessment of explanations: The forgery explanations generated by the large language model are extracted and compared with manually annotated ground truth values ​​using standard natural language generation evaluation metrics (including SPICE, BLEU-4, CIDEr, and ROUGE-L) for similarity and semantic matching. Furthermore, to visually demonstrate the credibility of the model-generated explanations, this application includes a qualitative case comparison analysis (such as...). Figure 3As shown in the figure, the comparison between the input image, ground truth (GT) annotations, baseline model (LLaVA), and the generated results of our method (Ours) is illustrated. It can be seen that the baseline model exhibits obvious hallucination phenomena (such as misidentifying asymmetrical eyebrows), while the interpretation generated by our method is highly consistent with the ground truth annotations, accurately pointing out forgery traces (such as abnormal skin texture). This dimension focuses on verifying the effectiveness of the early semantic injection mechanism in aligning visual features with linguistic concepts, eliminating the "hallucination" phenomenon of model fabrication artifacts, and providing reliable visual evidence.

[0072] Accuracy Assessment: The true / false judgments from the output text of the large language model are extracted, and the accuracy (Acc) of the true / false binary classification is used as the core evaluation criterion. This assessment aims to verify that the model provides fine-grained semantic interpretation without compromising end-to-end discrimination accuracy.

[0073] Table 1 presents the quantitative evaluation comparison results of this application and existing mainstream interpretable face forgery detection systems on the DD-VQA dataset. The system proposed in this invention achieves the best performance in all interpretation quality metrics. In particular, on the high-dimensional metric CIDEr, which measures the consensus between the generated text and human annotations, this invention scores as high as 3.82, achieving a significant improvement over the suboptimal method M2F2-Det (score 2.87). This fully demonstrates that the early semantic injection mechanism adopted in this application can effectively eliminate the "illusion" phenomenon of large language models and generate reliable explanations that highly match real visual artifacts. In terms of detection performance, this invention also achieves an accuracy of 0.926 and an F1-score of 0.943, proving that this method significantly improves interpretability while maintaining extremely high accuracy in face forgery identification.

[0074] Table 1. Evaluation of Explainable Face Forgery Detection Systems

[0075] S43: To verify the enhancement effect of the early interpretation injection mechanism proposed in this application on the ability to identify low-level visual features, the facial feature visual extraction network trained in the first stage was tested separately. Evaluation was conducted. The predicted forgery probability output by the dedicated forgery classifier in the feature extraction network was extracted. The area under the receiver operating feature curve (AUC) was calculated as a quantification metric on multiple cross-domain deep forgery detection benchmark datasets. As shown in Table 2, even with parameter freezing and without dependence on downstream language models, the visual extractor optimized through multidimensional learnable cues and contrastive learning (pre-alignment) still extracts highly discriminative and cross-domain generalization visual representations.

[0076]

[0077] Table 2 Evaluation of the facial feature visual extraction network Based on the same technical concept, in other embodiments of this application, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the above-described visual encoder training method and face forgery detection and interpretation method.

[0078] Based on the same technical concept, in other embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, can be used to perform the above-described visual encoder training method and face forgery detection and interpretation method.

[0079] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0080] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0081] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0082] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0083] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0084] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A visual encoder training method based on early semantic injection, characterized in that, include: Obtain a face image and its corresponding true / false labels and explanatory text, and serialize the face image into blocks to obtain an image feature sequence; Initialize a learnable cue vector that includes classification, general forgery, and interpretation-aware cue vectors, and concatenate it with the image feature sequence to form a composite input sequence; The composite input sequence is input into a visual encoder to extract classification features, general pattern features, and interpretive features; The semantic embeddings corresponding to the true and false tags and the explanatory text are extracted using a frozen text encoder; Construct a comprehensive loss function, which includes a classification loss based on the classification features and a contrastive loss that constrains the similarity between the general pattern features, the interpretive features, and the semantic embedding; The learnable cue vector is updated based on the comprehensive loss function to obtain a visual encoder configured with the trained learnable cue vector.

2. The visual encoder training method based on early semantic injection according to claim 1, characterized in that, The step of extracting the semantic embeddings corresponding to the true / false tags and the explanatory text using a frozen text encoder includes: Construct corresponding prompt text based on the true / false labels, and input it into the text encoder to obtain the classification semantic embedding; The explanatory text is input into the text encoder to obtain the explanatory semantic embedding.

3. The visual encoder training method based on early semantic injection according to claim 2, characterized in that, The comparative loss constraining the similarity between the general pattern features, the interpretive features, and the semantic embedding includes: Calculate the first cosine similarity between the general pattern features and the classification semantic embedding; The second cosine similarity between the explanatory features and the explanatory semantic embedding; Calculate the first difference between 1 and the first cosine similarity; Calculate the second difference between 1 and the second cosine similarity; The contrast loss is obtained by summing the product of the first difference and the preset first weight coefficient, and the product of the second difference and the preset second weight coefficient.

4. The visual encoder training method based on early semantic injection according to claim 1, characterized in that, The classification loss based on the classification features includes: The classification features, general pattern features, and explanatory features are input into the classifier to obtain the true / false prediction results; The difference between the true / false prediction result and the true / false label is calculated to obtain the classification loss.

5. An interpretable face forgery detection method based on early semantic injection, characterized in that, include: Using the visual encoder trained by the method described in any one of claims 1-4 and the trained learnable cue vector, the face image to be detected is segmented and serialized, and then concatenated with the trained learnable cue vector and input into the visual encoder to extract pre-aligned visual features. The pre-aligned visual features are mapped by a pre-trained projector and concatenated with the query text, then input into a large language model that integrates a pre-optimized low-rank adaptive module. The large language model is used to generate a response sequence to obtain the output results; wherein, the output results include the face authenticity determination results and the corresponding natural language interpretation report.

6. The interpretable face forgery detection method based on early semantic injection according to claim 5, characterized in that, The large language model integrating a pre-optimized low-rank adaptive module specifically involves inserting a trainable low-rank decomposition matrix only into the query and key-value projection layers of the self-attention module of the large language model.

7. The interpretable face forgery detection method based on early semantic injection according to claim 5, characterized in that, The projector is used to connect visual features with the language space, mapping the dimensions of the pre-aligned visual features to the language embedding space dimension of the large language model.

8. The interpretable face forgery detection method based on early semantic injection according to claim 5, characterized in that, The projector and the low-rank adaptive module are jointly optimized, including: Based on a deepfake visual question answering dataset, we calculate the standard language modeling loss between the generated response sequence and the real natural language interpretation label; Based on the standard language modeling loss, the parameters of the projector and the low-rank adaptive module are jointly updated using gradients.

9. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, this program can be used to perform the method of any one of claims 1-8.

Images