A text-guided face spoofing detection method and system

Abstract

The application provides a text-guided face forgery detection method and system, which comprises obtaining a face image to be detected for authenticity, inputting the face image to be detected for authenticity into a face detection model, and obtaining a face authenticity detection result; the face detection model comprises: constructing a text prompt library covering multiple granularities to generate multi-dimensional text prototypes; extracting visual features, optimizing the feature distribution of the visual features, and obtaining global visual features; performing feature separation and enhancement on the global visual features; mapping the global visual features after the feature separation and enhancement into predicted text features; performing similarity constraint to obtain a cross-modal prototype matching result; performing distinctiveness constraint on different-dimensional text prototypes to obtain a feature metric learning result; and outputting a detection result based on the cross-modal prototype matching result and the feature metric learning result. Through the method, the problem of poor generalization of face forgery detection is solved, and the detection precision and generalization ability are improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a text-guided method and system for detecting face forgery. Background Technology

[0002] With the rapid development of generative AI technologies such as generative adversarial networks and diffusion models, the technical threshold for face spoofing has continued to decrease. The realism of spoofed content generated by means of identity replacement, attribute editing, and full-face synthesis has been greatly improved. It has been maliciously used for illegal activities such as spreading fake news and defamation, which seriously threatens personal rights and public safety.

[0003] Existing methods are mostly based on deep learning models, which realize the authenticity of a product by mining features such as spatial domain texture, edge and frequency domain spectrum anomaly. The mainstream frameworks include convolutional neural networks, Transformer and pre-trained large visual models, which can be divided into single-modal visual detection and multimodal fusion detection. Video-level detection also introduces temporal consistency constraints.

[0004] However, existing methods still have many shortcomings: First, they mostly focus on binary classification of true and false, lacking fine-grained attribute mining of forgery type and generator type, and cannot achieve forgery source tracing; second, single-modal methods lack high-level semantic guidance, resulting in low detection accuracy for forgery of local attribute manipulation type; and third, they have weak feature differentiation ability for different generators and forgery types, and lack cross-scenario generalization and robustness when facing new forgery methods. Summary of the Invention

[0005] This application provides a text-guided face forgery detection method and system to address the problem of poor cross-scene generalization of existing methods when facing new forgery techniques, and to improve detection accuracy, robustness and cross-dataset generalization ability.

[0006] Firstly, this application provides a text-guided method for detecting face forgery, which is executed by a computing device. The computing device can be understood as a computer or similar device, and is not limited thereto in this application. The method includes:

[0007] The process involves obtaining a face image to be tested for authenticity, inputting it into a face detection model, and obtaining the face authenticity detection result. The face detection model's workflow includes: constructing a multi-granularity text prompt lexicon to generate multi-dimensional text prototypes; extracting visual features from the input face image using a homologous visual encoder, and optimizing the feature distribution of these visual features through a visual adaptation module to obtain global visual features; performing feature separation and enhancement on the global visual features through a grouping type extraction module; mapping the feature-separated and enhanced global visual features to predicted text features using a text predictor; applying similarity constraints to the multi-dimensional text prototypes and predicted text features through a multi-task learning module to obtain cross-modal prototype matching results; applying discriminative constraints to different dimensions of the multi-dimensional text prototypes to obtain feature measurement learning results; and performing face authenticity detection based on the cross-modal prototype matching results and feature measurement learning results, outputting the face authenticity detection result.

[0008] Through the above methods, this application obtains face authenticity detection results based on a face detection model. The face detection model first generates text prototypes covering multiple granularities, constructing a multi-dimensional, fine-grained set of text prototypes to provide a unified benchmark for cross-modal forgery feature matching. Subsequently, it extracts multi-dimensional visual features of the face image and maps them to the text feature space to achieve cross-modal alignment. Based on the text prototypes in the input text prototype library and the predicted text features output by the text predictor, similarity constraints are applied between the two. For text prototypes of different dimensions, internal discriminative constraints are applied to provide loss signals for subsequent model parameter optimization. Based on the similarity distance between the global text prediction output by the text predictor and the true and false global text prototypes in the text prototype library, the authenticity probability of the input image is calculated and predicted, improving detection accuracy, robustness, and cross-dataset generalization ability. It can be applied to identity verification and forgery tracing in fields such as social media, finance, and the judiciary.

[0009] In the aforementioned text-guided face forgery detection method, a multi-granular text prompt lexicon is constructed to generate multi-dimensional text prototypes, including:

[0010] Obtain multi-dimensional fine-grained text prompts with semantics related to face forgery detection from a multi-granularity text prompt lexicon; wherein, the multi-granularity text prompt lexicon includes dimensions of authenticity attributes, forgery type, generator type, and facial component; convert the multi-dimensional fine-grained text prompts provided by the multi-granularity text prompt lexicon into text feature vectors of fixed size; construct a multi-dimensional text prototype set based on the text feature vectors.

[0011] Through the above methods, this application constructs a multi-granular text prompt lexicon covering authenticity attributes, forgery types, generator types, and facial components from the perspective of visual-language cross-modal modeling, and generates multi-dimensional text prototypes as cross-modal matching benchmarks.

[0012] In the aforementioned text-guided face forgery detection method, visual features are extracted through a homologous visual encoder, and the feature distribution of the visual features is optimized through a visual adaptation module to obtain global visual features, specifically including:

[0013] Acquire a face image and locate facial key points using a 68 keypoint detector; extract and output fixed-size general visual features based on the face image using a visual encoder; adapt the general visual features to the feature space of the face forgery detection task using a visual adaptation module for feature learning; perform residual connections on the general visual features after feature learning to preserve the original visual semantics and output the adapted global visual features.

[0014] Through the above method, this application extracts image features by using a homologous visual encoder, optimizes the feature distribution by a visual adaptation module, and transforms fixed-size general visual features into global visual features that are adapted to the whole world, thereby increasing the application scenarios of face recognition and improving recognition accuracy.

[0015] In the aforementioned text-guided face forgery detection method, the global visual features are separated and enhanced through a grouping type extraction module. This includes: grouping the input global visual features into channels according to forgery type and generator type, applying spatial attention weighting to each group, strengthening the fine-grained feature expression of different dimensions, and obtaining the type visual features corresponding to the group.

[0016] In the above manner, this application groups the input global visual features by channel according to forgery type and generator type, applies spatial attention weighting to each group, strengthens the fine-grained feature expression of different dimensions, obtains the type visual features corresponding to the group, and combines the group type extraction module to realize feature separation and enhancement of different forgery types, strengthens the fine-grained features of different dimensions, and improves classification accuracy.

[0017] In the aforementioned text-guided face forgery detection method, a text predictor maps the global visual features after feature separation and enhancement to predicted text features, including: mapping the input global visual features, component-level visual features, and type visual features to text features; wherein, global visual features are mapped to global text prediction, component-level visual features are mapped to component-level text prediction, and type visual features are mapped to type text prediction.

[0018] In this way, the application maps the input global visual features, component-level visual features, and type visual features to text features; the global visual features are mapped to global text prediction; the component-level visual features are mapped to component-level text prediction; and the type visual features are mapped to type text prediction. By using a text predictor to complete the feature mapping from vision to text, and with the help of general semantic understanding capabilities, it can accurately capture the semantic information of prompts at different granularities, so that the output text features have a good cross-modal alignment foundation, providing text support for the subsequent construction of text prototypes.

[0019] The aforementioned text-guided face forgery detection method further includes: training a face detection model; the model training method includes: obtaining a training set, which includes real and face images generated by existing forgery techniques, and corresponding labels; wherein the label value is 0 or 1, where 0 represents a real face and 1 represents a forged face; inputting the real and face images generated by existing forgery techniques into the pre-built detection model to obtain the model prediction result; calculating the total loss function based on the model prediction result and the corresponding label; iteratively updating and training the detection model based on the Adam optimization algorithm until the detection model loss function is minimized, thus completing the model training.

[0020] In the above manner, this application trains a face recognition model. During the model training process, the text prototype serves as a high-level semantic prior. Through the joint guidance of cross-modal alignment constraints of the multi-task learning module and strong supervision signals in three dimensions (global, component, and type), the various channels of this module are driven to gradually converge to the corresponding type's exclusive feature extraction mode. Ultimately, this enables the module to selectively mine fine-grained feature differences between different forgery types and generator types.

[0021] In a second aspect, this application provides a text-guided face forgery detection system for performing the method in the first aspect of this application. The system includes: a detection result acquisition module, a text prototype generation module, a visual-text prediction module, a multi-task learning module, and a classification module.

[0022] The detection result acquisition module is used to obtain the face image to be detected (whether it is real or fake), input the face image to be detected (whether it is real or fake) into the face detection model, and obtain the face authenticity detection result. The face detection model includes: a text prototype generation module, used to construct a text prompt lexicon covering multiple granularities and generate multi-dimensional text prototypes; a visual-text prediction module, used to extract visual features from the input face image through a homologous visual encoder, and to optimize the feature distribution of the visual features through a visual adaptation module to obtain global visual features; the visual-text prediction module is also used to extract global visual features through a grouping type extraction module. The system performs feature separation and enhancement on visual features; the visual-text prediction module is also used to map the global visual features after feature separation and enhancement to predicted text features through a text predictor; the multi-task learning module is used to impose similarity constraints on multi-dimensional text prototypes and predicted text features to obtain cross-modal prototype matching results; the multi-task learning module is also used to impose discriminative constraints on different dimensions of text prototypes in multi-dimensional text prototypes to obtain feature measurement learning results; the classification module is used to perform face authenticity detection based on cross-modal prototype matching results and feature measurement learning results, and output face authenticity detection results.

[0023] Thirdly, this application also provides a computing device, comprising: a memory for storing program instructions; and a processor for calling the program instructions stored in the memory and executing the method described in the first aspect according to the obtained program instructions.

[0024] Fourthly, this application also provides a computer-readable storage medium storing computer-readable instructions, which, when read and executed by a computer, implement the method of the first aspect described above.

[0025] Fifthly, this application provides a computer program product including a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the method described in the first aspect.

[0026] Beneficial Effects: The text-guided multi-task face forgery detection method proposed in this application aims to construct a multi-dimensional, fine-grained text prototype covering authenticity attributes, forgery type, generator type, and facial components from the perspective of visual-language cross-modal modeling, combining the ideas of prototype learning and metric learning. This allows text semantics to guide the in-depth mining of visual forgery features. The visual feature distribution is optimized through a visual adaptation module, and feature separation and semantic enhancement for different forgery types and generator types are achieved by combining a grouping type extraction module. Furthermore, a multi-task learning module containing three type-dimensional supervised losses is used to simultaneously complete cross-modal prototype similarity matching and multi-dimensional feature internal metric learning, maximizing the discriminative power of different types of forgery features in the feature space, and effectively improving the model's ability to identify different forgery types, cross-scene generalization, and robustness. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A flowchart illustrating the first text-guided face forgery detection method provided in Embodiment 1 of this application;

[0029] Figure 2 A flowchart illustrating the second text-guided face forgery detection method provided in Embodiment 2 of this application;

[0030] Figure 3 This is a schematic diagram of a text prototype generation module for a text-guided face forgery detection method provided in Embodiment 2 of this application;

[0031] Figure 4 This is a schematic diagram of the visual-text prediction module of a text-guided face forgery detection method provided in Embodiment 2 of this application;

[0032] Figure 5 This is a schematic diagram of the structure of a computing device provided in Embodiment 4 of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0034] In the following embodiments of the present application, "and / or" describes the association relationship of associated objects, which means that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone, wherein A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects are in an "or" relationship. The expression "at least one of the following" or similar expressions refers to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b or c can mean: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b and c can be single or plural. The singular expressions "a", "an", "the", "the aforesaid", "said" and "this" are intended to also include the expression like "one or more" unless explicitly indicated otherwise in the context. In addition, unless otherwise stated, the ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the order, timing, priority or importance of multiple objects.

[0035] References to "one embodiment" or "some embodiments" described in the specification of the present application mean that one or more embodiments of the present application include the specific features, structures or characteristics described in combination with the embodiments. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in additional embodiments" and the like appearing in different parts of this specification do not necessarily all refer to the same embodiment, but instead mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "comprise", "include", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0036] Example 1

[0037] Example 1 of the present application provides a text-guided face forgery detection method, which is executed by a computing device. The computing device can be understood as a computer and other devices, which is not limited herein. The overall flow of the method is as follows Figure 1 shown, comprising:

[0038] Step 101, obtaining a face image to be detected for authenticity.

[0039] Step 102, inputting the face image to be detected for authenticity into a face detection model.

[0040] Wherein, the flow executed by the face detection model comprises:

[0041] Step 103, constructing a multi-granularity covering text prompt lexicon, and generating multi-dimensional text prototypes.

[0042] Specifically, the process involves acquiring multi-dimensional fine-grained text prompts with semantics related to face forgery detection from a multi-granularity text prompt lexicon. The multi-granularity text prompt lexicon includes dimensions of authenticity attributes, forgery type, generator type, and facial components. The multi-dimensional fine-grained text prompts provided by the multi-granularity text prompt lexicon are then converted into fixed-size text feature vectors. A multi-dimensional text prototype set is constructed based on the text feature vectors.

[0043] In one possible implementation, this step is performed by a detection module, and the detection model includes:

[0044] The text prototype generation branch is used to generate text prototypes covering multiple granularities, providing a unified benchmark for cross-modal forgery feature matching. It includes the following three sub-modules: a multi-granularity text prompt lexicon, which provides multi-dimensional fine-grained text prompts with semantics related to face forgery detection, providing basic input for the subsequent construction of text prototypes; a text encoder, which converts the input multi-granularity text prompts into fixed-size text feature vectors; and a text prototype library, which constructs a multi-dimensional, fine-grained set of text prototypes based on the input text feature vectors.

[0045] The multi-granularity text prompt library has four core dimensions: First, the global dimension, also known as the authenticity attribute dimension, includes the two basic prompt words "real" and "fake"; second, the facial component dimension, encompassing prompt words corresponding to key facial components such as "eyes," "nose," and "lips"; the facial component dimension prompt words are combined with the global dimension prompt words to achieve further refinement, such as "real eyes"; third, the forgery type dimension, covering prompt words corresponding to mainstream forgery types such as "identity replacement," "attribute editing," and "full face replacement"; and fourth, the generator type dimension, including prompt words corresponding to generator types such as "generative adversarial network" and "diffusion model."

[0046] The text encoder uses existing pre-trained visual-language large models, such as CLIP's text encoder. The pre-trained visual-language large models, with the general semantic understanding capabilities learned during their pre-training process, can accurately capture the semantic information of cue words at different granularities, enabling the output text features to have a good cross-modal alignment foundation and providing text support for the construction of subsequent text prototypes.

[0047] The text prototype library is built during model initialization and remains fixed during training, serving as a constant benchmark for cross-modal matching. It categorizes and aggregates the text features output by the text encoder according to four preset dimensions, forming a dedicated text prototype for each dimension. (Text prototypes for each dimension are shown below.) ,Include:

[0048] Global Dimension: Real Global Text Prototype Fake global text prototype .

[0049] Facial component dimension: Real component-level text prototype Includes a real eye text prototype Real nose text prototype and real lip text prototype etc.; forging component-level text prototypes Contains a text prototype of fake eyes Fake nose text prototype and fake lip text prototype wait.

[0050] Forgery type dimension: Forgery type text prototype Includes identity replacement text prototype Attribute editing text prototype Full face replacement text prototype .

[0051] Generator Dimension: Generator Text Prototype Includes a prototype of generative adversarial network text. diffusion model text prototype .

[0052] Among them, the forgery type dimension and the generator dimension are both fine-grained attribute dimensions of forged faces, belonging to the category of global forgery attributes.

[0053] Step 104: Extract visual features from the input face image using a homologous visual encoder, and optimize the feature distribution of the visual features using a visual adaptation module to obtain global visual features.

[0054] In one possible implementation, a face image is acquired, and facial key points are located in the face image using a 68 keypoint detector; a visual encoder extracts and outputs a fixed-size general visual feature based on the face image; a visual adaptation module adapts the general visual feature to the feature space of the face forgery detection task for feature learning; the general visual feature after feature learning is residually connected to retain the original visual semantics and output the adapted global visual feature.

[0055] The 68 keypoint detector takes a face image as input and uses existing pre-trained facial keypoint detection models, such as the Dlib face feature point detector. The feature point detector can accurately identify and output a 68 keypoint map containing the coordinate information of 68 standard key feature points in the face image. The coordinate information can be used to further segment facial components such as eyes, nose, and lips.

[0056] Furthermore, the visual encoder employs a pre-trained visual-language large model derived from the same source as the text encoder, such as CLIP's visual encoder. The pre-trained visual-language large model, relying on its pre-trained general feature extraction capabilities, can accurately capture the visual features of the input image, providing visual support for subsequent text feature prediction.

[0057] The visual adaptation module takes the general visual features obtained from the visual encoder as input. The general visual features are transformed for task adaptation by passing through two 3×3 depthwise separable convolutional layers, a normalization layer, and a GELU activation layer. Finally, the global visual features are obtained through residual connections. Global visual features are mapped to coordinate information to obtain component-level visual features; the size of the component-level visual features is the same as that of the global visual features, and they include eye visual features. Nose visual characteristics and visual features of the lips wait.

[0058] Step 105: Perform feature separation and enhancement on global visual features through the grouping type extraction module.

[0059] Specifically, the input global visual features are grouped by channel according to forgery type and generator type, and spatial attention weighting is applied to each group to enhance the fine-grained feature expression of different dimensions, thereby obtaining the type visual features corresponding to the group.

[0060] The grouping type extraction module consists of three sub-layers: a type channel grouping sub-layer, a type feature enhancement sub-layer, and a global type feature integration layer. At the beginning of training, this module only has the ability to extract general features with random initialization and has not yet formed the ability to represent specific features for different forgery types and generator types. However, during the model training process, the text prototype, as a high-level semantic prior, is driven by the cross-modal alignment constraints of the multi-task learning module and the joint guidance of strong supervision signals in the three dimensions of global, component, and type. This drives the grouping of each channel in this module to gradually converge to the specific feature extraction mode of the corresponding type, and finally has the ability to specifically mine the fine-grained feature differences of different forgery types and generator types.

[0061] The type channel grouping sublayer divides the global visual features into 5 independent channel groups according to type attributes. The feature dimension is reshaped from [previous layer] to [new layer], while the batch dimension remains unchanged. Only the channel dimension is split, and then the features of each group are extracted separately. Each group focuses on the feature expression of a single type, ensuring that different types of features do not interfere with each other. The 5 independent channel groups are: Forgery type: Identity replacement group, Forgery type: Attribute manipulation group, Forgery type: Fully synthesized group, Generator type: Generative Adversarial Network group, and Generator type: Diffusion model group.

[0062] The type feature enhancement sublayer performs spatial enhancement based on global feature semantics for each group of type features. The type features are first passed through a group normalization layer and a global average pooling layer to obtain initial features. Then, the dot product similarity between the feature vector at each spatial location within the group and the global feature vector of the group is calculated to generate a spatial attention map, which locates the spatial regions related to the type attribute in the global range. The spatial attention map is mapped to the [0,1] interval through the Sigmoid activation function and used as a spatial weight to modulate the initial features of the group position by position, amplifying the features of the type-related regions and suppressing irrelevant noise to obtain enhanced features. Finally, the enhanced features are residually connected with the initial features of the group.

[0063] The global type feature integration layer first concatenates the five enhanced type features by channel dimension. Since there is a natural cross-relationship between the forged type and the generator type, the same forged type can be implemented by multiple generator architectures, and the same generator can also generate forged content with different attribute types. Therefore, a 1×1 convolutional layer is used to achieve feature interaction fusion and dimension unification between different channel groups, outputting type visual features with the same size as the original global visual features. .

[0064] Step 106: Using a text predictor, the global visual features after feature separation and enhancement are mapped to predicted text features.

[0065] Specifically, the input global visual features, component-level visual features, and type visual features are mapped to text features; among them, global visual features are mapped to global text prediction, component-level visual features are mapped to component-level text prediction, and type visual features are mapped to type text prediction.

[0066] In one possible implementation, the text predictor employs a three-branch parallel architecture, corresponding to text space mappings of global, component-level, and type visual features, respectively. Each branch consists of two fully connected layers, a GELU activation layer, and a normalization layer concatenated, with an output dimension of [missing information]. It remains consistent with the text prototype.

[0067] Global text prediction branch: Input global visual features First, the size is compressed using global average pooling. Then, it is mapped to global text prediction via MLP. , used to match true and false global text prototypes.

[0068] Component-level text prediction branch: Input component-level visual features such as eyes, nose, and lips respectively. After global average pooling, each component is mapped to its corresponding component-level text prediction through its own independent MLP mapping. , used to match true and false component-level text prototypes.

[0069] Type-based text prediction branch: Input type visual features First, divide it according to the channel dimension into sizes corresponding to the 5 type groups. The sub-features are then mapped to five types of text prediction using independent MLP mappings. This is used to match the text prototype of the corresponding forgery type and generator type.

[0070] Step 107: Through the multi-task learning module, similarity constraints are applied based on multi-dimensional text prototypes and predicted text features to obtain cross-modal prototype matching results.

[0071] Step 108: Through the multi-task learning module, discriminative constraints are applied to the text prototypes of different dimensions in the multi-dimensional text prototype to obtain the feature measurement learning results.

[0072] The multi-task learning module performs both prototype learning (matching the similarity of text prototypes generated by the text prototype generation branch with text prediction features obtained by the visual-text prediction branch) and metric learning (keeping global and component-level features of true / false, as well as different types of features within the dimensions of fake type and generator type, as far apart as possible in the feature space). It utilizes a strongly supervised learning mechanism. When the model is input with samples containing complete annotation information of global true / false, facial components, fake type, and generator type during the training phase, it forces each channel group to converge to the corresponding type's exclusive feature extraction mode through joint strong supervision constraints of cosine similarity loss, contrast loss, and KL divergence loss. This ensures that the feature expressions of different type attributes do not interfere with each other and have strong discriminative power.

[0073] Cosine similarity loss To constrain the maximization of the similarity between predicted text features and their corresponding text prototypes, it is defined as follows:

[0074] ;

[0075] ;

[0076] ;

[0077] ;

[0078] in, This represents the global cosine similarity loss; Batch size; Sample label Where "0" represents a real face and "1" represents a fake face; The cosine similarity function is defined as follows: Where u and v are intermediate variables, as shown in the formula above:

[0079]

[0080] ;

[0081] ;

[0082] ;

[0083] ;

[0084] in: The number of facial components divided; Let i represent the predicted text features of the i-th global text. This indicates the forgery of the global text prototype. Represents the true global text prototype; This represents the local cosine similarity loss. This represents a predefined local region, where eye represents the eye and nose represents the nose; Represents the type cosine similarity loss. The text represents the type dimension: is for identity replacement text, am for attribute editing text, es for full-face replacement text, gan for generative adversarial network text, and ddpm for diffusion model text; type labels. Indicates the first Does each sample belong to type? "0" indicates no, and "1" indicates yes;

[0085] For component 1 Text prediction, Components The real text prototype and the fake text prototype For type The Text prediction, For type The text prototype.

[0086] Comparative loss The InfoNCE loss is used to maximize the similarity of features of the same class and minimize the distance of features of different classes, as defined below:

[0087] ;

[0088] ;

[0089] ;

[0090] ;

[0091] in: Global comparison loss, This indicates local contrast loss. Indicates type-contrast loss; Temperature coefficient; It is a natural exponential function; For batch and sample Positive samples with the same label This refers to all samples within the batch; Let i represent the predicted text features of the i-th global text. This represents the predicted text features of the j-th global text. This represents the predicted text feature of the k-th global text; For component 1 Text prediction, For component 1 Text prediction, For type The Text prediction, For type The Text prediction.

[0092] Furthermore, KL divergence loss To constrain the consistency between the predicted text feature distribution and the text prototype distribution, and to improve the effectiveness of cross-modal matching, the feature vector needs to be normalized to a probability distribution using softmax before calculation, as defined below:

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] in, Represents the global divergence loss. This represents the local divergence loss. The KL divergence function represents the type divergence loss. Let p represent the target distribution, q represent the model distribution, and x represent the random variable. It refers to the lips.

[0098] Step 109: Based on the cross-modal prototype matching results and feature metric learning results, perform face authenticity detection and output the face authenticity detection results.

[0099] The classifier is used to perform face authenticity detection based on cross-modal prototype matching results and feature measurement learning results. The classifier consists of a similarity calculation sub-layer and a probability mapping sub-layer connected in series.

[0100] The similarity calculation sublayer calculates the cosine similarity between the input global text prediction and the real global text prototype and the forged global text prototype, respectively, to obtain a similarity vector:

[0101] ;

[0102] The probability mapping sublayer uses the Softmax function to transform the similarity vector... Mapped to probability distribution ,in: This represents the probability that the predicted face is a real human face. Let represent the probability of predicting a fake face, and satisfy . .

[0103] In one possible implementation, a face detection model is trained; the training method includes:

[0104] Obtain a training set, which includes real and face images generated by existing forgery techniques, along with their corresponding labels; where the label value is 0 or 1, where 0 represents a real face and 1 represents a forged face.

[0105] Real and face images generated using existing forgery techniques are input into a pre-built detection model to obtain the model's prediction results.

[0106] Calculate the total loss function based on the model prediction results and corresponding labels.

[0107] The detection model is iteratively updated and trained using the Adam optimization algorithm until the loss function of the detection model is minimized, thus completing the model training.

[0108] The training set can be obtained by directly downloading public datasets or building a private dataset; public datasets and private datasets can be divided into training sets and test sets respectively, with the division ratio set to 70% and 30% respectively. The training set is used for model training, and the test set is used for the final evaluation of the model's robustness and generalization ability.

[0109] Public datasets include CelebFaces Attributes (CelebA), FaceForensics++ (FF++), and DeepFake Detection Challenge (DFDC), all of which contain labels for real and fake faces. Among them, mainstream face forgery datasets such as FF++ and DFDC come with annotations for forgery types and generator types, providing complete type annotations. For the FF++ dataset, the official documentation provides annotations for the four mainstream face forgery methods it covers. Deepfakes and FaceSwap belong to the identity replacement type, while Face2Face and NeuralTextures belong to the attribute manipulation type, and all forgery methods are based on generative adversarial networks. For the DFDC dataset, the official documentation provides source annotations for the forgery methods to distinguish samples generated by generative adversarial networks, diffusion models, etc. The training and test sets can be constructed by creating a hybrid dataset using public datasets.

[0110] Building a private dataset involves the following steps:

[0111] Step 1: Collect real face images, such as personal photos, group photos from schools or companies, etc.; select real face images that meet the requirements from some open-source face datasets that allow secondary use, such as the Labeled Faces in the Wild (LFW) dataset; and, while complying with the relevant platform terms of use and laws and regulations, use web crawling tools to scrape real face images from platforms such as social media and news websites.

[0112] Step two involves generating fake face images using face synthesis tools such as DeepFakeGenerator and FaceSwap; or using generative adversarial networks and diffusion models to generate fake face images.

[0113] Step 3: Data annotation. Use professional image annotation tools, such as image annotation tools (LabelImg) and image annotation software (RectLabel), to annotate each image to indicate whether it is a real face or a fake face. "0" represents a real face and "1" represents a fake face.

[0114] Step four, image processing: using tools from the open-source computer vision library (OpenCV), detect and align faces, then crop and scale the image to a uniform size, such as 256×256 pixels.

[0115] Furthermore, the total loss function Combining the cross-entropy loss of the classifier and cosine similarity loss Comparative loss and KL divergence loss Its expression is as follows:

[0116] ;

[0117] in, These are the weighting coefficients for the cross-entropy loss. These are the weighting coefficients for the cosine similarity loss. To compare the weighting coefficients of the loss, The weights of the KL divergence loss are used to balance the learning intensity of different tasks by adjusting the weights of each loss term, so that the model can achieve optimal performance in dimensions such as true / false classification, fine-grained type recognition and cross-modal alignment.

[0118] Cross-entropy loss The expression is as follows:

[0119] ;

[0120] in, For the model to the first Each sample is predicted to be the probability of a fake face.

[0121] Step 110: Obtain the face authenticity detection result.

[0122] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0123] This invention proposes a text-guided, multi-task, text-guided face forgery detection method to address the problem of poor cross-scene generalization of existing methods when facing new forgery techniques:

[0124] First, from the perspective of visual-language cross-modal modeling, a multi-granular text prompt lexicon covering true / false attributes, forgery type, generator type, and facial components is constructed. Multi-dimensional fine-grained text prototypes are generated with the help of a pre-trained text encoder, providing a benchmark for cross-modal forgery feature matching.

[0125] Secondly, the input face image is processed by a pre-trained visual encoder to extract global visual features. The visual adaptation module optimizes the feature distribution and enhances the sensitivity to forgery features. Then, the grouping type extraction module completes feature separation and semantic enhancement for different forgery types and generator types, and explores potential differences in forgery features.

[0126] Next, the enhanced visual features are mapped to text features of the corresponding dimensions through the text predictor. Combined with the multi-task learning module, cross-modal prototype similarity matching and multi-dimensional feature measurement learning are completed simultaneously to maximize the distinguishability of different types of fake features in the feature space and enhance the representation ability of fine-grained fake features.

[0127] Finally, based on the similarity between text prediction and text prototype, a classifier is used to accurately predict the authenticity of face images, effectively improving the model's detection accuracy, robustness, and cross-dataset generalization ability.

[0128] Example 2

[0129] Embodiment 2 of this application provides a specific implementation method for a text-guided face forgery detection method based on Embodiment 1. This method is executed by a computing device and includes:

[0130] like Figure 2 As shown, it includes the following steps:

[0131] like Figure 2 As shown, step 1 involves obtaining the face image to be detected as genuine or fake;

[0132] Step 2 involves inputting the face image to be tested for authenticity into the detection model that has been trained to its optimal performance, and finally predicting its authenticity.

[0133] The detection models include:

[0134] The text prototype generation branch is used to generate text prototypes covering multiple granularities, providing a unified benchmark for cross-modal forgery feature matching. It includes the following three sub-modules:

[0135] A multi-granular text prompt lexicon is used to provide fine-grained text prompts with multi-dimensional semantics related to face forgery detection, providing basic input for the construction of subsequent text prototypes;

[0136] A text encoder is used to convert input multi-granular text prompts into a fixed-size text feature vector;

[0137] The text prototype library is used to construct a multi-dimensional, fine-grained collection of text prototypes based on the input text feature vector.

[0138] The visual-text prediction branch is used to extract multi-dimensional visual features from face images and map them to the text feature space to achieve cross-modal alignment. It includes the following 5 sub-modules:

[0139] A 68-keypoint detector is used to locate facial keypoints in an input face image.

[0140] A visual encoder is used to extract and output fixed-size general visual features from an input face image.

[0141] The visual adaptation module is used to adapt the input general visual features to the feature space of the face forgery detection task. It performs feature learning through operations such as convolution and normalization, and then preserves the original visual semantics through residual connections, outputting the adapted global visual features.

[0142] The grouping type extraction module is used to group the input global visual features by channel according to forgery type and generator type, apply spatial attention weighting to each group, enhance the fine-grained feature expression of different dimensions, and obtain the type visual features corresponding to the group.

[0143] A text predictor is used to map input global visual features, component-level visual features, and type visual features to text features; global visual features are mapped to global text prediction; component-level visual features are mapped to component-level text prediction; and type visual features are mapped to type text prediction.

[0144] The multi-task learning module is used to perform similarity constraints between text prototypes in the input text prototype library and predicted text features output by the text predictor; it also performs discriminative constraints on text prototypes of different dimensions, providing loss signals for subsequent model parameter optimization. The multi-task learning module includes three loss functions: cosine similarity loss, contrastive loss, and KL divergence loss. The multi-task learning module performs both prototype learning, which is to match the similarity between text prototypes generated by the text prototype generation branch and text predicted features obtained by the visual-text prediction branch; and metric learning, which is to keep true / false global and local features, i.e., positive / negative samples, and different types of features within the dimensions of fake type and generator type as far apart as possible in the feature space.

[0145] A classifier is used to calculate and predict the probability of an input image being true or false based on the similarity distance between the global text prediction output by the text predictor and the true and false global text prototypes in the text prototype library.

[0146] The concept of this invention is as follows: First, from the perspective of visual-language cross-modal modeling, a multi-granular text prompt lexicon covering authenticity attributes, forgery types, generator types, and facial components is constructed to generate multi-dimensional text prototypes as cross-modal matching benchmarks. Then, image features are extracted through a homologous visual encoder, and the feature distribution is optimized by a visual adaptation module. Combined with a grouping type extraction module, feature separation and enhancement for different forgery types are achieved. Next, a text predictor is used to complete the visual-to-text feature mapping, and a multi-task learning module is used to simultaneously achieve cross-modal prototype matching and feature measurement learning. Finally, authenticity prediction is completed based on the similarity between global text prediction and global text prototypes, significantly improving detection accuracy, robustness, and cross-dataset generalization ability.

[0147] like Figure 3 As shown, the multi-granularity text prompt lexicon, text encoder, and text prototype library together constitute the text prototype generation branch;

[0148] The multi-granularity text prompt library pre-defines four core dimensions: First, the global dimension, also known as the authenticity attribute dimension, includes the two basic prompt words "real" and "fake"; second, the facial component dimension, encompassing prompt words corresponding to key facial components such as "eyes," "nose," and "lips," which are further refined by combining prompt words from the facial component dimension with those from the global dimension, such as "real eyes"; third, the forgery type dimension, covering prompt words corresponding to mainstream forgery types such as "Identity Swapping (IS)," "Attribute Manipulation (AM)," and "Entire Synthesis (ES); and fourth, the generator type dimension, including prompt words corresponding to generator types such as "Generative Adversarial Network (GAN)" and "Denoising Diffusion Probabilistic Model (DDPM)."

[0149] The text encoder uses existing pre-trained visual-language large models, such as CLIP's text encoder; it leverages the general semantic understanding capabilities learned during its pre-training process to accurately capture the semantic information of cue words at different granularities, enabling the output text features to have a good cross-modal alignment foundation and providing text support for the construction of subsequent text prototypes.

[0150] The text prototype library categorizes and aggregates the text features output by the text encoder according to four preset dimensions, forming a unique text prototype for each dimension. ,Include:

[0151] Global Dimension: Real Global Text Prototype ; Fake global text prototype ;

[0152] Facial component dimension: Real component-level text prototype Includes a real eye text prototype Real nose text prototype and real lip text prototype etc.; forging component-level text prototypes Contains a text prototype of fake eyes Fake nose text prototype and fake lip text prototype wait;

[0153] Forgery type dimension: Forgery type text prototype Includes identity replacement text prototype Attribute editing text prototype Full face replacement text prototype ;

[0154] Generator Dimension: Generator Text Prototype Includes a prototype of generative adversarial network text. diffusion model text prototype .

[0155] Where Maximize and Max represent keeping the global and local features of true / false, i.e., positive / negative samples, as well as the different types of features within the dimensions of fake type and generator type, as far apart as possible in the feature space;

[0156] The multi-task learning module completes prototype learning, which means matching the text prototypes generated by the text prototype generation branch with the text prediction features obtained by the visual-text prediction branch based on similarity.

[0157] like Figure 4 As shown, the 68 keypoint detector, visual encoder, visual adaptation module, grouping type extraction module, and text predictor together constitute the visual-text prediction branch.

[0158] 68 keypoint detector for face images As input, existing pre-trained facial landmark detection models, such as the Dlib face landmark detector, are used; these models can accurately identify and output a 68-landmark map containing the coordinate information of 68 standard key landmarks in a face image. The coordinate information can be used to further segment facial components such as eyes, nose, and lips.

[0159] The visual encoder uses a pre-trained visual-language large model that is the same as the text encoder, such as CLIP's visual encoder. Relying on its pre-trained general feature extraction capabilities, it can accurately capture the visual features of the input image and provide visual support for the subsequent prediction of text features.

[0160] The visual adaptation module takes the general visual features obtained from the visual encoder as input, which are then processed through two 3×3 depthwise separable convolutional layers, a normalization layer, and a GELU activation layer, and finally obtained through residual connections to obtain global visual features. Global visual features are mapped to coordinate information to obtain component-level visual features of the same size as the global visual features, including eye visual features. Nose visual characteristics and visual features of the lips wait.

[0161] The grouping type extraction module consists of three sub-layers: a type channel grouping sub-layer, a type feature enhancement sub-layer, and a global type feature integration layer. At the beginning of training, this module only has the ability to extract general features with random initialization and has not yet formed the ability to represent specific features for different forgery types and generator types. However, during the model training process, the text prototype, as a high-level semantic prior, is driven by the cross-modal alignment constraints of the multi-task learning module and the joint guidance of strong supervision signals in the three dimensions of global, component, and type. This drives the grouping of each channel in this module to gradually converge to the specific feature extraction mode of the corresponding type, and finally has the ability to specifically mine the fine-grained feature differences of different forgery types and generator types.

[0162] The type-channel grouping sublayer divides the global visual features into 5 independent channel groups based on type attributes, and increases the feature dimension from... Remodeling The batch dimension remains unchanged, and the system is split only from the channel dimension. Each group of features is then extracted separately, with each group focusing on a single type of feature expression to ensure that different types of attributes do not interfere with each other. The five independent channel groups are: forgery type: identity replacement group, forgery type: attribute manipulation group, forgery type: fully synthesized group, generator type: generative adversarial network group, and generator type: diffusion model group.

[0163] The type feature enhancement sublayer performs spatial enhancement based on global feature semantics for each group of type features. The type features are first processed through a group normalization layer and a global average pooling layer to obtain initial features. Then, the dot product similarity between the feature vector at each spatial location within the group and the global feature vector of that group is calculated to generate a spatial attention map, locating spatial regions related to the type attribute globally. The spatial attention map is mapped to the [0,1] interval using a Sigmoid activation function, serving as a spatial weight to modulate the initial features of the group position-by-position, amplifying type-related regional features and suppressing irrelevant noise to obtain enhanced features. Finally, the enhanced features are residually connected to the initial features of the group. The global type feature integration layer concatenates the five enhanced type features by channel dimension. Since there is a natural cross-relationship between the forged type and the generator type, the same forged type can be implemented by multiple generator architectures, and the same generator can also generate forged content with different semantic types. Therefore, a 1×1 convolutional layer is used to achieve feature interaction fusion and dimension unification between different channel groups, outputting type visual features with the same size as the original global visual features. .

[0164] The text predictor employs a three-branch parallel architecture, corresponding to text space mappings for global, component-level, and type-level visual features, respectively. Each branch consists of two fully connected layers, a GELU activation layer, and a normalization layer concatenated together, with an output dimension of [missing information]. It remains consistent with the text prototype.

[0165] Global text prediction branch: Input global visual features First, the size is compressed using global average pooling. Then, it is mapped to global text prediction via MLP. , used to match true and false global text prototypes.

[0166] Component-level text prediction branch: Input component-level visual features such as eyes, nose, and lips respectively. After global average pooling, each component is mapped to its corresponding component-level text prediction through its own independent MLP mapping. , used to match true and false component-level text prototypes.

[0167] Type-based text prediction branch: Input type visual features First, divide it according to the channel dimension into sizes corresponding to the 5 type groups. The sub-features are then mapped to five types of text prediction using independent MLP mappings. This is used to match the text prototype of the corresponding forgery type and generator type.

[0168] The training method for the detection model in this embodiment includes the following steps:

[0169] 1. Obtain the training set, which includes real and face images generated using existing forgery techniques. and and the corresponding tags, among which The corresponding label is "0". The corresponding label is "1".

[0170] Specifically, obtaining the training set includes the following steps:

[0171] Training sets were obtained from public datasets, including the Celebrity Face Attributes dataset (CelebA), the FaceForensics++ dataset (FF++), and the DeepFake Detection Challenge (DFDC). A private dataset was constructed by first collecting real face images, then using face synthesis tools or generative adversarial networks and diffusion models to generate fake face images from the real face images as input. Next, professional image annotation tools were used to label each image as either a real or fake face, with "0" representing a real face and "1" representing a fake face. Finally, tools from the Open Source Computer Vision Library (OpenCV) were used to detect and align faces before cropping and scaling the images to standardize their size; in this example, the image size was set to 256×256 pixels. The public and private datasets were divided into training and testing sets, with a ratio of 70% and 30%, respectively. The training set was used for model training, and the testing set was used to evaluate the model's robustness and generalization ability.

[0172] II. Comparing real facial images with those generated using existing forgery techniques and The input is fed into a pre-built detection model, yielding two types of output results: one is the probability of predicting whether a face is real or fake, output by the classifier. Secondly, the predicted text features output by the text predictor include global, component-level, and type-level features.

[0173] III. Calculate the total loss function based on the model output and corresponding labels. Its cross-entropy loss combined with the classifier and cosine similarity loss Comparative loss KL divergence loss The expression is as follows:

[0174] ;

[0175] in, These are the weighting coefficients for the cross-entropy loss. These are the weighting coefficients for the cosine similarity loss. To compare the weighting coefficients of the loss, The weights of the KL divergence loss are used to balance the learning intensity of different tasks, enabling the model to achieve optimal performance in dimensions such as true / false classification, fine-grained type recognition, and cross-modal alignment.

[0176] Cross-entropy loss The expression used to constrain the accuracy of true / false classification is as follows:

[0177] ;

[0178] in, Batch size; Sample label Where "0" represents a real face and "1" represents a fake face; For the model to the first Each sample is predicted to be the probability of a fake face.

[0179] Fourth, the detection model is iteratively updated and trained based on the Adam optimization algorithm until the loss function of the detection model is minimized. At this point, the detection model is considered to have reached its optimal performance.

[0180] Example 3

[0181] This application provides a text-guided face forgery detection system based on embodiments 1 and 2, which is used to execute the methods provided in embodiments 1 and 2 of this application. The system includes: a detection result acquisition module, a text prototype generation module, a visual-text prediction module, a multi-task learning module, and a classification module.

[0182] The detection result acquisition module is used to obtain the face image to be detected (whether it is real or fake), input the face image to be detected (whether it is real or fake) into the face detection model, and obtain the face authenticity detection result. The face detection model includes: a text prototype generation module, used to construct a text prompt lexicon covering multiple granularities and generate multi-dimensional text prototypes; a visual-text prediction module, used to extract visual features from the input face image through a homologous visual encoder, and to optimize the feature distribution of the visual features through a visual adaptation module to obtain global visual features; the visual-text prediction module is also used to extract global visual features through a grouping type extraction module. The system performs feature separation and enhancement on visual features; the visual-text prediction module is also used to map the global visual features after feature separation and enhancement to predicted text features through a text predictor; the multi-task learning module is used to impose similarity constraints on multi-dimensional text prototypes and predicted text features to obtain cross-modal prototype matching results; the multi-task learning module is also used to impose discriminative constraints on different dimensions of text prototypes in multi-dimensional text prototypes to obtain feature measurement learning results; the classification module is used to perform face authenticity detection based on cross-modal prototype matching results and feature measurement learning results, and output face authenticity detection results.

[0183] Example 4

[0184] Having introduced the text-guided face forgery detection system in an exemplary embodiment of this application, we will now introduce a computing device in another exemplary embodiment of this application.

[0185] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0186] In some possible implementations, the computing device according to this application may include at least one processor and at least one memory. The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the cross-device assisted interaction method for off-site clearing according to various exemplary embodiments of this application described above.

[0187] The following reference Figure 5 To describe a computing device 130 according to this embodiment of the present application. Figure 5 The computing device 130 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this application. Figure 5As shown, the computing device 130 is presented in the form of a general-purpose smart terminal (or Bluetooth headset). The components of the computing device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0188] Bus 133 represents one or more of several bus architectures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus architectures. Memory 132 may include readable media in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323. Memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0189] The computing device 130 can also communicate with one or more external devices 134 (e.g., a keyboard, pointing device, etc.), and / or with any device that enables the computing device 130 to communicate with one or more other smart terminals (e.g., a router, modem, etc.). This communication can be performed via the input / output (I / O) interface 135. Furthermore, the computing device 130 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 136. Figure 5 As shown, network adapter 136 communicates with other modules for computing device 130 via bus 133. It should be understood that, although... Figure 5 As not shown, other hardware and / or software modules may be used in conjunction with computing device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0190] In some possible implementations, various aspects of the text-guided face forgery detection method provided in this application can also be implemented in the form of a program product, which includes a computer program that, when run on a computer device, causes the computer device to perform the steps in the text-guided face forgery detection method according to the various exemplary embodiments of this application described above.

[0191] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0192] The program product for text-guided face forgery detection according to embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on a smart terminal. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0193] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0194] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0195] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0196] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable access frequency prediction device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable access frequency prediction device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0197] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable access frequency predictive device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0198] These computer program instructions can also be loaded onto a computer or other programmable access device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0199] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0200] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A text-guided method for detecting face forgery, characterized in that, include: Obtain a face image to be tested for authenticity, input the face image to be tested for authenticity into a face detection model, and obtain the face authenticity detection result; The process executed by the face detection model includes: Build a text suggestion dictionary covering multiple granularities and generate multi-dimensional text prototypes; Visual features are extracted from the input face image using a homologous visual encoder, and the feature distribution of the visual features is optimized by a visual adaptation module to obtain global visual features. Obtaining global visual features includes: acquiring a face image; locating facial key points in the face image using a 68-keypoint detector; extracting and outputting fixed-size general visual features based on the face image using a visual encoder; adapting the general visual features to the feature space of a face forgery detection task using a visual adaptation module for feature learning; and performing residual connections on the learned general visual features to preserve the original visual semantics and outputting the adapted global visual features. The global visual features are separated and enhanced by the grouping type extraction module. The feature separation and enhancement includes grouping the input global visual features by channel according to the forgery type and generator type, applying spatial attention weighting to each group, strengthening the fine-grained feature expression of different dimensions, and obtaining the type visual features corresponding to the group. The text predictor maps the global visual features, after feature separation and enhancement, to predicted text features. This mapping includes mapping the input global visual features, component-level visual features, and type-level visual features to text features. Specifically, the global visual features are mapped to global text prediction, the component-level visual features are mapped to component-level text prediction, and the type-level visual features are mapped to type-level text prediction. By using a multi-task learning module, similarity constraints are applied to the multi-dimensional text prototype and the predicted text features to obtain cross-modal prototype matching results. Discriminative constraints are applied to the text prototypes of different dimensions in the multi-dimensional text prototypes to obtain feature measurement learning results; Based on the cross-modal prototype matching results and the feature metric learning results, face authenticity detection is performed, and the face authenticity detection results are output.

2. The text-guided face forgery detection method according to claim 1, characterized in that, The construction of a multi-granular text prompt lexicon and the generation of multi-dimensional text prototypes include: Obtain multi-dimensional fine-grained text prompts with semantics related to face forgery detection provided by a multi-granularity text prompt lexicon; wherein, the multi-granularity text prompt lexicon includes a true / false attribute dimension, a forgery type dimension, a generator type dimension, and a facial component dimension; The multi-dimensional fine-grained text prompts provided by the multi-granularity text prompt dictionary are converted into text feature vectors of a fixed size; A multi-dimensional text prototype set is constructed based on the text feature vectors.

3. The method according to claim 1, characterized in that, The method further includes: The face detection model is trained; the training method of the model includes: Obtain a training set, which includes real and face images generated by existing forgery techniques, and corresponding labels; wherein the label value is 0 or 1, where 0 represents a real face and 1 represents a forged face; The real and the face images generated by existing forgery techniques are input into a pre-built detection model to obtain the model prediction results; Calculate the total loss function based on the model prediction results and the corresponding labels; The detection model is iteratively updated and trained using the Adam optimization algorithm until the loss function of the detection model is minimized, thus completing the model training.

4. A text-guided face forgery detection system, characterized in that, include: The detection result acquisition module is used to obtain the face image to be detected as genuine or fake, input the face image to be detected as genuine or fake into the face detection model, and obtain the face authenticity detection result; The face detection model includes: The text prototype generation module is used to build a text prompt dictionary covering multiple granularities and generate multi-dimensional text prototypes; The visual-text prediction module is used to extract visual features from the input face image through a homologous visual encoder, and optimize the feature distribution of the visual features through a visual adaptation module to obtain global visual features. Obtaining global visual features includes acquiring a face image, locating facial key points in the face image using a 68-keypoint detector, extracting and outputting fixed-size general visual features based on the face image through a visual encoder, adapting the general visual features to the feature space of a face forgery detection task through the visual adaptation module for feature learning, and performing residual connections on the general visual features after feature learning to preserve the original visual semantics and output the adapted global visual features. The visual-text prediction module is also used to perform feature separation and enhancement on global visual features through the grouping type extraction module; wherein, performing feature separation and enhancement includes grouping the input global visual features into channels according to forgery type and generator type, performing spatial attention weighting on each group, strengthening the fine-grained feature expression of different dimensions, and obtaining the type visual features corresponding to the group. The visual-text prediction module is further configured to map the global visual features after feature separation and enhancement to predicted text features using a text predictor; wherein, mapping the global visual features after feature separation and enhancement to predicted text features includes mapping the input global visual features, component-level visual features, and type visual features to text features; wherein, the global visual features are mapped to global text prediction, the component-level visual features are mapped to component-level text prediction, and the type visual features are mapped to type text prediction. A multi-task learning module is used to constrain the similarity between the multi-dimensional text prototype and the predicted text features to obtain cross-modal prototype matching results; The multi-task learning module is also used to apply discriminative constraints to the text prototypes of different dimensions in the multi-dimensional text prototypes to obtain feature measurement learning results. The classification module is used to perform face authenticity detection based on the cross-modal prototype matching results and the feature metric learning results, and output the face authenticity detection results.

5. A computing device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method as described in any one of claims 1-3 according to the obtained program instructions.

6. A computer-readable storage medium, characterized in that, Includes computer-readable instructions that, when read and executed by a computer, cause the method as described in any one of claims 1 to 3 to be implemented.

7. A computer program product, characterized in that, It includes a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the method according to any one of claims 1 to 3.

Images