A method for deep fake detection and localization based on visual transformer

Abstract

The present application relates to the field of intelligent detection, and proposes a method for deep fake detection and positioning based on visual Transformer, which specifically comprises: obtaining target data to be detected and performing preprocessing; performing feature extraction according to a DF-ViT feature extraction network to obtain global deep fake artifact features and patch-level local features; inputting the global deep fake artifact features and the patch-level local features into a double-branch output module to obtain classification results and segmentation results; performing video time sequence enhancement processing according to a video time sequence ViT enhancement module; visually marking the fake area and giving explainability analysis. The present application improves artifact capturing capability and detection accuracy, solves the problem that the prior art can only classify but cannot position, and is suitable for multi-scene deployment, thereby enhancing explainability and generalizability.

Description

Technical Field

[0001] This invention relates to the field of intelligent detection, and in particular to a method for deepfake detection and localization based on visual Transformer. Background Technology

[0002] With the rapid development of artificial intelligence generation technology, deepfake technologies such as GAN, Diffusion, and NeRF can quickly generate highly realistic fake videos and images, achieving effects such as face replacement, expression alteration, and local feature modification. The visual realism of the fake content has reached a level that is difficult to distinguish with the naked eye.

[0003] In existing deepfake detection technologies, traditional convolutional neural networks (CNNs) are limited by their local receptive fields, making it difficult to capture global semantic artifacts and fine-grained local texture anomalies in deepfakes. In contrast, the Visual Transformer (ViT), with its multi-head self-attention mechanism, can model global image dependencies and capture local detail features through block embedding, making it one of the best backbone networks for deepfake detection.

[0004] Existing ViT-based deepfake detection methods primarily focus on binary classification of forged content, i.e., determining whether a video / image is a deepfake. They cannot achieve precise localization of forged regions, and ViT is not customized for deepfake artifacts, resulting in insufficient robustness to high-resolution and re-edited forged content. Furthermore, for temporal artifacts in video deepfakes, existing ViT methods do not achieve joint modeling of temporal and spatial features, limiting video detection performance.

[0005] Therefore, there is an urgent need for an integrated detection solution based on ViT, which can achieve the collaborative completion of deepfake authenticity determination and pixel-level localization of forged areas through architectural modification, taking into account both detection accuracy and result interpretability, and adapting to the detection needs of different types of deepfake content such as images and videos. Summary of the Invention

[0006] Based on this, this invention proposes a method for deepfake detection and localization based on Visual Transformer. First, the image / video to be detected is preprocessed to obtain patch feature sequences and time-series patch features. Then, a DeepFake Artifact Awareness ViT Network (DF-ViT) is constructed to extract global deepfake artifact features and patch-level fine-grained features. Next, a dual-branch structure of "classification + segmentation" is used to achieve deepfake authenticity determination and pixel-level localization of forged regions, combined with facial keypoint matching to clarify the type of forged region. For video data, a TimeSformer temporal ViT module is introduced to model inter-frame temporal dependencies and smooth the localization results. A joint loss function is constructed to perform end-to-end training and optimization of the model, ultimately achieving the labeling of forged regions. This invention solves the problem that existing ViT methods can only classify but not locate, effectively improving artifact capture capability and detection accuracy, and is adaptable to multi-scenario deployment, enhancing interpretability and generalization.

[0007] This invention proposes a method for depth forgery detection and localization based on visual Transformer, comprising: Acquire and preprocess the target data to be detected, wherein the target data to be detected includes image data to be detected and video data to be detected; Feature extraction is performed using the DF-ViT feature extraction network to obtain global deep forgery artifact features and patch-level local features. The DF-ViT feature extraction network is based on the DF-ViT encoder block constructed from the artifact-enhanced multi-head self-attention layer and the feedforward network. The artifact-enhanced multi-head self-attention layer is based on the artifact attention mask. Global deep forgery artifact features and patch-level local features are input into a dual-branch output module to obtain classification and segmentation results. The dual-branch output module includes a classification branch and a segmentation branch. The classification branch is based on a single fully connected layer, and the segmentation branch is based on a probability heatmap of forgery regions and a binary mask of forgery regions. Video temporal enhancement processing is performed based on the video temporal ViT enhancement module. The video temporal enhancement includes temporal artifact feature extraction processing, spatiotemporal feature fusion processing, and temporal smoothing processing. The forged areas are visually marked and an interpretability analysis is provided. The visual marking includes contour extraction and outlining, type labeling and video smoothing. The interpretability analysis is based on attention weight extraction and key region and channel recognition.

[0008] Furthermore, the step of acquiring and preprocessing the target data to be detected specifically includes: Acquire the image data and video data to be detected; The image data to be detected is subjected to pixel normalization processing. The specific algorithm for pixel normalization processing is as follows: , in, This represents the pixel-normalized image data to be detected. This represents the image data to be detected. , ; The pixel-normalized image data to be detected is divided into multiple non-overlapping patches. Each patch is then embedded with features through a convolutional layer to generate a patch feature sequence. The stride of the convolutional layer is 16 and the output channels are 768. The specific algorithm for generating the patch feature sequence is as follows: , , in, Indicates a single patch characteristic. Indicates a convolutional layer. Indicates the first i The pixel-normalized image region to be detected corresponding to each patch. Represents the patch feature sequence; The video data to be detected is subjected to frame sequence extraction, and then the cosine similarity between adjacent frames in the frame sequence is calculated for deduplication. The specific algorithm for the cosine similarity between adjacent frames is as follows: , in, Indicates the cosine similarity between adjacent frames. and They represent the first Frame and the The adjacent frames of a frame; Keyframes are filtered based on the cosine similarity between adjacent frames, and redundant frames with a cosine similarity greater than a preset similarity threshold are deleted. Each keyframe undergoes resolution unification, normalization, and patch embedding processing to obtain a keyframe patch feature sequence. The keyframe patch feature sequences are then stacked in time sequence to generate time sequence patch features.

[0009] Furthermore, the step of extracting features using the DF-ViT feature extraction network to obtain global deep forgery artifact features and patch-level local features specifically includes: The preprocessed patch feature sequence is used to construct initial input features by adding learnable location embeddings and deep forgery artifact-aware category embeddings to the patch feature sequence. The specific algorithm for the initial input features is as follows: , in, Represents the initial input features. This indicates a category embedding for deepfake artifact perception. This represents a learnable positional embedding. Represents the patch feature sequence; The initial input features are input into the DF-ViT encoder block to obtain global deep forgery artifact features and patch-level local features. The DF-ViT encoder block is constructed based on an artifact-enhanced multi-head self-attention layer and a feedforward network. The global deep forgery artifact features are the 0th features of the top-level DF-ViT encoder block, and the patch-level local features are multi-scale fine-grained features. The specific algorithm for the artifact enhancement multi-head self-attention layer is as follows: , in, This indicates that artifacts enhance the output features of the multi-head self-attention layer. express Activation function This indicates that artifacts enhance the input features of the multi-head self-attention layer. , , These represent the query features, key features, and value features of multi-head self-attention, respectively. This indicates the single-head attention dimension. Indicates the artifact attention mask. This indicates element-wise multiplication; The specific algorithm of the DF-ViT encoder block is as follows: , , in, Indicates the first l Artifact enhancement of multi-head self-attention layer output features Indicates the first l Layer patch characteristics, This indicates that artifacts enhance the bullish self-attention layer. Representation layer normalization, Indicates a feedforward network; The patch-level local features are obtained by weighted fusion of the patch features output by the DF-ViT encoder block of each layer. The specific algorithm for the patch-level local features is as follows: , in, Indicates patch-level local features, Indicates the number of floors. Represents the learnable layer weights. Indicates the first lThe patch feature sequence output by the layer DF-ViT encoder block, excluding the category embedding.

[0010] Furthermore, the step of inputting global deep forgery artifact features and patch-level local features into the dual-branch output module to obtain classification and segmentation results specifically includes: Global deep forgery artifact features are input into the classification branch to obtain the deep forgery prediction probability. The classification head of the classification branch is based on a single fully connected layer and a Sigmoid activation function. The specific algorithm for predicting the probability of deepfake is as follows: , in, Indicates the probability of deepfake prediction. This represents the Sigmoid activation function. This represents the weight matrix of the fully connected layer. This indicates the characteristics of global depth-fake artifacts. Indicates the bias parameter; The authenticity of a deepfake is determined based on the predicted probability, and the specific algorithm for this determination is as follows: , in, Indicates the threshold for determining authenticity; The patch-level local features are input into the segmentation branch, rearranged into a two-dimensional feature map, and then upsampled through two layers of transposed convolution to restore the original resolution. The transposed convolution includes a first layer of transposed convolution and a second layer of transposed convolution. The first layer of transposed convolution has a kernel of 16×16, a stride of 16, and 384 output channels. The second layer of transposed convolution has a kernel of 16×16, a stride of 16, and 1 output channel. The upsampled features are processed using the Sigmoid activation function to generate a heatmap of forged region probability. Then, the heatmap is binarized according to a segmentation threshold to obtain a binary mask for the forged region. The specific algorithm for this binary mask is as follows: , in, This indicates a forged region binary mask. This represents the probability that a pixel in the heatmap is a fake region. Indicates the segmentation threshold; Multiple target single-class key points are obtained, and the average Euclidean distance between pixels within the binary mask of the forged region and the target single-class key points is calculated. The specific algorithm for the average Euclidean distance is as follows: , in, This represents the average Euclidean distance between pixels within the binary mask of the forged region and keypoints of the target single class. Represents a single set of target key points. This represents the binary mask region for the forged area. This represents the set of pixels with a mask value of 1. and These represent the x and y coordinates of a pixel, respectively. and These represent the x and y coordinates of a single target key point, respectively. The current mask region is determined to be a forged region of the corresponding feature based on a preset distance threshold. If the average Euclidean distance between the pixels in the binary mask of the forged region and the target single-class key point is less than the preset distance threshold, the current mask region is determined to be a forged region of the corresponding feature.

[0011] Furthermore, the step of performing video temporal enhancement processing based on the video temporal ViT enhancement module specifically includes: The temporal patch features are input into the video temporal ViT enhancement module. Based on the spatiotemporal attention mechanism, the dependencies between the spatial and temporal dimensions are modeled separately to extract temporal artifact features. The specific algorithm for extracting the temporal artifact features is as follows: , in, Indicates temporal artifact features, This refers to the video timing ViT enhancement module. Indicates the characteristics of timing patches. Represents the spatial self-attention layer. Represents a temporal self-attention layer; Temporal artifact features are fused with global deepfake artifact features to obtain spatiotemporal joint artifact features. The specific algorithm for the spatiotemporal joint artifact features is as follows: , in, Indicates the spatiotemporal joint artifact characteristics, This indicates the characteristics of global depth-fake artifacts. Represents the time-series feature weight matrix; Based on the deepfake prediction probability, the binary mask of the forged region in the keyframe is subjected to temporal smoothing. The specific algorithm for the temporal smoothing is as follows: , in, This represents the binary mask of the forged region after timing smoothing. , , These represent the binary masks of the forged regions in the current keyframe, the previous keyframe, and the next keyframe, respectively.

[0012] Furthermore, the steps of visually marking the forged areas and providing interpretability analysis specifically include: The Canny edge detection algorithm is used to extract the contour point set from the binary mask of the fake region. The contour point set is outlined by contour lines in the original image and video frames to circle the fake region. Add text labels next to the outlines to indicate the type of fake region, and perform inter-frame linear interpolation on the labeling results of all keyframes. Generate the labeling results of intermediate frames based on the outline positions and types of adjacent keyframes. An attention weight matrix is ​​constructed based on the artifact enhancement attention weights in the DF-ViT feature extraction network and the feature fusion weights in the dual-branch output module. Based on the attention weight matrix, the top-20 patch regions that contribute the most to the authenticity determination and the top-50 feature channels with the highest weights are selected. The locations of key patch regions in the original image are visualized to obtain a heatmap of key regions in the image, so as to generate a ranking of the contribution values ​​of artifact feature channels. Output an interpretability analysis report, which includes a heatmap of key image regions, an intermodal interaction path map, and a heatmap of the location reliability of forged regions.

[0013] Furthermore, after the step of performing video temporal enhancement processing based on the video temporal ViT enhancement module, the method further includes: Optimization is performed based on the joint loss function, the specific algorithm of which is as follows: , , , , , , in, Indicates joint loss, , , These represent the loss weights for classification loss, segmentation loss, and temporal consistency loss, respectively. Represents classification loss, Indicates the partition loss. Indicates the time-series consistency loss. Represents pixel cross-entropy loss. This indicates Dice's loss. Indicates the number of samples in the batch. Represents ordinal numbers. Indicates the true label of the sample. Indicates the probability of deepfake prediction. and These represent the x and y coordinates of a pixel, respectively. This indicates a fake region's true mask. This indicates a forged region binary mask. Indicates the minimum value. This represents the binary mask region for the forged area. Indicates the keyframe sequence number. Indicates the first Patch-level local features of keyframes.

[0014] This invention proposes a system for depth forgery detection and localization based on visual Transformer, comprising: The preprocessing module is used to acquire and preprocess the target data to be detected, which includes image data and video data to be detected. The feature extraction module is used to extract features based on the DF-ViT feature extraction network to obtain global deep forgery artifact features and patch-level local features. The DF-ViT feature extraction network is based on the DF-ViT encoder block constructed based on the artifact enhancement multi-head self-attention layer and the feedforward network. The artifact enhancement multi-head self-attention layer is based on the artifact attention mask. A dual-branch output module is used to input global deep forgery artifact features and patch-level local features into the dual-branch output module to obtain classification results and segmentation results. The dual-branch output module includes a classification branch and a segmentation branch. The classification branch is based on a single fully connected layer, and the segmentation branch is based on a forgery region probability heatmap and a forgery region binary mask. The video temporal ViT enhancement module is used to perform video temporal enhancement processing, which includes temporal artifact feature extraction processing, spatiotemporal feature fusion processing, and temporal smoothing processing. The visualization analysis module is used to visually mark the forged areas and provide interpretability analysis. The visualization marks include contour extraction and outlining processing, type labeling and video smoothing processing. The interpretability analysis is based on attention weight extraction and key region and channel recognition.

[0015] The present invention also provides a storage medium that stores one or more programs, which, when executed by a processor, implement the method for depth forgery detection and localization based on visual Transformer as described above.

[0016] The present invention also provides a computer device, the computer device including a memory and a processor, wherein: The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the method for depth forgery detection and localization based on visual Transformer as described above.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. Integrated detection and localization: Through a customized ViT architecture with a dual branch of "classification + segmentation", deepfake authenticity determination and pixel-level localization of forgery areas can be completed simultaneously in a single forward inference, which solves the core defect of existing methods that can only classify and cannot accurately mark forgery areas, and the detection results have strong interpretability.

[0018] 2. Comprehensive artifact capture capability: By enhancing the self-attention mechanism through artifact capture and fusing multi-scale features, it can capture both global semantic artifacts of deepfakes and perceive local fine-grained artifacts; the introduction of the temporal ViT module for video data effectively captures inter-frame temporal artifacts, significantly improving detection robustness in complex scenes.

[0019] 3. Strong dynamic adaptability: Through adaptive weighted fusion and temporal smoothing, the model can dynamically adjust the feature contribution weights according to the modal features and forgery types of different samples, adapting to low-resolution, second-edited deepfake content, as well as samples generated by various mainstream forgery algorithms, and has excellent generalization ability.

[0020] 4. High Practical Value: The visual marking of forged areas can directly provide interpretable visual evidence for online content review, anti-fraud supervision, and judicial evidence collection. The interpretability module further enhances the transparency and credibility of model decisions. At the same time, the model supports lightweight modification and can be deployed in various scenarios such as batch review on the server side and real-time detection on the edge, adapting to actual application needs. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method for depth forgery detection and localization based on visual Transformer proposed in the first embodiment of the present invention; Figure 2 This is a schematic diagram of the system for depth forgery detection and localization based on visual Transformer proposed in the second embodiment of the present invention.

[0022] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0023] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0024] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0026] Please see Figure 1 The diagram shows a flowchart of a depth forgery detection and localization method based on visual Transformer proposed in the first embodiment of the present invention. This depth forgery detection and localization method based on visual Transformer includes steps S01 to S05, wherein: Step S01: Acquire the target data to be detected and perform preprocessing; It should be noted that in this embodiment, the target data to be detected includes image data to be detected and video data to be detected, and the image data to be detected and video data to be detected are obtained; The image data to be detected is subjected to pixel normalization processing. The specific algorithm for pixel normalization processing is as follows: , in, This represents the pixel-normalized image data to be detected. This represents the image data to be detected. , ; The pixel-normalized image data to be detected is divided into multiple non-overlapping patches. Each patch is then embedded with features through a convolutional layer to generate a patch feature sequence. The stride of the convolutional layer is 16 and the output channels are 768. The specific algorithm for generating the patch feature sequence is as follows: , , in, Indicates a single patch characteristic. Indicates a convolutional layer. Indicates the first i The pixel-normalized image region to be detected corresponding to each patch. Represents the patch feature sequence; The video data to be detected is subjected to frame sequence extraction, and then the cosine similarity between adjacent frames in the frame sequence is calculated for deduplication. The specific algorithm for the cosine similarity between adjacent frames is as follows: , in, Indicates the cosine similarity between adjacent frames. and They represent the first Frame and the The adjacent frames of a frame; Keyframes are filtered based on the cosine similarity between adjacent frames, and redundant frames with a cosine similarity greater than a preset similarity threshold are deleted. Each keyframe undergoes resolution unification, normalization, and patch embedding processing to obtain a keyframe patch feature sequence. The keyframe patch feature sequences are then stacked in time sequence to generate time sequence patch features.

[0027] Step S02: Perform feature extraction based on the DF-ViT feature extraction network to obtain global deep forgery artifact features and patch-level local features; It should be noted that in this embodiment, the DF-ViT feature extraction network is based on a DF-ViT encoder block constructed from an artifact-enhanced multi-head self-attention layer and a feedforward network. The artifact-enhanced multi-head self-attention layer is based on an artifact attention mask and constructs initial input features from the preprocessed patch feature sequence. Learnable position embeddings and deep spoofing artifact-aware category embeddings are added to the patch feature sequence to obtain initial input features. The specific algorithm for the initial input features is as follows: , in, Represents the initial input features. This indicates a category embedding for deepfake artifact perception. This represents a learnable positional embedding. Represents the patch feature sequence; The initial input features are input into the DF-ViT encoder block to obtain global deep forgery artifact features and patch-level local features. The DF-ViT encoder block is constructed based on an artifact-enhanced multi-head self-attention layer and a feedforward network. The global deep forgery artifact features are the 0th features of the top-level DF-ViT encoder block, and the patch-level local features are multi-scale fine-grained features. The specific algorithm for the artifact enhancement multi-head self-attention layer is as follows: , in, This indicates that artifacts enhance the output features of the multi-head self-attention layer. express Activation function This indicates that artifacts enhance the input features of the multi-head self-attention layer. , , These represent the query features, key features, and value features of multi-head self-attention, respectively. This indicates the single-head attention dimension. Indicates the artifact attention mask. This indicates element-wise multiplication; The specific algorithm of the DF-ViT encoder block is as follows: , , in, Indicates the first l Artifact enhancement of multi-head self-attention layer output features Indicates the first l Layer patch characteristics, This indicates that artifacts enhance the bullish self-attention layer. Representation layer normalization, Indicates a feedforward network; The patch-level local features are obtained by weighted fusion of the patch features output by the DF-ViT encoder block of each layer. The specific algorithm for the patch-level local features is as follows: , in, Indicates patch-level local features, Indicates the number of floors. Represents the learnable layer weights. Indicates the first l The patch feature sequence output by the layer DF-ViT encoder block, excluding the category embedding.

[0028] Step S03: Input the global deep forgery artifact features and patch-level local features into the dual-branch output module to obtain classification and segmentation results; It should be noted that in this embodiment, the dual-branch output module includes a classification branch and a segmentation branch. The classification branch is based on a single fully connected layer, and the segmentation branch is based on a probability heatmap of the fake region and a binary mask of the fake region. The global deep forgery artifact features are input into the classification branch to obtain the deep forgery prediction probability. The classification head of the classification branch is based on a single fully connected layer and a Sigmoid activation function. The specific algorithm for predicting the probability of deepfake is as follows: , in, Indicates the probability of deepfake prediction. This represents the Sigmoid activation function. This represents the weight matrix of the fully connected layer. Indicates global depth forgery artifact features. Indicates the bias parameter; The authenticity of a deepfake is determined based on the predicted probability, and the specific algorithm for this determination is as follows: , in, This represents the threshold for determining authenticity; the threshold for determining authenticity is set to 0.5. The patch-level local features are input into the segmentation branch, rearranged into a two-dimensional feature map, and then upsampled through two layers of transposed convolution to restore the original resolution. The transposed convolution includes a first layer of transposed convolution and a second layer of transposed convolution. The first layer of transposed convolution has a kernel of 16×16, a stride of 16, and 384 output channels. The second layer of transposed convolution has a kernel of 16×16, a stride of 16, and 1 output channel. The upsampled features are processed using the Sigmoid activation function to generate a heatmap of forged region probability. Then, the heatmap is binarized according to a segmentation threshold to obtain a binary mask for the forged region. The specific algorithm for the binary mask is as follows: , in, This indicates a forged region binary mask. This represents the probability that a pixel in the heatmap is a fake region. This represents the segmentation threshold; the segmentation threshold is set to 0.6. Multiple target single-class key points are obtained, and the average Euclidean distance between pixels within the binary mask of the forged region and the target single-class key points is calculated. The specific algorithm for the average Euclidean distance is as follows: , in, This represents the average Euclidean distance between pixels within the binary mask of the forged region and keypoints of the target single class. Represents the set of key points of a single target class. This represents the binary mask area for the forged region. This represents the set of pixels with a mask value of 1. and These represent the x and y coordinates of a pixel, respectively. and These represent the x and y coordinates of a single target key point, respectively. The current mask region is determined to be a forged region of the corresponding feature based on a preset distance threshold. If the average Euclidean distance between the pixels in the binary mask of the forged region and the target single-class key point is less than the preset distance threshold, the current mask region is determined to be a forged region of the corresponding feature.

[0029] Step S04: Perform video temporal enhancement processing based on the video temporal enhancement module; It should be noted that in this embodiment, the video temporal enhancement includes temporal artifact feature extraction, spatiotemporal feature fusion, and temporal smoothing. The temporal patch features are input into the video temporal ViT enhancement module. Based on the separation of spatiotemporal attention mechanism, the dependency relationship between the spatial and temporal dimensions is modeled to extract temporal artifact features. The specific algorithm for extracting the temporal artifact features is as follows: , in, Indicates temporal artifact features, This refers to the video timing ViT enhancement module. Indicates the characteristics of timing patches. Represents the spatial self-attention layer. Represents a temporal self-attention layer; Temporal artifact features are fused with global deepfake artifact features to obtain spatiotemporal joint artifact features. The specific algorithm for the spatiotemporal joint artifact features is as follows: , in, Indicates the spatiotemporal joint artifact characteristics, Indicates global depth forgery artifact features. Represents the time-series feature weight matrix; Based on the deepfake prediction probability, the binary mask of the forged region in the keyframe is subjected to temporal smoothing. The specific algorithm for the temporal smoothing is as follows: , in, This represents the binary mask of the forged region after timing smoothing. , , These represent the binary masks of the forged regions in the current keyframe, the previous keyframe, and the next keyframe, respectively.

[0030] Optimization is performed based on the joint loss function, the specific algorithm of which is as follows: , , , , , , in, Indicates joint loss, , , These represent the loss weights for classification loss, segmentation loss, and temporal consistency loss, respectively. Represents classification loss. Indicates the partition loss. Indicates the time-series consistency loss. Represents pixel cross-entropy loss. This indicates Dice's loss. Indicates the number of samples in the batch. Represents ordinal numbers. Indicates the true label of the sample. Indicates the probability of deepfake prediction. and These represent the x and y coordinates of a pixel, respectively. This indicates a fake region's true mask. This indicates a forged region binary mask. Indicates the minimum value. This represents the binary mask area for the forged region. Indicates the keyframe sequence number. Indicates the first Patch-level local features of keyframes.

[0031] Step S05: Visualize and mark the forged areas and provide an interpretability analysis; It should be noted that in this embodiment, the visualization markers include contour extraction and outlining processing, type labeling and video smoothing processing. The interpretability analysis is based on attention weight extraction and key region and channel recognition. The Canny edge detection algorithm is used to extract the contour point set of the binary mask of the fake region. The contour point set is outlined by contour lines in the original image and video frames to circle the fake region. Add text labels next to the outlines to indicate the type of fake region, and perform inter-frame linear interpolation on the labeling results of all keyframes. Generate the labeling results of intermediate frames based on the outline positions and types of adjacent keyframes. An attention weight matrix is ​​constructed based on the artifact enhancement attention weights in the DF-ViT feature extraction network and the feature fusion weights in the dual-branch output module. Based on the attention weight matrix, the top-20 patch regions that contribute the most to the authenticity determination and the top-50 feature channels with the highest weights are selected. The locations of key patch regions in the original image are visualized to obtain a heatmap of key regions in the image, so as to generate a ranking of the contribution values ​​of artifact feature channels. Output an interpretability analysis report, which includes a heatmap of key image regions, an intermodal interaction path map, and a heatmap of the location reliability of forged regions.

[0032] Please see Figure 2 The figure shown is a schematic diagram of the system for depth forgery detection and localization based on visual Transformer proposed in the second embodiment of the present invention. The system includes: Preprocessing module 10 is used to acquire and preprocess the target data to be detected, wherein the target data to be detected includes image data to be detected and video data to be detected; Feature extraction module 20 is used to extract features based on DF-ViT feature extraction network to obtain global deep forgery artifact features and patch-level local features. The DF-ViT feature extraction network is a DF-ViT encoder block constructed based on artifact enhancement multi-head self-attention layer and feedforward network. The artifact enhancement multi-head self-attention layer is based on artifact attention mask. The dual-branch output module 30 is used to input global deep forgery artifact features and patch-level local features into the dual-branch output module to obtain classification results and segmentation results. The dual-branch output module includes a classification branch and a segmentation branch. The classification branch is based on a single fully connected layer, and the segmentation branch is based on a forgery region probability heatmap and a forgery region binary mask. The video temporal ViT enhancement module 40 is used to perform video temporal enhancement processing, which includes temporal artifact feature extraction processing, spatiotemporal feature fusion processing, and temporal smoothing processing. The visualization analysis module 50 is used to visually mark the forged area and provide interpretability analysis. The visualization marking includes contour extraction and outlining processing, type labeling and video smoothing processing. The interpretability analysis is based on attention weight extraction and key area and channel recognition.

[0033] The present invention also proposes a computer storage medium storing one or more programs that, when executed by a processor, implement the above-described method for depth forgery detection and localization based on visual Transformer.

[0034] The present invention also proposes a computer device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to realize the above-described method for depth forgery detection and localization based on visual Transformer.

[0035] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain stored, communicated, propagated, or transmitted programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0036] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0037] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0038] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0039] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for depth forgery detection and localization based on visual Transformer, characterized in that, include: Acquire and preprocess the target data to be detected, wherein the target data to be detected includes image data to be detected and video data to be detected; Feature extraction is performed using the DF-ViT feature extraction network to obtain global deep forgery artifact features and patch-level local features. The DF-ViT feature extraction network is based on the DF-ViT encoder block constructed from the artifact-enhanced multi-head self-attention layer and the feedforward network. The artifact-enhanced multi-head self-attention layer is based on the artifact attention mask. Global deep forgery artifact features and patch-level local features are input into a dual-branch output module to obtain classification and segmentation results. The dual-branch output module includes a classification branch and a segmentation branch. The classification branch is based on a single fully connected layer, and the segmentation branch is based on a probability heatmap of forgery regions and a binary mask of forgery regions. Video temporal enhancement processing is performed based on the video temporal ViT enhancement module. The video temporal enhancement includes temporal artifact feature extraction processing, spatiotemporal feature fusion processing, and temporal smoothing processing. The forged areas are visually marked and an interpretability analysis is provided. The visual marking includes contour extraction and outlining, type labeling and video smoothing. The interpretability analysis is based on attention weight extraction and key region and channel recognition.

2. The method for depth forgery detection and localization based on visual Transformer according to claim 1, characterized in that, The steps of acquiring and preprocessing the target data to be detected specifically include: Acquire the image data and video data to be detected; The image data to be detected is subjected to pixel normalization processing. The specific algorithm for pixel normalization processing is as follows: ， in, This represents the pixel-normalized image data to be detected. This represents the image data to be detected. , ; The pixel-normalized image data to be detected is divided into multiple non-overlapping patches. Each patch is then embedded with features through a convolutional layer to generate a patch feature sequence. The stride of the convolutional layer is 16 and the output channels are 768. The specific algorithm for generating the patch feature sequence is as follows: ， ， in, Indicates a single patch characteristic. Indicates a convolutional layer. Indicates the first i The pixel-normalized image region to be detected corresponding to each patch. Represents the patch feature sequence; The video data to be detected is subjected to frame sequence extraction, and then the cosine similarity between adjacent frames in the frame sequence is calculated for deduplication. The specific algorithm for the cosine similarity between adjacent frames is as follows: ， in, Indicates the cosine similarity between adjacent frames. and They represent the first Frame and the The adjacent frames of a frame; Keyframes are filtered based on the cosine similarity between adjacent frames, and redundant frames with a cosine similarity greater than a preset similarity threshold are deleted. Each keyframe undergoes resolution unification, normalization, and patch embedding processing to obtain a keyframe patch feature sequence. The keyframe patch feature sequences are then stacked in time sequence to generate time sequence patch features.

3. The method for depth forgery detection and localization based on visual Transformer according to claim 1, characterized in that, The step of extracting features using the DF-ViT feature extraction network to obtain global deep forgery artifact features and patch-level local features specifically includes: The preprocessed patch feature sequence is used to construct initial input features by adding learnable location embeddings and deep forgery artifact-aware category embeddings to the patch feature sequence. The specific algorithm for the initial input features is as follows: ， in, Represents the initial input features. This indicates a category embedding for deepfake artifact perception. This represents a learnable positional embedding. Represents the patch feature sequence; The initial input features are input into the DF-ViT encoder block to obtain global deep forgery artifact features and patch-level local features. The DF-ViT encoder block is constructed based on an artifact-enhanced multi-head self-attention layer and a feedforward network. The global deep forgery artifact features are the 0th features of the top-level DF-ViT encoder block, and the patch-level local features are multi-scale fine-grained features. The specific algorithm for the artifact enhancement multi-head self-attention layer is as follows: ， in, This indicates that artifacts enhance the output features of the multi-head self-attention layer. express Activation function This indicates that artifacts enhance the input features of the multi-head self-attention layer. , , These represent the query features, key features, and value features of multi-head self-attention, respectively. This indicates the single-head attention dimension. Indicates the artifact attention mask. This indicates element-wise multiplication; The specific algorithm of the DF-ViT encoder block is as follows: ， ， in, Indicates the first l Artifact enhancement of multi-head self-attention layer output features Indicates the first l Layer patch characteristics, This indicates that artifacts enhance the bullish self-attention layer. Representation layer normalization, Indicates a feedforward network; The patch-level local features are obtained by weighted fusion of the patch features output by the DF-ViT encoder block of each layer. The specific algorithm for the patch-level local features is as follows: ， in, Indicates patch-level local features, Indicates the number of floors. Represents the learnable layer weights. Indicates the first l The patch feature sequence output by the layer DF-ViT encoder block, excluding the category embedding.

4. The method for depth forgery detection and localization based on visual Transformer according to claim 1, characterized in that, The step of inputting global deep forgery artifact features and patch-level local features into the dual-branch output module to obtain classification and segmentation results specifically includes: Global deep forgery artifact features are input into the classification branch to obtain the deep forgery prediction probability. The classification head of the classification branch is based on a single fully connected layer and a Sigmoid activation function. The specific algorithm for predicting the probability of deepfake is as follows: ， in, Indicates the probability of deepfake prediction. This represents the Sigmoid activation function. This represents the weight matrix of the fully connected layer. This indicates the characteristics of global depth-fake artifacts. Indicates the bias parameter; The authenticity of a deepfake is determined based on the predicted probability, and the specific algorithm for this determination is as follows: ， in, Indicates the threshold for determining authenticity; The patch-level local features are input into the segmentation branch, rearranged into a two-dimensional feature map, and then upsampled through two layers of transposed convolution to restore the original resolution. The transposed convolution includes a first layer of transposed convolution and a second layer of transposed convolution. The first layer of transposed convolution has a kernel of 16×16, a stride of 16, and 384 output channels. The second layer of transposed convolution has a kernel of 16×16, a stride of 16, and 1 output channel. The upsampled features are processed using the Sigmoid activation function to generate a heatmap of forged region probability. Then, the heatmap is binarized according to a segmentation threshold to obtain a binary mask for the forged region. The specific algorithm for this binary mask is as follows: ， in, This indicates a forged region binary mask. This represents the probability that a pixel in the heatmap is a fake region. Indicates the segmentation threshold; Multiple target single-class key points are obtained, and the average Euclidean distance between pixels within the binary mask of the forged region and the target single-class key points is calculated. The specific algorithm for the average Euclidean distance is as follows: ， in, This represents the average Euclidean distance between pixels within the binary mask of the forged region and keypoints of the target single class. Represents a single set of target key points. This represents the binary mask region for the forged area. This represents the set of pixels with a mask value of 1. and These represent the x and y coordinates of a pixel, respectively. and These represent the x and y coordinates of a single target key point, respectively. The current mask region is determined to be a forged region of the corresponding feature based on a preset distance threshold. If the average Euclidean distance between the pixels in the binary mask of the forged region and the target single-class key point is less than the preset distance threshold, the current mask region is determined to be a forged region of the corresponding feature.

5. The method for depth forgery detection and localization based on visual Transformer according to claim 1, characterized in that, The steps for performing video temporal enhancement processing based on the video temporal ViT enhancement module specifically include: The temporal patch features are input into the video temporal ViT enhancement module. Based on the spatiotemporal attention mechanism, the dependencies between the spatial and temporal dimensions are modeled separately to extract temporal artifact features. The specific algorithm for extracting the temporal artifact features is as follows: ， in, Indicates temporal artifact features, This refers to the video timing ViT enhancement module. Indicates the characteristics of timing patches. Represents the spatial self-attention layer. Represents a temporal self-attention layer; Temporal artifact features are fused with global deepfake artifact features to obtain spatiotemporal joint artifact features. The specific algorithm for the spatiotemporal joint artifact features is as follows: ， in, Indicates the spatiotemporal joint artifact characteristics, This indicates the characteristics of global depth-fake artifacts. Represents the time-series feature weight matrix; Based on the deepfake prediction probability, the binary mask of the forged region in the keyframe is subjected to temporal smoothing. The specific algorithm for the temporal smoothing is as follows: ， in, This represents the binary mask of the forged region after timing smoothing. , , These represent the binary masks of the forged regions in the current keyframe, the previous keyframe, and the next keyframe, respectively.

6. The method for depth forgery detection and localization based on visual Transformer according to claim 1, characterized in that, The steps of visually marking the forged areas and providing interpretability analysis specifically include: The Canny edge detection algorithm is used to extract the contour point set from the binary mask of the fake region. The contour point set is outlined by contour lines in the original image and video frames to circle the fake region. Add text labels next to the outlines to indicate the type of fake region, and perform inter-frame linear interpolation on the labeling results of all keyframes. Generate the labeling results of intermediate frames based on the outline positions and types of adjacent keyframes. An attention weight matrix is ​​constructed based on the artifact enhancement attention weights in the DF-ViT feature extraction network and the feature fusion weights in the dual-branch output module. Based on the attention weight matrix, the top-20 patch regions that contribute the most to the authenticity determination and the top-50 feature channels with the highest weights are selected. The locations of key patch regions in the original image are visualized to obtain a heatmap of key regions in the image, so as to generate a ranking of the contribution values ​​of artifact feature channels. Output an interpretability analysis report, which includes a heatmap of key image regions, an intermodal interaction path map, and a heatmap of the location reliability of forged regions.

7. The method for depth forgery detection and localization based on visual Transformer according to claim 1, characterized in that, Following the step of performing video temporal enhancement processing based on the video temporal ViT enhancement module, the following is also included: Optimization is performed based on the joint loss function, the specific algorithm of which is as follows: ， ， ， ， ， ， in, Indicates joint loss, , , These represent the loss weights for classification loss, segmentation loss, and temporal consistency loss, respectively. Represents classification loss, Indicates the loss from partitioning. Indicates the time-series consistency loss. Represents pixel cross-entropy loss. This indicates Dice's loss. Indicates the number of samples in the batch. Represents ordinal numbers. Indicates the true label of the sample. Indicates the probability of deepfake prediction. and These represent the x and y coordinates of a pixel, respectively. This indicates a fake region's true mask. This indicates a forged region binary mask. Indicates the minimum value. This represents the binary mask region for the forged area. Indicates the keyframe sequence number. Indicates the first Patch-level local features of keyframes.

8. A system for depth forgery detection and localization based on visual Transformer, characterized in that, include: The preprocessing module is used to acquire and preprocess the target data to be detected, which includes image data and video data to be detected. The feature extraction module is used to extract features based on the DF-ViT feature extraction network to obtain global deep forgery artifact features and patch-level local features. The DF-ViT feature extraction network is based on the DF-ViT encoder block constructed based on the artifact enhancement multi-head self-attention layer and the feedforward network. The artifact enhancement multi-head self-attention layer is based on the artifact attention mask. A dual-branch output module is used to input global deep forgery artifact features and patch-level local features into the dual-branch output module to obtain classification results and segmentation results. The dual-branch output module includes a classification branch and a segmentation branch. The classification branch is based on a single fully connected layer, and the segmentation branch is based on a forgery region probability heatmap and a forgery region binary mask. The video temporal ViT enhancement module is used to perform video temporal enhancement processing, which includes temporal artifact feature extraction processing, spatiotemporal feature fusion processing, and temporal smoothing processing. The visualization analysis module is used to visually mark the forged areas and provide interpretability analysis. The visualization marks include contour extraction and outlining processing, type labeling and video smoothing processing. The interpretability analysis is based on attention weight extraction and key region and channel recognition.

9. A storage medium, characterized in that, The storage medium stores one or more programs that, when executed by a processor, implement the method for depth forgery detection and localization based on visual Transformer as described in any one of claims 1-7.

10. A computer device, characterized in that, The computer device includes a memory and a processor, wherein: The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the method for depth forgery detection and localization based on visual Transformer as described in any one of claims 1-7.

Images