Face spoofing detection model-based detection method and detection device

Abstract

The present disclosure provides a detection method and a detection device based on a face forgery detection model, the face forgery detection model comprising a BAM module, a ViR backbone network and a classifier module, the detection method comprising: obtaining an original spatial feature map by performing spatial feature extraction on a face image to be detected; obtaining an original frequency domain feature map by performing discrete cosine transform on the original spatial feature map; determining a frequency band weight for each of a plurality of frequency bands divided from the original frequency domain feature map by the BAM module, and modulating the frequency domain features in the corresponding frequency band based on the determined frequency band weight to obtain a modulated frequency domain feature map; obtaining a modulated spatial feature map based on the original spatial feature map and the modulated frequency domain feature map; performing reserved feature extraction on the modulated spatial feature map by the ViR backbone network to obtain visual reserved features; and generating a face forgery detection result of the face image based on the visual reserved features by the classifier module.

Description

Technical Field

[0001] This disclosure relates to the fields of deep learning, computer vision and / or multimedia content security technology, and more specifically, to a detection method and detection device based on a face forgery detection model. Background Technology

[0002] With the rapid development of deep synthesis technology, images and videos generated by face forgery technology are becoming increasingly realistic, which can easily lead to information security risks such as information leakage and data misuse. To address this challenge, face forgery detection technology has emerged. Although related face forgery detection methods have achieved certain results for specific datasets, they still exhibit significant limitations when facing complex real-world application scenarios, such as poor robustness, limited detection accuracy, and low detection efficiency.

[0003] The above information is presented as background technical information only to aid in understanding this disclosure. No decision or assertion has been made regarding whether any of the above content is applicable to the relevant technical aspects of this disclosure. Summary of the Invention

[0004] To at least address the problems in the related technologies, various embodiments of this disclosure propose detection methods and devices based on face forgery detection models, which can adaptively modulate and extract features according to the input image, thereby achieving efficient and accurate face forgery detection.

[0005] According to a first aspect of the embodiments of this disclosure, a detection method based on a face forgery detection model is proposed. The face forgery detection model includes a frequency band attention modulation module, a visual preservation backbone network, and a classifier module. The method includes: extracting spatial features from a face image to be detected to obtain an original spatial feature map; performing a discrete cosine transform on the original spatial feature map to obtain an original frequency domain feature map; using the frequency band attention modulation module, determining frequency band weights for multiple frequency bands divided from the original frequency domain feature map, and modulating the frequency domain features in the corresponding frequency bands based on the determined frequency band weights to obtain a modulated frequency domain feature map; obtaining a modulated spatial feature map based on the original spatial feature map and the modulated frequency domain feature map; extracting preserved features from the modulated spatial feature map using the visual preservation backbone network to obtain visually preserved features with local focus and global context awareness, wherein the visually preserved backbone network is a network based on a self-attention mechanism with spatial distance attenuation prior; and generating a face forgery detection result for the face image based on the visually preserved features using the classifier module.

[0006] Optionally, the step of determining frequency band weights for multiple frequency bands divided from the original frequency domain feature map using the frequency band attention modulation module, and modulating the frequency domain features in the corresponding frequency bands based on the determined frequency band weights to obtain a modulated frequency domain feature map includes: dividing the original frequency domain feature map into multiple frequency bands by dividing at least one frequency domain feature in the original frequency domain feature map into a frequency band by dividing it into a frequency band; determining the global statistical features of the frequency domain features for each of the multiple frequency bands, wherein, for each frequency band, the global maximum feature and global average feature of at least one frequency domain feature in the frequency band are determined, and the global statistical features of the frequency band are obtained by adding the global maximum feature and the global average feature of the at least one frequency domain feature in the frequency band; generating multiple frequency band weights corresponding to the multiple frequency bands based on the global statistical features of the multiple frequency bands using a multilayer perceptron; and modulating the frequency domain features in the original frequency domain feature map using the multiple frequency band weights to obtain the modulated frequency domain feature map.

[0007] Optionally, the multilayer perceptron is configured to: reshape all global statistical features of multiple face image samples within a batch of inputs into a single global statistical feature input; and perform weight generation processing on the single global statistical feature input using parameters shared across samples and channels to generate a frequency band weight matrix, wherein each frequency band weight in the frequency band weight matrix is ​​determined individually for each frequency band of the frequency domain feature map of each channel of each face image sample.

[0008] Optionally, the step of modulating the frequency domain features in the original frequency domain feature map using the plurality of frequency band weights to obtain the modulated frequency domain feature map includes: generating a frequency domain attention map with the same size as the original frequency domain feature map based on the plurality of frequency band weights, wherein each frequency band weight in the frequency domain attention map is mapped to a value greater than 0 and less than 1 by a sigmoid function; and performing residual modulation on the original frequency domain feature map using the frequency domain attention map to obtain the modulated frequency domain feature map, wherein the residual modulation includes multiplying the original frequency domain feature map and the frequency domain attention map element by element and adding the result of the multiplication to the original frequency domain feature map element by element.

[0009] Optionally, the step of obtaining the modulated spatial feature map based on the original spatial feature map and the modulated frequency domain feature map includes: obtaining a spatial attention map by performing inverse discrete cosine transform and normalization on the modulated frequency domain feature map; and obtaining the modulated spatial feature map by performing residual modulation on the original spatial feature map using the spatial attention map, wherein the residual modulation includes multiplying the original spatial feature map and the spatial attention map element by element and adding the result of the multiplication element by element to the original spatial feature map.

[0010] Optionally, the step of extracting retained features from the modulated spatial feature map through the visual retention backbone network to obtain visual retention features with local focus and global context awareness includes: calculating a retained self-attention map of the modulated spatial feature map based on the linear projection result and spatial distance attenuation bias of the modulated spatial feature map through the visual retention backbone network; obtaining the visual retention features based on the linear projection result and the retained self-attention map, wherein the spatial distance attenuation bias is a bias determined based on the Manhattan distance between spatial locations.

[0011] Optionally, the step of calculating the retained self-attention of the modulated spatial feature map based on the linear projection result and spatial distance attenuation bias of the modulated spatial feature map includes: calculating the width retained self-attention of each spatial feature in the m-th row of the modulated spatial feature map based on the linear projection result and the spatial distance attenuation bias; calculating the height retained self-attention of each spatial feature in the n-th column of the modulated spatial feature map based on the linear projection result and the spatial distance attenuation bias; and obtaining the retained self-attention map of the modulated spatial feature map by multiplying each width retained self-attention in the m-th row by each height retained self-attention in the n-th column.

[0012] Optionally, the step of obtaining the visually preserved feature based on the linear projection result and the preserved self-attention map includes: performing local enhancement on the values ​​in the linear projection result based on depthwise separable convolution, and obtaining the visually preserved feature by adding the preserved self-attention map to the result of the local enhancement.

[0013] According to a second aspect of the embodiments of this disclosure, a detection device based on a face forgery detection model is proposed. The face forgery detection model includes a frequency band attention modulation module, a visual preservation backbone network, and a classifier module. The detection device includes: a spatial feature acquisition unit configured to extract spatial features from a face image to be detected to obtain an original spatial feature map; a frequency domain feature acquisition unit configured to obtain an original frequency domain feature map by performing a discrete cosine transform on the original spatial feature map; and a frequency band attention modulation unit configured to determine frequency band weights for multiple frequency bands divided from the original frequency domain feature map using the frequency band attention modulation module, and based on the determined frequency band weights... The frequency domain features in the corresponding frequency band are re-modulated to obtain a modulated frequency domain feature map; a spatial feature modulation unit is configured to obtain a modulated spatial feature map based on the original spatial feature map and the modulated frequency domain feature map; a visual preservation unit is configured to extract preserved features from the modulated spatial feature map through the visual preservation backbone network to obtain visual preservation features with local focus and global context awareness, wherein the visual preservation backbone network is a network based on a preservation self-attention mechanism with spatial distance attenuation prior; and a detection result generation unit is configured to generate a face forgery detection result of the face image based on the visual preservation features through the classifier module.

[0014] According to a third aspect of the embodiments of this disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the detection method as described above.

[0015] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein instructions in the computer-readable storage medium, when executed by at least one processor, cause the at least one processor to perform the detection method as described above.

[0016] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising computer executable instructions, wherein the computer executable instructions, when executed by at least one processor, implement the detection method as described above.

[0017] In the detection method and apparatus based on the face forgery detection model according to exemplary embodiments of the present disclosure, the frequency band attention modulation of the frequency band attention modulation module enables dynamic mining and enhancement of compression-robust and forgery-related spectral components, simulating a data-driven "inverse compression" process. Furthermore, through visually preserved feature extraction from the ViR backbone network, a spatial distance attenuation prior based on Manhattan distance can be explicitly introduced into the attention mechanism, and the computational complexity can be linearized using an axial decomposition strategy, thereby achieving efficient and accurate capture of local forgery traces. This frequency-spatial domain collaborative architecture design enables the present disclosure to significantly improve its generalization ability across datasets, compression techniques, and forgery technologies while maintaining high detection accuracy.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The above and other aspects, features and advantages of certain embodiments of this disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a detection method based on a face forgery detection model according to an embodiment of the present disclosure.

[0021] Figure 2 This is a schematic diagram illustrating the structure of a face forgery detection model according to an embodiment of the present disclosure.

[0022] Figure 3 This is a schematic diagram illustrating the structure of a band attention modulation (BAM) module according to an embodiment of the present disclosure.

[0023] Figure 4 This is a schematic diagram illustrating the computational processing of a (Vision Retentive, ViR) backbone network according to an embodiment of the present disclosure.

[0024] Figure 5 This is a schematic diagram illustrating an example structure of a visually preserved ViR backbone network according to an embodiment of the present disclosure.

[0025] Figure 6 This is a structural block diagram illustrating a detection apparatus based on a face forgery detection model according to an embodiment of the present disclosure.

[0026] Figure 7 This is a structural block diagram illustrating an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0028] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are only used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0029] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0030] In recent years, face spoofing detection technology has become crucial for ensuring the authenticity and security of digital content. As described in the background section, while related methods perform well on specific datasets, their generalization ability and robustness face severe challenges when dealing with complex real-world scenarios such as unknown spoofing algorithms and strong image compression.

[0031] Specifically, the relevant face forgery detection methods are mainly divided into two categories: spatial domain (also referred to as "spatial domain" in this paper) detection and frequency domain (also referred to as "frequency domain" in this paper) detection.

[0032] In spatial domain detection, traditional methods often rely on convolutional neural networks (CNNs) or vision transformers to extract local texture inconsistencies or global semantic anomalies. However, these methods tend to overfit to visual features generated by specific tampering algorithms, resulting in poor generalization ability to unseen manipulation techniques. In other words, spatial domain detection models have poor adaptability to unknown new data. Furthermore, when the input image undergoes intensive degradation processing (such as Joint Photographic Experts Group (JPEG) compression, multiple rounds of resampling, etc.), subtle spatial domain forgery traces are often erased or covered, leading to poor robustness of spatial domain detection methods.

[0033] To compensate for the limitations of spatial domain features, frequency domain detection methods capture tampering traces by analyzing the spectral distribution of images, and have attracted attention for their ability to reveal periodic artifacts that are not observable at the pixel level. However, relevant frequency domain detection methods typically rely on predefined fixed filters (such as high-pass or low-pass filters), lacking the ability to adaptively extract task-related spectral clues. In real-world scenarios, due to the uneven distribution of feature loss caused by compression interference, this fixed frequency domain feature extraction method struggles to adaptively compensate for information loss of varying intensities, thus significantly limiting the detection accuracy of relevant methods in highly compressed environments.

[0034] Furthermore, the high-performance spatial backbone networks (such as the Transformer series) incur huge computational overhead due to the quadratic computational complexity when processing high-resolution face images. Moreover, these models often ignore the local clustering characteristics of forgery traces in spatial distribution and lack effective spatial prior guidance, making it difficult to balance efficiency and accuracy when extracting subtle forgery clues.

[0035] In summary, among the relevant face forgery detection technologies, spatial domain detection methods are susceptible to compression interference, while frequency domain detection methods lack adaptability due to the use of fixed filtering strategies. The computational overhead of high-performance visual models also limits their application on high-resolution images.

[0036] In view of the problems existing in the related technologies, including at least the above and / or other aspects, according to the embodiments of this disclosure, a face forgery detection model (or framework) based on the Band Attention Modulation (BAM) mechanism and the Vision Retentive (ViR) backbone network with spatial distance attenuation attention is proposed, as well as a detection method and detection device based on the face forgery detection model.

[0037] The face forgery detection model according to embodiments of this disclosure can adaptively weigh and apply the weights of different frequency bands based on the frequency domain features (also referred to herein as "frequency domain characteristics" or "spectral features (or characteristics)") of the input face image to achieve dynamic enhancement of key forgery clues, and can also simulate an "inverse compression" process to compensate for information loss caused by degradation. Furthermore, the face forgery detection model according to embodiments of this disclosure combines a ViR backbone network with spatial distance attenuation attention (hereinafter also referred to as a "ViR network (or module)") to guide the face forgery detection model to focus on local abnormal regions using spatial priors, and can also reduce computational complexity through axial decomposition calculations. Through the implementation of a detection method or detection device based on a face forgery detection model that combines learnable adaptive frequency domain enhancement with efficient spatial domain modeling with spatial priors, embodiments of this disclosure can improve the generalization ability to unknown forgery techniques, maintain extremely high detection robustness under high-intensity compression environments, and achieve efficient and accurate face forgery detection. The following will combine... Figures 1 to 7 To describe various exemplary embodiments according to this disclosure.

[0038] Figure 1 This is a flowchart illustrating a detection method based on a face forgery detection model according to an embodiment of the present disclosure. Figure 2 This is a schematic diagram illustrating the structure of a face forgery detection model according to an embodiment of the present disclosure. (Refer to...) Figure 2 The face forgery detection model according to embodiments of this disclosure may include a BAM module 300, a ViR backbone network 400, and a classifier module 500, which will be described below in conjunction with... Figure 2 right Figure 1 The detection method will be described in detail.

[0039] Reference Figure 1 In operation S110, the face image to be detected (such as, Figure 2 Spatial feature extraction is performed on the face image (200) to obtain the original spatial feature map (such as, Figure 2 Multiple original spatial features in Figure 210).

[0040] As an example, an image or video frame to be detected can be acquired, and a face image to be detected can be obtained based on the image or video frame. For example, an input image or video frame can be acquired via user input or any other means, and a face region can be located by using relevant face detection algorithms (such as the single-stage face detection model RetinaFace, Multi-Task Cascaded Convolutional Network (MTCNN, etc.) to detect the face region. Then, the located face region can be processed by keypoint alignment, affine transformation, and / or image cropping to obtain a standardized face image to be detected with uniform size and pose normalization.

[0041] As an example, spatial feature extraction may include extracting raw spatial features from a face image using a spatial domain encoder and obtaining a raw spatial feature map that includes the raw spatial features, where the spatial features may include spatial feature information such as the image's location and structural features.

[0042] As an example, before spatial feature extraction, a face image can be separated into face images for C channels, and a spatial domain encoder can be used to obtain the original spatial feature maps for each of the C channels, such as... Figure 2 The original spatial feature map 210 for the three color channels of red (R), green (G), and blue (B) is shown in the figure. As another example, after spatial feature extraction, the single original spatial feature map extracted from the face image can be separated into original spatial feature maps for C channels, such as... Figure 2 The original spatial feature maps 210 for the three color channels of red (R), green (G), and blue (B) are shown in the figure. In other words, channel separation can be achieved before or after spatial feature extraction, and this disclosure is not limited thereto. According to embodiments of this disclosure, the face forgery detection model can perform the same processing on the spatial feature maps corresponding to each channel of each face image. Therefore, the various processing details of the spatial feature map of a single channel of a single face image described below can be applied to the feature maps of each channel without needing to be repeated. It should be understood that the corresponding descriptive details can be applied to the spatial feature maps of other face images and / or other channels.

[0043] As an example, the original spatial feature map can be in the form of a tensor (or understood as a "multidimensional array" or "multidimensional feature matrix"), and can be represented as... ,in, R This can represent that each element in tensor data is a real number. B It can represent the batch size, that is, the number of samples in a batch (i.e., the number of input face images). C It could be the number of channels. HIt can be the height of the image (or the number of pixels in the vertical direction). W It can be the width of the image (or the number of pixels in the horizontal direction).

[0044] As an example, the height of a standardized face image H The width W can be the same. In the following text, for ease of description, it is assumed that the original feature map is... W and H Similarly, the corresponding feature maps in subsequent processing... W It can also be expressed as H Additionally, for ease of description, unless specifically for a batch description, B can be omitted or omitted. B Assuming 1, based on this assumption, the original spatial feature map can also be represented as .

[0045] According to embodiments of the present disclosure, the BAM module 300 can adaptively enhance the most effective components for face forgery detection in the image spectrum in a data-driven manner. Therefore, the BAM module 300 can be used as a front-end processor of the backbone network for face forgery detection instead of conventional manual frequency domain filters. However, before the front-end processing based on the BAM module 300, the original spatial feature map is first transformed so that it can be processed by the BAM module 300.

[0046] In operation S120, the original frequency domain feature map (such as, ...) is obtained by performing a Discrete Cosine Transform (DCT) on the original spatial feature map. Figure 2 The original spectral characteristics (Figure 220).

[0047] As an example, DCT can be performed on the original spatial feature map of each channel to obtain the DCT spectrogram as the original frequency domain feature map. Therefore, in this disclosure, the frequency domain feature map is also referred to as a "spectral map" or "spectral representation". In the frequency domain feature map, the value of the horizontal axis can represent the width or the frequency index in the horizontal direction (simply referred to as the horizontal frequency index). u The value on the vertical axis can represent the frequency index in the height or vertical direction (referred to as the vertical frequency index). v And the location of the frequency domain feature map ( u, v The value at ) S(u, v) Can represent AND ( u, v The magnitude of the DCT coefficients of the corresponding frequency components, where, u =0,1,…,W-1, v =0, 1,…, H-1. Therefore, in this disclosure, the frequency domain feature map... S(u, v) It is also known as "frequency domain coefficient", "bandwidth coefficient" or "frequency domain feature".

[0048] As an example, the original frequency domain feature map can be in tensor form and can be represented as ,in, D(x) This can represent a feature map x Perform DCT processing. Furthermore, as mentioned above, W It can also be represented as H .

[0049] Once a frequency domain feature map that can be processed by the BAM module 300 is obtained in operation S120, the obtained frequency domain feature map can be input into the BAM module 300 to obtain a frequency domain feature map with enhanced frequency domain features.

[0050] In operation S130, the BAM module determines the frequency band weights for each of the multiple frequency bands divided from the original frequency domain feature map, and modulates the frequency domain features in the corresponding frequency bands based on the determined frequency band weights to obtain the modulated frequency domain feature map. Figure 3 This is a schematic diagram illustrating the structure of the BAM module according to this disclosure. The following is in conjunction with... Figure 3 The operation S130 is described in detail.

[0051] BAM Operations of the BAM Module Reference Figure 3 When the original frequency domain feature map 220 obtained through operation S120 is input into the BAM module 300, the original frequency domain feature map 220 of each channel is divided into multiple frequency bands through frequency band division. The frequency band division is described in detail below.

[0052] BAM frequency band division As an example, embodiments of this disclosure propose dividing the frequency domain feature map into frequency bands according to the anti-diagonal direction of the frequency domain feature map in spectrum analysis.

[0053] As an example, the frequency domain feature map that satisfies the "horizontal frequency index" u +Vertical Frequency Index v =constant k Frequency domain characteristics of " S(u, v) They are assigned to the same frequency band, among which, k = 0, 1, ... , W + H -2. For example, when W = H hour, k = 0, 1,... , 2 H -2, in this case, as a result of frequency band division, each original frequency domain feature map 220 is divided into N =2 H -1 frequency band.

[0054] Each frequency band determined by the above frequency band division method can contain frequency components with the same spatial frequency sum (i.e., similar energy characteristics), and the frequency domain characteristics contained in each frequency band can be represented as a matrix. ,in, L k It can represent the first k The number of frequency domain features contained in the frequency band.

[0055] Therefore, according to an embodiment of the present disclosure, operation S130 may include: dividing the original frequency domain feature map into multiple frequency bands by dividing at least one frequency domain feature in the original frequency domain feature map into a frequency band by dividing it into a frequency band by dividing at least one frequency domain feature in the original frequency domain feature map into a frequency band with the same sum of the frequency index in the horizontal direction and the frequency index in the vertical direction.

[0056] As an example, when dividing the frequency bands as described above, an overlapping frequency band strategy can also be introduced. For example, by allowing partial frequency overlap between adjacent frequency bands, the smoothness and robustness of the features at the frequency domain boundary can be further enhanced.

[0057] Subsequently, in order to characterize the global statistical features of the frequency domain features of multiple frequency bands, embodiments of this disclosure further propose extracting complementary global maximum features and global average features from multiple frequency bands to obtain the global statistical features of the frequency bands based on both. This will be described in detail below.

[0058] Feature extraction of BAM Reference Figure 3 In operation S321, the feature map 220 can be divided from an original frequency domain feature map. N The first frequency band k Perform max pooling (MaxPool) on the frequency band to obtain the first... k Global max pooling features of the frequency band 311.

[0059] As an example, the first k The largest frequency domain feature among all frequency domain features included in the frequency band is determined as the global max pooling feature. Therefore, in this disclosure, the global max pooling feature is also referred to as the "global maximum feature".

[0060] For example, the first equation can be determined by the following equation 1. k Global max pooling features of the frequency band : ...Equation 1 in, j It is greater than or equal to 1 and less than or equal to 1. L k an integer, and .

[0061] In operation S322, the N frequency bands divided from an original frequency domain feature map 220 can be processed. k Band average pooling (AvgPool) is used to obtain the first... k Global average pooling characteristics of frequency bands 312.

[0062] As an example, the first k The average frequency domain feature of all frequency domain features included in the frequency band is determined as the global average pooling feature. Therefore, in this disclosure, the global average pooling feature is also referred to as the "global average feature".

[0063] For example, the first equation can be determined by the following equation 2. k Global average pooling characteristics of frequency bands : ...Equation 2 in, j It is greater than or equal to 1 and less than or equal to 1. L k an integer, and .

[0064] By using a global average pooling layer to compress the frequency domain features within each frequency band, feature vectors reflecting the global energy distribution of the corresponding frequency band can be extracted.

[0065] In operation S323, global max pooling features are used. 311 and Global Average Pooling Features 312 are added together to obtain global statistical features. .

[0066] As an example, it can be achieved by taking the first k The global maximum feature of the frequency band and the first k The global average characteristics of the frequency bands are added together to form the first k Compact descriptors of the frequency band (or "descriptor vectors"), i.e., For example, the first equation can be determined by the following equation 3. k Global statistical characteristics of the frequency band : ...Equation 3 As an example, the frequency band feature matrix for the original frequency domain feature map 220 can be obtained by stacking the descriptors of N frequency bands divided from the original frequency domain feature map 220. F This is used for subsequent processing. For example, the frequency band feature matrix for an original frequency domain feature map 220 can be determined by the following equation 4. F : ...Equation 4 Therefore, according to embodiments of this disclosure, operation S130 may further include: determining global statistical features of frequency domain features for each of the plurality of frequency bands, wherein, for each frequency band, a global maximum feature and a global average feature of at least one frequency domain feature in the frequency band are determined, and the global statistical features of the frequency band are obtained by adding the global maximum feature and the global average feature of at least one frequency domain feature in the frequency band.

[0067] As an example, in addition to using global pooling features from a global pooling layer, more complex statistical features (such as variance or higher-order moments) can be used as global statistical features, for example, based on the first... k The maximum variance and average variance of the frequency domain characteristics of the frequency band are used to determine the first... k The variance characteristics of the frequency bands, etc., can be used to more comprehensively characterize the coefficient distribution of each frequency band. That is, this disclosure does not restrict the global statistical characteristics.

[0068] Continue to refer to Figure 3 To learn the importance of each frequency band, embodiments of this disclosure also propose designing a lightweight weight generation network in the BAM module to adaptively learn the weights. This is described below.

[0069] BAM's Adaptive Weight Learning According to embodiments of the present disclosure, the weight generation network 330 can generate weights based on the global statistical features obtained in operation S323. Or based on global statistical characteristics The determined frequency band feature matrix F For each frequency band, a weight of 331 is generated.

[0070] As an example, the weight generation network 330 may be a two-layer multilayer perceptron (MLP) that can share parameters between different input samples (i.e., samples of face images) and different channels, and generates independent frequency band weights for the feature maps of each channel of each sample. Therefore, according to embodiments of this disclosure, operation S130 may further include: generating multiple frequency band weights corresponding to the multiple frequency bands based on global statistical features of multiple frequency bands through the multilayer perceptron.

[0071] Specifically, preprocessing can be performed on multiple inputs (such as multiple global statistical features or frequency band feature matrices) before weight generation is performed by the multilayer perceptron. For example, during model training, the multilayer perceptron can perform weight generation on the inputs in batches. In this case, the multiple inputs can first be reshaped into a single input that is easy for the multilayer perceptron to process, for example, a full frequency band feature matrix based on all global statistical features of all channels of all frequency bands of multiple face image samples within a batch. F Reshape into a feature matrix ,in, B For batch size (such as, B =32), C For the number of channels (such as, C =3), N This represents the number of frequency bands in a frequency domain feature map.

[0072] Then, the reshaped feature matrix can be... Inputting data into a multilayer perceptron generates frequency band weights. For example, the weight generation process for a multilayer perceptron can be represented by the following expression 5: ...expression 5 Where MLP() can represent the weight generation process of a multilayer perceptron, FC can represent the processing of a fully connected layer, FC1 can correspond to the processing of the first fully connected layer, FC2 can correspond to the processing of the second fully connected layer, BN can represent the batch normalization process, and GELU can represent the processing of the Gaussian error linear unit activation function.

[0073] In other words, the feature matrix The weight matrix can be generated by sequentially processing the first fully connected layer, batch normalization layer, activation layer, and fully connected layer. .

[0074] As an example, in consideration of computational costs, the MLP according to embodiments of this disclosure can further increase its depth or introduce skip connections to enhance its nonlinear modeling capabilities. In other words, the above-described number of layers and structure of the multilayer perceptron are merely illustrative examples, and this disclosure does not impose any limitations on them.

[0075] As an example, in the above adaptive weight generation, a cross-band joint learning strategy can also be adopted to improve the accuracy of the model in identifying abnormal spectral shifts by modeling the correlation between different frequency bands.

[0076] Then, the output of the weight generation network 330 can be used. Reshape the weight matrix into tensor form ,in, W elements inw b,c,k Representation matrix W The first in b Sample (i.e., the first sample in the current batch) b (personal face image) c The first frequency domain feature map of the channel k Frequency band weighting.

[0077] Therefore, according to embodiments of this disclosure, a multilayer perceptron can be configured to: reshape all global statistical features of multiple face image samples within an input batch into a single global statistical feature input; and perform weight generation processing on the single global statistical feature input using parameters shared across samples and channels to generate a frequency band weight matrix, wherein each frequency band weight in the frequency band weight matrix is ​​determined individually for each frequency band of the frequency domain feature map for each channel of each face image sample. In other words, the multilayer perceptron according to embodiments of this disclosure can process descriptors of all frequency bands under all channels of all samples in a batch processing manner, the parameters of the multilayer perceptron are shared across samples and channels, but can output independent frequency band weights for each frequency band of each channel of each sample.

[0078] According to embodiments of this disclosure, operation S130 may further include: modulating the frequency domain features in the original frequency domain feature map by using multiple frequency band weights to obtain a modulated frequency domain feature map. This will be described in detail below.

[0079] Frequency Domain Feature Modulation and Reconstruction of BAM To apply band weights to a frequency domain feature map to modulate that feature map, refer to Figure 3 In operation S341, a frequency domain attention map 342 with the same size as the original frequency domain feature map can be generated based on multiple frequency band weights 331, and the frequency band weights can be normalized by mapping each frequency band weight in the frequency domain attention map 342 to the (0, 1) interval using a sigmoid function.

[0080] As an example, the weight matrix W Expanded into a feature map similar to the original frequency domain feature map S Two-dimensional frequency domain attention maps with the same size (or dimensions, shape) The expansion rule could be, for example, adjusting the weights. w b,c,k Assigned to the two-dimensional frequency domain attention map M The first in Elements at all positions on the anti-diagonal (i.e., the anti-diagonal corresponding to the k-th frequency band).

[0081] Then, a residual modulation strategy can be employed, utilizing an extended two-dimensional frequency domain attention map.M For the original frequency domain feature map S Frequency domain information enhancement is performed. For example, residual modulation can be achieved using the following expression 6: ...Expression 6 in, It can represent element-wise multiplication. This can represent the normalization process of the Sigmoid function, which is used to map the learned frequency band weights to the (0, 1) interval.

[0082] For example, refer to Figure 3 In operation S343, the frequency domain attention map normalized by the sigmoid function is multiplied element-wise with the original frequency domain feature map 220. In operation S344, the result of the element-wise multiplication is added element-wise to the original frequency domain feature map 220 to output the modulated frequency domain feature map after modulation by the BAM module. 230.

[0083] Therefore, according to embodiments of this disclosure, the step of modulating the frequency domain features in the original frequency domain feature map by using multiple frequency band weights to obtain a modulated frequency domain feature map may include: generating a frequency domain attention map with the same size as the original frequency domain feature map based on multiple frequency band weights, wherein each frequency band weight in the frequency domain attention map is mapped to a value greater than 0 and less than 1 by a sigmoid function; and performing residual modulation on the original frequency domain feature map by using the frequency domain attention map to obtain a modulated frequency domain feature map, wherein the residual modulation includes multiplying the original frequency domain feature map and the frequency domain attention map element by element and adding the result of the multiplication to the original frequency domain feature map element by element.

[0084] In the above modulation process, the weight generation network consisting of two linear layers can perform nonlinear mapping of frequency domain features and use the Sigmoid activation function to output the modulation frequency band weights of each frequency band, so that the output frequency band weights are dynamically adjusted according to the degradation of the input face image, in order to simulate the "inverse compression" compensation process for compression loss.

[0085] Furthermore, in the aforementioned frequency domain feature enhancement based on residual modulation, the generated dynamic frequency band weights are multiplied element-wise with the original frequency domain coefficients to highlight the statistical features left by the forgery algorithm on the spectrum and suppress low-discrimination noise components.

[0086] Modulating the frequency domain features in the original frequency domain feature map as described above means that for frequency bands with weights close to 1, their spectral components are enhanced, while for frequency bands with weights close to 0, their spectral components remain essentially unchanged. This modulation method based on dynamic weights ensures the stability of the modulation and avoids gradient vanishing or exploding.

[0087] In this way, the BAM module can divide the frequency domain feature map into multiple independent frequency bands along the anti-diagonal direction. It extracts global statistical information for each frequency band using a lightweight MLP, dynamically calculates the weight of each band, and thus adaptively enhances the mid-to-high frequency components associated with forgery traces and compensates for spectral signal loss caused by image compression. In other words, BAM aims to highlight forgery traces that are compressed or masked by noise from a frequency domain perspective.

[0088] Return to reference Figure 1 In operation S140, a spatial feature map of modulation is obtained based on the original spatial feature map and the frequency domain feature map of modulation. This will be described in detail below.

[0089] Spatial Feature Modulation As an example, in order to seamlessly integrate the enhanced frequency domain features and spatial features, a frequency-domain enhanced image representation (i.e., a spatial attention map) can be obtained by reconstructing or mapping the modulated frequency domain feature map back to the spatial domain. The frequency-domain enhanced image representation is then applied to the original spatial feature map to integrate the modulated frequency domain features into the spatial features, thereby forming a frequency-space dual-domain discriminative feature (i.e., a modulated spatial feature map) that can be used as input to the subsequent backbone network.

[0090] Reference Figure 2 The frequency domain feature map 230 of the modulation output by the BAM module 300 can be subjected to inverse discrete cosine transform (iDCT) and normalization to obtain the corresponding spatial attention map 240 in the spatial domain.

[0091] For example, the modulated frequency domain feature map 230 can be transformed into an intermediate spatial attention map (not shown) using iDCT. Then, the intermediate spatial attention map after iDCT can be normalized using a sigmoid function to map each spatial attention in the intermediate spatial feature map to the (0, 1) interval, thereby obtaining a spatial attention map 240 for modulating the spatial feature map. Here, the spatial attention map 240 can indicate the potential forgery regions in the original spatial feature map that need to be enhanced.

[0092] Then, the spatial attention map 240 is used to perform residual modulation on the corresponding original spatial feature map 210 to obtain the modulated spatial feature map 241.

[0093] As an example, residual modulation may include element-wise multiplying the spatial attention map 240 with the corresponding original spatial feature map 210, and adding the result of the element-wise multiplication with the corresponding original spatial feature map to obtain the modulated spatial feature map 241.

[0094] As an example, the modulated frequency domain feature map 230 can also be mapped to the spatial domain dimension and fused with the spatial feature map through methods such as feature concatenation or residual addition to achieve spatial feature modulation. This disclosure does not limit this.

[0095] As an example, when channel separation is performed in operation S110, residual modulation is performed on the corresponding original spatial feature map 210 using the spatial attention map 240 for each channel to obtain the modulated spatial feature map of that channel, and based on... C The modulation spatial feature map 241 is obtained by merging the modulation spatial feature maps of each channel. This modulation spatial feature map can be represented as... For example, the original spatial feature map of the R channel can be residually modulated using the spatial attention map of the R channel to obtain the modulated spatial feature map of the R channel. Similarly, the modulated spatial feature maps of the G channel and the B channel can be obtained separately, and then the modulated spatial feature maps of the three channels can be merged to obtain the modulated spatial feature map that will be input into the subsequent ViR backbone network for processing.

[0096] Therefore, according to embodiments of this disclosure, operation S140 may include: obtaining a spatial attention map by performing an inverse discrete cosine transform and normalization on the modulated frequency domain feature map; and obtaining a modulated spatial feature map by performing residual modulation on the original spatial feature map using the spatial attention map, wherein the residual modulation includes multiplying the original spatial feature map and the spatial attention map element by element and adding the result of the multiplication element by element to the original spatial feature map.

[0097] By introducing a gated mechanism through this frequency-space feature fusion method, the detection method can dynamically adjust the degree of intervention of frequency domain features according to the quality of spatial features, emphasizing spatial domain features in high-quality images and emphasizing frequency domain features in low-quality and / or heavily compressed images.

[0098] After obtaining the modulated spatial feature map, this disclosure also proposes to use the ViR backbone network to extract deep spatial context features.

[0099] Continue to refer to Figure 1 In operation S150, the ViR backbone network is used to extract preserved features from the modulated spatial feature map to obtain visually preserved features with local focus and global context awareness. The ViR backbone network is a network based on a self-attention mechanism with spatial distance attenuation prior. This is described below.

[0100] Perceptual Preservation Feature Extraction Operations in ViR Backbone Networks Self-attention mechanism with spatial distance decay prior As an example, a preservative self-attention mechanism with a spatial distance decay prior can indicate that the preservative self-attention computation of the ViR backbone network can use a predetermined spatial distance decay bias.

[0101] As an example, spatial distance attenuation bias can be used. B ij Defined as position in a spatial feature map p i =( x i , y i ) and another position p j =( x j , y j The Manhattan distance between them is a function and can be based on a spatial distance decay bias. B ij Determine the spatial distance attenuation bias matrix B .

[0102] For example, the spatial distance attenuation bias can be defined by the following Equation 7. B ij : ...Equation 7 in, It is a learnable decay factor.

[0103] As an example, for a specific location p i Spatial distance attenuation bias matrix B In and position p i Correspondingly B i Can be defined as position With all other positions p i Spatial distance attenuation bias B ij The sum of all spatial distance attenuation biases at a given location and all other locations can be defined as the spatial distance attenuation bias at that location in the spatial distance attenuation bias matrix. In this way, the spatial distance attenuation bias matrix can be determined by traversing the locations in the spatial feature map. B As a priori for spatial distance attenuation.

[0104] As an example, in order for the different attention heads used for self-attention scoring to focus on contexts at different scales, each attention head... h They can have independent attenuation factors. For example, the attenuation factor can be set using the following equation 8: ...Equation 8 in, and This is a hyperparameter.

[0105] Therefore, according to embodiments of this disclosure, the spatial distance attenuation bias can be an bias determined based on the Manhattan distance between spatial locations, such as the product of the Manhattan distance between spatial locations and the logarithm of the attenuation factor. However, the spatial distance attenuation bias can also be based on a function of Euclidean distance or other distances, and this disclosure is not limited thereto.

[0106] Furthermore, according to embodiments of this disclosure, the multiple attention heads used to calculate the self-attention can have different decay factors set exponentially, allowing different attention heads to have differentiated spatial receptive fields. In other words, the model can adaptively adjust the learnable decay rate according to the semantic depth at different levels, thereby optimizing feature extraction performance under different receptive fields.

[0107] After describing the definition of spatial distance decay prior, the following describes the Retentive Self-Attention (ReSA) mechanism with spatial distance decay prior.

[0108] According to an embodiment of this disclosure, operation S150 may include: calculating the ReSA map of the modulated spatial feature map based on the linear projection result of the modulated spatial feature map and the spatial distance attenuation bias through the ViR backbone network.

[0109] As an example, the input spatial feature map (such as the modulated spatial feature map mentioned above) can be processed. X A linear projection is performed to obtain the corresponding linear projection result, which includes a matrix of query (Q), key (K), and value (V). To incorporate positional information, rotational position embedding (RoPE) is also performed on Q and V to obtain a matrix of rotated-encoded query. Q r and rotation encoding key K r .

[0110] For example, the self-attention retention can be calculated based on the linear projection of the input spatial feature map, with the introduction of a spatial distance attenuation bias, using the following expression 9 regarding the ReSA computation process: ...Expression 9 in, It is the dimension of the key vector. X It can be the input spatial feature map, such as the modulated spatial feature map mentioned above.

[0111] By introducing a spatial distance attenuation bias into the ReSA calculation process as described above. B Attention weights between distant locations are exponentially suppressed, forcing the model to focus more on neighboring regions. This aligns well with the physical property that forgery traces (such as seams, texture inconsistencies, etc.) tend to cluster locally. In other words, this explicit encoding of the spatial prior of "focusing on local" in the attention scoring mechanism causes the model to explicitly favor local features when modeling the global context.

[0112] After obtaining the ReSA map in the manner described above, according to an embodiment of this disclosure, operation S150 may further include: obtaining visually preserved features based on the linear projection result and the ReSA map.

[0113] As an example, to further enhance the ability to model local details, this disclosure also introduces local enhancements to the spatial feature map after ReSA computation. This is described below.

[0114] Local feature enhancement According to embodiments of this disclosure, the step of obtaining visually preserved features based on linear projection results and ReSA maps includes: performing local enhancement on the values ​​in the linear projection results based on depth-separable convolution, and obtaining visually preserved features by adding the ReSA map to the results of the local enhancement.

[0115] For example, visually preserved features can be obtained through the following expression 10. Output : ...expression 10 in, DWConv() This can represent performing depthwise separable convolutions on the values ​​of the input spatial feature map, such as convolution kernels of 5×5. ReSA() can be a self-attention computation process preserved as defined in expression 9. X It can be the input spatial feature map.

[0116] The aforementioned processing for obtaining visually preserved features can be understood as local augmented location coding, which allows the model to possess strong long-range dependency modeling capabilities without sacrificing sensitivity to capturing subtle local patterns.

[0117] Furthermore, the two-dimensional ReSA map of the spatial feature map of the above-mentioned computational input can have The complexity of this process can make it difficult to handle high-resolution input images. To address this, this disclosure also proposes an Axial Decomposition computational strategy, which is described below.

[0118] Axial decomposition calculation strategy (ReSA-A) Specifically, to reduce computational complexity, instead of calculating the ReSA of every spatial feature in the input spatial feature map element by element, only the ReSA of a row of spatial features in the width (or horizontal) direction and the ReSA of a column of spatial features in the height (or vertical) direction are calculated. The ReSA of the spatial feature at each location in the spatial feature map is obtained based on the products of each ReSA in the row and each ReSA in the column. In other words, the two-dimensional ReSA calculation process can be axially decomposed into two one-dimensional ReSA calculation processes in two directions. In this disclosure, this axially decomposed ReSA calculation process can be simply referred to as ReSA-A. Figure 4 An example of ReSA-A processing according to an embodiment of this disclosure is shown.

[0119] Reference Figure 4 Using Expression 9, based on the linear projection result and the spatial distance attenuation bias, the ReSA of each element in the m-th row of the input spatial feature map is calculated to obtain the width-preserving self-attention W_att. Here, the spatial distance attenuation bias can also be simplified accordingly to... .

[0120] Then, using Expression 9, based on the linear projection result and the spatial distance attenuation bias, the ReSA (Retained Self-Attention) of each element in the nth column of the input spatial feature map is calculated, i.e., the height-direction preserved self-attention H_att. Here, the spatial distance attenuation bias can also be simplified accordingly to... .

[0121] Finally, matrix multiplication is performed by multiplying each width-direction self-attention in the m-th row with the height-direction self-attention in the n-th column, thus obtaining the final matrix form of the ReSA map. For example, by multiplying the ReSA of the first column of pixels from left to right of W_att with the ReSA of the first row of pixels from top to bottom of H_att, the ReSA at the intersection of the first column from left to right and the first row from top to bottom in ReSA map 410 (i.e., the top-left pixel position) is obtained. Similarly, all ReSAs in ReSA map 410 can be obtained.

[0122] Here, it is assumed that a higher ReSA value (i.e., score) corresponds to a higher brightness in the corresponding pixel area. (Refer to...) Figure 4After ReSA-A calculation processing of the modulated spatial feature map 241, the brightness of the middle pixel region of the generated ReSA map 410 is the highest, so the middle pixel region is the most concerned.

[0123] Furthermore, although the above only shows the ReSA plot after ReSA processing and... DWConv(V) The results are added together; however, the ReSA image after ReSA-A processing can also be added to... DWConv(V) The results are summed, and this disclosure will not repeat the description of this.

[0124] Therefore, according to embodiments of this disclosure, the step of calculating the retained self-attention of the modulated spatial feature map based on the linear projection result and spatial distance attenuation bias of the modulated spatial feature map includes: calculating the width retained self-attention of each spatial feature in the m-th row of the modulated spatial feature map based on the linear projection result and spatial distance attenuation bias; calculating the height retained self-attention of each spatial feature in the n-th column of the modulated spatial feature map based on the linear projection result and spatial distance attenuation bias; and obtaining the retained self-attention map of the modulated spatial feature map by multiplying each width retained self-attention in the m-th row by each height retained self-attention in the n-th column.

[0125] In this way, ReSA-A performs one-dimensional ReSA calculations in two directions. Each pixel can ultimately interact indirectly with all other pixels in the image through "row propagation" and "column propagation," preserving the global receptive field. At the same time, ReSA-A's axial decomposition can reduce the computational complexity to ,when and When the time complexity is approximately linear, it is approximately equal to that of a linear time complexity. This significantly improved efficiency.

[0126] Furthermore, although the above example illustrates the process of first calculating the width-direction self-attention and then calculating the height-direction self-attention, this disclosure does not restrict the order of the directions for calculating one-dimensional self-attention. It is also possible to first calculate the height-direction self-attention and then calculate the width-direction self-attention, or to perform both calculations simultaneously.

[0127] To capture more complex global interactions while controlling computational load, this disclosure also proposes to implement the ViR backbone network using a four-stage pyramid architecture. Figure 5 This is a schematic diagram illustrating an example structure of a ViR backbone network according to an embodiment of the present disclosure.

[0128] Reference Figure 5According to embodiments of this disclosure, the ViR backbone network 400 can be a four-stage pyramid structure network, for example, it may include a patch embedding module 510, three consecutive ReSA-A modules 520, a ReSA module 530, and a feature projection module 540. As an example, the processing of each module can reduce the data size by half.

[0129] As an example, the block coding module 510 can be configured to block and encode the input modulated spatial feature map 241, so that the input spatial feature map 241 is converted into a form that can be processed by the subsequent ReSA-A module 520.

[0130] As an example, each ReSA-A module 520 may include a ReSA-A block and a convolutional block. The ReSA-A block may perform one-dimensional ReSA computation and matrix multiplication in two directions as described above on the input (i.e., the output of the block encoding module 510), and the convolutional block may perform local enhancement based on depthwise separable convolution as described above on the value obtained based on the linear projection of the input. Finally, the ReSA-A module 520 may add the output of the ReSA-A block to the output of the convolutional block. Output Output. Although not shown, the output of the first ReSA module 530 can be input to the second ReSA module 530, and similarly, the output of the second ReSA module 530 can be input to the third ReSA module 530.

[0131] As an example, ReSA module 530 may include a ReSA block and a convolutional block, wherein the ReSA block may perform the two-dimensional ReSA calculation processing as described above on the input (i.e., the output of the third ReSA module 530), and the convolutional block may perform local enhancement based on depth-separable convolution on the value obtained based on the linear projection of the input; finally, ReSA module 540 may add the output of the ReSA block to the output of the convolutional block. Output Output.

[0132] As an example, the feature projection module 540 can be configured to map the features of the output of the ReSA module 530 to the original dimension.

[0133] In the above structure, the computational load can be controlled using an efficient ReSA-A module in the first three high-resolution stages, and in the last low-resolution stage, a full 2D ReSA can be used to capture more complex global interactions. Furthermore, downsampling and channel expansion are performed between stages through stride convolution.

[0134] As an example, the axial linearization process of the three ReSA-A modules 520 can employ alternating direction transformations, for example, by alternately performing the computation order of "horizontal first then vertical" or "vertical first then horizontal" in different network layers (i.e., each ReSA-A module 520) to enhance the model's ability to cover irregularly shaped fake regions.

[0135] It should be understood that this disclosure does not limit the number of ReSA-A modules and / or the execution order of the axial decomposition directions of the ReSA-A modules.

[0136] As mentioned above, axial decomposition of the two-dimensional ReSA computation process can reduce the computational complexity from quadratic to linear, achieving linearized time complexity feature extraction on large-size feature maps, thereby significantly improving the inference speed of the model in high-resolution face detection scenarios.

[0137] In addition, the ViR backbone network integrates multi-scale features output by multi-layer ReSA-A and / or ReSA modules to form a unified expression of the contradiction between local forgery artifacts and global structure, which serves as the basis for final classification.

[0138] Furthermore, by using the preserved self-attention mechanism of the ViR backbone network to replace the traditional self-attention mechanism, the processing efficiency can be improved by utilizing the cyclic properties of the preserved operators while capturing the global context.

[0139] Continue to refer to Figure 1 In operation S160, the classifier module generates face forgery detection results for face images based on visually preserved features.

[0140] As an example, the visually preserved features output by the ViR backbone network can be fed into a lightweight classification head after being aggregated by global average pooling. For example, the classification head can be composed of fully connected layers and a Softmax function, and the classification head can output a prediction probability value between 0 and 1. This prediction probability value can be used to represent the credibility of a face image as a fake image, thereby realizing face forgery detection.

[0141] Model training According to embodiments of this disclosure, the training of the face spoofing model can employ a standard supervised learning paradigm. During model training, for a batch of training data, a loss function consisting of binary cross-entropy loss can be used. L End-to-end training of the face spoofing model: The face image to be detected is sequentially enhanced in the frequency domain by the BAM module, spatial context modeling is performed by the ViR backbone network, and finally the predicted probability is output by the classification head.

[0142] For example, loss function LAs shown in expression 11: ...Expression 11 in, It can represent the true label of the input face image (where 0 is real and 1 is fake). The human element can represent the probability of forgery predicted by the face forgery model. It can indicate the batch size.

[0143] As an example, the model can be trained using the Adam optimizer and cosine annealing learning rate scheduling. By being fully trained on large-scale mixed datasets (such as different compressed versions of FF++, Celeb-DF, and subsets of the DeepFake Detection Challenge (DFDC), the face spoofing model learns compression-robust, multi-spoofing-technique-sensitive generalized feature representations, thus achieving excellent cross-domain generalization capabilities. In particular, the dedicated face spoofing detection dataset used for model training combines various mainstream face spoofing detection benchmarks (such as FaceForensics++, Celeb-DF, DFDC, etc.) and covers various spoofing techniques such as deepfakes, face swapping, face2Face, and neural textures. Data augmentation is performed by applying JPEG compression with different quality factors, Gaussian blur, and multi-scale resampling to simulate the degradation environment in the actual social media platform propagation process, ensuring the model's generalization and robustness to different compression levels and unknown spoofing algorithms.

[0144] It is understood that the BAM module 300, ViR backbone network 400, and classifier module 500 described in the embodiments of the present invention can be implemented as software components (such as software algorithm modules) executing on a general-purpose or special-purpose processor, or as hardware components (such as dedicated hardware acceleration modules (such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs))) to achieve higher detection efficiency, or as a combination of software and hardware components. Furthermore, the BAM module 300, ViR backbone network 400, and classifier module 500 can be communicatively connected to each other to transmit signals or data.

[0145] It should be noted that the images (including but not limited to input images, video frames, or facial images) and data (including but not limited to data used for display, analysis, and training) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0146] The above description is merely illustrative of embodiments of this disclosure and is not intended to limit the scope of this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. For example, the frequency band division method can be adjusted; the distance attenuation function can adopt other forms such as Euclidean distance; the specific number of layers and channels in the backbone network can be scaled according to actual resource constraints. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the protection scope of this invention.

[0147] In the embodiments of this disclosure, by executing the BAM module of the above-described detection method based on a face forgery model, dynamic mining and enhancement of compression-robust and forgery-related spectral components can be achieved, simulating a data-driven "inverse compression" process. Furthermore, visually preserving feature extraction from the ViR backbone network can be performed, explicitly introducing a spatial distance attenuation prior based on Manhattan distance into the attention mechanism, and linearizing computational complexity using an axial decomposition strategy, thereby achieving efficient and accurate capture of local forgery traces. This frequency-spatial domain collaborative architecture design enables this disclosure to significantly improve generalization capabilities across datasets, compression techniques, and forgery technologies while maintaining high detection accuracy.

[0148] Figure 6 This is a structural block diagram illustrating a detection apparatus based on a face forgery detection model according to an embodiment of the present disclosure.

[0149] Reference Figure 6 The detection device 600 based on the face forgery detection model may include a spatial feature acquisition unit 610, a frequency domain feature acquisition unit 620, a frequency band attention modulation unit 630, a spatial feature modulation unit 640, a visual preservation unit 650, and a detection result generation unit 660.

[0150] According to embodiments of this disclosure, the spatial feature acquisition unit 610 can be configured to extract spatial features from a face image to be detected to obtain an original spatial feature map.

[0151] According to embodiments of the present disclosure, the frequency domain feature acquisition unit 620 can be configured to obtain the original frequency domain feature map by performing a discrete cosine transform on the original spatial feature map.

[0152] According to embodiments of the present disclosure, the frequency band attention modulation unit 630 can be configured to determine frequency band weights for multiple frequency bands divided from the original frequency domain feature map through the frequency band attention modulation module, and modulate the frequency domain features in the corresponding frequency bands based on the determined frequency band weights to obtain a modulated frequency domain feature map.

[0153] According to embodiments of the present disclosure, the spatial feature modulation unit 640 can be configured to obtain a modulated spatial feature map based on the original spatial feature map and the modulated frequency domain feature map.

[0154] According to embodiments of the present disclosure, the visual retention unit 650 can be configured to perform retention feature extraction on the modulated spatial feature map through a visual retention backbone network to obtain visual retention features with local focus and global context awareness, wherein the visual retention backbone network is a network based on a retention self-attention mechanism with spatial distance attenuation prior.

[0155] According to embodiments of this disclosure, the detection result generation unit 660 can be configured to generate a face forgery detection result of a face image based on visually preserved features through a classifier module.

[0156] Optionally, the frequency band attention modulation unit 630 can be configured to determine frequency band weights for multiple frequency bands divided from the original frequency domain feature map through the frequency band attention modulation module by the following operations, and modulate the frequency domain features in the corresponding frequency bands based on the determined frequency band weights to obtain a modulated frequency domain feature map: dividing the original frequency domain feature map into multiple frequency bands by dividing at least one frequency domain feature in the original frequency domain feature map into a frequency band by dividing it into a frequency band; determining the global statistical features of the frequency domain features of each of the multiple frequency bands, wherein, for each frequency band, the global maximum feature and global average feature of at least one frequency domain feature in the frequency band are determined, and the global statistical features of the frequency band are obtained by adding the global maximum feature and global average feature of at least one frequency domain feature in the frequency band; generating multiple frequency band weights corresponding to the multiple frequency bands based on the global statistical features of the multiple frequency bands through a multilayer perceptron; and modulating the frequency domain features in the original frequency domain feature map by using the multiple frequency band weights to obtain a modulated frequency domain feature map.

[0157] Optionally, the multilayer perceptron is configured to: reshape all global statistical features of multiple face image samples within a batch of input into a single global statistical feature input; and perform weight generation processing on the single global statistical feature input to generate a frequency band weight matrix by using parameters shared across samples and channels, wherein each frequency band weight in the frequency band weight matrix is ​​determined individually for each frequency band of the frequency domain feature map for each channel of each face image sample.

[0158] Optionally, the frequency band attention modulation unit 630 can be configured to modulate the frequency domain features in the original frequency domain feature map by using multiple frequency band weights to obtain a modulated frequency domain feature map by: generating a frequency domain attention map with the same size as the original frequency domain feature map based on multiple frequency band weights, wherein each frequency band weight in the frequency domain attention map is mapped to a value greater than 0 and less than 1 by a sigmoid function; and performing residual modulation on the original frequency domain feature map by using the frequency domain attention map to obtain a modulated frequency domain feature map, wherein the residual modulation includes multiplying the original frequency domain feature map and the frequency domain attention map element by element and adding the result of the multiplication to the original frequency domain feature map element by element.

[0159] Optionally, the spatial feature modulation unit 640 can be configured to obtain a modulated spatial feature map based on the original spatial feature map and the modulated frequency domain feature map by: obtaining a spatial attention map by performing an inverse discrete cosine transform and normalization on the modulated frequency domain feature map; and obtaining a modulated spatial feature map by performing residual modulation on the original spatial feature map using the spatial attention map, wherein the residual modulation includes multiplying the original spatial feature map and the spatial attention map element-wise and adding the result of the multiplication element-wise to the original spatial feature map.

[0160] Optionally, the visual preservation unit 650 can be configured to perform feature preservation extraction on the modulated spatial feature map through the visual preservation backbone network to obtain visually preserved features with local focus and global context awareness by: calculating a preserved self-attention map of the modulated spatial feature map based on the linear projection result and spatial distance attenuation bias of the modulated spatial feature map through the visual preservation backbone network; and obtaining visually preserved features based on the linear projection result and the preserved self-attention map, wherein the spatial distance attenuation bias is a bias determined based on the Manhattan distance between spatial locations.

[0161] Optionally, the visual retention unit 650 can be configured to calculate the retention self-attention of the modulated spatial feature map based on the linear projection result and spatial distance attenuation bias of the modulated spatial feature map by the following operations: calculating the width retention self-attention of each spatial feature in the m-th row of the modulated spatial feature map based on the linear projection result and spatial distance attenuation bias; calculating the height retention self-attention of each spatial feature in the n-th column of the modulated spatial feature map based on the linear projection result and spatial distance attenuation bias; and obtaining the retention self-attention map of the modulated spatial feature map by multiplying each width retention self-attention in the m-th row by each height retention self-attention in the n-th column.

[0162] Optionally, the visual retention unit 650 can be configured to obtain visual retention features based on the linear projection result and the retention self-attention map by performing local enhancements on the values ​​in the linear projection result based on depth-separable convolution, and obtaining the visual retention features by adding the retention self-attention map to the result of the local enhancements.

[0163] In other words, according to embodiments of this disclosure, the spatial feature acquisition unit 610 can be configured to perform the operation S110 as described above; the frequency domain feature acquisition unit 620 can be configured to perform the operation S120 as described above; the frequency band attention modulation unit 630 can be configured to perform the operation S130 as described above; the spatial feature modulation unit 640 can be configured to perform the operation S140 as described above; the visual retention unit 650 can be configured to perform the operation S150 as described above; and the detection result generation unit 660 can be configured to perform the operation S160 as described above. (As already referred to above...) Figures 1 to 5 The description of the specific operation details of each unit of the detection device 600 is described in the previous section, so it will not be described in detail here.

[0164] Furthermore, it should be understood that the various units in the detection apparatus 600 according to embodiments of this disclosure may be implemented as hardware components, software components, or a combination of hardware and software components, and may be communicatively connected to each other to transmit signals or data. Those skilled in the art, based on the processes performed by the defined various units, may implement the various units, for example, using a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0165] Figure 7 This is a structural block diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0166] Reference Figure 7 The electronic device 700 may include at least one processor 710 and at least one memory 720 storing computer-executable instructions, wherein the computer-executable instructions, when executed by at least one processor 710, cause at least one processor 710 to perform the detection method as described above.

[0167] As already referenced above Figures 1 to 5 The description provides specific operational details regarding at least one processor 710, so they will not be described in detail here.

[0168] As an example, electronic device 700 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, electronic device 700 is not necessarily a single computer device, but may be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 700 may also be part of an integrated control system or system manager, or may be configured to interface with a portable computer device locally or remotely (e.g., via wireless transmission).

[0169] In electronic device 700, at least one processor 710 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, at least one processor 710 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0170] At least one processor 710 can execute instructions or code stored in at least one memory 720, wherein the at least one memory 720 can also store data. Instructions and data can also be sent and received via a network through a network interface device, wherein the network interface device can employ any known transmission protocol.

[0171] At least one memory 720 may be integrated with at least one processor 710, for example, by arranging RAM or flash memory within an integrated circuit microprocessor. Alternatively, at least one memory 720 may include a separate device, such as an external disk drive, a storage array, or other storage device that can be used by any database system. At least one memory 720 and at least one processor 710 may be operatively coupled or may communicate with each other, for example, via I / O ports, network connections, etc., enabling at least one processor 710 to read files stored in the memory.

[0172] In addition, the electronic device 700 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 700 can be interconnected via a bus and / or network.

[0173] According to embodiments of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions in the computer-readable storage medium are executed by at least one processor, they cause at least one processor to perform the detection method described above.

[0174] As already referenced above Figures 1 to 5 The description provides specific operational details regarding at least one processor, so they will not be described in detail here.

[0175] According to embodiments of this disclosure, a computer-readable storage medium may include: a read-only memory (ROM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), flash memory, non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, a BD-R, and a BD-R. LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program or instructions in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program or instructions and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0176] According to exemplary embodiments of the present disclosure, a computer software or computer program product may also be provided, wherein the instructions in the computer software or computer program product, when executed by at least one processor, can implement the detection method as described above.

[0177] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0178] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A detection method based on a face forgery detection model, characterized in that, The face forgery detection model includes a frequency band attention modulation module, a visual preservation backbone network, and a classifier module; the method includes: By extracting spatial features from the face image to be detected, the original spatial feature map is obtained; The original frequency domain feature map is obtained by performing a discrete cosine transform on the original spatial feature map. The frequency band attention modulation module determines the frequency band weights for each of the multiple frequency bands divided from the original frequency domain feature map, and modulates the frequency domain features in the corresponding frequency bands based on the determined frequency band weights to obtain a modulated frequency domain feature map. Based on the original spatial feature map and the frequency domain feature map of the modulation, a spatial feature map of the modulation is obtained; The visual preservation backbone network is used to extract preserved features from the modulated spatial feature map to obtain visual preservation features with local focus and global context awareness. The visual preservation backbone network is a network based on a preservation self-attention mechanism with spatial distance decay prior. The classifier module generates a face forgery detection result for the face image based on the visually preserved features.

2. The detection method according to claim 1, characterized in that, The steps of determining frequency band weights for multiple frequency bands divided from the original frequency domain feature map using the frequency band attention modulation module, and modulating the frequency domain features in the corresponding frequency bands based on the determined frequency band weights to obtain a modulated frequency domain feature map include: The original frequency domain feature map is divided into multiple frequency bands by dividing at least one frequency domain feature in the original frequency domain feature map into a frequency band where the sum of the frequency indices in the horizontal direction and the vertical direction is the same. Determine the global statistical features of frequency domain features for each of the plurality of frequency bands, wherein, for each frequency band, determine the global maximum feature and global average feature of at least one frequency domain feature in the frequency band, and obtain the global statistical features of the frequency band by adding the global maximum feature and the global average feature of the at least one frequency domain feature in the frequency band. Using a multilayer perceptron, multiple frequency band weights are generated based on the global statistical features of the multiple frequency bands, each corresponding to one of the multiple frequency bands. The frequency domain features in the original frequency domain feature map are modulated by using the multiple frequency band weights to obtain the modulated frequency domain feature map.

3. The detection method according to claim 2, characterized in that, The multilayer sensor is configured as follows: Reshape all global statistical features of multiple face image samples in a batch into a single global statistical feature input; By using parameters shared across samples and channels, a frequency band weight matrix is ​​generated by weighting the single global statistical feature input. In this context, each frequency band weight in the frequency band weight matrix is ​​determined individually for each frequency band of the frequency domain feature map of each channel of each face image sample.

4. The detection method according to claim 2, characterized in that, The step of obtaining the modulated frequency domain feature map by modulating the frequency domain features in the original frequency domain feature map using the multiple frequency band weights includes: A frequency domain attention map with the same size as the original frequency domain feature map is generated based on the multiple frequency band weights, wherein each frequency band weight in the frequency domain attention map is mapped to a value greater than 0 and less than 1 by a sigmoid function. The original frequency domain feature map is residually modulated using the frequency domain attention map to obtain a modulated frequency domain feature map. The residual modulation includes multiplying the original frequency domain feature map element-wise with the frequency domain attention map and adding the result of the multiplication element-wise to the original frequency domain feature map.

5. The detection method according to claim 1, characterized in that, The steps for obtaining the spatial feature map of modulation based on the original spatial feature map and the frequency domain feature map of modulation include: A spatial attention map is obtained by performing inverse discrete cosine transform and normalization on the frequency domain feature map of the modulation. The original spatial feature map is residually modulated using the spatial attention map to obtain a modulated spatial feature map. The residual modulation includes multiplying the original spatial feature map element-wise with the spatial attention map and adding the result of the multiplication element-wise to the original spatial feature map.

6. The detection method according to claim 1, characterized in that, The steps of extracting preserved features from the modulated spatial feature map through the visual preservation backbone network to obtain visual preservation features with local focus and global context awareness include: Using the visual preservation backbone network, based on the linear projection result of the modulated spatial feature map and the spatial distance attenuation bias, the preservation self-attention map of the modulated spatial feature map is calculated. Based on the linear projection result and the retained self-attention map, the visual retention feature is obtained. The spatial distance attenuation bias is a bias determined based on the Manhattan distance between spatial locations.

7. The detection method according to claim 6, characterized in that, The steps for calculating the self-attention preserved in the modulated spatial feature map, based on the linear projection result of the modulated spatial feature map and the spatial distance attenuation bias, include: Based on the linear projection result and the spatial distance attenuation bias, the width of each spatial feature in the m-th row of the modulated spatial feature map is calculated to retain self-attention. Based on the linear projection result and the spatial distance attenuation bias, the height retention self-attention of each spatial feature in the nth column of the modulated spatial feature map is calculated. The retained self-attention map of the modulated spatial feature map is obtained by multiplying each width retained self-attention of the m-th row with each height retained self-attention of the n-th column.

8. The detection method according to claim 6, characterized in that, The steps for obtaining the visually preserved features based on the linear projection result and the retained self-attention map include: The values ​​in the linear projection result are locally enhanced based on depthwise separable convolution, and The visually preserved feature is obtained by adding the retained self-attention map to the result of local enhancement.

9. A detection device based on a face forgery detection model, characterized in that, The face forgery detection model includes a frequency band attention modulation module, a visual preservation backbone network, and a classifier module; the detection device includes: The spatial feature acquisition unit is configured to extract spatial features from the face image to be detected to obtain the original spatial feature map; The frequency domain feature acquisition unit is configured to obtain the original frequency domain feature map by performing a discrete cosine transform on the original spatial feature map; The frequency band attention modulation unit is configured to determine frequency band weights for multiple frequency bands divided from the original frequency domain feature map through the frequency band attention modulation module, and modulate the frequency domain features in the corresponding frequency bands based on the determined frequency band weights to obtain a modulated frequency domain feature map. A spatial feature modulation unit is configured to obtain a modulated spatial feature map based on the original spatial feature map and the modulated frequency domain feature map; The visual retention unit is configured to extract retained features from the modulated spatial feature map through the visual retention backbone network to obtain visual retention features with local focus and global context awareness, wherein the visual retention backbone network is a network based on a retention self-attention mechanism with spatial distance decay prior. The detection result generation unit is configured to generate a face forgery detection result of the face image based on the visually preserved features through the classifier module.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; At least one memory that stores computer-executable instructions. Wherein, when the computer-executable instructions are executed by the at least one processor, they cause the at least one processor to perform the detection method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by at least one processor, they cause the at least one processor to perform the detection method according to any one of claims 1 to 8.

12. A computer program product comprising computer-executable instructions, characterized in that, The computer-executable instructions, when executed by at least one processor, implement the detection method according to any one of claims 1 to 8.

Images