Face forgery detection system and method based on enhancement of spatio-temporal and frequency domain cues

Abstract

The present invention discloses a face forgery detection system and method based on spatio-temporal-frequency domain clue enhancement, including: a face image acquisition module for extracting multiple frames of face images of a video to be detected; a backbone network module for obtaining a sequence of feature maps of the multiple frames of face images; a frequency domain conversion module for mapping the sequence of feature maps to the frequency domain to obtain a discrete cosine transform spectrum sequence; an enhancement module for performing feature enhancement on the discrete cosine transform spectrum sequence by using a set weight matrix; a spatio-temporal feature extraction module for obtaining spatial-frequency domain clues and a time-frequency domain attention map from the enhanced spectrum sequence and forming spatio-temporal-frequency domain features; and a authenticity discrimination module for determining the authenticity of the face in the video to be detected according to the spatio-temporal-frequency domain features. The present invention introduces the frequency domain into the spatio-temporal feature clues and uses the attention mechanism to capture the time information between multiple frames, and the obtained features are more discriminative and robust.

Description

Technical Field

[0001] The present invention belongs to the technical field of face forgery detection, and particularly relates to a face forgery detection system and method based on the enhancement of spatio-temporal frequency domain cues. Background Art

[0002] In recent years, more and more researchers have begun to explore face forgery detection methods. Existing methods define video forgery detection as a binary classification problem and explore image frame-level and video-level solutions.

[0003] For frame-level forgery detection, due to the uncontrollable characteristics of early face generation technologies, obvious visual artifacts and inconsistencies often appear directly in the facial area. Many previous works use facial or head statistical inconsistencies for image forgery detection. However, with the development of face forgery technologies, these image-based detectors may not be able to capture temporal inconsistencies across multiple frames. Therefore, a large number of current research works have begun to focus on exploring video-level features.

[0004] For video-level forgery detection, early forgery generation technologies were relatively limited and might visually show obvious inter-frame temporal inconsistencies. In recent years, with the wide application of deep neural network and generative adversarial network technologies, face generation and editing technologies have become more and more realistic and controllable, but have also brought more security problems to digital forensics.

[0005] Considering that synthetic images in the frequency domain often better reflect subtle visual artifacts, research in the field of forgery detection in the frequency domain has become increasingly popular, such as discrete Fourier transform and discrete cosine transform. Tarik Dzanic et al. found that the attenuation rate of high-frequency components in the Fourier spectrum is clearly distinguishable between real and deep network-generated images. Using this clue, it proposed a new model to detect synthetic images. Jiaming Li et al. believe that fixed filters and handcrafted features are not sufficient to extract forgery features from the frequency domain. Therefore, a single-center loss was proposed to make the intra-class features of the real class more compact. All of the above methods are limited to exploring subtle spatial artifacts and rarely pay attention to the temporal frequency domain cues between different frames.

[0006] Existing image-based face forgery detection technologies only focus on detecting in a single frame of an image or video, which often ignores the inter-frame inconsistencies for forged videos. Although existing image-based technologies have explored forgery clues in the frequency domain on a single image, they have not further explored the frequency domain inconsistencies between frames of a video sequence. Summary of the Invention

[0007] To solve the above problems existing in the prior art, the present invention provides a face forgery detection system and method based on the enhancement of spatio-temporal-frequency domain cues. The technical problems to be solved by the present invention are realized through the following technical solutions:

[0008] One aspect of the present invention provides a face forgery detection system based on the enhancement of spatio-temporal-frequency domain cues, including a face image acquisition module, a backbone network module, a frequency domain conversion module, an enhancement module, a spatio-temporal feature extraction module, and a authenticity discrimination module. Among them,

[0009] The face image acquisition module is used to extract multiple frames of images of the video to be detected, and intercept the face regions of each frame of image to form multiple frames of face images;

[0010] The backbone network module is used to extract the feature vectors of the face images and obtain a sequence of feature maps of the multiple frames of face images; [[ID=?]]

[0011] The frequency domain conversion module is used to map the sequence of feature maps to the frequency domain by using the discrete cosine transform, and obtain a sequence of discrete cosine transform spectra corresponding to the sequence of feature maps;

[0012] The enhancement module is used to perform feature enhancement on the sequence of discrete cosine transform spectra by using a set weight matrix to obtain an enhanced spectrum sequence;

[0013] The spatio-temporal feature extraction module is used to obtain spatial frequency domain cues and temporal frequency domain attention maps from the enhanced spectrum sequence, and form spatio-temporal-frequency domain features;

[0014] The authenticity discrimination module is used to determine the authenticity of the face in the video to be detected according to the spatio-temporal-frequency domain features.

[0015] In an embodiment of the present invention, the face image acquisition module includes an image frame extraction unit and a Facenet network unit. Among them,

[0016] The image frame extraction unit is used to randomly extract N frames of images containing faces from the video to be detected;

[0017] The Facenet network unit is used to obtain the coordinate points of the faces in the images containing faces, and intercept the face regions in each frame of image to form N frames of face images.

[0018] In an embodiment of the present invention, the backbone network module is a trained ResNet50 network or XceptionNet network.

[0019] In an embodiment of the present invention, the spatio-temporal feature extraction module includes a compact spatial feature extraction unit, a frequency domain temporal attention unit, and a synthesis unit. Among them, It should be noted that there seems to be a format error in the original text where the "? " is marked in the translation of item . Please check and correct it if necessary.

[0020] The compact space feature extraction unit is used to evenly divide the enhanced spectrum of each frame into K blocks, select the maximum value of the intra-block pixel points in each block as the feature value after compression of the current block, and form the spatial feature map after compression of each frame of image;

[0021] The frequency-domain time attention unit is used to obtain the time-frequency domain attention map of each frame of image according to the enhanced spectrum sequence;

[0022] The synthesis unit is used to merge the compressed spatial feature map and the time-frequency domain attention map of each frame of image to form the final spatio-temporal-frequency domain feature.

[0023] In an embodiment of the present invention, the frequency-domain time attention unit specifically is used for:

[0024] Compress each frame of spectrum map F in the enhanced spectrum sequence in the channel dimension by using the method of L2 norm to obtain the attention matrix after compression of each frame of spectrum map F:

[0025]

[0026] where C is the number of channels of the feature map, and A(n, h, w) represents the attention score of the nth frame of spectrum feature map at the position (h, w);

[0027] Normalize the attention matrix to obtain the normalized attention matrix:

[0028]

[0029] where H′ and W′ are the height and width of the feature map respectively;

[0030] Evenly divide the normalized attention matrix into K blocks, and use the sum of the scores of the intra-block feature points as the attention score of the current block for each block area to obtain the representation of the attention matrix after partitioning:

[0031] A″(n) = {A″(n, 1), A″(n, 2) … A″(n, K)}

[0032] Perform a normalization operation on the partitioned attention matrix A″, and finally obtain the time-frequency domain attention map A FDA , which is expressed as:

[0033]

[0034] On the other hand, the present invention provides a face forgery detection method based on spatio-temporal-frequency domain cue enhancement, including:

[0035] S1: Extract multiple frames of images of the video to be detected, and intercept the face regions of each frame of image to form multiple frames of face images;

[0036] S2: Extract the feature vectors of the face images by using the trained backbone network to obtain a sequence of feature maps of the multi-frame face images;

[0037] S3: Map the sequence of feature maps to the frequency domain by using the discrete cosine transform to obtain a discrete cosine transform frequency spectrum sequence corresponding to the sequence of feature maps;

[0038] S4: Perform feature enhancement on the discrete cosine transform frequency spectrum sequence by using a set weight matrix to obtain an enhanced frequency spectrum sequence;

[0039] S5: Use the enhanced frequency spectrum sequence to obtain the time-frequency domain attention maps of each frame of image and the time-frequency domain attention maps, and form spatio-temporal-frequency domain features;

[0040] S6: According to the spatio-temporal-frequency domain features, use a pre-trained classifier to obtain the authenticity of the face in the video to be detected.

[0041] In an embodiment of the present invention, the expression of the discrete cosine transform is:

[0042]

[0043]

[0044] where m(i, j) represents a frame of feature map extracted by the backbone network, and u, v represent the coordinates of the discrete cosine transform spectrum The height and width of the feature map extracted by the backbone network are H' and W' respectively.

[0045] In an embodiment of the present invention, the S4 includes:

[0046] After multiplying the discrete cosine transform frequency spectrum sequence D(u, v) after the discrete cosine transform by a set coefficient matrix, an enhanced frequency spectrum sequence F = {F(1), F(2)... F(N)} is obtained, where F(N) represents the enhanced frequency spectrum corresponding to the Nth frame of face image, where,

[0047]

[0048] where, β α Represents the weight value at the position corresponding to the coordinate in the weight matrix.

[0049] In an embodiment of the present invention, the S5 includes:

[0050] Divide the enhanced frequency spectrum of each frame into K blocks evenly, select the maximum value of the in-block pixel points in each block as the compressed feature value of the current block, and form a compressed spatial feature map of each frame of image;

[0051] Obtain the time-frequency domain attention map of each frame of image according to the enhanced spectrum sequence;

[0052] Merge the compressed spatial feature map and the time-frequency domain attention map of each frame of image to form the final spatio-temporal-frequency domain feature.

[0053] In one embodiment of the present invention, the method further includes:

[0054] Use the training data set to train the backbone network and the classifier together, and update the parameters of the classifier and the backbone network using cross-entropy loss. The training data set includes multiple video data sets, and each video data set contains multiple frames of images with true or false labels annotated in the current video.

[0055] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0056] 1. The face forgery detection system and method based on spatio-temporal-frequency domain clue enhancement of the present invention analyzes the original image in the frequency domain, and proposes a spatial frequency domain clue enhancement matrix, which increases the difference in the middle and high frequency components of the generated forged image, and can effectively improve the accuracy. By using a block-based strategy, a compact spatial feature representation is obtained, which reasonably addresses the discrete characteristics of the discrete Fourier transform.

[0057] 2. The present invention first introduces the frequency domain into the time feature clues, and uses the attention mechanism to capture the time forgery information between multiple frames. At the same time, it considers the frequency domain forgery clues in time and space, and the finally obtained features are more discriminative and robust.

[0058] 3. The present invention can address the problem of multi-modal images or videos. By using the discrete cosine transform to convert the video frame sequence into the frequency domain, and by exploring the differences in different components in the frequency domain to determine the authenticity of the video, the spectral differences of different modal images are eliminated.

[0059] The following will further describe the present invention in detail with reference to the drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a module block diagram of a face forgery detection system based on spatio-temporal-frequency domain clue enhancement provided by an embodiment of the present invention;

[0061] Figure 2 is a specific structural schematic diagram of a face forgery detection system based on spatio-temporal-frequency domain clue enhancement provided by an embodiment of the present invention;

[0062] Figure 3 is a schematic diagram of a weight matrix provided by an embodiment of the present invention;

[0063] Figure 4 It is a flowchart of a face forgery detection method based on the enhancement of time-space-frequency domain cues provided by an embodiment of the present invention. Detailed implementation manners

[0064] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following, in combination with the accompanying drawings and specific implementation manners, will detail a face forgery detection system and method based on the enhancement of time-space-frequency domain cues proposed according to the present invention.

[0065] The foregoing and other technical contents, features and effects of the present invention can be clearly presented in the following detailed description in conjunction with the accompanying drawings. Through the description of the specific implementation manners, a more in-depth and specific understanding of the technical means and effects adopted by the present invention to achieve the intended purpose can be obtained. However, the accompanying drawings are only for reference and illustration, and are not used to limit the technical solution of the present invention.

[0066] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant is intended to cover non-exclusive inclusion, so that an article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the article or device including the said element.

[0067] Embodiment 1

[0068] Please refer to Figure 1 and Figure 2, the face forgery detection system of this embodiment includes a face image acquisition module 1, a backbone network module 2, a frequency domain conversion module 3, an enhancement module 4, a spatio-temporal feature extraction module 5, and a authenticity discrimination module 6. The face image acquisition module 1 is used to extract multiple frames of images from the video to be detected, and intercept the face regions of each frame of image to form multiple frames of face images. The face image acquisition module 1 of this embodiment includes an image frame extraction unit and a Facenet network unit. Among them, the image frame extraction unit is used to randomly extract N frames of images containing faces from the video to be detected; the Facenet network unit is used to obtain the coordinate points of the faces in the images containing faces, and intercept the face regions in each frame of image to form N frames of face images. Specifically, first randomly extract N frames of images containing faces from the video to be detected. Since the tampered regions in forged videos or pictures are usually limited to the face regions, in order to prevent other regions from affecting the final detection result, then use the Facenet network to extract the coordinate points of the faces in the face images, and extract the face regions in each frame of image, and intercept the face regions of each frame of image to form N frames of face images where H and W are the height and width of the face image after cropping, respectively. The N frames of face images together constitute the video input sequence I = {f1, f2, f3... f N} of the video to be detected.

[0069] The backbone network module 2 of this embodiment is used to extract the feature vectors of the face images and obtain the feature map sequence of multiple frames of face images. Specifically, each frame of face image is input into the backbone network module 2 and the feature map of each frame of face image is output through the backbone network, which together constitute the feature map sequence where H' and W' are the height and width of the feature map respectively, and C represents the number of channels of the feature map.. In this embodiment, the ResNet50 network or the XceptionNet network can be used as the backbone network module. In other embodiments, other deep learning network models capable of performing face feature extraction can also be used.

[0070] The frequency domain conversion module 3 of this embodiment is used to map the feature map sequence to the frequency domain by using the discrete cosine transform to obtain the discrete cosine transform spectrum sequence corresponding to the feature map sequence. The discrete cosine transform (Discrete Cosine Transform, DCT) is similar to the discrete Fourier transform (DFT). The point sequence in the DCT space is the sum of different frequency domain cosine functions. Given a spatial pixel matrix, such as the feature map of a frame of face image its two-dimensional discrete cosine transform can be expressed as:

[0071]

[0072]

[0073] Among them, m(i, j) represents the feature map extracted by the backbone network model 2, and u, v represent the coordinates of the discrete cosine transform frequency spectrum sequence D(u, v). Compared with the DFT, the DCT has better energy concentration performance in the frequency domain and can directly filter out unimportant frequency domain regions and coefficients. Therefore, it is widely used in image compression tasks. In summary, let represent the sequence of feature maps extracted by the backbone network. According to the above equation, the obtained two-dimensional spatial pixel feature map is converted to the frequency domain to obtain a two-dimensional frequency spectrum sequence spectrum, denoted as

[0074] Furthermore, the enhancement module 4 is used to perform feature enhancement on the discrete cosine transform frequency spectrum sequence by using a set weight matrix to obtain an enhanced frequency spectrum sequence.

[0075] It is generally considered that the components on the medium and high frequency band spectra are very inconsistent between real images and forged or synthetic images. Therefore, as much as possible, the medium and high frequency components should be utilized, and the influence degree of the medium and high frequency components in the spectrum should be increased. Therefore, this embodiment provides a weight matrix to increase the influence degree of the medium and high frequency components. Please refer to Figure 3 , Figure 3 which is a schematic diagram of a weight matrix provided by an embodiment of the present invention. This embodiment performs feature enhancement on the discrete cosine transform frequency spectrum sequence by using a set weight matrix to obtain an enhanced frequency spectrum sequence. The size of the weight matrix is equal to the size of each discrete cosine transform frequency spectrum. For two-dimensional DCT, the equation can be expressed as:

[0076]

[0077] where u, v are the coordinates of the two-dimensional DCT frequency spectrum diagram, and β α represents the weight value at the corresponding position of the coordinate (u, v) in the weight matrix, which is taken as

[0078] After multiplying the two-dimensional frequency spectrum sequence spectrum after discrete cosine transform by the above weight matrix, a two-dimensional DCT enhanced frequency spectrum sequence F = {F(1), F(2)…F(N)} is obtained, denoted as

[0079] Then, the spatio-temporal feature extraction module 5 can obtain spatial frequency domain clues and temporal frequency domain attention maps by using the enhanced frequency spectrum sequence and form spatio-temporal frequency domain features.

[0080] Continue to refer to Figure 2, the spatio-temporal feature extraction module 5 of this embodiment includes a compact spatial feature extraction unit, a frequency-domain time attention unit, and a synthesis unit. Among them, the compact spatial feature extraction unit is used to evenly divide the enhanced spectrum of each frame into K blocks, select the maximum value of the intra-block pixel points in each block as the feature value after compression of the current block, and form the spatial feature map after compression of each frame of image; the frequency-domain time attention unit is used to obtain the time-frequency domain attention map of each frame of image according to the enhanced spectrum sequence; the synthesis unit is used to merge the compressed spatial feature map and the time-frequency domain attention map of each frame of image to form the final spatio-temporal frequency domain feature.

[0081] Specifically, considering the discreteness of the discrete cosine transform, that is, some of the most valuable and effective information is often distributed in different parts, this embodiment adopts a block strategy to obtain the final compact spatial feature. First, the discrete cosine transform spectrum obtained for each frame is divided into K blocks, where n = 1, 2,... N represents the nth frame image of the video. In this embodiment, F(n) is evenly divided into 16 blocks, and the height and width of each block are H' / 4 and W' / 4 respectively. Subsequently, the maximum value of the intra-block pixel points in each block is taken as the feature expression of this block, so that the enhanced spectrum map F(n) is compressed from the H'×W' dimension to the K dimension, and the compressed feature map is used as the final compact spatial feature representation, that is, the spatial frequency domain forgery clue.

[0082] Furthermore, the frequency-domain time attention unit is specifically used for:

[0083] Compress each frame of spectrum map F in the channel dimension of the enhanced spectrum sequence by using the L2 norm method to obtain the attention matrix after compression of each frame of spectrum map F:

[0084]

[0085] Among them, C is the number of channels of the feature map, and A(n, h, w) represents the attention score of the nth frame spectrum feature map at the position (h, w);

[0086] Normalize the attention matrix to obtain the normalized attention matrix:

[0087]

[0088] Among them, H' and W' are the height and width of the feature map respectively;

[0089] Evenly divide the normalized attention matrix into K blocks, and use the sum of the scores of the intra-block feature points as the attention score of the current block for each block area to obtain the block-based attention matrix representation:

[0090] A″(n) = {A″(n, 1), A″(n, 2) … A″(n, K)}

[0091] Perform a normalization operation on the partitioned attention matrix A″, and finally obtain the time-frequency domain attention map A FDA , which is expressed as:

[0092]

[0093] The synthesis unit is specifically used to combine the spatial-frequency domain forgery clues and the time-frequency domain attention map together to form the final spatio-temporal-frequency domain forgery feature

[0094]

[0095] Furthermore, the authenticity discrimination module 6 of this embodiment is used to determine the authenticity of the face in the video to be detected according to the spatio-temporal-frequency domain features. The authenticity discrimination module 6 includes a trained classifier. Specifically, the above spatio-temporal-frequency domain forgery feature f c is input into a classifier composed of three fully connected layers. The classifier outputs the probability of each category (true or false) through the softmax function. The formula of the softmax function is:

[0096]

[0097] where, x i represents the value of the i-th category in the output vector of the classifier, x j represents the value of the j-th dimension in the output vector, and n is the dimension of the output vector.

[0098] It should be noted that in the actual process of using this face forgery detection system for face image forgery detection, first, the backbone network and classifier in this face forgery detection system need to be trained. In this embodiment, the open-source dataset WildDeepfake is used as the training set to train the backbone network and classifier. This dataset includes picture frames of processed videos, and the authenticity of these picture frames has been labeled. During the training process, the picture frames in the training set are sequentially input into this face forgery detection system to obtain the probability values of the picture being true or false. Subsequently, the obtained probability values of being true or false are used with the cross-entropy loss function to obtain the misclassification loss of the current face forgery detection system, and the parameters of the classifier and the backbone network are updated using this loss. The cross-entropy loss function is:

[0099]

[0100] where, x i represents the value of the i-th category in the output vector of the model, σ(xi ) The probability of being true or false, y i represents the true class label of the input image. When the face in the image is true, y i = 1. When the face in the image is false, y i = 0. n is the dimension of the output vector, and n = 2.

[0101] Use the training set to repeat multiple rounds of training. During the training process, continuously update the parameters in the backbone network and the classifier network, and continuously improve the accuracy of the system in identifying the authenticity of images. To prevent overfitting, when the accuracy does not increase for 5 consecutive rounds of training, stop the training, and then the trained backbone network and classifier can be obtained, and further the trained face forgery detection system can be obtained.

[0102] The face forgery detection system based on time-space-frequency domain clue enhancement in the embodiments of the present invention analyzes the original image in the frequency domain, and proposes a spatial frequency domain clue enhancement matrix, which increases the difference in the mid-high frequency components of the generated forged images, can effectively improve the accuracy, and obtains a compact spatial feature representation through a block strategy, reasonably dealing with the discrete characteristics of the discrete Fourier transform.

[0103] Embodiment 2

[0104] Please refer to Figure 4 , Figure 4 which is the flowchart of the face forgery detection method based on time-space-frequency domain clue enhancement provided by the embodiments of the present invention. The method includes:

[0105] S1: Extract multiple frames of images of the video to be detected, and intercept the face regions of each frame of image to form multiple frames of face images.

[0106] First, randomly extract N frames of images containing faces from the video to be detected, use the Facenet network to extract the coordinate points of the faces in the face images, then extract the face regions in each frame of image, and intercept the face regions of each frame of image to form N frames of face images where H and W are the height and width of the face image after interception respectively. The N frames of face images together form the video input sequence I = {f1, f2, f3... f N}.

[0107] S2: Extract the feature vectors of the face images to obtain the feature map sequences of each frame of face image.

[0108] In this embodiment, the backbone network is used to extract the feature maps of each frame of face image Among them, H' and W' are the height and width of the feature map respectively, and C represents the number of channels of the feature map. Specifically, each frame of face image is input into the backbone network, and the feature map of each frame of face image is output through the backbone network. In this embodiment, the ResNet50 network or the XceptionNet network can be used as the backbone network.

[0109] S3: Use the discrete chord transform to map the obtained sequence of feature maps to the frequency domain to obtain the discrete cosine transform spectrum corresponding to each sequence of feature maps.

[0110] In this embodiment, the discrete cosine transform (Discrete Cosine Transform, DCT) is used to map the obtained sequence of feature maps to the frequency domain. Let

[0111] denote the sequence of feature maps extracted by the backbone network. According to the discrete cosine transform equation in Embodiment 1, the obtained two-dimensional spatial pixel feature map is converted to the frequency domain to obtain a two-dimensional frequency spectrum sequence spectrum, denoted as

[0112] S4: Use the set weight matrix to perform feature enhancement on the discrete cosine transform spectrum sequence to obtain an enhanced spectrum sequence. Figure 3 In this embodiment, a weight matrix is provided to increase the influence degree of the middle and high frequency components. As

[0113] shown, use the set weight matrix to perform feature enhancement on the discrete cosine transform spectrum sequence to obtain an enhanced spectrum sequence. The size of the weight matrix is equal to the size of each discrete cosine transform spectrum. For two-dimensional DCT,

[0114]

[0115] where u and v are the coordinates of the two-dimensional DCT frequency spectrum diagram, and β α represents the weight value at the position corresponding to the coordinates (u, v) in the weight matrix. Here, it is taken as

[0116] S5: Use the enhanced spectrum sequence to obtain the time-frequency domain attention map and the time-frequency domain attention map of each frame of image, and form spatio-temporal-frequency domain features.

[0117] To fill the gap in analyzing the entire video sequence rather than a single image in the frequency domain, this embodiment introduces a forgery clue enhancement network based on the discrete cosine transform for forgery detection. The forgery clue enhancement network consists of a compact spatial feature extraction module and a frequency-domain temporal attention module. The goal is to make full use of the rich information in the frame image sequence that is not considered in existing methods, mine the spatio-temporal-frequency domain clues between multiple frames, and ultimately achieve a more comprehensive feature representation in the frequency domain.

[0118] Furthermore, this embodiment uses a compact spatial feature extraction module to present a compact spatial feature representation. The compact spatial feature extraction module is used to compress the above DCT-enhanced spectrum sequence F to obtain a more compact feature representation.

[0119] Specifically, this embodiment adopts a block strategy to obtain the final compact spatial features. First, the discrete cosine transform spectrum obtained for each frame is divided into K blocks, where n = 1, 2, … N. In this embodiment, F(n) is evenly divided into 16 blocks, and the height and width of each block are H' / 4 and W' / 4 respectively. Then, the maximum value of the intra-block pixel points in each block is taken as the feature expression of this block. Therefore, the enhanced spectrogram F(n) is compressed from the H'×W' dimension to the K dimension, and the compressed feature map is used as the final compact spatial feature representation, that is, the spatial-frequency domain forgery clue.

[0120] To further explore the temporal-frequency domain clues between multiple frames, this embodiment also designs a frequency-domain temporal attention module. Similar to the compact spatial feature extraction module, the input of the frequency-domain temporal attention module is also the enhanced two-dimensional DCT-enhanced spectrum The goal of this module is to construct an attention map based on the discrete cosine transform spectrum First, we compress the enhanced DCT spectrogram F using the L2 norm method in the channel dimension. The channel compression representation of the feature map for each frame obtained is:

[0121]

[0122] where C is the number of channels, and A(n, h, w) represents the attention score of the nth frame spectrogram feature map at the position (h, w). Subsequently, in order to normalize all the values in the attention matrix composed of all attention scores and control the values between 0 and 1, the normalized result is:

[0123]

[0124] Furthermore, following the same principle in the compact space feature extraction module, in this embodiment, the normalized attention map matrix is also divided into K blocks. For each region, the sum of the scores of the feature points within the block is used as the attention score of the current block. The attention matrix after partitioning is expressed as:

[0125] A″(n)={A″(n,1),A″(n,2)…A″(n,K)}

[0126] Since the sum of the feature points within the block is used to represent the feature representation of each block, in order to avoid the phenomenon of gradient explosion caused by excessive data during the training process, this embodiment uses the method based on the L1 norm to further normalize the partitioned attention matrix A″, and finally obtain the time-frequency domain attention map A FDA , which is expressed as:

[0127]

[0128] Finally, the spatial-frequency domain forgery clue and the time-frequency domain attention map are merged together to form the final spatio-temporal-frequency domain forgery feature

[0129]

[0130] This feature is the final true / false feature representation of a video. This feature is input into a classifier composed of three fully connected layers, and the classifier outputs the probability of each category (true or false) through the softmax function. The formula of the softmax function is:

[0131]

[0132] where, x i represents the value of the i-th category in the output vector of the classifier, x j represents the value of the j-th dimension in the output vector, and n is the dimension of the output vector.

[0133] It should be noted that during the actual process of using this face forgery detection system for face image forgery detection, first, the backbone network and the classifier in this face forgery detection system need to be trained. In this embodiment, the open-source dataset WildDeepfake is used as the training set to train the backbone network and the classifier. This dataset includes the picture frames of the processed videos, and the true / false of these picture frames have been labeled. For the specific training process, please refer to Embodiment 1, which will not be elaborated here.

[0134] The face forgery detection method based on the enhancement of spatio-temporal frequency domain clues in the embodiments of the present invention analyzes the original image in the frequency domain, and proposes a spatial frequency domain clue enhancement matrix, which increases the difference in the high and middle frequency components of the generated forged image, and can effectively improve the accuracy. Through the strategy of block division, a compact spatial feature representation is obtained, which reasonably deals with the discrete characteristics of the discrete Fourier transform. The embodiments of the present invention introduce the frequency domain into the time feature clues for the first time, and use the attention mechanism to capture the time forgery information between multiple frames. At the same time, the frequency domain forgery clues in time and space are considered, and the finally obtained features are more discriminative and robust. The embodiments of the present invention can handle the problems of multi-modal images or videos. By using the discrete cosine transform, the video frame sequence is converted into the frequency domain, and the authenticity of the video is judged by exploring the differences of different components in the frequency domain, thus eliminating the spectral differences of different modal images.

[0135] In several embodiments provided by the present invention, it should be understood that the devices and methods disclosed by the present invention can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0136] In addition, each functional module in the various embodiments of the present invention can be integrated into a processing module, or each module can exist physically alone, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware, or in the form of a combination of hardware and software functional modules.

[0137] Another embodiment of the present invention provides a storage medium, in which a computer program is stored, and the computer program is used to execute the steps of the face forgery detection method based on enhanced spatio-temporal frequency domain clues in the above embodiment. Another aspect of the present invention provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the face forgery detection method based on enhanced spatio-temporal frequency domain clues as described in the above embodiment are implemented. Specifically, the integrated module implemented in the form of a software functional module can be stored in a computer-readable storage medium. The above software functional module is stored in a storage medium and includes several instructions for causing an electronic device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute some steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0138] The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several simple deductions or substitutions can still be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A face forgery detection system based on the enhancement of spatio-temporal frequency domain cues, characterized in that, It includes a face image acquisition module, a backbone network module, a frequency domain conversion module, an enhancement module, a spatio-temporal feature extraction module, and a authenticity discrimination module. Among them, the face image acquisition module is used to extract multiple frames of images from the video to be detected, and intercept the face regions of each frame of image to form multiple frames of face images; the backbone network module is used to extract the feature vectors of the face images and obtain a sequence of feature maps of the multiple frames of face images; the frequency domain conversion module is used to map the sequence of feature maps to the frequency domain by using the discrete cosine transform to obtain a discrete cosine transform spectrum sequence corresponding to the sequence of feature maps; the enhancement module is used to perform feature enhancement on the discrete cosine transform spectrum sequence by using a set weight matrix to obtain an enhanced spectrum sequence; the spatio-temporal feature extraction module is used to obtain spatial frequency domain clues and temporal frequency domain attention maps from the enhanced spectrum sequence and form spatio-temporal frequency domain features; the authenticity discrimination module is used to determine the authenticity of the face in the video to be detected according to the spatio-temporal frequency domain features.

2. The face forgery detection system based on spatio-temporal frequency domain clue enhancement according to claim 1, wherein The face image acquisition module includes an image frame extraction unit and a Facenet network unit. Among them, the image frame extraction unit is used to randomly extract N frames of images containing faces from the video to be detected; the Facenet network unit is used to obtain the coordinate points of the faces in the images containing faces and intercept the face regions in each frame of image to form N frames of face images.

3. The face forgery detection system based on enhancing spatio-temporal-frequency domain cues according to claim 1, wherein The backbone network module is a trained ResNet50 network or XceptionNet network.

4. The face forgery detection system based on the enhancement of spatio-temporal frequency domain cues according to claim 1, wherein, The spatio-temporal feature extraction module includes a compact spatial feature extraction unit, a frequency domain temporal attention unit, and a synthesis unit. Among them, the compact spatial feature extraction unit is used to evenly divide the enhanced spectrum of each frame into K blocks, select the maximum value of the intra-block pixel points in each block as the compressed feature value of the current block, and form a compressed spatial feature map of each frame of image; the frequency domain temporal attention unit is used to obtain the temporal frequency domain attention map of each frame of image according to the enhanced spectrum sequence; the synthesis unit is used to merge the compressed spatial feature map and the temporal frequency domain attention map of each frame of image to form the final spatio-temporal frequency domain features.

5. The face forgery detection system based on the enhancement of spatio-temporal-frequency domain cues according to claim 4, wherein The frequency domain temporal attention unit is specifically used for: Compressing each frame of spectrogram F in the enhanced spectrum sequence in the channel dimension by using the L2 norm method to obtain the compressed attention matrix of each frame of spectrogram F: where C is the number of channels of the feature map, and A(n,h,w) represents the attention score of the nth frame of spectrogram feature map at the position (h,w); Normalize the attention matrix to obtain the normalized attention matrix: where H′ and W′ are the height and width of the feature map respectively; Evenly divide the normalized attention matrix into K blocks, and use the sum of the scores of the feature points within each block as the attention score of the current block for each block region to obtain the block-based attention matrix representation: A″(n) = {A″(n,1), A″(n,2)…A″(n,K)} Perform a normalization operation on the partitioned attention matrix A″, and finally obtain the time-frequency domain attention map A FDA , which is expressed as:

6. A face forgery detection method based on the enhancement of spatio-temporal frequency domain cues, characterized in that, including: S1: Extract multiple frames of images from the video to be detected, and intercept the face regions of each frame of image to form multiple frames of face images; S2: Extract the feature vectors of the face images by using the trained backbone network to obtain a sequence of feature maps of the multi-frame face images; S3: Map the sequence of feature maps to the frequency domain by using the discrete cosine transform to obtain a discrete cosine transform frequency spectrum sequence corresponding to the sequence of feature maps; S4: Enhance the features of the discrete cosine transform frequency spectrum sequence by using a set weight matrix to obtain an enhanced frequency spectrum sequence; S5: Obtain the temporal-frequency domain attention map and the temporal-frequency domain attention map of each frame of image by using the enhanced frequency spectrum sequence, and form spatio-temporal-frequency domain features; S6: According to the spatio-temporal-frequency domain features, use a pre-trained classifier to obtain the authenticity of the face in the video to be detected.

7. The face forgery detection method based on the enhancement of spatio-temporal frequency domain clues according to claim 6, characterized in that The expression of the discrete cosine transform is: Among them, m(i, j) represents a frame of feature map extracted by the backbone network, and u, v represent the coordinates of the discrete cosine transform spectrum , and H′, W′ are the height and width of the feature map extracted by the backbone network respectively.

8. The face forgery detection method based on the enhancement of spatio-temporal frequency domain cues according to claim 7, wherein The S4 includes: After multiplying the discrete cosine transform frequency spectrum sequence D(u, v) after the discrete cosine transform by a set coefficient matrix, an enhanced frequency spectrum sequence F = {F(1), F(2)…F(N)} is obtained, where F(N) represents the enhanced frequency spectrum corresponding to the Nth frame of face image, where, F(u,v) = β α D(u,v), Among them, β α represents the weight value at the position corresponding to the coordinates in the weight matrix.

9. The face forgery detection system based on enhancement of spatio-temporal-frequency domain cues according to claim 8, wherein The S5 includes: Evenly divide the enhanced frequency spectrum of each frame into K blocks, select the maximum value of the in-block pixel points in each block as the eigenvalue after compression of the current block, and form a spatially compressed feature map of each frame of image; Obtain the temporal-frequency domain attention map of each frame of image according to the enhanced frequency spectrum sequence; Merge the spatially compressed feature map and the temporal-frequency domain attention map of each frame of image to form the final spatio-temporal-frequency domain features.

10. The face forgery detection system based on enhanced spatio-temporal frequency domain cues according to any one of claims 6 to 9, characterized in that, The method further includes: Use the training data set to train the backbone network and the classifier together, and update the parameters of the classifier and the backbone network by using the cross-entropy loss. The training data set includes multiple video data sets, and each video data set contains multiple frames of images with authenticity labels in the current video.

Images