Face forgery detection method and device, electronic equipment and storage medium

Abstract

The present application relates to the technical field of machine learning and computer vision, and provides a face forgery detection method and device, electronic equipment and storage medium, the method first acquires a face image to be detected, and extracts image features of the face image to be detected, the image features include content features and texture features; then the content features and the texture features are fused respectively, content domain fusion results and texture domain fusion results are obtained, and the content domain fusion results and the texture domain fusion results are fused, feature cross-domain fusion results are obtained; finally, based on the feature cross-domain fusion results, a forgery detection result of the face image to be detected is determined. The method extracts content features and texture features, represents forgery evidence of the face in the face image to be detected from multiple dimensions, which can improve the accuracy of the forgery detection result. Moreover, the method can further improve the accuracy and reliability of the forgery detection result through feature intra-domain fusion and feature cross-domain fusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning and computer vision, and particularly relates to a face forgery detection method and device, electronic equipment and storage medium. BACKGROUND

[0002] Face forgery detection is to determine whether the face contained in a given picture is generated by forgery.

[0003] At present, there are two ways for face forgery detection. One is to use artificially designed deep semantic features for forgery detection, such as head posture consistency, abnormal blinking frequency, etc. The other is to use data-driven facial defect features for forgery detection, such as regional texture inconsistency, generated artifact abnormality, spectral domain distribution abnormality, etc.

[0004] However, the above methods only use deep semantic features or facial defect features, which will result in low accuracy of face forgery detection. SUMMARY

[0005] The present application provides a face forgery detection method, device, electronic equipment and storage medium to solve the defects in the prior art.

[0006] The present application provides a face forgery detection method, comprising:

[0007] Obtaining a face image to be detected and extracting image features of the face image to be detected, wherein the image features include content features and texture features;

[0008] Fusing the content features and the texture features respectively to obtain content domain fusion results and texture domain fusion results, and fusing the content domain fusion results and the texture domain fusion results to obtain a feature cross-domain fusion result;

[0009] Based on the feature cross-domain fusion result, determining a forgery detection result of the face image to be detected.

[0010] According to the face forgery detection method provided by the present application, the image features include multiple levels;

[0011] Correspondingly, the fusing of the content features and the texture features to obtain the content domain fusion results and the texture domain fusion results, and the fusing of the content domain fusion results and the texture domain fusion results to obtain the feature cross-domain fusion result, comprises:

[0012] Fusing the content features of each level to obtain the content domain fusion results of each level, and fusing the texture features of each level to obtain the texture domain fusion results of each level;

[0013] The content domain fusion result and the texture domain fusion result of each level are fused to obtain a feature cross-domain fusion result of each level.

[0014] According to the face forgery detection method provided by the application, the image features of the face image to be detected are extracted, including:

[0015] For the lowest level, the image features of the lowest level are determined based on the initial features of the lowest level of the face image to be detected;

[0016] For any level except the lowest level, the image features of the level are determined based on the feature cross-domain fusion result of the previous level of the level and the initial features of the level.

[0017] According to the face forgery detection method provided by the application, the image features of the level are determined based on the feature cross-domain fusion result of the previous level of the level and the initial features of the level, including:

[0018] The image features of the previous level are element-wise superimposed with the feature cross-domain fusion result to obtain a feature superposition result of the previous level;

[0019] The feature superposition result of the previous level is merged with the initial features of the level in the channel dimension to obtain the image features of the level.

[0020] According to the face forgery detection method provided by the application, the content features of each level are fused to obtain a content domain fusion result of each level, and the texture features of each level are fused to obtain a texture domain fusion result of each level, including:

[0021] Based on the corresponding feature interaction vector of each level, the content features and the texture features of each level are fused respectively to obtain the content domain fusion result and the texture domain fusion result of each level.

[0022] According to the face forgery detection method provided by the application, the content domain fusion result and the texture domain fusion result of each level are fused to obtain a feature cross-domain fusion result of each level, including:

[0023] Based on a bilinear pooling method, the content domain fusion result and the texture domain fusion result of each level are embedded to obtain a content-texture semantic relationship of each level;

[0024] The content-texture semantic relationship of each level is normalized nonlinearly, and the normalized result is aggregated in similarity;

[0025] Based on the similarity aggregation result, the feature cross-domain fusion result of each level is determined.

[0026] According to the face forgery detection method provided by the application, the feature cross-domain fusion result is used to determine the forgery detection result of the to-be-detected face image, which comprises:

[0027] The feature cross-domain fusion result of the highest level is respectively element-wise superimposed with the content feature and the texture feature of the highest level to obtain the content superposition result and the texture superposition result of the highest level.

[0028] Based on the content superposition result and the texture superposition result of the highest level, the to-be-detected face image is classified into true and false to obtain the forgery detection result.

[0029] According to the face forgery detection method provided by the application, the texture feature is extracted based on the difference difference convolution operator or based on the difference difference convolution operator and the center difference convolution operator.

[0030] According to the face forgery detection method provided by the application, the difference difference convolution operator is represented based on the following formula:

[0031]

[0032] Wherein, f SDC (F l , k 2 ) represents the difference difference convolution operator, F l represents the texture feature of the lth level, k 2 represents the difference difference convolution kernel, z n and z m both represent the position of the current receptive field of k 2 , n and m both represent the position index of the current receptive field of k 2 , represents the set of position pairs of the current receptive field of k 2 , k i 2 is the parameter of the i th position in k 2 , i=n.

[0033] The application also provides a face forgery detection device, comprising:

[0034] The feature extraction module is used for acquiring the to-be-detected face image and extracting the image features of the to-be-detected face image, wherein the image features comprise content features and texture features.

[0035] a feature fusion module configured to fuse the content features and the texture features respectively to obtain a content domain fusion result and a texture domain fusion result, and fuse the content domain fusion result and the texture domain fusion result to obtain a feature cross-domain fusion result;

[0036] a forgery detection module configured to determine a forgery detection result of the face image to be detected based on the feature cross-domain fusion result.

[0037] The present application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the face forgery detection method according to any one of the above when executing the program.

[0038] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the face forgery detection method according to any one of the above.

[0039] The present application also provides a computer program product comprising a computer program, wherein the computer program is executable on a processor to implement the face forgery detection method according to any one of the above.

[0040] The face forgery detection method, device, electronic device, and storage medium provided by the present application first acquire a face image to be detected and extract image features of the face image to be detected, wherein the image features comprise content features and texture features; then fuse the content features and the texture features respectively to obtain a content domain fusion result and a texture domain fusion result, and fuse the content domain fusion result and the texture domain fusion result to obtain a feature cross-domain fusion result; and finally determine a forgery detection result of the face image to be detected based on the feature cross-domain fusion result. The method can improve the accuracy of the forgery detection result by extracting the content features and the texture features to represent the forgery evidence of the face in the face image to be detected from multiple dimensions. Moreover, the method can further improve the accuracy and reliability of the forgery detection result by feature intra-domain fusion and feature cross-domain fusion, and has good generalization for multiple data sets, multiple forgery types, and multiple tampering modes, and has good robustness for high noise and strong compression in real-world scenarios. The method can be effectively deployed to terminal devices for forgery detection of media data or real-time face images on the Internet. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description can also be obtained by those skilled in the art without creative effort.

[0042] Figure 1 is a flowchart of a face forgery detection method provided by the present application;

[0043] Figure 2 is an operation schematic diagram of the CDC operator in the face forgery detection method provided by the present application;

[0044] Figure 3 is an operation schematic diagram of the SDC operator in the face forgery detection method provided by the present application;

[0045] Figure 4 is a principle schematic diagram of the CrossGFI of the lth level in the face forgery detection method provided by the present application;

[0046] Figure 5 is a structure schematic diagram of the face forgery detection model in the face forgery detection method provided by the present application;

[0047] Figure 6 is a structure schematic diagram of the face forgery detection device provided by the present application;

[0048] Figure 7 is a structure schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0049] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0050] In recent years, with the development of deep learning, especially neural networks, deep learning has begun to exceed humans in many visual tasks. One type of neural network is called a generative adversarial network (GAN), which is mainly used for image generation and editing tasks and can generate very realistic images. Many forgery picture and forgery video generation methods have been derived from the generative adversarial network, among which the most eye-catching and most dangerous one is the face replacement forgery video.

[0051] On the one hand, face deepfake technology represented by Deepfake, Zao, etc. can be used to make interesting short video applications, such as replacing the user's face into movie clips using face replacement technology, and driving static portraits using expression replay technology. On the other hand, due to its easy generation, short generation time, high fidelity and other characteristics, face deepfake technology is also easy to be used to make movies, fake news and rumors by synthesizing false voices. Therefore, how to detect face fake videos in the Internet has become a research hotspot and a problem to be solved in the computer vision community.

[0052] In order to minimize the impact of deepfake technology and eliminate the spread of fake videos, the academic and industrial communities have begun to explore different deepfake detection technologies and have proposed a series of defense methods from different aspects. These methods cover multiple modalities such as spatial domain, time domain, frequency domain, and have achieved a series of successes on some specific data sets. In addition, scholars have constructed data sets and carried out multi-angle research on Deepfakes detection.

[0053] Currently, there are two ways to detect face forgery. One is to use artificially designed high-level semantic features for forgery detection, such as head pose consistency, abnormal blinking frequency, etc. The other is to use data-driven facial defect features for forgery detection, such as regional texture inconsistency, generated artifact abnormalities, spectral domain distribution abnormalities, etc.

[0054] However, the above methods only use deep semantic features or facial defect features, which will result in low accuracy of face forgery detection.

[0055] Based on this, the embodiment of the present application provides a face forgery detection method.

[0056] Figure 1 The flowchart of the face forgery detection method provided in the embodiment of the present application is shown in Figure 1 The method comprises the following steps:

[0057] S1, obtaining a face image to be detected, and extracting image features of the face image to be detected, the image features comprising content features and texture features;

[0058] S2, respectively fusing the content features and the texture features to obtain a content domain fusion result and a texture domain fusion result, and fusing the content domain fusion result and the texture domain fusion result to obtain a feature cross-domain fusion result;

[0059] S3, determining a forgery detection result of the face image to be detected based on the feature cross-domain fusion result.

[0060] Specifically, the face forgery detection method provided in the embodiments of the present application has a subject of face forgery detection device, which can be configured in a computer, which can be a local computer or a cloud computer, and the local computer can be a computer, a tablet, etc., which is not specifically limited here.

[0061] Firstly, step S1 is performed to obtain a face image to be detected. The face image to be detected refers to an image in which it is needed to determine whether the face therein is a real face or a fake face. The face image to be detected can be a grayscale image or a color image, which is not specifically limited here.

[0062] Inspired by the image decomposition theory, the face image to be detected I is decomposed into a structured content space (s) and a fine-grained texture space (t) by using an energy minimization model of energy variation:

[0063]

[0064] Among them, s can model isomorphic content, and t contains noise, texture and other oscillation modes.

[0065] Since only the content features are insufficient to distinguish the subtle differences between real faces and fake faces, in the embodiments of the present application, image features of the face image to be detected are extracted. The image features can include content features and texture features, the content features can be features representing the content semantic information of the face in the face image to be detected, and the texture features can be features representing the texture detail information of the face in the face image to be detected.

[0066] The image features can include one level or multiple levels with different granularities, that is, the content features and the texture features can each include one level or multiple levels with different granularities. When the image features include multiple levels, the number of levels of the content features is the same as that of the texture features, and each level of the content features corresponds to each level of the texture features.

[0067] The content features can be extracted by a content feature extraction module, which can be a conventional feature extraction module or a content attention map learning (CAML) module based on a channel attention mechanism and a spatial attention mechanism.

[0068] The texture features can be extracted by a texture feature extraction module. The texture feature extraction module can be a conventional feature extraction module or a texture attention map learning (TAML) module based on a channel attention mechanism and a spatial attention mechanism.

[0069] The CAML module and the TAML module can be collectively referred to as a Convolutional Block Attention Module (CBAM) based on a channel attention mechanism and a spatial attention mechanism.

[0070] The introduction of the content feature extraction module and the texture feature extraction module can realize independent mining of the content features and the texture features in the face image to be detected. Here, the content features and the texture features are features in two feature domains of the face image to be detected, and are respectively used to represent the forged evidence of the two feature domains.

[0071] Then, step S2 is performed to fuse the content features and the texture features respectively to obtain a content domain fusion result and a texture domain fusion result, that is, the content features are fused to obtain the content domain fusion result, and the texture features are fused to obtain the texture domain fusion result.

[0072] Both of the above two fusion processes are implemented in the two independent feature domains of the content domain and the texture domain, and can be implemented through the content feature interaction module and the texture feature interaction module. The fusion mode can be directly implemented in a self-convolution manner, or can be implemented by introducing a feature interaction vector, which is not limited here.

[0073] The content feature interaction module and the texture feature interaction module can both be an Intra-domain Attention-based Feature Interaction (IntraAFI) based on an attention mechanism.

[0074] Thereafter, the content domain fusion result and the texture domain fusion result can be fused to obtain a feature cross-domain fusion result. The fusion process is implemented across the feature domains, and can be implemented through a cross-domain feature interaction module to grasp the high-order semantic relationship between the features of the content domain and the texture domain. The obtained feature cross-domain fusion result is a correlation interaction feature between the two feature domains of the content domain and the texture domain, and is used to represent the forged evidence across the feature domains.

[0075] Since the graph convolutional neural network has a natural modeling advantage in the correlation inference of multi-source heterogeneous data, a cross-domain feature interaction module can be constructed based on the graph convolutional neural network to mine the high-order correlation between the content features and the texture features, explore the high-order semantic relationship between the content features and the texture features, and perform discriminative forged relationship learning. That is, the cross-domain feature interaction module can be a Cross-domain Graph-based Feature Interaction (CrossGFI) based on a graph convolution, so as to realize cross-domain fusion of the features through a graph convolution algorithm.

[0076] When the content feature and the texture feature each only have one level, then the content domain fusion result and the texture domain fusion result are directly fused; when the content feature and the texture feature each include multiple levels, then the content domain fusion result and the texture domain fusion result of the same level are fused.

[0077] The feature cross-domain fusion result can include a content domain feature cross-domain fusion result and a texture domain feature cross-domain fusion result, the content domain feature cross-domain fusion result being a fusion result obtained after the texture domain fusion result is fused to the content domain fusion result, and the texture domain feature cross-domain fusion result being a fusion result obtained after the content domain fusion result is fused to the texture domain fusion result.

[0078] Finally, step S3 is performed to determine a forgery detection result of the to-be-detected face image by using the feature cross-domain fusion result. This process can be implemented by a forgery detection module, that is, the feature cross-domain fusion result can be input to the forgery detection module, and the forgery detection module can perform true and false binary classification on the to-be-detected face image by using the feature cross-domain fusion result to obtain the final forgery detection result.

[0079] The forgery detection module can include a classifier to perform true and false binary classification on the to-be-detected face image and output a forgery detection result. The forgery detection result can include that the to-be-detected face image is a real face (i.e., true) and that the to-be-detected face image is a fake face (i.e., false).

[0080] In the embodiment of the present application, steps S1-S3 can be implemented by means of a face forgery detection model, the face forgery detection model including a feature extraction module, a feature fusion module, and a forgery detection module, the feature extraction module including a content feature extraction module and a texture feature extraction module, and the feature fusion module including a content feature interaction module, a texture feature interaction module, and a cross-domain feature interaction module.

[0081] The face forgery detection method provided in the embodiment of the present application first acquires a face image to be detected, and extracts image features of the face image to be detected, which include content features and texture features; then fuses the content features and the texture features respectively to obtain content domain fusion results and texture domain fusion results, and fuses the content domain fusion results and the texture domain fusion results to obtain feature cross-domain fusion results; finally, based on the feature cross-domain fusion results, a forgery detection result of the face image to be detected is determined. This method represents the forgery evidence of the face in the face image to be detected from multiple dimensions by extracting the content features and the texture features, which can improve the accuracy of the forgery detection result. Moreover, this method can further improve the accuracy and reliability of the forgery detection result by feature intra-domain fusion and feature cross-domain fusion, has good generalization for multiple data sets, multiple forgery types and multiple tampering modes, has good robustness for high noise and strong compression in real-world scenarios, and can be effectively deployed to terminal devices for forgery detection of media data or real-time face images on the Internet.

[0082] On the basis of the above-mentioned embodiment, the face forgery detection method provided in the embodiment of the present application, the image features include multiple levels;

[0083] Correspondingly, the fusing the content features and the texture features respectively to obtain content domain fusion results and texture domain fusion results, and the fusing the content domain fusion results and the texture domain fusion results to obtain feature cross-domain fusion results, include:

[0084] The content features of each level are fused to obtain content domain fusion results of each level, and the texture features of each level are fused to obtain texture domain fusion results of each level;

[0085] The content domain fusion results and the texture domain fusion results of each level are fused to obtain feature cross-domain fusion results of each level.

[0086] Specifically, the image features include multiple levels, that is, the content features and the texture features both include multiple levels, the feature granularity of each level is different, for example, three levels or more levels can be included, which is not limited here.

[0087] Furthermore, the content feature extraction module and the texture feature extraction module are both hierarchical feature extraction modules, and the content feature fusion module is a hierarchical feature fusion module.

[0088] The content feature extraction module can include content feature extraction modules at different levels, respectively, to extract content features at different levels, and the content features at different levels can be represented by content feature maps. The texture feature extraction module can include texture feature extraction modules at different levels, respectively, to extract texture features at different levels, and the texture features at different levels can be represented by texture feature maps.

[0089] By using hierarchical semantic embedding, rich content information and texture information with different receptive fields can be obtained, and the forgery evidence based on content perception and texture perception can be enhanced hierarchically.

[0090] The content feature interaction module can include content feature interaction modules at different levels, respectively, to fuse content features at different levels to obtain content domain fusion results at different levels; the texture feature interaction module can include texture feature interaction modules at different levels, respectively, to fuse texture features at different levels to obtain texture domain fusion results at different levels; and the cross-domain feature interaction module can include cross-domain feature interaction modules at different levels, respectively, to fuse the content domain fusion results and the texture domain fusion results at different levels to obtain feature cross-domain fusion results at different levels.

[0091] Therefore, when the content features and the texture features include multiple levels, the CAML module and the TAML module in the embodiment of the application can be collectively referred to as a hierarchical attention map learning (HAML) module. The face forgery detection model is a hierarchical content-texture correlation learning (HCTER) network framework, which can greatly improve the generalization and robustness of the face detection method.

[0092] At this time, after the content features and the texture features are fused to obtain the content domain fusion results and the texture domain fusion results, and the content domain fusion results and the texture domain fusion results are fused to obtain the feature cross-domain fusion results, the content feature interaction module at each level is used to fuse the content features at each level to obtain the content domain fusion results at each level, and the texture feature interaction module at each level is used to fuse the texture features at each level to obtain the texture domain fusion results at each level. Thereafter, the cross-domain feature interaction module at each level is used to fuse the content domain fusion results and the texture domain fusion results at each level to obtain the feature cross-domain fusion results at each level.

[0093] In the embodiment of the present application, the image features include multiple levels, which greatly improves the generalization and robustness of the face detection method. Moreover, through the image feature fusion method of hierarchical levels, the feature domain fusion and cross-domain fusion are performed at each level, which can realize the fine fusion of content features and texture features, improve the fusion accuracy, and improve the accuracy of the forgery detection result.

[0094] On the basis of the above-mentioned embodiment, the face forgery detection method provided in the embodiment of the present application is based on a successive difference convolution operator, or based on a successive difference convolution operator and a central difference convolution operator.

[0095] Specifically, in the embodiment of the present application, the texture features of the face image to be detected are extracted by using a successive difference convolution (SDC) operator, or by using an SDC operator and a central difference convolution (CDC) operator. Due to the existence of the SDC operator, the continuity of the pixel level details can be captured to obtain potential discriminative forgery evidence.

[0096] Here, the SDC operator and the CDC operator can be collectively referred to as an extended difference convolution operator (EDC). Furthermore, the texture features can be extracted by a texture feature extraction module integrated with the SDC operator or the EDC operator. For example, the texture feature extraction module can be a TAML module integrated with the SDC operator or the EDC operator. The texture feature extraction module can mine fine-grained texture features that are robust to complex changing scenes, which is beneficial to capture the internal forgery patterns of the face image, and thus can significantly improve the generalization ability of the face forgery detection method.

[0097] On the basis of the above-mentioned embodiment, the face forgery detection method provided in the embodiment of the present application is based on a successive difference convolution operator, or based on a successive difference convolution operator and a central difference convolution operator.

[0098]

[0099] wherein f CDC (F l , k 1 ) represents the CDC operator, F l represents the texture features of the lth level, k 1 represents the central difference convolution kernel, z0 represents the current position of the current receptive field of k 1 , z n represents the positions other than z0 in the current receptive field of k 1 , and n represents k1 The current receptive field location index, Indicates k 1 The set of current receptive field locations, For k 1 The parameter at the i-th position, where i = n.

[0100] like Figure 2 The diagram shows the operation of the CDC operator. In this embodiment of the invention, the texture features of the eight surrounding positions are selected, and the difference between these differences and the texture feature of the current position is calculated. Then, the difference is compared with k. 1 Multiplying the parameters at the corresponding positions yields the CDC feature map after the CDC operator operation.

[0101] Because the CDC operator generates additional gradient-level texture information, it is beneficial for depth forgery detection compared to ordinary convolution.

[0102] Based on the above embodiments, the face forgery detection method provided in this embodiment of the invention, wherein the SDC operator is expressed based on the following formula:

[0103]

[0104] Among them, f SDC (F l k 2 ) represents the SDC operator, F l Represents the texture features of the l-th level, k 2 This represents the difference-by-difference convolution kernel, z n and z m All represent k 2 The current position of the receptive field, where n and m both represent the position of k. 2 The current receptive field location index, Indicates k 2 The set of current receptive field location pairs. For k 2 The parameter at the i-th position, where i = n.

[0105] The SDC operator can capture continuous pixel-level texture details, better optimize the robustness of the texture feature extraction module, and obtain potentially discriminative evidence of forgery.

[0106] like Figure 3 The diagram shows the operation of the SDC operator. In this embodiment of the invention, texture features from eight locations surrounding the current location are selected. The difference between texture features from any two locations spaced one location apart is calculated and then compared with k. 2 Multiplying the parameters at the corresponding positions yields the SDC feature map after the SDC operator operation.

[0107] On the basis of the above-mentioned embodiments, the face forgery detection method provided in the embodiments of the present application, the image feature of the to-be-detected face image is extracted, including:

[0108] For the lowest level, based on the initial feature of the lowest level of the to-be-detected face image, the image feature of the lowest level is determined;

[0109] For any level except the lowest level, based on the image feature of the previous level of the any level, the feature cross-domain fusion result and the initial feature of the any level, the image feature of the any level is determined.

[0110] Specifically, the content feature extraction module of each level can be a CAML module, and the CAML module can include a plurality of residual blocks, a batch normalization (BN) layer and a nonlinear activation (ReLU, sigmond, etc.) layer.

[0111] For the lth level, the input of the CAML module of the lth level includes the initial feature F l of the lth level and the feature cross-domain fusion result from the (l-1)th level. The CAML module of each level extracts the content feature of each level in an iterative manner, and the expression form of the content feature of the lth level can be:

[0112]

[0113] Wherein, β and σ represent the batch normalization layer and the nonlinear activation layer respectively. l represents the content feature of the lth level. is the feature cross-domain fusion result of the content domain of the (l-1)th level.

[0114] The initial feature of each level can be extracted by a multi-level backbone network encoder.

[0115] In particular, since the lowest level has no previous level, for the lowest level, the content feature thereof is directly determined according to the initial feature of the lowest level of the to-be-detected face image.

[0116] That is:

[0117] C l = σ(β(ResBlock(F l ))).

[0118] Similarly, the texture feature extraction module of each level can be a TAML module, and the TAML module integrates an EDC operator, that is, integrates an SDC operator and a CDC operator.

[0119] For the lth level, the input of the TAML module of the lth level includes the initial feature F of the lth level l , and the feature cross-domain fusion result of the (l-1)th level. The TAML module of each level extracts the texture feature of each level in an iterative manner, and the expression form of the texture feature of the lth level can be:

[0120]

[0121] Wherein, T l represents the texture feature of the lth level. is the feature cross-domain fusion result of the texture domain of the (l-1)th level.

[0122] In particular, since the lowest level has no previous level, for the lowest level, its texture feature is directly determined according to the initial feature of the lowest level of the face image to be detected.

[0123] That is:

[0124] T 1 = f CDC (F 1 , k 1 ) + f SDC ; (F 1 , k 2 ).

[0125] In the embodiment of the application, the image features include multiple levels, and through the image feature extraction mode of hierarchical levels, the feature extraction module of each level can be used to deeply explore the content semantic information and the texture detail information, and then the forged evidence of each feature domain in the face image to be detected can be mastered from coarse to fine.

[0126] On the basis of the above-mentioned embodiment, the face forgery detection method provided in the embodiment of the application includes the following steps:

[0127] Element-by-element superposition is performed on the image feature of the previous level and the feature cross-domain fusion result to obtain the feature superposition result of the previous level;

[0128] The feature superposition result of the previous level and the initial feature of the any level are merged in the channel dimension to obtain the image feature of the any level.

[0129] Specifically, in the embodiment of the present application, the feature superposition result of each level includes the content feature superposition result and the texture feature superposition result, so when determining the image feature of any level except the lowest level, for the content feature of any level, the content feature of the previous level can be first element-wise superimposed with the cross-domain fusion result of the feature of the content domain to obtain the content feature superposition result of the previous level, and then the content feature superposition result of the previous level is merged with the initial feature of any level in the channel dimension to obtain the content feature of any level.

[0130] Similarly, for the texture feature of any level, the texture feature of the previous level can be first element-wise superimposed with the cross-domain fusion result of the feature of the texture domain to obtain the texture feature superposition result of the previous level, and then the texture feature superposition result of the previous level is merged (Concat) with the initial feature of any level in the channel dimension to obtain the texture feature of any level.

[0131] In the embodiment of the present application, when determining the image feature of any level, the image feature of the previous level is introduced, so that more effective information can be used to extract the image feature of any level, and the obtained image feature of any level is more accurate.

[0132] On the basis of the above-mentioned embodiment, the face forgery detection method provided in the embodiment of the present application, the content feature of each level is fused to obtain the content domain fusion result of each level, and the texture feature of each level is fused to obtain the texture domain fusion result of each level, comprising:

[0133] Based on the feature interaction vector corresponding to each level, the content feature and the texture feature of each level are fused respectively to obtain the content domain fusion result and the texture domain fusion result of each level.

[0134] Specifically, the feature interaction vector corresponding to each level can include a content feature interaction vector and a texture feature interaction vector.

[0135] The content feature interaction module of each level can be an IntraAFI of the content domain, which is used to generate the content feature interaction vector corresponding to each level and fuse the content feature of each level by using the content feature interaction vector corresponding to each level to obtain the content domain fusion result of each level.

[0136] The texture feature interaction module of each level can be an IntraAFI of the texture domain, which is used to generate the texture feature interaction vector corresponding to each level and fuse the texture feature of each level by using the texture feature interaction vector corresponding to each level to obtain the texture domain fusion result of each level.

[0137] The content feature interaction vector corresponding to each level includes a one-dimensional content attention vector and a two-dimensional content gate feature vector The texture feature interaction vector corresponding to each level includes a one-dimensional texture attention vector and a two-dimensional texture gate feature vector The content feature interaction vector corresponding to each level includes a one-dimensional content attention vector and a two-dimensional content gate feature vector The texture feature interaction vector corresponding to each level includes a one-dimensional texture attention vector and a two-dimensional texture gate feature vector M l is the number of channels of the content feature of the lth level, N l is the number of channels of the texture feature of the lth level, H l is the total height of the content feature and the texture feature of the lth level, and W l is the total width of the content feature and the texture feature of the lth level.

[0138] The content domain fusion result of the lth level and the texture domain fusion result of the lth level can be determined by the following formula:

[0139]

[0140] wherein, represents Hadamard product of element-by-element multiplication, C l is the content feature of the lth level, and T l is the texture feature of the lth level.

[0141] On the basis of the above embodiment, the face forgery detection method provided in the embodiment of the application fuses the content domain fusion result and the texture domain fusion result of each level to obtain the feature cross-domain fusion result of each level, which comprises the following steps:

[0142] Based on the bilinear pooling method, the content domain fusion result and the texture domain fusion result of each level are subjected to feature embedding to obtain the content-texture semantic relationship of each level;

[0143] The content-texture semantic relationship of each level is subjected to nonlinear normalization, and the result obtained by the normalization is subjected to similarity aggregation;

[0144] Based on the result obtained by the similarity aggregation, the feature cross-domain fusion result of each level is determined.

[0145] Specifically, each level can correspond to a cross-domain feature interaction module, which can be CrossGFI for example. The CrossGFI of each level first utilizes the bilinear pooling method to perform feature embedding on the content domain fusion result and the texture domain fusion result of each level to obtain the content-texture semantic relationship of each level. The content-texture semantic relationship of each level can be represented by a content-texture semantic relationship matrix.

[0146] Bilinear pooling can be expressed as:

[0147]

[0148] wherein, is the component in the kth channel, ith height, and jth width of the lth layer, is the vector of the kth row in the content-texture semantic relationship matrix of the lth layer.

[0149] Stacking all together, the content-texture semantic relationship matrix is formed. The content-texture semantic relationship matrix has good interpretability: It can be regarded as a projection from the content domain to the texture domain, where the kth row represents the kth (k = 1, 2,..., N l ) texture-related quantization basis function, which has M l dimensional content features to be fused. In order to fully explore the high-order semantic relationship between different quantization basis functions, a graph convolutional network (GCN) is used for pseudo-evidence reasoning on the content-texture semantic relationship matrix, and the content features to be fused contained in each texture-related quantization basis function are regarded as a node in the graph.

[0150] Thereafter, the content-texture semantic relationship of each layer is nonlinearly normalized, which is shown in the following formula:

[0151]

[0152] wherein, A l is the result of normalization of the lth layer, representing the affinity matrix, and represent 1x1 convolutional layers for dimension conversion, and Softmax is a nonlinear normalization function.

[0153] Thereafter, the result of normalization of the lth layer is similarity aggregated. Here, the similarity aggregation can be performed by a dense fully connected graph convolutional neural network operator, i.e.,

[0154]

[0155] wherein, Y l is the result of similarity aggregation of the lth layer, represents the similarity matrix obtained after regularization of A l , and is a learnable graph convolutional weight.

[0156] Finally, according to the correlation between the characteristics of the content domain and the texture domain, Y l The texture domain fusion result applied to the lth level is refined through the cross-attention mechanism to obtain the cross-domain fusion result of the characteristics of the content domain of the lth level

[0157]

[0158] Similarly, the content domain fusion result applied to the lth level is refined through the cross-attention mechanism to obtain the cross-domain fusion result of the characteristics of the texture domain of the lth level l The texture domain fusion result applied to the lth level is refined through the cross-attention mechanism to obtain the cross-domain fusion result of the characteristics of the texture domain of the lth level

[0159]

[0160] The IntraAFI and CrossGAI modules are staggered at different levels to gradually enhance the content-texture representation with semantic relationships.

[0161] The principle of the CrossGFI of the lth level is as shown in Figure 4 The content domain fusion result of the lth level and the texture domain fusion result of the lth level are subjected to feature embedding to obtain the content-texture semantic relationship of the lth level The content-texture semantic relationship of the lth level is subjected to nonlinear normalization, and the result A l obtained by the normalization of the lth level is subjected to similarity aggregation to obtain the cross-domain fusion result of the characteristics of the content domain of the lth level l and the cross-domain fusion result of the characteristics of the texture domain of the lth level

[0162] In the embodiment of the application, the introduction of the bilinear pooling method, the nonlinear normalization operation and the similarity aggregation method can make the cross-domain fusion result of the characteristics of each level more accurate and improve the reliability thereof.

[0163] On the basis of the above embodiment, the face forgery detection method provided in the embodiment of the application comprises:

[0164] The cross-domain fusion result of the characteristics of the highest level is respectively element-wise superimposed with the content features and the texture features of the highest level to obtain the content superposition result and the texture superposition result of the highest level; ​

[0165] Based on the highest level content superposition result and the texture superposition result, the to-be-detected face image is classified into true and false to obtain the forgery detection result.

[0166] Specifically, in the embodiment of the present application, when the feature cross-domain fusion result is used to determine the forgery detection result of the to-be-detected face image, if the content feature and the texture feature both include one level, the feature cross-domain fusion result of the level is directly used to classify the to-be-detected face image into true and false to obtain the forgery detection result.

[0167] If the content feature and the texture feature both contain multiple levels, the feature cross-domain fusion result of the highest level is actually used to classify the to-be-detected face image into true and false to obtain the forgery detection result. At this time, the feature cross-domain fusion result of the highest level of the content domain is element-wise superimposed with the content feature of the highest level to obtain the content superposition result of the highest level, and the feature cross-domain fusion result of the highest level of the texture domain is element-wise superimposed with the texture feature of the highest level to obtain the texture superposition result.

[0168] After that, the to-be-detected face image is classified into true and false based on the highest level content superposition result and the texture superposition result to obtain the forgery detection result.

[0169] In the embodiment of the present application, the feature cross-domain fusion result of the highest level is used to superimpose the content feature and the texture feature to classify the to-be-detected face image into true and false, which can ensure the accuracy of the true and false classification and further improve the accuracy of the forgery detection result.

[0170] Figure 5 is a structural schematic diagram of the face forgery detection model used in the embodiment of the present application, Figure 5 For example, in the embodiment of the present application, the content feature and the texture feature both include three levels, which are low level (Low-level), intermediate level (Mid-level) and high level (High-level), respectively corresponding to coarse-grained features, medium-grained features and fine-grained features.

[0171] The Low-level IntraAFI, the Mid-level IntraAFI and the High-level IntraAFI both include the IntraAFI of the content domain and the IntraAFI of the texture domain.

[0172] The cross-domain feature interaction module includes Low-level CrossGFI, Mid-level CrossGFI and High-level CrossGFI.

[0173] Figure 5The backbone network encoders of each level include a backbone network encoder of a content domain and a backbone network encoder of a texture domain, and the backbone network encoder of the content domain of each level shares weights with the backbone network encoder of the texture domain. Figure 5 The element-wise superposition operation is represented by an operator and the merging operation is represented by a node C.

[0174] To sum up, the embodiment of the present application provides a method for general face forgery detection through content-texture correlation mining. The method performs fine-grained face forgery detection through content-texture attention map learning. In addition, a hierarchical content-texture correlation learning (HCTER) network framework is provided, which greatly improves the generalization and robustness of the face detection scheme. The HCTER network framework is an end-to-end face deep forgery detection scheme, which has good generalization for multiple datasets, multiple forgery types and multiple tampering modes.

[0175] At the same time, the method has good robustness to high noise and strong compression in real-world scenarios, and can be effectively deployed to terminal devices for forgery detection of media data or real-time face images on the Internet. The network model first uses a content-texture attention map learning (CTAML) model to deeply mine multi-level content-texture high-level semantic features in a coarse-to-fine hierarchical learning manner. Next, the HCTER network framework explores the high-order semantic correlation of content-texture features through an effective progressive multi-domain feature fusion module (PFMI), and interacts and fuses the intra-domain and cross-domain features in a reasonable manner. Finally, the method is fully experimentally verified and algorithmically evaluated on 6 commonly used public academic sets. A large number of experiments prove that the detection accuracy of the HCTER network framework is significantly improved compared with existing face forgery detectors. At the same time, in the cross-domain experimental evaluation and robustness analysis experiment, it is proved that the HCTER network framework has excellent generalization performance and model robustness.

[0176] As shown in Figure 6 On the basis of the above embodiment, the embodiment of the present application provides a face forgery detection device, which comprises:

[0177] The feature extraction module 61 is configured to acquire a face image to be detected and extract image features of the face image to be detected, wherein the image features include content features and texture features.

[0178] The feature fusion module 62 is configured to fuse the content features and the texture features respectively to obtain a content domain fusion result and a texture domain fusion result, and fuse the content domain fusion result and the texture domain fusion result to obtain a feature cross-domain fusion result.

[0179] The forgery detection module 63 is configured to determine a forgery detection result of the face image to be detected based on the feature cross-domain fusion result.

[0180] On the basis of the above-mentioned embodiments, the face forgery detection device provided in the embodiments of the present application, the image features include multiple levels;

[0181] Correspondingly, the feature fusion module is specifically configured to:

[0182] The content features of each level are fused to obtain the content domain fusion result of each level, and the texture features of each level are fused to obtain the texture domain fusion result of each level.

[0183] The content domain fusion result and the texture domain fusion result of each level are fused to obtain the feature cross-domain fusion result of each level.

[0184] On the basis of the above-mentioned embodiments, the face forgery detection device provided in the embodiments of the present application, the feature extraction module is specifically configured to:

[0185] For the lowest level, the image features of the lowest level are determined based on the initial features of the lowest level of the face image to be detected.

[0186] For any level other than the lowest level, the image features of the any level are determined based on the feature cross-domain fusion result of the previous level of the any level and the initial features of the any level.

[0187] On the basis of the above-mentioned embodiments, the face forgery detection device provided in the embodiments of the present application, the feature extraction module is further specifically configured to:

[0188] The image features of the previous level and the feature cross-domain fusion result are element-wise superimposed to obtain the feature superimposition result of the previous level.

[0189] The feature superimposition result of the previous level and the initial features of the any level are merged in the channel dimension to obtain the image features of the any level.

[0190] On the basis of the above-mentioned embodiments, the face forgery detection device provided in the embodiments of the present application, the feature fusion module is further specifically configured to:

[0191] The content features and the texture features of each level are fused based on the feature interaction vector corresponding to each level to obtain the content domain fusion result and the texture domain fusion result of each level.

[0192] On the basis of the above-mentioned embodiments, the face forgery detection device provided in the embodiments of the present application, the feature fusion module is further specifically configured to:

[0193] Based on the bilinear pooling method, the content domain fusion result and the texture domain fusion result of each level are embedded into features to obtain the content-texture semantic relationship of each level;

[0194] The content-texture semantic relationship of each level is normalized in a nonlinear manner, and the normalized results are aggregated in similarity;

[0195] Based on the similarity aggregation result, the feature cross-domain fusion result of each level is determined.

[0196] On the basis of the above-mentioned embodiments, the face forgery detection device provided in the embodiments of the present application, the forgery detection module, is specifically used for:

[0197] The feature cross-domain fusion result of the highest level is respectively element-wise superimposed with the content feature and the texture feature of the highest level to obtain the content superposition result and the texture superposition result of the highest level;

[0198] Based on the content superposition result and the texture superposition result of the highest level, the to-be-detected face image is classified into true or false to obtain the forgery detection result.

[0199] On the basis of the above-mentioned embodiments, the face forgery detection device provided in the embodiments of the present application, the texture feature is extracted based on the difference difference convolution operator or based on the difference difference convolution operator and the center difference convolution operator.

[0200] On the basis of the above-mentioned embodiments, the face forgery detection device provided in the embodiments of the present application, the difference difference convolution operator is represented based on the following formula:

[0201]

[0202] wherein, f SDC (F l , k 2 ) represents the difference difference convolution operator, F l represents the texture feature of the lth level, k 2 represents the difference difference convolution kernel, z n and z m both represent the position of the current receptive field of k 2 , n and m both represent the position index of the current receptive field of k 2 , represents the set of position pairs of the current receptive field of k 2 , is the parameter of the i th position in k 2 , i=n.

[0203] Specifically, the functions of each module in the face forgery detection device provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0204] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute the face forgery detection method provided in the above embodiments. This method includes: acquiring a face image to be detected and extracting image features from the face image to be detected, the image features including content features and texture features; fusing the content features and the texture features respectively to obtain a content domain fusion result and a texture domain fusion result, and fusing the content domain fusion result and the texture domain fusion result to obtain a cross-domain feature fusion result; and determining the forgery detection result of the face image to be detected based on the cross-domain feature fusion result.

[0205] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0206] In another aspect, the present application also provides a computer program product comprising a computer program, the computer program being stored in a non-transitory computer-readable storage medium, and the computer program being executable by a processor to enable a computer to perform the face forgery detection method provided in each of the above embodiments, the method comprising: obtaining a to-be-detected face image, and extracting image features of the to-be-detected face image, the image features comprising content features and texture features; fusing the content features and the texture features respectively to obtain content domain fusion results and texture domain fusion results, and fusing the content domain fusion results and the texture domain fusion results to obtain a feature cross-domain fusion result; and determining a forgery detection result of the to-be-detected face image based on the feature cross-domain fusion result.

[0207] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement a face forgery detection method provided in each of the above embodiments, the method comprising: obtaining a to-be-detected face image, and extracting image features of the to-be-detected face image, the image features comprising content features and texture features; fusing the content features and the texture features respectively to obtain content domain fusion results and texture domain fusion results, and fusing the content domain fusion results and the texture domain fusion results to obtain a feature cross-domain fusion result; and determining a forgery detection result of the to-be-detected face image based on the feature cross-domain fusion result.

[0208] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0209] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus a necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0210] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting face forgery, characterized in that, include: Acquire a face image to be detected and extract image features from the face image to be detected, the image features including content features and texture features; The content features and the texture features are fused separately to obtain a content domain fusion result and a texture domain fusion result. The content domain fusion result and the texture domain fusion result are then fused together to obtain a cross-domain feature fusion result. Based on the cross-domain fusion results of the features, the forgery detection result of the face image to be detected is determined.

2. The face forgery detection method according to claim 1, characterized in that, The image features include multiple levels; Accordingly, the process of fusing the content features and the texture features respectively to obtain a content domain fusion result and a texture domain fusion result, and then fusing the content domain fusion result and the texture domain fusion result to obtain a cross-domain feature fusion result, includes: The content features of each level are fused to obtain the content domain fusion result of each level, and the texture features of each level are fused to obtain the texture domain fusion result of each level. The content domain fusion results and texture domain fusion results of each level are fused to obtain the feature cross-domain fusion results of each level.

3. The face forgery detection method according to claim 2, characterized in that, The extraction of image features from the face image to be detected includes: For the lowest level, the image features of the lowest level are determined based on the initial features of the lowest level of the face image to be detected; For any level other than the lowest level, the image features of that level are determined based on the cross-domain fusion result of the features of the previous level and the initial features of that level.

4. The face forgery detection method according to claim 3, characterized in that, The process of determining the image features of any level based on the cross-domain fusion result of the features of the previous level and the initial features of any level includes: The image features of the previous level and the feature cross-domain fusion result are superimposed element by element to obtain the feature superposition result of the previous level; The feature overlay result of the previous level is merged with the initial feature of any level in the channel dimension to obtain the image feature of any level.

5. The face forgery detection method according to claim 2, characterized in that, The process of fusing content features at each level to obtain a content domain fusion result for each level, and fusing texture features at each level to obtain a texture domain fusion result for each level, includes: Based on the feature interaction vector corresponding to each level, the content features and texture features of each level are fused to obtain the content domain fusion result and the texture domain fusion result of each level.

6. The face forgery detection method according to claim 2, characterized in that, The process of fusing the content domain fusion results and texture domain fusion results at each level to obtain the feature cross-domain fusion results at each level includes: Based on the bilinear pooling method, feature embedding is performed on the content domain fusion result and texture domain fusion result of each level to obtain the content-texture semantic relationship of each level. Nonlinear normalization is performed on the semantic relationships of content texture at each level, and the normalization results are then aggregated for similarity. Based on the results obtained from similarity aggregation, the cross-domain fusion results of features at each level are determined.

7. The face forgery detection method according to claim 2, characterized in that, The step of determining the forgery detection result of the face image to be detected based on the cross-domain fusion result of the features includes: The cross-domain fusion result of the highest-level feature is superimposed element-wise with the content feature and texture feature of the highest-level feature to obtain the content superposition result and texture superposition result of the highest-level feature; Based on the content overlay result and texture overlay result at the highest level, the face image to be detected is classified into real and fake categories to obtain the forgery detection result.

8. The face forgery detection method according to any one of claims 1-7, characterized in that, The texture features are extracted based on the successive difference convolution operator, or based on the successive difference convolution operator and the central difference convolution operator.

9. The face forgery detection method according to claim 8, characterized in that, The successive difference convolution operator is expressed based on the following formula: Among them, f SDC (F l k 2 ) represents the difference-by-difference convolution operator, F l Represents the texture features of the l-th level, k 2 This represents the difference-by-difference convolution kernel, z n and z m All represent k 2 The current position of the receptive field, where n and m both represent the position of k. 2 The current receptive field location index, Indicates k 2 The set of current receptive field location pairs. For k 2 The parameter at the i-th position, i = n.

10. A face forgery detection device, characterized in that, include: The feature extraction module is used to acquire a face image to be detected and extract the image features of the face image to be detected, the image features including content features and texture features; The feature fusion module is used to fuse content features and texture features separately to obtain content domain fusion results and texture domain fusion results, and then fuse the content domain fusion results and the texture domain fusion results to obtain cross-domain feature fusion results; The forgery detection module is used to determine the forgery detection result of the face image to be detected based on the cross-domain fusion result of the features.

Images