A face forgery detection method and system based on multi-view collaborative fusion

Abstract

The application discloses a face forgery detection method and system based on multi-view cooperative fusion, and relates to the technical field of computer vision, and the method comprises the following steps: acquiring video frame data to be detected; pre-processing the video frame data to be detected to obtain a standardized input image tensor; inputting the standardized input image tensor into a pre-trained face forgery detection model, processing the face forgery detection model, and generating a face forgery detection result. The application solves the technical problem of poor detection performance of the model in cross-library and complex environments, significantly improves the detection robustness of the model in complex scenes such as blurring and compression, and significantly improves the detection accuracy.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of computer vision, and particularly relates to a face forgery detection method and system based on multi-view collaborative fusion. BACKGROUND

[0002] In recent years, with the explosive growth of deep generation technologies such as generative adversarial networks (GAN) and diffusion models, the generation threshold of deepfake videos is lowered, and the realism is greatly improved. Such forged content not only infringes on personal privacy, but also poses a serious challenge to judicial evidence, identity authentication and national information security. Therefore, developing a deepfake detection algorithm with high precision and high generalization has become a research hotspot.

[0003] Existing deepfake detection methods mainly fall into two categories. The first category is the method based on spatial domain, which mainly extracts the RGB pixel features of the image through convolutional neural network (CNN). However, with the progress of generation technology, it is extremely difficult to distinguish the forged image visually, and simply relying on spatial domain features is easy to ignore the subtle high-frequency artifacts left in the generation process, resulting in poor robustness when facing highly compressed or post-processed videos. The second category is the method based on frequency domain, which uses Fourier transform or DCT transform to analyze the spectral abnormalities. Although this method can capture some forgery traces, it often lacks precise positioning ability for local spatial features, and it is difficult to effectively separate the content information and the forgery noise of the image. The existing fusion method of deepfake detection is mainly simple feature splicing, and lacks modeling of the internal correlation between "global semantics" and "local texture". In complex scenarios, due to the lack of global prior guidance, the model is easily disturbed by environmental noise, and it is difficult to focus on the real tampering boundary, thereby affecting the detection accuracy in complex environments. Therefore, there is an urgent need for a collaborative fusion detection algorithm that can effectively decouple the image frequency band information and use global prior to guide local texture feature verification, in order to improve the detection performance of the model in cross-library and complex environments.

[0004] Therefore, there is a need for a face forgery detection method and system based on multi-view collaborative fusion. SUMMARY

[0005] The embodiment of the present specification provides a face forgery detection method and system based on multi-view collaborative fusion, which is used to solve the following technical problem: the existing fusion method of deepfake detection is mainly simple feature splicing, and lacks modeling of the internal correlation between "global semantics" and "local texture". In complex scenarios, due to the lack of global prior guidance, the model is easily disturbed by environmental noise, and it is difficult to focus on the real tampering boundary, thereby affecting the detection accuracy in complex environments.

[0006] To solve the above technical problems, the embodiment of the present specification is implemented as follows:

[0007] The embodiment of the present specification provides a face forgery detection method based on multi-view collaborative fusion, comprising:

[0008] Obtaining video frame data to be detected;

[0009] Preprocessing the video frame data to be detected to obtain a standardized input image tensor;

[0010] Inputting the standardized input image tensor into a pre-trained face forgery detection model, and generating a face forgery detection result through the face forgery detection model processing, wherein the face forgery detection model processing process comprises: based on the standardized input image tensor, using a recursive three-level discrete wavelet transform and a wavelet multi-band construction module to obtain a style flow tensor and a texture flow tensor; inputting the standardized input image tensor, the style flow tensor and the texture flow tensor into a multi-view hybrid feature extraction network to obtain a global semantic vector, a style feature map and a texture feature map, the multi-view hybrid feature extraction network comprising: a global feature extraction branch, a style encoding branch and a texture encoding branch; performing feature checking on the style feature map and the texture feature map through a mutual attention mechanism to generate an interaction feature; taking a spatial gate mask as an adaptive adjustment coefficient, performing weighted summation on the splicing feature and the interaction feature to generate a final feature, the splicing feature being generated by splicing the interaction feature and the global semantic vector, and the spatial gate mask being formed by mapping the splicing feature through a convolution layer; the final feature is inputted into a spatial pyramid confidence decision module to generate a forgery prediction score of each scale and a corresponding confidence weight of each scale; based on the forgery prediction score of each scale and the confidence weight of each scale, a face forgery prediction probability value is generated for face forgery detection.

[0011] The embodiment of the present specification also provides a face forgery detection system based on multi-view collaborative fusion, comprising:

[0012] An acquisition module for acquiring video frame data to be detected;

[0013] A preprocessing module for preprocessing the video frame data to be detected to obtain a standardized input image tensor;

[0014] The detection module inputs the standardized input image tensor into a pre-trained face forgery detection model, and generates a face forgery detection result through processing of the face forgery detection model, wherein the processing of the face forgery detection model comprises: based on the standardized input image tensor, a recursive three-level discrete wavelet transform and a wavelet multi-band construction module are used to obtain a style flow tensor and a texture flow tensor; the standardized input image tensor, the style flow tensor and the texture flow tensor are input into a multi-view mixed feature extraction network to obtain a global semantic vector, a style feature map and a texture feature map, the multi-view mixed feature extraction network comprises: a global feature extraction branch, a style encoding branch and a texture encoding branch; the style feature map and the texture feature map are subjected to feature checking through an interaction attention mechanism to generate interaction features; a spatial gate mask is used as an adaptive adjustment coefficient to perform weighted summation on the interaction features and spliced features to generate final features, the spliced features are generated by splicing the interaction features and the global semantic vector, and the spatial gate mask is formed by mapping the spliced features through a convolution layer; the final features are subjected to a spatial pyramid confidence decision module to generate forgery prediction scores of each scale and corresponding confidence weights of each scale; and based on the forgery prediction scores of each scale and the confidence weights of each scale, a face forgery prediction probability value is generated for face forgery detection.

[0015] The face forgery detection method based on multi-view collaborative fusion provided by the embodiments of the present specification can obtain video frame data to be detected; the video frame data to be detected is preprocessed to obtain a standardized input image tensor; the standardized input image tensor is input into a pre-trained face forgery detection model, and a face forgery detection result is generated through processing of the face forgery detection model, which can avoid the influence of background environment changes on face forgery detection, realize the acquisition of face image all-around and multi-view deep representation, and make up for the deficiency caused by high-frequency information loss, significantly improve the detection robustness of the model in complex scenes such as blurring and compression, and significantly improve the detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present specification, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0017] Figure 1 A system architecture schematic diagram of a face forgery detection method based on multi-view collaborative fusion provided by the embodiments of the present specification is shown in the following figure:

[0018] Figure 2A flowchart of a face forgery detection method based on multi-view collaborative fusion is provided for the embodiments of the present specification.

[0019] Figure 3 A wavelet multi-band construction module network diagram is provided for the embodiments of the present specification.

[0020] Figure 4 A style encoding and texture encoding branch network diagram in a multi-view hybrid feature extraction network is provided for the embodiments of the present specification.

[0021] Figure 5 A spatial pyramid confidence decision module network diagram is provided for the embodiments of the present specification.

[0022] Figure 6 A training diagram of a face forgery detection model is provided for the embodiments of the present specification.

[0023] Figure 7 A schematic diagram of a face forgery detection system based on multi-view collaborative fusion is provided for the embodiments of the present specification.

[0024] Legend of reference signs:

[0025] LL represents row low pass and column low pass, LH represents vertical direction column high pass output, HL represents horizontal direction column high pass output, HH represents diagonal direction column high pass output, CDC represents center difference convolution, SRM represents spatial rich model steganalysis, system architecture 100, terminal device 101, network 102, server 103, acquisition module 701, preprocessing module 703, detection module 705. DETAILED DESCRIPTION

[0026] In order for those skilled in the art to better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be described clearly and completely below in conjunction with the drawings in the embodiments of the present specification. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments of the present specification, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0027] Figure 1 A system architecture diagram of a face forgery detection method based on multi-view collaborative fusion is provided for the embodiments of the present specification. As shown in Figure 1 The system architecture 100 can include at least one terminal device 101, a network 102 and a server 103. The network 102 is used as a medium to provide a communication link between the terminal device 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0028] The terminal device 101 interacts with the server 103 through the network 102 to receive or send messages, etc. Various client applications can be installed on the terminal device 101. For example, a special program based on multi-view collaborative fusion face spoofing detection method.

[0029] The terminal device 101 can be hardware or software. When the terminal device 101 is hardware, it can be various special or general electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When the terminal device 101 is software, it can be installed in the above-mentioned electronic devices. It can be implemented as multiple software or software modules (for example, multiple software or software modules used to provide distributed services), or as a single software or software module.

[0030] The server 103 can be a server that provides various services, such as a backend server that provides services for client applications installed on the terminal device 101. For example, the server can perform face spoofing detection based on multi-view collaborative fusion to display face spoofing detection results based on multi-view collaborative fusion on the terminal device 101.

[0031] The server 103 can be hardware or software. When the server 103 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server 103 is software, it can be implemented as multiple software or software modules (for example, multiple software or software modules used to provide distributed services), or as a single software or software module.

[0032] Figure 2 A flowchart of a face spoofing detection method based on multi-view collaborative fusion provided by an embodiment of the present specification. From a program perspective, the execution subject of the flowchart can be a program loaded on an application server or an application terminal. It can be understood that the method can be executed by any device, equipment, platform, device cluster with computing and processing capabilities. As shown in the figure, the face spoofing detection method includes: Figure 2

[0033] Step S201: Obtain video frame data to be detected.

[0034] In the embodiment of the present specification, the video frame data to be detected is a video frame that may have face spoofing. Specifically, it can be a live scene or other scenes. The scene where the video frame data to be detected is located does not constitute a limitation of the present application, and therefore will not be described again.

[0035] Step S203: Preprocess the video frame data to be detected to obtain a standardized input image tensor.​

[0036] Detecting the face of each video frame to be detected one by one by using a face key point detection tool (such as DLIB), and locating the face key points, including: the lower jaw contour, the eyebrow area, the eye area, the nose area and the mouth area; performing affine transformation alignment and cutting on the face image based on the key point coordinates, eliminating the differences in posture and angle, and then performing standardization preprocessing on the aligned face image to obtain a standardized input image tensor , wherein N is the batch size, C is the channel size, is the image space resolution, and a unified data basis is provided for subsequent model training and inference.

[0037] Step S205: inputting the standardized input image tensor into a pre-trained face forgery detection model, and generating a face forgery detection result through the face forgery detection model, wherein the face forgery detection model processing process includes: based on the standardized input image tensor, using a recursive three-level discrete wavelet transform and a wavelet multi-band construction module to obtain a style flow tensor and a texture flow tensor; inputting the standardized input image tensor, the style flow tensor and the texture flow tensor into a multi-view mixed feature extraction network to obtain a global semantic vector, a style feature map and a texture feature map, the multi-view mixed feature extraction network includes: a global feature extraction branch, a style encoding branch and a texture encoding branch; performing feature verification on the style feature map and the texture feature map through a mutual attention mechanism to generate an interaction feature; taking a spatial gate mask as an adaptive adjustment coefficient to perform weighted summation on the spliced feature and the interaction feature to generate a final feature, the spliced feature is generated by splicing the interaction feature and the global semantic vector, and the spatial gate mask is formed by mapping the spliced feature through a convolution layer; the final feature is input into a spatial pyramid confidence decision module to generate a forgery prediction score and a corresponding confidence weight of each scale; based on the forgery prediction score and the confidence weight of each scale, a face forgery prediction probability value is generated for face forgery detection.

[0038] In the embodiments of the present specification, based on the standardized input image tensor, a recursive three-level discrete wavelet transform and a wavelet multi-band construction module are used to obtain a style flow tensor and a texture flow tensor, specifically including:

[0039] Taking Haar wavelet basis as a decomposition kernel, a low-pass filter and a high-pass filter are used to perform recursive three-level discrete wavelet transform on the standardized input image tensor to generate first to third level low-frequency approximation subbands and first to third level high-frequency detail subbands;

[0040] Performing bilinear interpolation up-sampling on the third level low-frequency approximation subband to a preset resolution to generate the style flow tensor;

[0041] The first-3-level high-frequency detail subbands are bilinearly interpolated and upsampled to the preset resolution, and then concatenated in the channel dimension depth to obtain the texture flow tensor.

[0042] In the embodiments of the present specification, the kernel coefficient vector of the low-pass filter is , and the kernel coefficient vector of the high-pass filter is ;

[0043] The first-3-level low-frequency approximation subband formula is:

[0044] ;

[0045] wherein,

[0046] ;

[0047] represents a spatial coordinate, represents a horizontal index, represents a vertical index;

[0048] represents the output variable of the spatial coordinate in the first-3-level low-frequency subband;

[0049] represents a row low-pass and a column low-pass;

[0050] represents a row direction coefficient of the low-pass filter;

[0051] represents a column direction coefficient of the low-pass filter;

[0052] represents a row direction offset index;

[0053] represents a column direction offset index;

[0054] The style flow tensor is: ;

[0055] wherein,

[0056] represents a style flow tensor;

[0057] R represents a data type;

[0058] represents the preset resolution;

[0059] ​​​ represents a preset height;

[0060] represents a preset width;

[0061] R represents a data type;

[0062] N represents a batch size;

[0063] 3 represents a number of channels;

[0064] the first to third level high-frequency detail subbands ;

[0065] wherein,

[0066] LH represents a vertical direction column high-pass output;

[0067] HL represents a horizontal direction column high-pass output;

[0068] HH represents a diagonal direction column high-pass output;

[0069] the texture flow tensor ;

[0070] wherein,

[0071] 9 represents a number of channels;

[0072] a calculation formula of the texture flow tensor is:

[0073] ;

[0074] wherein,

[0075] represents connection;

[0076] represents up-sampling;

[0077] represents fine high frequency;

[0078] represents medium frequency detail;

[0079] represents coarse outline;

[0080] , .

[0081] The wavelet transform realizes physical level depth decoupling of image style and texture, effectively solves the problem that the traditional convolutional neural network is easy to lose high frequency in the down-sampling process and fake traces, and strengthens the capture ability of the model to subtle tampering artifacts.

[0082] In the embodiment of the present specification, the standardized input image tensor, the style flow tensor and the texture flow tensor are input into the multi-view mixed feature extraction network to obtain a global semantic vector, a style feature map and a texture feature map, specifically comprising:

[0083] The standardized input image tensor is input into the global feature extraction branch to obtain the global semantic vector; the style flow tensor is input into the style encoding branch to obtain the style feature map; and the texture flow tensor is input into the texture encoding branch to obtain the texture feature map.

[0084] Figure 3 A wavelet multi-band construction module network diagram is provided for the embodiment of the present specification. As shown in Figure 3 , the low-frequency approximation sub-band obtained by selecting the 3rd level decomposition , the dimension of which is . Bilinear interpolation upsampling is performed on it to the specified resolution , and the style flow tensor representing the global illumination and topological structure is constructed.

[0085] All high-frequency detail sub-bands of the 1st to 3rd levels are extracted. The high-frequency feature set of the 3rd level is defined as , wherein belongs to fine high frequency, is medium frequency detail, and is coarse outline.

[0086] Further, all of are upsampled to a unified resolution . Then, a deep cascade is performed in the channel dimension to construct a texture flow tensor with a channel number of 9, and the calculation formula is as follows: ;

[0087] wherein, represents a bilinear interpolation operation.

[0088] In the embodiment of the present specification, the standardized input image tensor is input into the global feature extraction branch to obtain the global semantic vector, specifically comprising:

[0089] The standardized input image tensor is input into the global feature extraction branch, and high-level abstract semantic features are obtained based on the deep convolution structure of the global feature extraction branch, wherein the global feature extraction branch adopts EfficientNet-B4 as the backbone network;

[0090] Global average pooling is performed based on the high-level abstract semantic features to obtain the global semantic vector; ​

[0091] The calculation formula of the global semantic vector is:

[0092] ;

[0093] Wherein,

[0094] The global semantic vector is represented as

[0095] The preset height is represented as

[0096] The preset width is represented as

[0097] The height is represented as

[0098] The width is represented as

[0099] The high-level abstract semantic feature is represented as

[0100] The style flow tensor is generated into a style feature map through the style encoding branch, specifically including:

[0101] The style flow tensor is input into the style encoding branch, mapped to a high-dimensional latent feature space through the first convolutional module of the style encoding branch, and an intermediate feature map is obtained;

[0102] Based on the intermediate feature map, the channel correlation is calculated to obtain a Gram matrix;

[0103] The Gram matrix is flattened and input into the multi-layer perceptron of the style encoding branch to obtain a style embedding vector;

[0104] The style embedding vector is injected into the intermediate feature map through element-wise multiplication to generate the style feature map;

[0105] The texture flow tensor is obtained through the texture encoding branch, specifically including:

[0106] The texture flow tensor is input into the texture encoding branch, and the first convolutional module of the texture encoding branch is used for convolution to generate a residual feature map;

[0107] The residual feature map is input into the non-pooling feature extraction network of the texture encoding branch to generate deep texture features;

[0108] The deep texture features are input into the CDC center difference convolution network, and the standard convolution item and the center difference item are weighted to generate CDC enhanced features;

[0109] The CDC enhancement feature is input into a convolutional layer to generate the texture feature map.

[0110] In the embodiments of the present specification, the feature checking of the style feature map and the texture feature map through the attention mechanism generates interaction features, and specifically includes:

[0111] The style feature map is taken as a condition to generate a query tensor, and the texture feature map is taken as a basis to generate a key tensor and a value tensor;

[0112] The similarity weight is calculated based on the query tensor and the key tensor through the attention mechanism to generate an attention weight matrix;

[0113] The value tensor is weighted and summed based on the attention weight matrix to generate the interaction features.

[0114] Figure 4 The style encoding and texture encoding branch network diagram of the multi-view hybrid feature extraction network provided in the embodiments of the present specification is shown. In the embodiments of the present specification, the multi-view hybrid feature extraction network refers to a network for extracting a global semantic vector, a style feature map and a texture feature map, which respectively extracts the macro-structure consistency of a face, environmental style features and high-fidelity forgery traces.

[0115] As shown in Figure 4 , a normalized input image tensor is input into a global feature extraction branch, and high-level abstract semantic features are obtained based on the deep convolutional structure of the global feature extraction branch, and then global average pooling is performed on the high-level abstract semantic features to compress and aggregate the information of the spatial dimension to obtain a global semantic vector .

[0116] In order to convert the color distribution of the physical space into an abstract representation of the feature space, the style flow tensor is further input into a style encoding branch, and the style flow tensor is mapped to a high-dimensional latent feature space through the first convolutional module of the style encoding branch to obtain an intermediate spatial feature map .

[0117] Further, in order to capture the global style pattern, the channel correlation of the intermediate spatial feature map M is calculated to obtain a Gram matrix . This step explicitly models the second-order covariance statistical characteristics between texture primitives by calculating the inner product between different channel responses, and the calculation formula is as follows:

[0118] ;

[0119] wherein represents the first the first channel and the second channel, i.e., the style fingerprint.

[0120] Further flattening the Gram matrix into a one-dimensional vector, inputting the style encoding branch multi-layer perception, obtaining a style embedding vector, so as to realize the compression of the high-dimensional covariance matrix into a compact, semantically discriminative style embedding vector .

[0121] Finally, by element-wise multiplication, the style embedding vector is injected into the intermediate feature map, and the local spatial features are adaptively weighted and calibrated by the global style prior, to generate a style feature map , the calculation formula is as follows:

[0122] ;

[0123] wherein, represents a convolutional layer with an output channel of , represents element-wise multiplication.

[0124] The texture flow tensor feature is input into the texture encoding branch, and the first layer of convolution kernel weights of the network are fixedly initialized as 30 high-pass filters of the spatial rich model and do not participate in the backward propagation update. The input is convolved by to generate a residual feature map .

[0125] Further, the residual feature map is input into a non-pooling feature extraction network composed of multiple residual blocks (ResBlock). The spatial resolution of the feature map is kept unchanged to avoid the loss of subtle fake traces caused by down-sampling, and deep texture features are extracted.

[0126] Further, the deep texture features are input into the CDC center difference convolution network, and the module not only aggregates local pixel intensity, but also calculates the gradient difference of the pixel and its neighborhood. For the center pixel on the feature map, its output is composed of a standard convolution term and a center difference term, aiming to strengthen the sensitivity of the feature map to the tampered boundary and heterogeneous texture by using gradient information, and its formula is: ;

[0127] wherein is a balance coefficient (usually set to 0.7), is an input feature value, is a local neighborhood, The shared weights are learnable.

[0128] The CDC-enhanced feature map is passed through a convolutional layer for channel fusion and dimension alignment, finally outputting a high-fidelity texture feature map containing rich high-frequency details and tampering traces .

[0129] Style feature map is generated as a conditional query tensor Q, and the texture feature map is used to generate key tensor K and value tensor V, calculate attention weight matrix and reconstruct local texture to obtain interaction feature , the calculation formula is:

[0130] ;

[0131] ;

[0132] ;

[0133] ;

[0134] wherein, is a learnable linear projection weight matrix, is a scaling factor of feature dimension.

[0135] Global semantic vector is broadcasted and expanded in the spatial dimension to construct an expanded global tensor consistent with the spatial resolution of , and is concatenated with in the channel dimension to obtain the concatenated feature , which is then mapped to a spatial gating mask through a convolutional layer , which is used to adaptively adjust the feature response: ;

[0136] wherein, represents a convolutional layer with an output channel of 1, represents a Sigmoid activation function.

[0137] Further, the spatial gating mask is used as an adaptive adjustment coefficient to perform weighted summation on the mixed feature mapped by dimension reduction and the original interaction feature . This step adaptively introduces global information repair features in areas affected by environmental interference (such as overexposure and blur), while forcing to retain the original high-frequency details in areas with clear texture, to obtain the final feature : ;

[0138] in, Indicates the output channel is convolutional layers, This indicates element-wise multiplication. This determines that at each pixel, the focus is more on hybrid features that incorporate global semantics. Or should the original verified texture features be retained? .

[0139] Furthermore, Input space pyramid confidence decision module, executed in parallel at the global level ( ), regional level ( ) and detail level ( Multi-granularity pooling, for the first Each scale ( The target spatial resolution is defined as follows: (in );right Perform adaptive average pooling and flatten the output tensor to obtain the first... Scaled eigenvectors : .

[0140] In the embodiments of this specification, the final features are processed by the spatial pyramid confidence decision module to generate forgery prediction scores at each scale and corresponding confidence weights at each scale, specifically including:

[0141] The final features are processed by the spatial pyramid confidence decision module, which performs multi-granularity pooling at the global, regional, and detail levels to generate feature vectors at each scale.

[0142] Each feature vector in the feature vectors of each scale is processed by the classification head network to generate a fake prediction score for each scale and a corresponding confidence weight for each scale.

[0143] In the embodiments described in this specification, the classification head network comprises: two fully connected layers, a BatchNorm layer, and an activation function;

[0144] The process of generating a face forgery prediction probability value based on the forgery prediction scores at each scale and the confidence weights at each scale specifically includes:

[0145] The forgery prediction scores at each scale are multiplied by the confidence weights at each scale, then summed and normalized to generate the face forgery prediction probability value.

[0146] For feature vectors at each scale Construct an independent classification head network The classification head network consists of two fully connected layers, a BatchNorm layer, and an activation function. Input the corresponding classification head network Obtain the fake prediction score and confidence weight at this scale. Its internal calculation logic is as follows:

[0147] ,

[0148] ;

[0149] in, This indicates an input mapping fully connected layer;

[0150] Indicates batch normalization;

[0151] This represents a fully connected layer for classification mapping, used to map hidden layer features to forgery discrimination values;

[0152] This indicates a fully connected layer that maps confidence level to hidden layer features, used to map them to confidence levels.

[0153] Figure 5 This is a network diagram of the spatial pyramid confidence decision module provided in the embodiments of this specification. Figure 5 As shown, the final feature After adaptive average pooling and flattening, the first... Scaled eigenvectors That is, P1, P2, and P3; input P1, P2, and P3 into the classification head network. ,Right now Then, the global-level forgery prediction score is obtained as y1, the global-level confidence weight is w1, the regional-level forgery prediction score is y2, the regional-level confidence weight is w2, and the detail-level forgery prediction score is y3, with the detail-level confidence weight being w3. The probability value of face forgery prediction is then calculated. It should be noted that, They belong to the same classifier network, but are represented differently for ease of description and understanding.

[0154] If the predicted probability of face spoofing is greater than 50%, it is considered to be face spoofing.

[0155] In the embodiments described in this specification, the loss function used in the training process of the face forgery detection model is a composite loss function;

[0156] The composite loss function is: ;

[0157] in, Primary classification loss, For orthogonal decoupling loss, For uncertainty regularization loss, and These are the hyperparameter weights used to balance the magnitudes of the various loss terms;

[0158] The main loss function is: ;

[0159] in, Indicates the first The true label of each sample;

[0160] This represents the probability of forgery predicted by the face forgery detection model;

[0161] The orthogonal decoupling loss is: ;

[0162] in, This means flattening a multidimensional feature map into a one-dimensional vector;

[0163] express Norm;

[0164] Uncertainty regularization loss is: ;

[0165] in, For the first Predicted scores for scale;

[0166] This is a real label;

[0167] This is the learnable confidence parameter corresponding to this scale.

[0168] It should be noted that the samples are for face spoofing detection.

[0169] In the embodiments of this specification, the concatenation feature is generated by concatenating the interaction feature and the global semantic vector, specifically including:

[0170] The global semantic vector is broadcast and expanded to generate an extended global vector;

[0171] The extended global vector is concatenated with the interaction feature to generate the concatenated feature.

[0172] To further understand the multi-view collaborative fusion face detection method provided in the embodiments of this specification, Figure 6 This is a training diagram of a face forgery detection model provided in an embodiment of this specification.Figure 6 As shown, the face video frame data is taken as a face video frame data set, data set preprocessing is performed, and a standardized data set is generated; the standardized data set passes through a global semantic feature extraction and expansion module to generate an expanded global tensor; the standardized data set passes through a wavelet multi-band construction module to generate a low-frequency style flow tensor and a high-frequency texture flow tensor; the style flow tensor passes through a style coding branch to generate a style feature map, and the texture flow tensor passes through a texture coding branch to generate a texture feature map; the style feature map and the texture feature map pass through an interaction attention mechanism module to generate an interaction feature, and the interaction feature and the global semantic feature pass through splicing to generate a spliced feature; the spliced feature and the interaction feature are weighted and summed through a spatial gating mask to generate a final feature; the final feature passes through a spatial pyramid confidence decision module to generate each scale forgery prediction score and a corresponding each scale confidence weight, in the face forgery detection model training process, the loss function selects a composite loss, and finally, based on each scale forgery prediction score and a corresponding each scale confidence weight, a face forgery prediction probability value is generated for face forgery detection.

[0173] The face forgery detection method based on multi-view collaborative fusion provided by the embodiments of the present specification can obtain video frame data to be detected; the video frame data to be detected is preprocessed to obtain a standardized input image tensor; the standardized input image tensor is input into a pre-trained face forgery detection model, and a face forgery detection result is generated through processing of the face forgery detection model, which can avoid the influence of background environment changes on face forgery detection, realize the acquisition of face image all-around and multi-view deep representation, and make up for the deficiency caused by high-frequency information loss, significantly improve the detection robustness of the model in complex scenes such as blur and compression, and significantly improve the detection accuracy.

[0174] The above describes a face forgery detection method based on multi-view collaborative fusion, and accordingly, the present specification also provides a face forgery detection system based on multi-view collaborative fusion, as shown in Figure 7 . Figure 7 A face forgery detection system based on multi-view collaborative fusion provided by the embodiments of the present specification is a schematic diagram of the face forgery detection system, which comprises:

[0175] The acquisition module 701 acquires video frame data to be detected;

[0176] The preprocessing module 703 pre-processes the video frame data to be detected to obtain a standardized input image tensor;

[0177] The detection module 705 inputs the standardized input image tensor into a pre-trained face forgery detection model, processes the face forgery detection model to generate a face forgery detection result, wherein the face forgery detection model processing process comprises: based on the standardized input image tensor, using a recursive three-level discrete wavelet transform and a wavelet multi-band construction module to obtain a style flow tensor and a texture flow tensor; inputting the standardized input image tensor, the style flow tensor and the texture flow tensor into a multi-view mixed feature extraction network to obtain a global semantic vector, a style feature map and a texture feature map, the multi-view mixed feature extraction network comprising: a global feature extraction branch, a style encoding branch and a texture encoding branch; performing feature verification on the style feature map and the texture feature map through an attention mechanism to generate an interaction feature; using a spatial gate mask as an adaptive adjustment coefficient to perform weighted summation on the interaction feature and a splicing feature to generate a final feature, the splicing feature being generated by splicing the interaction feature and the global semantic vector, and the spatial gate mask being formed by mapping the splicing feature through a convolution layer; the final feature is input into a spatial pyramid confidence decision module to generate a forgery prediction score and a corresponding confidence weight of each scale; based on the forgery prediction score and the confidence weight of each scale, a face forgery prediction probability value is generated for face forgery detection.

[0178] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in the embodiments and still achieve the desired result. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible.

[0179] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device, electronic device, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0180] The device, electronic device, and non-volatile computer storage medium provided by the embodiments of the present specification correspond to the method, and therefore, the device, electronic device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding device, electronic device, and non-volatile computer storage medium will not be described here.

[0181] In the 1990s, it was possible to distinguish whether an improvement in a technology was a hardware improvement (e.g., an improvement in the circuit structure of a diode, transistor, switch, etc.) or a software improvement (an improvement in a method flow). However, as technology has advanced, many improvements in method flows today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into a hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A designer programs a digital system "integrated" on a PLD by himself / herself, without having to ask a chip manufacturer to design and manufacture a special integrated circuit chip. Moreover, instead of manually manufacturing an integrated circuit chip, this programming is now mostly implemented using "logic compiler" software, which is similar to a software compiler used when developing a program, and the original code before compilation must also be written in a specific programming language, which is called a hardware description language (HDL), and there are many types of HDL, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that it is only necessary to logically program a method flow using the above-mentioned hardware description languages and program it into an integrated circuit to easily obtain a hardware circuit that implements the logical method flow.

[0182] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to implementing the controller in pure computer readable program code, it is also possible to implement the controller in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. to perform the same functions by logically programming the method steps. Such a controller can therefore be considered as a hardware component, and the means included therein for performing various functions can also be considered as structures within the hardware component. Alternatively, the means for performing various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0183] The systems, apparatuses, modules or units illustrated by the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0184] For the sake of description, the above apparatuses are described in functional division and are described respectively. Of course, the functions of each unit can be implemented in the same or more software and / or hardware when implementing one or more embodiments of the present specification.

[0185] Those skilled in the art will understand that the embodiments of the present specification can be provided as a method, a system or a computer program product. Therefore, the embodiments of the present specification can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0187] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0188] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0189] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0190] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0191] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0192] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to encompass non-exclusive inclusion, such that processes, methods, articles or devices that comprise a list of elements not only include those elements, but also include other elements not expressly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0193] The specification can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0194] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0195] The above merely provides a description of the embodiments of the present application, but should not be taken as a limitation. Various changes and modifications can be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall fall within the scope of claims of the present application.

Claims

1. A face forgery detection method based on multi-view collaborative fusion, characterized in that, The face forgery detection method includes: Acquire the video frame data to be detected; The video frame data to be detected is preprocessed to obtain a standardized input image tensor; The standardized input image tensor is input into a pre-trained face forgery detection model. The model processes the image to generate face forgery detection results. The face forgery detection model's processing includes: based on the standardized input image tensor, a recursive three-level discrete wavelet transform and a wavelet multi-band construction module are used to obtain style flow tensors and texture flow tensors. Specifically, this includes: using a Haar wavelet basis as the decomposition kernel, performing a recursive three-level discrete wavelet transform on the standardized input image tensor using low-pass and high-pass filters respectively to generate first- to third-level low-frequency approximation subbands and first- to third-level high-frequency detail subbands; performing bilinear interpolation upsampling on the third-level low-frequency approximation subband to a preset resolution to generate the style flow tensor; performing bilinear interpolation upsampling on the first- to third-level high-frequency detail subbands to the preset resolution, and then depth-concatenating them in the channel dimension to obtain the texture flow tensor; and then processing the standardized input image tensor... Image tensors, style flow tensors, and texture flow tensors are input into a multi-view hybrid feature extraction network to obtain a global semantic vector, style feature map, and texture feature map. The multi-view hybrid feature extraction network includes a global feature extraction branch, a style encoding branch, and a texture encoding branch. The style feature map and the texture feature map are verified through a mutual attention mechanism to generate interactive features. A spatial gating mask is used as an adaptive adjustment coefficient to perform a weighted summation of the concatenated features and the interactive features to generate the final features. The concatenated features are generated by concatenating the interactive features and the global semantic vector, and the spatial gating mask is formed by mapping the concatenated features through a convolutional layer. The final features are processed by a spatial pyramid confidence decision module to generate forgery prediction scores at each scale and corresponding confidence weights at each scale. Based on the forgery prediction scores at each scale and the confidence weights at each scale, a face forgery prediction probability value is generated for face forgery detection.

2. The face forgery detection method as described in claim 1, characterized in that, The kernel coefficient vector of the low-pass filter for The kernel coefficient vector of the high-pass filter for ; The formulas for the first to third levels of low-frequency approximate sub-bands are as follows: ； in, ； Represents spatial coordinates, Indicates a horizontal index. Indicates a vertical index; Indicates the first In the low-frequency approximate sub-band, spatial coordinates The output variable; Indicates row low-pass and column low-pass; This represents the row direction coefficient of the low-pass filter; Represents the column direction coefficients of a low-pass filter; Indicates the row direction offset index; Indicates the column direction offset index; The style flow tensor is: ; in, Represents the style flow tensor; R represents the data type; This indicates the preset resolution; Indicates the preset height; Indicates the preset width; N represents the batch size; 3 indicates the number of channels; The high-frequency detail subbands of levels 1-3 ; in, LH indicates vertical column high-pass output; HL indicates a horizontal column high-pass output; HH indicates a column high-pass output in the diagonal direction; The texture flow tensor ; in, 9 indicates the number of channels; The formula for calculating the texture flow tensor is: ； in, Indicates a connection; Indicates upsampling; Represents fine high frequency; Indicates mid-frequency details; Indicates a rough outline; ， 。 3. The face forgery detection method as described in claim 1, characterized in that, The step of inputting the standardized input image tensor, the style flow tensor, and the texture flow tensor into a multi-view hybrid feature extraction network to obtain a global semantic vector, style feature map, and texture feature map specifically includes: The normalized input image tensor is input to the global feature extraction branch to obtain the global semantic vector; the style flow tensor is input to the style encoding branch to obtain the style feature map; and the texture flow tensor is processed through the texture encoding branch to obtain the texture feature map.

4. The face forgery detection method as described in claim 3, characterized in that, The step of inputting the standardized input image tensor into the global feature extraction branch to obtain the global semantic vector specifically includes: The standardized input image tensor is input into the global feature extraction branch, and high-level abstract semantic features are obtained based on the deep convolutional structure of the global feature extraction branch. The global feature extraction branch uses EfficientNet-B4 as the backbone network. Global average pooling is performed based on the high-level abstract semantic features to obtain the global semantic vector; The formula for calculating the global semantic vector is: ； in, This represents the global semantic vector; Indicates the preset height; Indicates the preset width; Indicates altitude; Indicates width; This represents the high-level abstract semantic features; The step of generating a style feature map from the style flow tensor via the style coding branch specifically includes: The style flow tensor is input into the style encoding branch, and mapped to a high-dimensional latent feature space through the first convolutional module of the style encoding branch to obtain an intermediate feature map; Based on the intermediate feature map, channel correlation is calculated to obtain the Gram matrix; Flatten the Gram matrix and input it into the style encoding branch multilayer perceptron to obtain the style embedding vector; The style embedding vector is injected into the intermediate feature map through element-wise multiplication to generate the style feature map; The step of obtaining the texture feature map by passing the texture stream tensor through the texture coding branch specifically includes: The texture flow tensor is passed through the texture coding branch and then convolved by the first convolution module of the texture coding branch to generate a residual feature map. The residual feature map is input into the unpooled feature extraction network of the texture coding branch to generate deep texture features; The deep texture features are input into the CDC central difference convolutional network, and after being weighted by standard convolutional terms and central difference terms, CDC enhanced features are generated. The CDC enhanced features are passed through a convolutional layer to generate the texture feature map.

5. The face forgery detection method as described in claim 1, characterized in that, The step of performing feature verification on the style feature map and the texture feature map through a mutual attention mechanism to generate interactive features specifically includes: The style feature map is used as a condition to generate a query tensor, and the texture feature map is used as a basis to generate a key tensor and a value tensor. Based on the query tensor and the key tensor, a similarity weight is calculated through a mutual attention mechanism to generate an attention weight matrix; Based on the attention weight matrix, the value tensor is weighted and summed to generate the interaction feature.

6. The face forgery detection method as described in claim 1, characterized in that, The final features are processed by the spatial pyramid confidence decision module to generate forgery prediction scores at each scale and corresponding confidence weights at each scale, specifically including: The final features are processed by the spatial pyramid confidence decision module, which performs multi-granularity pooling at the global, regional, and detail levels to generate feature vectors at various scales. Each feature vector in the feature vectors of each scale is processed by the classification head network to generate a fake prediction score for each scale and a corresponding confidence weight for each scale.

7. The face forgery detection method as described in claim 6, characterized in that, The classification head network consists of two fully connected layers, a BatchNorm layer, and an activation function. The process of generating a face forgery prediction probability value based on the forgery prediction scores at each scale and the confidence weights at each scale specifically includes: The forgery prediction scores at each scale are multiplied by the confidence weights at each scale, then summed and normalized to generate the face forgery prediction probability value.

8. The face forgery detection method as described in claim 1, characterized in that, The loss function used in the training process of the face forgery detection model is a composite loss function; The composite loss function is: ; in, Primary classification loss, For orthogonal decoupling loss, For uncertainty regularization loss, and These are the hyperparameter weights used to balance the magnitudes of the various loss terms; The main loss function is: ; in, Indicates the first The true label of each sample; This represents the probability of forgery predicted by the face forgery detection model; The orthogonal decoupling loss is: ; in, This means flattening a multidimensional feature map into a one-dimensional vector; Represents style feature diagram; Represents a texture feature map; express Norm; Uncertainty regularization loss is: ; in, For the first Predicted scores for scale; This is a real label; This is the learnable confidence parameter corresponding to this scale.

9. A face forgery detection system based on multi-view collaborative fusion, characterized in that, The face forgery detection system includes: The acquisition module acquires the video frame data to be detected; The preprocessing module preprocesses the video frame data to be detected to obtain a standardized input image tensor. The detection module inputs the standardized input image tensor into a pre-trained face forgery detection model. The model processes the input tensor to generate face forgery detection results. Specifically, the face forgery detection model's processing includes: based on the standardized input image tensor, using a recursive three-level discrete wavelet transform and a wavelet multi-band construction module to obtain style flow tensors and texture flow tensors. This specifically includes: using a Haar wavelet basis as the decomposition kernel, performing a recursive three-level discrete wavelet transform on the standardized input image tensor using low-pass and high-pass filters respectively to generate first- to third-level low-frequency approximation subbands and first- to third-level high-frequency detail subbands; performing bilinear interpolation upsampling on the third-level low-frequency approximation subband to a preset resolution to generate the style flow tensor; performing bilinear interpolation upsampling on the first- to third-level high-frequency detail subbands to the preset resolution, and then deep concatenating them in the channel dimension to obtain the texture flow tensor; and then processing the standardized input image tensor... The input image tensor, the style flow tensor, and the texture flow tensor are fed into a multi-view hybrid feature extraction network to obtain a global semantic vector, a style feature map, and a texture feature map. The multi-view hybrid feature extraction network includes a global feature extraction branch, a style encoding branch, and a texture encoding branch. The style feature map and the texture feature map are verified through a mutual attention mechanism to generate interactive features. A spatial gating mask is used as an adaptive adjustment coefficient to perform a weighted summation of the concatenated features and the interactive features to generate the final features. The concatenated features are generated by concatenating the interactive features and the global semantic vector, and the spatial gating mask is formed by mapping the concatenated features through a convolutional layer. The final features are processed by a spatial pyramid confidence decision module to generate forgery prediction scores at each scale and corresponding confidence weights at each scale. Based on the forgery prediction scores at each scale and the confidence weights at each scale, a face forgery prediction probability value is generated for face forgery detection.

Images