Face counterfeit video intelligent detection method based on multi-feature fusion

Abstract

The invention discloses a face counterfeit video intelligent detection method based on multi-feature fusion, and the method comprises the steps: carrying out the video key frame extraction processing of an input to-be-detected face counterfeit video, and obtaining a preliminary processing result; feature extraction based on DSSIM is carried out, and features based on a DSSIM value are obtained; integrity feature extraction based on noise fingerprints is carried out to obtain integrity features; feature extraction based on multiple views is carried out, and multi-view features are obtained; fusing the features to obtain a comprehensive feature; judging an authenticity result of the to-be-detected face counterfeit video, inputting the comprehensive characteristics into a priority judgment device based on combined evidence, and judging the authenticity result according to a priority judgment principle; and completing the intelligent detection of the face counterfeit video based on the fusion of the multiple features. According to the method, the problem that detection of a plurality of forged videos with different forged traces by a single method fails is solved, and the overall detection accuracy is greatly improved.

Description

Technical Field

[0001] The invention relates to a method for detecting forged face videos, in particular to an intelligent method for detecting forged face videos based on fusion of multiple features. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] In the digital media era, video has become the core carrier of information and entertainment, but the accompanying deep fake technology uses AI algorithms to tamper with or create realistic video content, threatening the authenticity of information and social order. Deep fakes use technologies such as generative adversarial networks to manipulate audio-visual information with a high degree of authenticity. Although its original intention was to be applied to film and television special effects, it has been abused due to the popularization of technology, causing a series of problems. To meet this challenge, the development of detection technology is crucial. Researchers are using machine learning to analyze video features, such as the naturalness of facial expressions and blinking patterns, to identify authenticity.

[0004] In today's digital and information age, video content, with its intuitive and vivid characteristics, has become an important tool for obtaining and disseminating information. However, with the rapid development of deep fake technology, the authenticity and credibility of video content are facing unprecedented challenges. Deep fake technology uses advanced image processing, machine learning, and generative adversarial networks (GANs) to easily tamper with and forge video content, which not only poses a serious threat to personal privacy and reputation rights, but also poses potential risks to social public security and order.

[0005] The existing "Saliency Target Detection Method Based on Multi-scale Feature Extraction and Multi-level Feature Fusion" by Li Lingli, Meng Lingbing, and Li Jinbao proposed a saliency detection method based on multi-scale feature extraction (MSFE) and multi-level feature fusion (MLFF). The multi-scale feature extraction module is embedded in the deep layer of the network to alleviate the problems of discontinuity between the salient target area and the background and blurred boundaries. The multi-level feature module fuses feature information at multiple different levels, which can not only suppress the noise transmitted from the shallow layer, but also more effectively restore the spatial structure detail information of the salient target in the decoding stage, thereby improving the detection effect of the model.

[0006] Therefore, researching and developing effective deep fake video detection technology has important practical significance and urgent social needs. This technology can prevent the spread of false information. At the same time, the research and development of deep fake video detection technology is also a challenge and improvement to existing scientific and technological capabilities. It will promote technological progress in image processing, video analysis, machine learning and other fields, and open up new paths for future scientific and technological development.

[0007] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0008] Purpose of the invention: The technical problem to be solved by the present invention is to provide an intelligent detection method for face forgery videos based on the fusion of multiple features in view of the shortcomings of the prior art.

[0009] In order to solve the above technical problems, the present invention discloses an intelligent detection method for face forgery videos based on multiple feature fusion, comprising the following steps:

[0010] Step 1, extracting key frames from the input video of the face forgery to be detected to obtain a preliminary processing result; the preliminary result includes a key frame image sequence;

[0011] Step 2, performing DSSIM-based feature extraction on the preliminary processing result to obtain features based on DSSIM values;

[0012] Step 3, extracting integrity features based on noise fingerprints from the preliminary processing results to obtain integrity features;

[0013] Step 4, performing multi-view feature extraction on the preliminary processing result to obtain multi-view features;

[0014] Step 5, fusing the features based on DSSIM values, integrity features, and multi-view features obtained in steps 2 to 4 to obtain comprehensive features;

[0015] Step 6, judging the authenticity of the face forgery video to be detected, and completing the intelligent detection of face forgery video based on the fusion of multiple features.

[0016] Further, the feature extraction based on DSSIM described in step 2 specifically includes the following steps:

[0017] Step 2-1, processing the input key frame image through a face detector, extracting the face borders in the key frame image, and performing a face cropping operation based on the borders;

[0018] Step 2-2, setting and extracting facial key points in the key frame image, including the left eye, the right eye, the nose, the left corner of the mouth and the right corner of the mouth;

[0019] Step 2-3, calculating the DSSIM value, according to the key frame image and the reference image or the template image, calculating the DSSIM value DSSIM (x, y) between them, the calculation method is as follows:

[0020]

[0021] Among them, x represents the key frame image, y represents the reference image or template image, μ x Represents the average pixel brightness of image x, μ y represents the average pixel brightness of image y, σ x represents the contrast of image x, σ y represents the contrast of image y, σ xy represents the constant factor, C1 and C2 are two parameters;

[0022] Step 2-4, input the cropping operation described in step 2-1, the key points described in step 2-2, and the DSSIM value described in step 2-3 into the convolutional network to obtain the feature X based on the DSSIM value S .

[0023] Furthermore, the extraction of integrity features based on noise fingerprint described in step 3 specifically includes the following steps:

[0024] Step 3-1, build a noise fingerprint extractor, as follows:

[0025] The noise fingerprint extractor adopts the DnCNN architecture, which includes 15 trainable layers, 3 input channels and 1 output channel;

[0026] Step 3-2, training the noise fingerprint extractor, is as follows:

[0027] The training is performed on randomly extracted 64×64 pixel blocks from the images in the pre-prepared training dataset, using contrast loss. The details are as follows:

[0028]

[0029] in, is a batch of patches, s(o, j) is the square of the Euclidean distance between the i-th and j-th residual patches, is the subset of patches that have the same origin, location, and editing history as the oth residual patch;

[0030] Step 3-3, using the trained noise fingerprint extractor to extract noise fingerprint features from the input key frame image x;

[0031] Step 3-4, inputting the key frame image x and the obtained noise fingerprint features into the encoding and decoding network at the same time, obtaining and outputting the predicted anomaly map of the key frame image x;

[0032] The encoder in the encoding and decoding network is based on a transformer segmentation architecture, with four transformer blocks, one CM-FRM block between each transformer block, and the transformer block is based on a Mix Transformer encoder B2 for semantic segmentation;

[0033] The decoder uses a lightweight multilayer perceptron in SegFormer;

[0034] Step 3-5, use the Sigmoid function to calculate the confidence of each pixel in the predicted anomaly map to form a confidence map, use the pooling block to calculate the four features of the predicted anomaly map, namely the maximum value, minimum value, average value and mean square, and perform weighted statistics, where the weight comes from the original anomaly area in the confidence map, and combine the four corresponding features extracted from the predicted anomaly map and their respective standard deviations to finally obtain an 8-component feature vector, namely the integrity feature X C , used to predict the completeness score.

[0035] Further, the multi-view feature extraction described in step 4, that is, using a multi-view edge feature extractor MEF and a multi-view noise feature extractor MNF to extract multi-view edge features and multi-view noise features respectively, and fusing the multi-view edge features and multi-view noise features to finally obtain multi-view features, specifically includes the following steps:

[0036] Step 4-1, use the multi-view edge feature extractor MEF to obtain multi-view edge features, which is expressed as follows:

[0037]

[0038] Among them, {f mef，1 , ..., f mef,k} represents the multi-view edge feature map, f mef,k represents the kth edge eigenvalue, S edge (x) represents the prediction operation edge map obtained by transforming the output with the sigmoid (σ) layer, ← represents the network output symbol, and MEF(x) represents the feature extraction of the key frame image x using the multi-view edge feature extractor;

[0039] Step 4-3, use the multi-view noise feature extractor MNF to obtain the multi-view noise feature, which is an array of k feature maps from the last ResNet block in the multi-view edge feature extractor MEF, expressed as follows:

[0040] {f mnf，1 , ..., f mnf，k}←ResNet(x)

[0041] Among them, {f mef，1 , ..., f mef,k} represents the multi-view noise feature map, f mnf，k represents the kth noise eigenvalue, and ResNet(x) represents the output of the feature extraction of the key frame image x by the ResNet block;

[0042] Step 4-4, using the element-by-element sigmoid function, convert the multi-view noise feature map S′(x) into the full-size segmentation map feature S(x), expressed as follows:

[0043]

[0044] Among them, σ() represents the sigmoid function, bilinear-upsampling() represents bilinear upsampling;

[0045] Step 4-5, using the generalized mean pooling GeM of image retrieval, performs global average pooling on the full-size segmentation map S(x), which is expressed as follows:

[0046]

[0047] Among them, GeM(S(x)) represents the mean graph feature, p is a trainable parameter, W represents the width of image x, H represents the height of image x, and x ii,jj Indicates that the coordinates of image x are (ii, jj) pixels;

[0048] Step 4-6, add the mean graph feature GeM(S(x)) to the multi-view feature X through a non-negative hyperparameter λ as a weighted decay skip connection P In, it is expressed as follows:

[0049] X P =λ·GeM(S(x))+(1-λ)·GeM(Conv(S(x)))

[0050] Among them, X P That is the multi-view feature, Conv() represents a convolution block, which is composed of Conv3x3, ReLU, BatchNorm, Conv1x1 and sigmoid functions.

[0051] Furthermore, the fusion described in step 5 specifically includes:

[0052] Step 5-1, using a time series feature alignment method to correspond the features based on the DSSIM value, the integrity features and the multi-view features in time series;

[0053] Step 5-2: Use linear interpolation to calculate the length difference d between the integrity feature and the feature based on the DSSIM value. u , and randomly d in the integrity feature u Interpolate the positions to fill the integrity feature length to the feature length based on the DSSIM value;

[0054] In step 5-3, a maximum-based downsampling method is adopted, and the average pooling principle is used to downsample the multi-view feature length to the feature length based on the DSSIM value, and finally the lengths of the features based on the DSSIM value, the integrity features, and the multi-view features are aligned.

[0055] Furthermore, the authenticity of the forged face video to be detected as determined in step 6 is determined by inputting the comprehensive features into a priority determiner based on combined evidence to determine whether the priority determination principle is satisfied; if the priority determination principle is satisfied, the authenticity of the forged face video to be detected is determined directly; otherwise, the authenticity is determined by a synthesizer determiner based on the optimal boundary.

[0056] Furthermore, the result of determining the authenticity of the video to be detected in step 6 specifically includes the following steps:

[0057] Step 6-1, judging the three features of the DSSIM value-based feature, the integrity feature and the multi-view feature separately, and calculating the normalized features of the three features A default judgment threshold θ is set, and when the normalized feature of the feature is less than θ, it is judged as a forged video, otherwise it is judged as a real video;

[0058] Step 6-2, according to the result obtained in step 6-1, the final judgment result is directly obtained according to two priority principles, otherwise the final judgment result is determined by using a comprehensive judge.

[0059] Furthermore, the two priority principles described in step 6-2 specifically include:

[0060] Principle 1, the maximum fraction principle, is as follows:

[0061] If any of the three features is less than the first threshold or greater than the second threshold, the feature is directly used to make a decision alone, and the decision result is taken as the final decision result;

[0062] Principle 2: Judgment is based on the principle of maximum authenticity, as follows:

[0063] Based on the threshold θ, a double-threshold method is used, that is, the upper threshold θ is set top and the lower threshold θ lb , the original binary result is changed into a trichotomous result, that is, the original two results of true and forged are changed into three results of true, forged and fuzzy, and the true, forged and fuzzy results are represented by R, F and O respectively. There are 27 different result combinations for the three features described in step 6-1 when judging the results separately;

[0064] Among the 27 combinations, it is stipulated that the two combinations of RRR and FFF meet the principle of maximum authenticity and are directly judged as authentic and forged respectively, and the judgment results are used as the final judgment results;

[0065] If Principle 1 and Principle 2 conflict, Principle 1 shall be used as the judgment standard.

[0066] Furthermore, the final decision result is determined by using the comprehensive decision device in step 6-2, specifically including:

[0067] Step 6-2-1, build a classifier, as follows:

[0068] For the data of fuzzy results in the priority principle, three features are constructed into a feature space; a support vector machine SVM classifier is constructed in the feature space;

[0069] Step 6-2-2, training the classifier, as follows:

[0070] The features based on DSSIM values, integrity features and multi-view features are used as input features, and the instances of fuzzy results and their corresponding true labels are used as training sets to train the support vector machine (SVM) classifier.

[0071] Step 6-2-3, use the trained support vector machine SVM classifier to make a decision and obtain the final decision result.

[0072] Furthermore, the loss function used in the encoding and decoding network described in step 3-4 is is the weighted cross entropy loss and dice loss The combination is as follows:

[0073]

[0074] Among them, λ ce is the weight coefficient, weighted cross entropy loss The details are as follows:

[0075]

[0076] Among them, g i and a i are the i-th pixel of the key frame image and the predicted anomaly image respectively, N is the number of pixels, and γ0 and γ1 are weights.

[0077] Beneficial effects:

[0078] 1. The present invention considers feature extraction methods at different angles, thereby avoiding the problem that a single method fails to detect forged videos with a variety of different forgery traces.

[0079] 2. The present invention proposes a priority decision model based on combined evidence, which preliminarily screens out forged video data suitable for a single method through two priority decision principles, thus avoiding interference of excellent data detected by a single model by other models.

[0080] 3. The optimal boundary-based comprehensive decision model proposed in the present invention uses a classifier to retrain the fuzzy data that does not meet the priority decision maker, comprehensively considering various aspects of forgery traces, and generally solving the problem of poor detection effect of this part of fuzzy data, thereby greatly improving the overall detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0082] Figure 1 It is a schematic diagram of the overall process of the method of the present invention.

[0083] Figure 2 Schematic diagram of feature extraction based on DSSIM.

[0084] Figure 3 Schematic diagram of feature extraction based on image integrity.

[0085] Figure 4 Schematic diagram of feature extraction based on multiple views.

[0086] Figure 5 It is the regional map of the independent feature authenticity detection results. DETAILED DESCRIPTION

[0087] The present invention proposes an intelligent detection method for face forgery video based on the fusion of multiple features. The overall idea is as follows: first, the video is subjected to key frame extraction processing to obtain multiple frames of images. The images are subjected to feature extraction based on structural difference to obtain DSSIM features; feature extraction based on image integrity to obtain integrity features; and feature extraction based on multi-view to obtain multi-view features. The fused features are passed through a priority judge based on combined evidence, and the video data that meets the priority judgment principle can directly judge the authenticity. The video data that does not meet the priority judgment result is then judged by a synthesizer judge based on the optimal boundary to determine its authenticity.

[0088] Specifically, if Figure 1 As shown, the present invention proposes an intelligent detection method for forged videos based on multiple feature fusion, comprising the following steps:

[0089] Step 1) extract key frames from the input video, and the commonly used frame numbers are 8, 16, 32, 64, and 128.

[0090] Step 2) extracting features based on DSSIM from the image obtained by dividing the video into frames in step 1) to obtain DSSIM features.

[0091] Step 3) extracting the image after the video is divided into frames in step 1) based on noise fingerprint features to obtain integrity features.

[0092] Step 4) extracting multi-view features from the image obtained by dividing the video into frames in step 1) to obtain multi-view features.

[0093] Step 5) The DSSIM features, integrity features and multi-view features obtained in steps 2), 3) and 4) are fused through temporal feature alignment, bilinear interpolation and maximum-based downsampling processes to obtain fused features based on image structure, integrity and multi-view. Since a single forgery detection method has limitations and can only solve forgery detection of specific forgery means, the present invention proposes a priority judge and a comprehensive judge to integrate multiple features for forgery detection.

[0094] Step 6) Input the comprehensive features in step 5) into the priority decision device based on combined evidence to determine whether the priority decision principle is met. Video data that meets the priority decision principle can directly determine the authenticity result. Video data that does not meet the priority decision result is then judged by the optimal boundary-based synthesizer decision device to determine its authenticity result.

[0095] Step 7) The above method is verified on the Celeb-DF dataset to obtain the forgery recognition results.

[0096] Further, as a preferred technical method of the present invention, the feature extraction based on DSSIM in the selected step 2) includes the following steps:

[0097] Step S211, in this model, when an image frame is received as input, the first step is for the face detector to process it, accurately extract the face borders in the image, and perform face cropping operations based on these borders.

[0098] Step S212, after the face cropping is completed, the next step is to extract the key points of the face. The key points of the face include the left eye, the right eye, the nose, the left corner of the mouth, and the right corner of the mouth. The accurate location of these key points is crucial for subsequent tasks such as face analysis and recognition.

[0099] Step S213, next, calculate the DSSIM value. DSSIM is an indicator for measuring the difference between two images based on the SSIM improvement, which takes into account the brightness, contrast and structural information of the image. The DSSIM between the original face image and a certain reference or template image is calculated by comparing them. The formula is as follows.

[0100]

[0101] in C1 and C2 are parameters, x i represents the i-th pixel in image x, y i represents the i-th pixel in image y, and N represents the number of pixels.

[0102] Step S214, as the last step, inputs the processed face image (including the results of the steps of cropping, key point extraction and DSSIM value generation) into the EfficientNet-B7 network (reference: EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks) to obtain the final DSSIM value-based feature X of the model. s This is an efficient and powerful convolutional neural network model that can reduce computation and memory consumption while maintaining high performance.

[0103] Further, as a preferred technical method of the present invention, the feature extraction based on noise fingerprint in the selected step 3) includes the following steps: Step S311, first, extract the noise fingerprint feature from the input RGB image x. This is a learned noise-sensitive feature with the same resolution as x. The feature extractor obtains the RGB image to obtain the learned noise-sensitive fingerprint. This is an improved image fingerprint that not only highlights the traces related to the in-camera processing, but also highlights the traces related to the out-camera processing.

[0104] Step S312, the noise fingerprint extractor proposed in step S311 learns patch-level self-similarity by contrastive learning, using a DnCNN architecture with 15 trainable layers, 3 input channels, and 1 output channel. The extractor is trained on 64×64 pixel blocks randomly extracted from the images in the dataset. The purpose of training is to obtain the same noise-sensitive fingerprint for small patches with the same attributes, and to obtain different noise residuals for small patches that are different in some aspects. The present invention uses the following contrastive loss:

[0105]

[0106] In the formula, is a batch of patches, s(i,j) is the square of the Euclidean distance between the i-th and j-th residual patches, is the subset of patches that have the same origin, location, and editing history as the ith patch.

[0107] Step S313, using the trained noise fingerprint extractor to extract noise fingerprint features from the input key frame image x; inputting the key frame image x and the obtained noise fingerprint features into the codec network at the same time, obtaining and outputting a predicted anomaly map of the key frame image x;

[0108] The encoder in the encoding and decoding network is based on a transformer segmentation architecture, with four transformer blocks, one CM-FRM block between each transformer block, and the transformer block is based on a Mix Transformer encoder B2 for semantic segmentation;

[0109] The decoder uses a lightweight multilayer perceptron in SegFormer; (the reference for this Transformer is Attention Is All You Need; the reference for the CM-FRM block is Toward reliable models for authenticating multimedia content: Detecting resampling artifacts with bayesian neural networks; the reference for the SegFormer and Mix Transformer encoder B2 is Learning rich features for image manipulation detection)

[0110] In step S314, the present invention uses a method for calculating pixel-by-pixel confidence estimates of the predicted anomaly map, which is used to calculate robust global statistics for detection. In the pooling block, this paper calculates four weighted statistics of the anomaly map, namely, maximum, minimum, average and mean square, where the weights come from the confidence map, which helps to weaken the original abnormal area of ​​the image.

[0111] The confidence of each pixel is calculated using the Sigmoid function to form a confidence map. The main purpose of this step is to convert the continuous values ​​output by the anomaly detection model into probability values ​​between 0 and 1, which can be interpreted as the probability of an anomaly at that location. The confidence map therefore represents the model's confidence in the degree of anomaly at each location in the image.

[0112] Next, by extracting features from the anomaly map through pooling operations (such as maximum pooling, minimum pooling, average pooling, and mean square pooling), some statistical information about the anomaly region can be obtained. These statistical information can provide the overall characteristics of the anomaly region, such as the most likely anomaly point (maximum value), the characteristics of the background or non-abnormal area (minimum value), and the distribution of the anomaly region (average value and mean square).

[0113] When we talk about using the weights in the confidence map to help weaken the original abnormal areas of the image, what we mean here is to use the information in the confidence map to adjust or influence the results of the pooling operation. Specifically, this means that when calculating statistics such as the maximum, minimum, average, and mean square of the abnormal map, instead of simply treating all pixel values ​​equally, we give each pixel a different weight based on the confidence map. The advantage of doing this is that we can pay more attention to those areas that the model believes are more likely to be abnormal (that is, areas with higher confidence), while relatively ignoring those areas with lower confidence.

[0114] The connection between this weighted statistical method and the confidence map is that it uses the additional information provided by the confidence map to optimize the results of anomaly detection. In this way, not only can the accuracy of detection be improved, but the false alarm rate can also be reduced, because those areas that are unlikely to be anomalies but show abnormal characteristics due to noise or other factors will be appropriately weakened. In short, this method combines confidence assessment and the statistical characteristics of abnormal areas to provide a more accurate and reliable anomaly detection result.

[0115] Step S315, the four corresponding features extracted from the confidence level and their respective standard deviations are combined to finally obtain an 8-component feature vector, namely the integrity feature X C , used to predict the completeness score.

[0116] Further, as a preferred technical method of the present invention, the feature extraction based on multiple views in the selected step 4) includes the following steps:

[0117] In step S411, this module uses multi-view edge (MEF) and multi-view noise feature (MNF) as features for deep fake detection. The edge supervision branch is specifically used to exploit subtle boundary artifacts around the tampered area, while the noise sensitive branch is used to capture noise inconsistencies between the tampered area and the real area.

[0118] Step 4-2, obtain the multi-view edge feature MEF, as follows:

[0119] It is represented by {f mef，1 , ..., f mef,k}, used for the main task, and the prediction operation edge map obtained by transforming the output with the sigmoid (σ) layer, denoted as S edge (x), the edge supervision branch feature ESB is expressed as follows:

[0120]

[0121] Step 4-3, obtain the noise supervision branch feature MNF, as follows:

[0122] A convolution filter parameterized by ω is used, with ω(0,0) as the central element, and a constraint is imposed on ω after each training iteration, that is,

[0123]

[0124] Step 4-4, the output of the MNF branch is an array of k feature maps, which comes from the last ResNet block, namely:

[0125] {f nsb，1 , ..., f nsb,k}←ResNet(x) Formula (10)

[0126] Steps 4-5, finally add an element-by-element sigmoid function to convert S′(x) into a full-size segmentation map S(x). The branch fusion based on data analysis can be expressed as

[0127]

[0128] Steps 4-6 use generalized mean pooling (GeM) for image retrieval. GeM uses a trainable positive parameter p to achieve balance in global average pooling (p=1):

[0129]

[0130] Steps 4-7, finally, GeM(S(x)) is added to C(x) through a non-negative hyperparameter λ as a weighted attenuation skip connection, where λ is initialized to a value close to 1 and decays nonlinearly with the increase in the number of iterations. The pixel-level features predicted by the model

[0131] X P =λ·GeM(S(x))+(1-λ)·GeM(Conv(S(x))) formula (13)

[0132] Further, as a preferred technical method of the present invention, the priority judge based on combined evidence and the comprehensive judge based on optimal boundary in the selected step 6) include the following steps:

[0133] Step S611: In this module, the images after the video is divided into frames are subjected to model analysis to obtain three model feature vectors X S , X C , X P . Perform temporal feature alignment, bilinear interpolation and maximum-based downsampling to obtain the fusion feature X based on image structure, integrity and pixels. In the original single model, there is a default threshold θ. This module continues to normalize the model fusion feature value. And the default threshold θ is studied, and a priority judge and a comprehensive judge are proposed to jointly complete the further judgment of the authenticity of the video.

[0134] Step S612: when judging the three models separately, the model fusion feature The default threshold θ, the feature average value is less than θ, it is judged as a forged video, and greater than θ, it is judged as a real video; for a single model, the closer the obtained model feature average value is to 0 or 1, the stronger the authenticity of the video is, indicating that the model has a better detection effect on whether there is forgery in the video. Therefore, the present invention uses a priority judge based on combined evidence to directly use the judgment result of this model as the final judgment result for videos with good detection effect.

[0135] Step S613, according to the result obtained by model prediction, the present invention implements a priority decision device according to the following two priority principles.

[0136] Step S614, principle 1: maximum score principle. The maximum probability principle means that the average value of the model features output by the model is infinitely close to 0 or 1. At this time, the prediction authenticity of the model for the video is higher than the general case. Therefore, if only this part of the data results are taken, its prediction accuracy will be higher than the prediction accuracy of all the data. Therefore, only the model results need to be used for this part of the data, and the result output can be given priority.

[0137] Step S615, principle 2: maximum authenticity principle. Based on the unique prediction threshold of the original model, the present invention uses a double-threshold method, that is, there is an upper threshold θ top and the lower threshold θ lb , the original two-part result is changed into three-part result, that is, the original two results of real (Real) and fake (Fake) are changed into three results of real (Real), fake (Fake) and fuzzy (Opacity). The abbreviations R, F, and O are used to represent real, fake, and fuzzy results respectively. There are 27 different result combinations in the three models. As shown in Table 1:

[0138] Table 1 Combination table of independent feature authenticity detection results

[0139]

[0140]

[0141] Step S616, among the 27 combinations, the present invention only stipulates that the RRR and FFF combinations meet the principle of maximum authenticity. These two cases are recorded as clear results, and the remaining 25 results are recorded as fuzzy results. The clear results can be directly outputted according to this principle, and the fuzzy results need to be further judged by a comprehensive judge. And if Principle 1 and Principle 2 conflict, Principle 1 is used as the judgment standard. At the same time, this article also draws the area in a three-dimensional coordinate system, such as Figure 2As shown in the figure, the blue spherical shell represents the lower threshold boundary, the red spherical shell represents the upper threshold boundary, the inner area of ​​the blue spherical shell represents the false area, the outer area of ​​the red spherical shell represents the true area, and the area between the two represents the fuzzy result area.

[0142] Step S617, in complex decision-making scenarios, facing the fuzzy results output by multiple models, the traditional single condition judgment mechanism may be difficult to achieve the ideal decision-making effect. To this end, this paper introduces an innovative comprehensive decision-maker design, which aims to improve the accuracy and stability of decision-making by integrating the prediction information of multiple models.

[0143] Step S618, with respect to step S617, specifically, the core strategy of the comprehensive judge is to use the features of the three independent models of the fuzzy result data obtained by the priority judge to construct a feature space, and then train a classifier in this space.

[0144] Step S619, under this framework, the model features of each model are regarded as important indicators reflecting different aspects of information, which together form the basis for comprehensive decision-making. By using these scores as input features, the classifier can learn the intrinsic connections and differences between the prediction results of different models, and then find an optimal decision boundary to distinguish and judge the final decision direction.

[0145] Step 6-2-1, constructing a classifier, specifically as follows: The core strategy of the comprehensive decision maker is to use the features of the three independent models of the fuzzy result data obtained by the priority decision maker to construct a feature space, and then train a SVM (Support Vector Machine) classifier in this space;

[0146] Step 6-2-2, training the classifier, as follows: Under the framework, the model features of each model are regarded as important indicators reflecting different aspects of information, which together form the basis for comprehensive decision-making. By using these scores as input features, a large number of fuzzy result instances and their corresponding true labels are used as training sets during the training process. The SVM classifier can learn the intrinsic connections and differences between the prediction results of different models, and then find an optimal decision boundary to distinguish and judge the final decision direction;

[0147] Step 6-2-3, use the trained SVM classifier to make a decision and obtain the final decision result.

[0148] Step S623, during the training process, a large number of examples of fuzzy results and their corresponding true labels are used as training sets. This not only teaches the classifier how to weigh the predictions of different models, but also enables it to learn how to make the most reasonable judgment when faced with similar fuzzy scenarios. The carefully tuned classifier model can quickly and accurately make the final decision based on the model characteristics of each model when new data arrives, greatly improving the flexibility and accuracy of the decision-making system.

[0149] Further, as a preferred technical method of the present invention, the selected step 7) verifies the priority judge based on combined evidence and the optimal boundary comprehensive judge based on the Celeb-DF dataset, including the following steps:

[0150] Step S711, performing key frame extraction processing on the videos in the data set through step 1), and randomly extracting 8 frames of images from each video.

[0151] Step S712, the image processed by step S711 is subjected to feature extraction based on DSSIM to obtain DSSIM features; feature extraction based on image integrity to obtain integrity features; and feature extraction based on multi-view to obtain multi-view features. Thus, the feature average value is obtained, and the detection accuracy of the individual models is obtained. For the convenience of recording, A, B, and C are used to represent the three models below. The detection accuracy is shown in Table 2:

[0152] Table 2 Independent feature authenticity detection results

[0153]

[0154] Step S713, for the priority decision maker, this experiment stipulates that the model feature meets the priority principle 1 when it is close to 0 or 1 with an error of ±0.1; the double-layer thresholds of the model feature division of the three prediction results in the priority principle 2 are respectively 0.25 and 0.75. The results of the data that meet the priority decision maker are shown in Table 3. As can be seen from Table 3, the priority decision maker can accurately lock those data subsets that are particularly beneficial to model prediction, and the inference accuracy of the target data has reached a nearly perfect 99%.

[0155] Table 3. Authenticity detection results of the priority judger based on combined evidence

[0156]

[0157]

[0158] Step S714, for the data eliminated by the priority judge, the comprehensive judge is used to process it. In this experiment, the model features of its three independent models are used as feature values, and a feature space is constructed using these features, and then a classifier is trained in this space. This part of the data is divided into a training set (80%) and a test set (20%). The data set contains 2219 videos, that is, the training set contains 1775 videos and the test set contains 444 videos. The training and test results are shown in Table 4. An accuracy rate of 83% can be achieved, which is higher than the prediction accuracy of each individual model for this part of the data.

[0159] Table 4. Authenticity detection results of the comprehensive decision maker based on the optimal boundary

[0160] Model Number of test set samples Test correct number Accuracy Authenticity detection based on DSSIM 444 337 75.90％ Authenticity detection based on image integrity 444 308 69.37％ Authenticity detection based on multiple views 444 269 60.58％ Comprehensive decision maker based on optimal margin 444 357 80.40％

[0161] In step S715, the final results are counted as shown in Table 5. The final accuracy of the model proposed in the present invention for the overall data set is 92.65%, which is higher than the accuracy of any model used alone. At the same time, the comprehensive judgment model of this paper also has a high accuracy rate on the priority judgment samples and fuzzy data samples. The success of this strategy is not only reflected in the significantly improved accuracy, but more importantly, it demonstrates the great potential of ensemble learning and multi-model collaboration in the complex and changeable deep fake video recognition task.

[0162] Table 5 Comparison of the accuracy of independent feature authenticity detection and fusion feature authenticity detection

[0163]

[0164] Example:

[0165] A specific embodiment of the present invention is given below in conjunction with the accompanying drawings.

[0166] like Figure 1 As shown, a specific embodiment of the present invention mainly includes several modules of video key frame extraction, independent feature extraction, multi-feature fusion and intelligent authenticity judgment; first, key frame extraction is performed on the input video; then, feature extraction based on DSSIM, feature extraction based on image integrity and feature extraction based on multi-view are performed on each key frame to obtain DSSIM features, integrity features and multi-view features; then, multi-feature fusion is achieved by aligning, interpolating and downsampling feature vectors of three different lengths; finally, priority judgment based on combined evidence and comprehensive judgment based on optimal boundaries are performed on the fused features in the uncertainty space, so as to obtain intelligent detection results for the authenticity of the video.

[0167] like Figure 2As shown, the DSSIM-based feature extraction in the present invention extracts 8 key frame images from the video, performs DSSIM-based feature extraction on each frame image, and obtains 6 feature scores, namely DSSIM features.

[0168] like Figure 3 As shown, the feature extraction based on image integrity in the present invention extracts 8 key frame images from the video, performs feature extraction based on image integrity on each frame image, and obtains 4 feature scores, namely, integrity features.

[0169] like Figure 4 As shown, in the present invention, based on multi-view feature extraction, 8 key frame images are extracted from the video, and multi-view feature extraction is performed on each frame image to obtain 8 feature scores, namely multi-view features.

[0170] like Figure 5 As shown, the authenticity intelligent judgment process in the present invention is to match the input fusion features in the uncertainty space, and make judgments based on two priority principles. If the conditions are met, the judgment results can be obtained directly, and if the conditions are not met, further comprehensive judgment based on the optimal boundary is required to obtain the final result.

[0171] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of the intelligent detection method for face forgery video based on multiple feature fusion provided by the present invention and some or all steps in each embodiment can be run. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0172] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on such an understanding, the technical solutions in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of computer programs, i.e., software products, which can be stored in a storage medium and include several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, an MCU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0173] The present invention provides a method and idea for intelligent detection of face forgery video based on multiple feature fusion. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. An intelligent detection method for face forgery videos based on multiple feature fusion, characterized in that: The following steps are involved: Step 1, extracting key frames from the input video of the face forgery to be detected to obtain a preliminary processing result; the preliminary result includes a key frame image sequence; Step 2, performing DSSIM-based feature extraction on the preliminary processing result to obtain features based on DSSIM values; Step 3, extracting integrity features based on noise fingerprints from the preliminary processing results to obtain integrity features; Step 4, performing multi-view feature extraction on the preliminary processing result to obtain multi-view features; Step 5, fusing the features based on DSSIM values, integrity features, and multi-view features obtained in steps 2 to 4 to obtain comprehensive features; Step 6, judging the authenticity of the face forgery video to be detected, and completing the intelligent detection of face forgery video based on the fusion of multiple features.

2. According to claim 1, a method for intelligent detection of forged face videos based on fusion of multiple features, characterized in that: The feature extraction based on DSSIM described in step 2 specifically includes the following steps: Step 2-1, processing the input key frame image through a face detector, extracting the face borders in the key frame image, and performing a face cropping operation based on the borders; Step 2-2, setting and extracting facial key points in the key frame image, including the left eye, the right eye, the nose, the left corner of the mouth and the right corner of the mouth; Step 2-3, calculating the DSSIM value, according to the key frame image and the reference image or the template image, calculating the DSSIM value DSSIM (x, y) between them, the calculation method is as follows: Among them, x represents the key frame image, y represents the reference image or template image, μ x Represents the average pixel brightness of image x, μ y represents the average pixel brightness of image y, σ x represents the contrast of image x, σ y represents the contrast of image y, σ xy represents the constant factor, C1 and C2 are two parameters; Step 2-4, input the cropping operation described in step 2-1, the key points described in step 2-2, and the DSSIM value described in step 2-3 into the convolutional network to obtain the feature X based on the DSSIM value S .

3. According to claim 1, a method for intelligent detection of forged face videos based on fusion of multiple features, characterized in that: The extraction of integrity features based on noise fingerprint described in step 3 specifically includes the following steps: Step 3-1, build a noise fingerprint extractor, as follows: The noise fingerprint extractor adopts the DnCNN architecture, which includes 15 trainable layers, 3 input channels and 1 output channel; Step 3-2, training the noise fingerprint extractor, is as follows: The training is performed on randomly extracted 64×64 pixel blocks from the images in the pre-prepared training dataset, using contrast loss. The details are as follows: in, is a batch of patches, s(i,j) is the square of the Euclidean distance between the i-th and j-th residual patches, is the subset of patches that have the same origin, location, and editing history as the i-th residual patch; Step 3-3, using the trained noise fingerprint extractor to extract noise fingerprint features from the input key frame image x; Step 3-4, inputting the key frame image x and the obtained noise fingerprint features into the encoding and decoding network at the same time, obtaining and outputting the predicted anomaly map of the key frame image x; The encoder in the encoding and decoding network is based on a transformer segmentation architecture, with four transformer blocks, one CM-FRM block between each transformer block, and the transformer block is based on a Mix Transformer encoder B2 for semantic segmentation; The decoder uses a lightweight multilayer perceptron in SegFormer; Step 3-5, use the Sigmoid function to calculate the confidence of each pixel in the predicted anomaly map to form a confidence map, use the pooling block to calculate the four features of the predicted anomaly map, namely the maximum value, minimum value, average value and mean square, and perform weighted statistics, where the weight comes from the original anomaly area in the confidence map, and combine the four corresponding features extracted from the predicted anomaly map and their respective standard deviations to finally obtain an 8-component feature vector, namely the integrity feature X C , used to predict the completeness score.

4. According to claim 1, a method for intelligent detection of forged face videos based on fusion of multiple features, characterized in that: The multi-view feature extraction described in step 4, that is, using a multi-view edge feature extractor MEF and a multi-view noise feature extractor MNF to extract multi-view edge features and multi-view noise features respectively, and fusing the multi-view edge features and multi-view noise features to finally obtain multi-view features, specifically includes the following steps: Step 4-1, use the multi-view edge feature extractor MEF to obtain multi-view edge features, which is expressed as follows: Among them, {f mef,1 ,…,f mef,k } represents the multi-view edge feature map, f mef,k represents the kth edge eigenvalue, S edge (x) represents the prediction operation edge map obtained by transforming the output with the sigmoid (σ) layer, ← represents the network output symbol, and MEF(x) represents the feature extraction of the key frame image x using the multi-view edge feature extractor; Step 4-3, use the multi-view noise feature extractor MNF to obtain the multi-view noise feature, which is an array of k feature maps from the last ResNet block in the multi-view edge feature extractor MEF, expressed as follows: {f mnf,1 ,…,f mnf,k }←ResNet(x) Among them, {f mef,1 ,…,f mef,k } represents the multi-view noise feature map, f mnf,k represents the kth noise eigenvalue, and ResNet(x) represents the output of the feature extraction of the key frame image x by the ResNet block; Step 4-4, use the element-by-element sigmoid function to map the multi-view noise feature S ′ (x) is converted into a full-size segmentation map feature S(x), which is expressed as follows: Among them, σ() represents the sigmoid function, bilinear-upsampling() represents bilinear upsampling; Step 4-5, using the generalized mean pooling GeM of image retrieval, performs global average pooling on the full-size segmentation map S(x), which is expressed as follows: Among them, GeM(S(x)) represents the mean graph feature, p is a trainable parameter, W represents the width of image x, H represents the height of image x, and x ii,jj Indicates that the coordinates of image x are pixel (ii, jj); Step 4-6, add the mean graph feature GeM(S(x)) to the multi-view feature X through a non-negative hyperparameter λ as a weighted decay skip connection P In, it is expressed as follows: X P =λ·GeM(S(x))+(1-λ)·GeM(Conv(S(x))) Among them, X P That is the multi-view feature, Conv() represents a convolution block, which is composed of Conv3x3, ReLU, Batch Norm, Conv1x1 and sigmoid functions.

5. According to claim 4, a method for intelligent detection of forged face videos based on fusion of multiple features, characterized in that: The fusion described in step 5 specifically includes: Step 5-1, using a time series feature alignment method to correspond the features based on the DSSIM value, the integrity features and the multi-view features in time series; Step 5-2: Use linear interpolation to calculate the length difference d between the integrity feature and the feature based on the DSSIM value. u , and randomly d in the integrity feature u Interpolate the positions to fill the integrity feature length to the feature length based on the DSSIM value; In step 5-3, a maximum-based downsampling method is adopted, and the average pooling principle is used to downsample the multi-view feature length to the feature length based on the DSSIM value, and finally the lengths of the features based on the DSSIM value, the integrity features, and the multi-view features are aligned.

6. According to claim 5, a method for intelligent detection of forged face videos based on fusion of multiple features, characterized in that: The authenticity determination result of the forged face video to be detected in step 6 is to input the comprehensive features into a priority determiner based on combined evidence to determine whether the priority determination principle is satisfied; if the priority determination principle is satisfied, the authenticity result of the forged face video to be detected is directly determined; otherwise, the authenticity result is determined by a synthesizer determiner based on the optimal boundary.

7. The method for intelligently detecting forged face videos based on multiple feature fusion according to claim 6 is characterized in that: The result of determining the authenticity of the video to be detected in step 6 specifically includes the following steps: Step 6-1, judging the three features of the DSSIM value-based feature, the integrity feature and the multi-view feature separately, and calculating the normalized features of the three features A default judgment threshold θ is set, and when the normalized feature of the feature is less than θ, it is judged as a forged video, otherwise it is judged as a real video; Step 6-2, according to the result obtained in step 6-1, the final judgment result is directly obtained according to two priority principles, otherwise the final judgment result is determined by using a comprehensive judge.

8. The method for intelligently detecting forged face videos based on multiple feature fusion according to claim 7 is characterized in that: The two priority principles described in step 6-2 include: Principle 1, the maximum fraction principle, is as follows: If any of the three features is less than the first threshold or greater than the second threshold, the feature is directly used to make a decision alone, and the decision result is taken as the final decision result; Principle 2: Judgment is based on the principle of maximum authenticity, as follows: Based on the threshold θ, a double-threshold method is used, that is, the upper threshold θ is set top and the lower threshold θ lb , the original binary result is changed into a trichotomous result, that is, the original two results of true and forged are changed into three results of true, forged and fuzzy, and the true, forged and fuzzy results are represented by R, F and O respectively. There are 27 different result combinations for the three features described in step 6-1 when judging the results separately; Among the 27 combinations, it is stipulated that the two combinations of RRR and FFF meet the principle of maximum authenticity and are directly judged as authentic and forged respectively, and the judgment results are used as the final judgment results; If Principle 1 and Principle 2 conflict, Principle 1 shall be used as the judgment standard.

9. The method for intelligently detecting forged face videos based on multiple feature fusion according to claim 8, characterized in that: The use of the comprehensive decision device to determine the final decision result described in step 6-2 specifically includes: Step 6-2-1, build a classifier, as follows: For the data of fuzzy results in the priority principle, three features are constructed into a feature space; a support vector machine SVM classifier is constructed in the feature space; Step 6-2-2, training the classifier, as follows: The features based on DSSIM values, integrity features and multi-view features are used as input features, and the instances of fuzzy results and their corresponding true labels are used as training sets to train the support vector machine (SVM) classifier. Step 6-2-3, use the trained support vector machine SVM classifier to make a decision and obtain the final decision result.

10. The intelligent detection method for face forgery video based on multiple feature fusion according to claim 9 is characterized in that: The loss function used in the codec network described in step 3-4 is is the weighted cross entropy loss and dice loss The combination is as follows: Among them, λ ce is the weight coefficient, weighted cross entropy loss The details are as follows: Among them, g i and a i are the i-th pixel of the key frame image and the predicted anomaly image respectively, N is the number of pixels, and γ0 and γ1 are weights.

Images