Face deepfake detection method based on matching of voice and expression and related device

Abstract

The application provides a face deep fake detection method based on voice and expression matching and related equipment, video files for training are respectively input into a constructed CNN-15-GRU-B model and an improved CNN model for training, a voice emotion analysis model and a face expression emotion analysis model are obtained, are used for feature extraction on a to-be-detected video file, voice emotion features and face expression emotion features are obtained to construct an emotion similarity matrix, and a face deep fake detection result is analyzed according to the emotion similarity matrix; compared with the prior art, voice emotion features and face expression emotion features are extracted through the voice emotion analysis model and the face expression emotion analysis model, the naturalness of a person's expression is deeply analyzed based on the similarity of two kinds of emotions, and then it is judged whether there is face deep fake, while the limitation of directly training face deep fake samples is got rid of, and the detection precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a face deepfake detection method based on speech and expression matching and related equipment. BACKGROUND

[0002] In recent years, deep learning technology has been developing rapidly and has been widely applied in the field of computer vision. On the one hand, deep learning technology leads a new round of artificial intelligence wave, but on the other hand, a series of security problems caused by deep learning have attracted more and more attention. At present, image and video recognition technology based on deep learning is widely used in various aspects of people's life, such as intelligent supervision of network content, automatic video monitoring analysis, face recognition-based access control system, face payment, etc. In these key application fields, the reliability and security of information and data should be valued and guaranteed. Since 2017, some false images and videos generated based on deepfake technology have attracted widespread attention on the Internet, especially when deepfake is used on influential figures, it often has greater influence with the influence of the person. False images and videos have become one of the most prominent information and data security risks, and their detection and supervision face great challenges.

[0003] AI-synthesized fake faces pose a huge threat, and can create videos that appear to show a target person doing or saying something with facial expressions and body movements close to reality, which subverts people's cognition of the concept of "seeing is believing". The industry urgently needs an effective technology to detect false face images or videos in the network environment, but it is very difficult, mainly because the forged area of the face forged image is weak and exists locally. When detecting, it is easy to be affected by image noise. In addition, the forged area is often unpredictable, and different areas are targeted for each forgery method, so it is difficult to correctly detect the forged area and classify it.

[0004] At present, in the field of deepfake detection, researchers have developed various deepfake detection methods, including texture analysis, light consistency, high-frequency detail analysis, etc. Through these methods, abnormal features in deepfake videos are identified, providing a powerful tool for detection. However, existing research often ignores in-depth analysis of the naturalness of the expression of the person in the video, resulting in low detection accuracy. SUMMARY

[0005] The present application provides a face deepfake detection method based on speech and expression matching and related equipment, which aims to improve detection accuracy.

[0006] In order to achieve the above purpose, the present application provides a face deepfake detection method based on speech and expression matching, comprising:

[0007] Step 1, obtaining a video file for training;

[0008] Step 2, inputting the video file into the constructed CNN-15-GRU-B model, training the CNN-15-GRU-B model, and obtaining a speech emotion analysis model;

[0009] Step 3, inputting the video file into the improved CNN model, training the improved CNN model, and obtaining a facial expression emotion analysis model;

[0010] Step 4, inputting a video file to be detected into the speech emotion analysis model and the facial expression emotion analysis model respectively for feature extraction, and obtaining speech emotion features and facial expression emotion features;

[0011] Step 5, constructing a sentiment similarity matrix through the speech emotion features and the facial expression emotion features, and analyzing the facial deepfake detection result according to the sentiment similarity matrix;

[0012] The CNN-15-GRU-B model comprises:

[0013] a first conversion module for converting a video file into a speech file, a second conversion module for converting a speech file into a text file, a speech feature extraction module, a BERT module, a splicing module, a GRU network module, and a first output module;

[0014] The first conversion module is an input end of the CNN-15-GRU-B model, the output end of the first conversion module is connected with the input end of the second conversion module and the input end of the speech feature extraction module respectively, the output end of the second conversion module is connected with the input end of the BERT module, the output end of the BERT module and the output end of the speech feature extraction module are connected with the input end of the splicing module, the output end of the splicing module is connected with the input end of the GRU network module, the output end of the GRU network module is connected with the input end of the first output module, and the output end of the first output module is an output end of the CNN-15-GRU-B model;

[0015] The improved CNN model comprises a face detection module, a feature extraction module, a feature integration module, and a second output module connected in sequence.

[0016] Further, the CNN-15-GRU-B model is used for:

[0017] inputting a video file into the first conversion module for conversion to obtain a speech file;

[0018] inputting the speech file into the second conversion module for conversion to obtain a text file;

[0019] The voice file is input into a voice feature extraction module for feature extraction to obtain a voice feature matrix;

[0020] The text file is input into a BERT module for multi-head self-attention calculation to obtain a text feature matrix;

[0021] The voice feature matrix and the text feature matrix are input into a splicing module for feature splicing to obtain joint features;

[0022] The joint features are input into a GRU network module for dependency relationship capturing to obtain classification features;

[0023] The classification features are input into a first output module for classification to obtain multiple voice emotion features.

[0024] Further, the voice feature extraction module comprises:

[0025] a first convolutional layer, a first max-pooling layer, a composite convolutional layer, and a second max-pooling layer connected in sequence.

[0026] Further, the composite convolutional layer comprises:

[0027] 14 convolutional layers connected in sequence, wherein the input end of the first convolutional layer is connected with the output end of the first max-pooling layer, and the output end of the last convolutional layer is connected with the input end of the second max-pooling layer.

[0028] Further, the first output module comprises a fully connected layer and a softmax activation function layer connected in sequence;

[0029] The input end of the fully connected layer is connected with the output end of the GRU network module;

[0030] The output end of the softmax activation function layer is the output end of the CNN-15-GRU-B model, used for outputting multiple voice emotion features.

[0031] Further, the improved CNN model is used for:

[0032] The video file is input into a face detection module for face part feature extraction to obtain a face part feature map;

[0033] The face part feature map is input into a feature extraction module for fine-grained feature extraction to obtain a face fine-grained feature map;

[0034] The face part feature map and the face fine-grained feature map are integrated to obtain a face global feature;

[0035] The face global feature is input into a second output module for classification to obtain multiple face expression emotion features.

[0036] Further, the face deep fake detection result is analyzed according to the emotion similarity matrix, including:

[0037] When the similarity between the speech emotion feature and the face expression emotion feature in the emotion similarity matrix is greater than a preset threshold, it is considered that the emotions are consistent, and the face deep fake detection result is no fake phenomenon;

[0038] When the similarity between the speech emotion feature and the face expression emotion feature in the emotion similarity matrix is less than a preset threshold, it is considered that the emotions are inconsistent, and the face deep fake detection result is a fake phenomenon.

[0039] The application also provides a face deep fake detection device based on speech and expression matching, including:

[0040] The acquisition module is configured to acquire a video file for training;

[0041] The first training module is configured to input the video file into the constructed CNN-15-GRU-B model, train the CNN-15-GRU-B model, and obtain a speech emotion analysis model;

[0042] The second training module is configured to input the video file into the improved CNN model, train the improved CNN model, and obtain a face expression emotion analysis model;

[0043] The extraction module is configured to input a video file to be detected into the speech emotion analysis model and the face expression emotion analysis model for feature extraction, respectively, and obtain a speech emotion feature and a face expression emotion feature;

[0044] The analysis module is configured to construct an emotion similarity matrix through the speech emotion feature and the face expression emotion feature, and analyze a face deep fake detection result according to the emotion similarity matrix;

[0045] The CNN-15-GRU-B model includes:

[0046] The CNN-15-GRU-B model includes:

[0047] The first conversion module is an input end of the CNN-15-GRU-B model, the output end of the first conversion module is connected with the input end of the second conversion module and the input end of the speech feature extraction module respectively, the output end of the second conversion module is connected with the input end of the BERT module, the output end of the BERT module and the output end of the speech feature extraction module are connected with the input end of the splicing module, the output end of the splicing module is connected with the input end of the GRU network module, the output end of the GRU network module is connected with the input end of the first output module, and the output end of the first output module is an output end of the CNN-15-GRU-B model.

[0048] The improved CNN model comprises a face detection module, a feature extraction module, a feature integration module and a second output module connected in sequence for extracting a face.

[0049] The application further provides a terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the face deep fake detection method based on speech and expression matching when executing the computer program.

[0050] The application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the face deep fake detection method based on speech and expression matching.

[0051] The above-mentioned scheme of the application has the following beneficial effects:

[0052] The application trains the CNN-15-GRU-B model by inputting the video files for training into the CNN-15-GRU-B model and the improved CNN model respectively, obtains a speech emotion analysis model by training the CNN-15-GRU-B model, obtains a face expression emotion analysis model by training the improved CNN model, extracts features from the video files to be detected by the speech emotion analysis model and the face expression emotion analysis model respectively to obtain speech emotion features and face expression emotion features, constructs an emotion similarity matrix by the speech emotion features and the face expression emotion features, analyzes the face deep fake detection result according to the emotion similarity matrix, and compares with the prior art.

[0053] Other beneficial effects of the application will be described in detail in the subsequent specific embodiment part. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1A flowchart of an embodiment of the present application;

[0055] Figure 2 A structural diagram of the CNN-15-GRU-B model in an embodiment of the present application;

[0056] Figure 3 A structural diagram of the improved CNN model in an embodiment of the present application;

[0057] Figure 4 A structural diagram of the LMS adaptive filtering processing model in an embodiment of the present application;

[0058] Figure 5 A structural diagram of the video face deepfake detection device in an embodiment of the present application;

[0059] Figure 6 A structural diagram of the terminal device in an embodiment of the present application. DETAILED DESCRIPTION

[0060] To make the technical problems, technical solutions and advantages of the present application clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0061] It should be understood that when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The terms "comprising", "including", "having" and their variants mean "including but not limited to", unless otherwise specifically stated.

[0062] It should also be understood that the term "and / or" used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0063] In addition, in the description of the present application and the appended claims, the terms "first", "second", "third" and the like are only used for differentiation in description, and cannot be understood as indicating or implying relative importance.

[0064] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be a locking connection, or a detachable connection, or an integral connection; it can be a mechanical connection, or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or the internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0065] In addition, the technical features involved in the different embodiments of the application described below can be combined with each other as long as there is no conflict.

[0066] The present application aims at the existing problems, and provides a face deep fake detection method based on voice and expression matching and related equipment.

[0067] As Figures 1-3 shown, the embodiment of the present application provides a face deep fake detection method based on voice and expression matching, comprising:

[0068] Step 1, acquiring a video file for training;

[0069] Step 2, inputting the video file into the constructed CNN-15-GRU-B model, training the CNN-15-GRU-B model, and obtaining a voice emotion analysis model;

[0070] Step 3, inputting the video file into the improved CNN model, training the improved CNN model, and obtaining a face expression emotion analysis model;

[0071] Step 4, inputting a video file to be detected into the voice emotion analysis model and the face expression emotion analysis model respectively for feature extraction, and obtaining voice emotion features and face expression emotion features;

[0072] Step 5, constructing an emotion similarity matrix through the voice emotion features and the face expression emotion features, and analyzing the face deep fake detection result according to the emotion similarity matrix;

[0073] The CNN-15-GRU-B model comprises:

[0074] a first conversion module for converting a video file into a voice file, a second conversion module for converting a voice file into a text file, a voice feature extraction module, a BERT module, a splicing module, a GRU network module, and a first output module;

[0075] The first conversion module is the input of the CNN-15-GRU-B model. The output of the first conversion module is connected to the input of the second conversion module and the input of the speech feature extraction module. The output of the second conversion module is connected to the input of the BERT module. The outputs of the BERT module and the speech feature extraction module are both connected to the input of the concatenation module. The output of the concatenation module is connected to the input of the GRU network module. The output of the GRU network module is connected to the input of the first output module. The output of the first output module is the output of the CNN-15-GRU-B model.

[0076] The improved CNN model includes a face detection module, a feature extraction module, a feature integration module, and a second output module connected in sequence.

[0077] Specifically, the CNN-15-GRU-B model is used for:

[0078] The video file is input into the first conversion module for conversion to obtain the audio file;

[0079] The audio file is input into the second conversion module for conversion, resulting in a text file;

[0080] The speech file is input into the speech feature extraction module for feature extraction, resulting in a speech feature matrix;

[0081] The text file is input into the BERT module for multi-head self-attention calculation to obtain the text feature matrix;

[0082] The speech feature matrix and the text feature matrix are input into the concatenation module for feature concatenation to obtain joint features;

[0083] The joint features are input into the GRU network module to capture dependencies and obtain classification features.

[0084] The classification features are input into the first output module for classification, resulting in various speech emotion features.

[0085] Preferably, after the first conversion module converts the video file into an audio file, it further includes preprocessing the audio file to obtain a preprocessed audio file, and then inputting the preprocessed audio file into the second conversion module and the audio feature extraction module respectively.

[0086] In this embodiment of the invention, the first conversion module uses Python's FFmpeg library to convert video files into audio files. FFmpeg is an open-source computer program that can record and convert digital audio and video and convert them into streams. The second conversion module uses Mozilla's DeepSpeech open-source ASR technology to convert audio files into text files. The core technology of DeepSpeech open-source ASR technology is based on deep learning and adopts a combination architecture of convolutional neural networks (CNN) and recurrent neural networks (RNN) to capture the temporal features in the audio signal.

[0087] In this embodiment of the invention, preprocessing of the voice file includes:

[0088] First, the audio file is processed by frame segmentation, dividing the audio file into multiple small segments with a width of 30ms and a frame shift of 10ms.

[0089] Then, the signal segment is windowed. The Hamming window function is applied to smooth each frame of the signal segment to obtain the windowed signal segment.

[0090] Then, a high-pass filter is used to enhance the high-frequency part of the signal in each frame segment after windowing, and the stop frequency is set to 300Hz and the filter order is 2 to obtain the enhanced signal segment.

[0091] Finally, the LMS adaptive filtering method is used for noise reduction to obtain the denoised speech file. The LMS adaptive filtering model diagram is shown below. Figure 4 As shown, the noise reduction process is as follows:

[0092] Let the input signal vector be X(n).

[0093] X(n) = [x(n)x(n-1)…x(nK)] T

[0094] The weight coefficient vector is W(n).

[0095] W(n) = [w0(n)w1(n-1)...w K (n)] T

[0096] Output

[0097]

[0098] The error e(n) is defined as d(n) - y(n), and the convergence condition of the function is defined as...

[0099]

[0100] Define the LMS function: Define the filter order as J = 25, and the convergence factor as 0.005; use the len function to obtain the length len_x of the input signal X(n); initialize the error signal e(n) with the same length as the input signal and an initial value of 0; initialize the weight matrix W with J rows and len_x columns, with an initial value of 0. Loop from index 25 until len_x ends. For each index k, retrieve the input signal vector x of length J in reverse order, calculate the filter output y using the weights and x from the previous time step, print the output value y, calculate the error signal en[k-1], and update the current column of the weight matrix W. Initialize the filter output vector yn, with all elements empty. Loop again, starting from index J until len(xn). For each index k, retrieve the input signal vector x in reverse order, calculate the filter output yn[k] using the latest weights and x. Finally, output the denoised speech file yn.

[0101] It should be noted that in the CNN-15-GRU-B model, the speech feature extraction module is the branch for processing speech, while the second conversion module and the BERT module constitute the branch for processing text.

[0102] Ideally, the speech feature extraction module includes:

[0103] The first convolutional layer, the first max pooling layer, the composite convolutional layer, and the second max pooling layer are connected in sequence.

[0104] It should be noted that in the speech branch, the first convolutional layer uses a one-dimensional convolution with a kernel size of 160 to perform convolution operations on the denoised speech file, with a default stride of 4. In this embodiment of the invention, a batch normalization layer is also provided after the first convolutional layer to accelerate the training process and reduce overfitting. A Leaky ReLU activation function is also applied to introduce non-linearity to enhance the model's expressive power. The expression for the Leaky ReLU activation function is:

[0105] Leaky ReLU(x)=max(0,x)+α·min(0,x)

[0106] Where x represents the input data, and α represents the slope that controls the negative input, with a default value of 0.01.

[0107] In this embodiment of the invention, both the first max pooling layer and the second max pooling layer are used to downsample the output of the first convolutional layer to reduce the spatial dimension of the features.

[0108] The most preferred embodiment is that the composite convolutional layer includes:

[0109] There are 14 sequentially connected convolutional layers, where the input of the first convolutional layer is connected to the output of the first max pooling layer, and the output of the last convolutional layer is connected to the input of the second max pooling layer.

[0110] In this embodiment of the invention, each convolutional layer in the composite convolutional layer uses a convolutional kernel of size 3 with a stride of 1. The initial number of output channels in the composite convolutional layer is 18, specifically, the first to fourth convolutional layers have 18 output channels, the fifth to eighth convolutional layers have 36 output channels, the ninth to eleventh convolutional layers have 72 output channels, and the twelfth to fourteenth convolutional layers have 144 output channels. Each convolutional layer is followed by a batch normalization layer and a Leaky ReLU activation function to accelerate the training process, reduce overfitting, and introduce nonlinearity to enhance the model's expressive power. Finally, a second max-pooling layer is used for downsampling to obtain a model with shape (T, D). s The speech feature matrix, where T represents the time step of the speech, and D... s This represents the dimension of the speech features at each time step.

[0111] Preferably, in the text branch, the second conversion module uses Mozilla's DeepSpeech open-source ASR technology to convert speech into corresponding text, and performs word segmentation and tokenization on the text to obtain a text representation. This representation is then input into the BERT module, which performs multi-head self-attention calculation on the input text representation to generate a high-dimensional vector containing contextual information to form a shape of (N,D). w The text feature matrix is ​​, where represents the number of words in the text, represents the dimension of the embedding for each word, which is usually 768, and represents the number of text representations, i.e. the number of words and segments.

[0112] It should be noted that the BERT module is the BERT model. This embodiment of the invention does not involve any improvement to the structure of the BERT model, therefore its specific structure is not described in detail.

[0113] Preferably, the first output module includes a fully connected layer and a softmax activation function layer connected in sequence;

[0114] The input of the fully connected layer is connected to the output of the GRU network module;

[0115] The output of the softmax activation function layer is the output of the CNN-15-GRU-B model, used to output various speech emotion features.

[0116] In this embodiment of the invention, to align speech and text features, a fixed-length alignment window method is used before inputting to the fully connected layer. Since the time axis of speech features is typically longer than that of the text sequence, the speech features need to be divided into multiple windows, each corresponding to one or more words in the text. Assuming the text sequence has N words and the speech sequence has T time steps, a fixed window length is set, dividing the speech sequence into N windows, each with a size of L = T / N. The speech feature matrix is ​​divided into N sub-intervals according to the time steps, with each sub-interval having a length of L, thus corresponding to each word in the text feature matrix. For each sub-interval, its features are compressed using average pooling to obtain a fixed-dimensional feature vector. This compresses the local features within each window into a single vector while ensuring alignment with each word in the text. The compression expression is:

[0117] speechfeature i =pooling(x L(i-1)；Li )

[0118] Where pooling(·) represents the pooling operation, i = 1, 2, 3, ..., N;

[0119] The concatenation module concatenates the speech and text features of the corresponding window, as shown in the expression:

[0120] joinfeature i =speechfeature i ;textfeature i

[0121] Among them, joinfeature i Speechfeature represents joint features. i Representing speech features, textfeature i Representing text features;

[0122] The joint feature is that the shape is (N,D) s +D w A matrix, where N represents the number of words, D s D w These represent the dimensions of speech features and text features, respectively.

[0123] After feature concatenation, the joint features can be passed as input to the GRU network module. The GRU network module, also known as the GRU network, can capture long-term dependencies, process sequence data, and capture temporal dependencies. The activation and update expression of the GRU network module is:

[0124]

[0125] in, Indicates candidate activation, z t Indicates an update to the door;

[0126]

[0127] Where r represents the reset gate and * represents element-wise multiplication.

[0128] The output of the last time step of the GRU layer is used as the feature representation for classification. The output of the GRU layer is passed through a fully connected layer to map to the required number of output categories, and then a softmax activation function layer is used to classify the output of the fully connected layer. The classification results mainly include four emotional states: neutral, happy, angry, and sad.

[0129] The softmax activation function is:

[0130]

[0131] Where, x i Let represent the i-th output of the fully connected layer, and C represent the number of sentiment categories, where C = 4.

[0132] In this embodiment of the invention, the CNN-15-GRU-B model uses the cross-entropy loss function, as shown in the following formula:

[0133]

[0134] Where M represents the number of categories, y o,c Represents a binary indicator (0 or 1), where y is the correct classification of observation o if category c is correct. o,c If y is 1, otherwise y o,c A value of 0 indicates that the model predicts the probability that observation o belongs to c.

[0135] Specifically, the improved CNN model is used for:

[0136] The video file is input into the face detection module to extract facial features and obtain a facial feature map;

[0137] The facial feature map is input into the feature extraction module for fine-grained feature extraction, resulting in a fine-grained facial feature map.

[0138] By integrating the partial facial feature map with the fine-grained facial feature map, the global facial features are obtained.

[0139] The global facial features are input into the second output module for classification, resulting in various facial expression and emotion features.

[0140] In this embodiment of the invention, the face portion is extracted using a face detection module, and then a 48*48 face portion feature map is obtained. The extraction process is as follows:

[0141] First, the VideoCapture method of the cv library is used to read each frame of the video. The read frame image is scaled to 800x600 pixels, and its color space is converted from BGR to RGB and then grayscaled.

[0142] Then, the face detector in the face detection module detects faces in the image and returns a list of rects containing face bounding boxes. The coordinates of two points in the face bounding box are Cp[(c1,y1),(x2,y2)], where (x1,y1) is the coordinate of the top left corner and (x2,y2) is the coordinate of the bottom right corner. At the same time, an empty list faces is initialized to store the detection results of face key points (full_object_detection). The face detector is provided by the Dlib library.

[0143] Next, the face bounding box in the list rects is traversed. The predictor in the face detection module predicts the feature points of the face within each bounding box and adds the prediction results to the faces list.

[0144] Finally, extract the image of each face from the faces list, normalize the pixel values ​​of the grayscale face image to between 0 and 1, and use linear interpolation to scale the normalized face image to a size of 48*48, thus obtaining a 48*48 face feature map.

[0145] In this embodiment of the invention, before inputting the facial feature map into the feature extraction module, image data enhancement is also included, and the specific process is as follows:

[0146] Initialize an empty list `resized_images` to store the augmented images;

[0147] Add the original scaled image to the list;

[0148] Crops a sub-image from rows 2 to 45 of the original image and adds this sub-image to the list;

[0149] Crops a sub-image from row 1 to row 47 and adds it to the list;

[0150] Flip the original image along the horizontal axis and add the flipped image to the list;

[0151] Iterate through each image in the list and scale them back to 48x48;

[0152] Finally, a new dimension is added to each image to convert it into a 3D tensor, and the augmented image array resized_images is returned.

[0153] The preferred method is to input the facial feature map into the feature extraction module for fine-grained feature extraction to obtain a fine-grained facial feature map, specifically including:

[0154] The initial convolutional layer in the feature extraction module extracts preliminary features from the augmented image array resized_images through a series of convolutional kernels;

[0155] Subsequently, these preliminary features are used to introduce nonlinearity through an activation function (ReLU), the expression of which is:

[0156] Leaky ReLU(x) = max(0,x)

[0157] Next, downsampling is performed through a pooling layer (Pool2) to reduce the spatial dimension of the features while increasing invariance to image displacement;

[0158] Then, the downsampled features are further abstracted through intermediate convolutional layers (Conv4, Conv5, and Conv6), and more advanced visual patterns are captured through complex feature combinations.

[0159] After passing through the activation function and pooling operation (Pool3) again, a more compact feature representation is formed.

[0160] Subsequently, the network further integrates feature information through multiple fully connected layers (fc1, fc2, and fc3) in the feature integration module;

[0161] Finally, the classification is output using the Softmax activation function, whose expression is:

[0162]

[0163] Where, x i Let represent the i-th output of the fully connected layer, and C be the number of sentiment categories, where C = 4.

[0164] The parameter settings for the improved CNN model are shown in Table 1 below:

[0165] Table 1

[0166] Name Number of convolution kernels Convolution kernel size Step Padding Dropout Feature map size Input 0 0 None None 0 (48,48,1) Conv1 32 1*1 1 0 0 (48,48,32) Conv2 64 3*3 1 1 0 (48,48,64) Conv3 64 5*5 1 2 0 (48,48,64) Pool2 0 2*2 2 0 0 (24,24,64) Conv4 64 3*3 1 1 0 (24,24,64) Conv5 64 5*5 1 2 0 (24,24,64) Conv6 64 5*5 1 2 0 (24,24,64) Pool3 0 2*2 2 0 0 (12,12,64) Fc1 None None None 0 50％ (1,1,2048) Fc2 None None None 0 50％ (1,1,1024) Output None None None 0 0 (1,1,8)

[0167] The improved CNN model uses the cross-entropy function, as shown in the following formula:

[0168]

[0169] Where C represents the number of categories, y a,b Represents a binary indicator (0 or 1), where y is the correct classification of observation a if category b is the correct classification of observation a. a,b If y is 1, otherwise y a,b A value of 0 indicates that the model predicts the probability that observation a belongs to b.

[0170] Specifically, the facial expression sentiment analysis model predicts the emotion of each facial image, resulting in a prediction vector. This vector contains the prediction probabilities of different emotion categories (4 types). The sum of all prediction vectors yields a final prediction vector. The linear weighted expression is as follows:

[0171] W=w1·r1+w2·r2+w3·r3+w4·r4

[0172] Wherein, weight w i All are 1;

[0173] Finally, the emotion category with the highest probability was selected as the facial expression emotion feature of this video file after weighting.

[0174] Specifically, the analysis of deepfake face detection results based on the sentiment similarity matrix includes:

[0175] When the similarity between the voice emotion features and the facial expression emotion features in the emotion similarity matrix is ​​greater than a preset threshold, the emotions are considered to be consistent, and the deep face forgery detection result is no forgery phenomenon;

[0176] When the similarity between the voice emotion features and the facial expression emotion features in the emotion similarity matrix is ​​less than a preset threshold, the emotions are considered inconsistent, and the deepfake detection result indicates that forgery exists.

[0177] Specifically, the emotion similarity matrix is ​​shown in Table 2 below, where S(i,j) represents the similarity between voice emotion feature i and facial expression emotion feature j (the value ranges from 0 to 1, where 1 indicates complete similarity and 0 indicates complete dissimilarity). When comparing two emotions, if the similarity exceeds a preset threshold of 0.7, it is considered that the emotions are consistent, indicating the absence of deepfake facial expressions; otherwise, it is judged that they are present.

[0178] Table 2

[0179] Neutral Happy Sad Angry Neutral 1.0 0.6 0.3 0.4 Happy 0. 1.0 0.2 0.3 Sad 0.3 0.2 1.0 0.5 Angry 0.4 0.3 0.5 1.0

[0180] In summary, this invention employs two methods: first, it inputs video files used for training into a constructed CNN-15-GRU-B model and an improved CNN model. The CNN-15-GRU-B model is trained to obtain a speech emotion analysis model, and the improved CNN model is trained to obtain a facial expression emotion analysis model. The video files to be detected are then input into both the speech emotion analysis model and the facial expression emotion analysis model for feature extraction, yielding speech emotion features and facial expression emotion features. An emotion similarity matrix is ​​constructed using these features, and the results of deepfake face detection are analyzed based on this matrix. Compared to existing technologies, this invention extracts speech emotion features and facial expression emotion features through the speech emotion analysis model and the facial expression emotion analysis model. Based on the similarity between these two emotions, it deeply analyzes the naturalness of facial expressions to determine whether deepfake faces exist. This approach overcomes the limitation of directly training deepfake face samples while improving detection accuracy.

[0181] like Figure 5 As shown, this embodiment of the invention also provides a deepfake face detection device 100 based on voice and facial expression matching. The video deepfake face detection device 100 includes:

[0182] Module 101 is used to acquire video files for training.

[0183] The first training module 102 is used to input video files into the constructed CNN-15-GRU-B model, train the CNN-15-GRU-B model, and obtain a speech sentiment analysis model.

[0184] The second training module 103 is used to input video files into the improved CNN model, train the improved CNN model, and obtain a facial expression emotion analysis model.

[0185] The extraction module 104 is used to input the video file to be detected into the speech emotion analysis model and the facial expression emotion analysis model for feature extraction, respectively, to obtain speech emotion features and facial expression emotion features;

[0186] Analysis module 105 is used to construct an emotion similarity matrix through voice emotion features and facial expression emotion features, and to analyze the deep face forgery detection results based on the emotion similarity matrix;

[0187] The CNN-15-GRU-B model includes:

[0188] A first conversion module for converting video files to audio files, a second conversion module for converting audio files to text files, an audio feature extraction module, a BERT module, a splicing module, a GRU network module, and a first output module;

[0189] The first conversion module is the input of the CNN-15-GRU-B model. The output of the first conversion module is connected to the input of the second conversion module and the input of the speech feature extraction module. The output of the second conversion module is connected to the input of the BERT module. The outputs of the BERT module and the speech feature extraction module are both connected to the input of the concatenation module. The output of the concatenation module is connected to the input of the GRU network module. The output of the GRU network module is connected to the input of the first output module. The output of the first output module is the output of the CNN-15-GRU-B model.

[0190] The improved CNN model includes a face detection module, a feature extraction module, a feature integration module, and a second output module connected in sequence.

[0191] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0192] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0193] This invention also provides a terminal device, such as... Figure 6 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 6 The diagram shows only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100. When the processor D100 executes the computer program D102, it implements the above-described deep face forgery detection method based on voice and facial expression matching.

[0194] The terminal device D10 can be a desktop computer, laptop, handheld computer, server, server cluster, or cloud server, etc. This terminal device may include, but is not limited to, a processor D100 and a memory D101. Those skilled in the art will understand that... Figure 6 This is merely an example of terminal device D10 and does not constitute a limitation on terminal device D10. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0195] The processor D100 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0196] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may be an external storage device of the terminal device D10, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device D10. Furthermore, the memory D101 may include both internal and external storage units of the terminal device D10. The memory D101 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory D101 can also be used to temporarily store data that has been output or will be output.

[0197] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0198] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0199] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a deep face forgery detection method based on voice and facial expression matching.

[0200] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a building device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0201] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A face deepfake detection method based on speech and expression matching, characterized in that, The method comprises the following steps: Step 1, obtaining a video file for training; Step 2, inputting the video file into a constructed CNN-15-GRU-B model, training the CNN-15-GRU-B model, and obtaining a voice emotion analysis model; Step 3, inputting the video file into an improved CNN model, training the improved CNN model, and obtaining a facial expression emotion analysis model; Step 4, inputting a video file to be detected into the voice emotion analysis model and the facial expression emotion analysis model respectively for feature extraction, and obtaining voice emotion features and facial expression emotion features; Step 5, constructing an emotion similarity matrix through the voice emotion features and the facial expression emotion features, and analyzing a facial deepfake detection result according to the emotion similarity matrix. The CNN-15-GRU-B model comprises: a first conversion module for converting a video file into a voice file, a second conversion module for converting a voice file into a text file, a voice feature extraction module, a BERT module, a splicing module, a GRU network module, and a first output module; the first conversion module is an input end of the CNN-15-GRU-B model, the output end of the first conversion module is connected with the input end of the second conversion module and the input end of the voice feature extraction module respectively, the output end of the second conversion module is connected with the input end of the BERT module, the output end of the BERT module and the output end of the voice feature extraction module are connected with the input end of the splicing module, the output end of the splicing module is connected with the input end of the GRU network module, the output end of the GRU network module is connected with the input end of the first output module, and the output end of the first output module is an output end of the CNN-15-GRU-B model; the improved CNN model comprises a face detection module, a feature extraction module, a feature integration module, and a second output module connected in sequence.

2. The face deepfake detection method based on voice and expression matching according to claim 1, characterized in that, The CNN-15-GRU-B model is used for: inputting the video file into the first conversion module for conversion to obtain a voice file; inputting the voice file into the second conversion module for conversion to obtain a text file; inputting the voice file into the voice feature extraction module for feature extraction to obtain a voice feature matrix; inputting the text file into the BERT module for multi-head self-attention calculation to obtain a text feature matrix; inputting the voice feature matrix and the text feature matrix into the splicing module for feature splicing to obtain a joint feature; inputting the joint feature into the GRU network module for dependency relationship capture to obtain a classification feature; inputting the classification feature into the first output module for classification to obtain a plurality of voice emotion features. 3.The face deepfake detection method based on voice and expression matching according to claim 2, characterized in that, The voice feature extraction module comprises: a first convolutional layer, a first max-pooling layer, a composite convolutional layer, and a second max-pooling layer connected in sequence.

4. The face deepfake detection method based on voice and expression matching according to claim 3, characterized in that, The composite convolutional layer comprises: 14 convolutional layers connected in sequence, wherein an input end of a first convolutional layer is connected with an output end of the first max-pooling layer, and an output end of a last convolutional layer is connected with an input end of the second max-pooling layer.

5. The face deepfake detection method based on voice and expression matching according to claim 4, characterized in that, The first output module comprises a fully connected layer and a softmax activation function layer connected in sequence; An input end of the fully connected layer is connected with an output end of the GRU network module; An output end of the softmax activation function layer is an output end of the CNN-15-GRU-B model, and is used for outputting multiple speech emotion features.

6. The face deepfake detection method based on voice and expression matching according to claim 5, characterized in that, The improved CNN model is used for: inputting the video file into a face detection module to extract face part features, and obtaining a face part feature map; inputting the face part feature map into the feature extraction module to extract fine-grained features, and obtaining a face fine-grained feature map; integrating the face part feature map and the face fine-grained feature map to obtain face global features; inputting the face global features into the second output module for classification to obtain multiple face expression emotion features.

7. The face deepfake detection method based on voice and expression matching according to claim 6, characterized in that, According to the emotion similarity matrix, the face deep fake detection result is analyzed, including: when the similarity between the speech emotion features and the face expression emotion features in the emotion similarity matrix is greater than a preset threshold, it is considered that the emotions are consistent, and the face deep fake detection result is no fake phenomenon; when the similarity between the speech emotion features and the face expression emotion features in the emotion similarity matrix is less than a preset threshold, it is considered that the emotions are inconsistent, and the face deep fake detection result is that there is a fake phenomenon.

8. A face deepfake detection device based on voice and expression matching, characterized in that, including: an acquisition module configured to acquire a video file for training; a first training module configured to input the video file into the constructed CNN-15-GRU-B model, train the CNN-15-GRU-B model, and obtain a speech emotion analysis model; a second training module configured to input the video file into the improved CNN model, train the improved CNN model, and obtain a face expression emotion analysis model; an extraction module configured to input a to-be-detected video file into the speech emotion analysis model and the face expression emotion analysis model respectively to extract features, and obtain speech emotion features and face expression emotion features; an analysis module configured to construct an emotion similarity matrix through the speech emotion features and the face expression emotion features, and analyze a face deep fake detection result according to the emotion similarity matrix. The CNN-15-GRU-B model comprises: a first conversion module configured to convert a video file into a speech file, a second conversion module configured to convert a speech file into a text file, a speech feature extraction module, a BERT module, a splicing module, a GRU network module, and a first output module. The first conversion module is an input end of the CNN-15-GRU-B model, output ends of the first conversion module are connected with input ends of the second conversion module and the speech feature extraction module respectively, an output end of the second conversion module is connected with an input end of the BERT module, output ends of the BERT module and the speech feature extraction module are connected with an input end of the splicing module, an output end of the splicing module is connected with an input end of the GRU network module, an output end of the GRU network module is connected with an input end of the first output module, and an output end of the first output module is an output end of the CNN-15-GRU-B model. The improved CNN model comprises a face detection module, a feature extraction module, a feature integration module and a second output module connected in sequence for extracting a face.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the face deep fake detection method based on matching of speech and expression according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the face deep fake detection method based on matching of speech and expression according to any one of claims 1 to 7.

Images