Face depth forgery detection method based on voice and expression matching and related equipment

Abstract

The invention provides a face deep counterfeiting detection method based on voice and expression matching and related equipment, and the method comprises the steps: inputting a video file for training into a constructed CNN-15-GRU-B model and an improved CNN model for training, and obtaining a voice emotion analysis model and a face expression emotion analysis model; the feature extraction module is used for performing feature extraction on the to-be-detected video file to obtain voice emotion features and facial expression emotion features to construct an emotion similarity matrix, and analyzing a face depth forgery detection result according to the emotion similarity matrix; compared with the prior art, the speech emotion feature and the facial expression emotion feature are extracted through the speech emotion analysis model and the facial expression emotion analysis model, the naturalness of the character expression is deeply analyzed based on the similarity of the two emotions, and then whether deep face counterfeiting exists or not is judged. And the detection precision is improved while the limitation of directly training a face deep forged sample is eliminated.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for detecting deep fake faces based on voice and expression matching and related equipment. Background Art

[0002] In recent years, deep learning technology has been developing continuously and has been widely used in the field of computer vision. On the one hand, deep learning technology has led a new wave of artificial intelligence, but on the other hand, a series of security issues caused by deep learning have also attracted more and more attention. At present, image and video recognition technology based on deep learning is widely used in all aspects of people's lives, such as intelligent supervision of network content, automatic video surveillance analysis, access control systems based on face recognition, face payment, etc. In these key application areas, the reliability and security of information and data should be taken seriously and guaranteed. Since 2017, some fake images and videos generated by deep fake (also known as DeepFake) technology have attracted widespread attention on the Internet, especially when deep fakes are used on people with great influence, they often use the influence of the person to have a greater impact. Fake images and videos have become one of the most significant information data security risks at present, and their detection and supervision face huge challenges.

[0003] Fake faces synthesized by AI pose a huge threat. They can create videos that make it appear that the target person is doing or saying something with facial expressions and body movements that are close to real, subverting people's perception of the concept of "seeing is believing". The industry urgently needs an effective technology to detect fake face images or videos in network environments, but it is extremely difficult, mainly because the fake areas of fake face images are weak and localized. Detection is easily affected by image noise. In addition, the fake areas are often unpredictable. Each fake method targets different areas, so it is very difficult to correctly detect and classify the fake areas.

[0004] Currently, researchers have developed a variety of deep fake detection methods in the field of deep fake detection, including those based on texture analysis, lighting consistency, and high-frequency detail analysis. These methods can identify abnormal features in deep fake videos and provide powerful tools for detection. However, existing research often ignores the in-depth analysis of the naturalness of the facial expressions of the characters in the video, resulting in low detection accuracy. Summary of the invention

[0005] The present invention provides a method and related equipment for detecting deep fake faces based on voice and expression matching, the purpose of which is to improve detection accuracy.

[0006] In order to achieve the above object, the present invention provides a method for detecting deep fake faces based on voice and expression matching, comprising:

[0007] Step 1, obtain the video file for training;

[0008] Step 2: Input the video file into the constructed CNN-15-GRU-B model, train the CNN-15-GRU-B model, and obtain a speech sentiment analysis model;

[0009] Step 3, inputting the video file into the improved CNN model, training the improved CNN model, and obtaining a facial expression emotion analysis model;

[0010] Step 4, inputting the video file to be detected into the speech emotion analysis model and the facial expression emotion analysis model for feature extraction, respectively, to obtain speech emotion features and facial expression emotion features;

[0011] Step 5, constructing an emotion similarity matrix through voice emotion features and facial expression emotion features, and analyzing the face deep fake detection results according to the emotion similarity matrix;

[0012] The CNN-15-GRU-B model includes:

[0013] A first conversion module for converting a video file into a voice file, a second conversion module for converting a voice file into a text file, a voice feature extraction module, a BERT module, a splicing module, a GRU network module, and a first output module;

[0014] The first conversion module is the input end of the CNN-15-GRU-B model, the output end of the first conversion module is respectively connected to the input end of the second conversion module and the input end of the speech feature extraction module, the output end of the second conversion module is connected to the input end of the BERT module, the output end of the BERT module and the output end of the speech feature extraction module are both connected to the input end of the splicing module, the output end of the splicing module is connected to the input end of the GRU network module, the output end of the GRU network module is connected to the input end of the first output module, and the output end of the first output module is the output end of the CNN-15-GRU-B model;

[0015] The improved CNN model includes a face detection module for extracting faces, a feature extraction module, a feature integration module, and a second output module which are connected in sequence.

[0016] Specifically, the CNN-15-GRU-B model is used to:

[0017] Inputting the video file into the first conversion module for conversion to obtain a voice file;

[0018] Input the voice file into the second conversion module for conversion to obtain a text file;

[0019] Input the voice file into the voice feature extraction module to extract features and obtain a voice feature matrix;

[0020] Input the text file into the BERT module for multi-head self-attention calculation to obtain the text feature matrix;

[0021] Input the speech feature matrix and the text feature matrix into the splicing module for feature splicing to obtain joint features;

[0022] The joint features are input into the GRU network module to capture dependencies and obtain classification features;

[0023] The classification features are input into the first output module for classification to obtain a variety of speech emotion features.

[0024] Specifically, the speech feature extraction module includes:

[0025] The first convolutional layer, the first maximum pooling layer, the compound convolutional layer, and the second maximum pooling layer are connected in sequence.

[0026] Specifically, the composite convolutional layer consists of:

[0027] 14 convolutional layers are connected in sequence, where the input of the first convolutional layer is connected to the output of the first maximum pooling layer, and the output of the last convolutional layer is connected to the input of the second maximum pooling layer.

[0028] Further, the first output module includes a fully connected layer and a softmax activation function layer connected in sequence;

[0029] The input of the fully connected layer is connected to the output of the GRU network module;

[0030] The output end of the softmax activation function layer is the output end of the CNN-15-GRU-B model, which is used to output a variety of speech emotion features.

[0031] Specifically, the improved CNN model is used to:

[0032] Input the video file into the face detection module to extract the facial features and obtain the facial feature map;

[0033] Inputting the partial feature map of the human face into the feature extraction module for fine-grained feature extraction to obtain a fine-grained feature map of the human face;

[0034] Integrate the partial face feature map with the fine-grained face feature map to obtain the global face feature;

[0035] The global facial features are input into the second output module for classification to obtain a variety of facial expression and emotion features.

[0036] Furthermore, the facial deepfake detection results are analyzed based on the emotion similarity matrix, including:

[0037] When the similarity between the speech emotion features and the facial expression emotion features in the emotion similarity matrix is ​​greater than the preset threshold, the emotions are considered consistent and the face deep fake detection result is no fake phenomenon;

[0038] When the similarity between the speech emotion features and the facial expression emotion features in the emotion similarity matrix is ​​less than the preset threshold, the emotions are considered inconsistent and the face deep fake detection result is that there is a fake phenomenon.

[0039] The present invention also provides a face deep fake detection device based on voice and expression matching, comprising:

[0040] An acquisition module, used to acquire video files for training;

[0041] The first training module is used to input the video file into the constructed CNN-15-GRU-B model, train the CNN-15-GRU-B model, and obtain a speech sentiment analysis model;

[0042] The second training module is used to input the video file into the improved CNN model, train the improved CNN model, and obtain a facial expression emotion analysis model;

[0043] An extraction module is used to input the video file to be detected into the speech emotion analysis model and the facial expression emotion analysis model for feature extraction, thereby obtaining speech emotion features and facial expression emotion features;

[0044] An analysis module is used to construct an emotion similarity matrix based on speech emotion features and facial expression emotion features, and analyze the face deep fake detection results based on the emotion similarity matrix;

[0045] The CNN-15-GRU-B model includes:

[0046] A first conversion module for converting a video file into a voice file, a second conversion module for converting a voice file into a text file, a voice feature extraction module, a BERT module, a splicing module, a GRU network module, and a first output module;

[0047] The first conversion module is the input end of the CNN-15-GRU-B model, the output end of the first conversion module is respectively connected to the input end of the second conversion module and the input end of the speech feature extraction module, the output end of the second conversion module is connected to the input end of the BERT module, the output end of the BERT module and the output end of the speech feature extraction module are both connected to the input end of the splicing module, the output end of the splicing module is connected to the input end of the GRU network module, the output end of the GRU network module is connected to the input end of the first output module, and the output end of the first output module is the output end of the CNN-15-GRU-B model;

[0048] The improved CNN model includes a face detection module for extracting faces, a feature extraction module, a feature integration module, and a second output module which are connected in sequence.

[0049] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method for detecting deep fake faces based on voice and expression matching is implemented.

[0050] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, a method for detecting deep fake faces based on voice and expression matching is implemented.

[0051] The above scheme of the present invention has the following beneficial effects:

[0052] The present invention inputs the video files for training into the constructed CNN-15-GRU-B model and the improved CNN model respectively, trains the CNN-15-GRU-B model to obtain a speech emotion analysis model, and trains the improved CNN model to obtain a facial expression emotion analysis model; inputs the video files to be detected into the speech emotion analysis model and the facial expression emotion analysis model for feature extraction, respectively, to obtain speech emotion features and facial expression emotion features; constructs an emotion similarity matrix through the speech emotion features and the facial expression emotion features, and analyzes the facial deep fake detection result according to the emotion similarity matrix; compared with the prior art, the present invention extracts speech emotion features and facial expression emotion features through the speech emotion analysis model and the facial expression emotion analysis model, deeply analyzes the naturalness of the character's expression based on the similarity of the two emotions, and then determines whether there is facial deep fake, while getting rid of the limitation of directly training facial deep fake samples, and improving the detection accuracy.

[0053] Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1A schematic diagram of a flow chart of an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the structure of the CNN-15-GRU-B model in an embodiment of the present invention;

[0056] Figure 3 It is a structural schematic diagram of an improved CNN model in an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of the structure of an LMS adaptive filtering processing model in an embodiment of the present invention;

[0058] Figure 5 Schematic diagram of the structure of a video face deep fake detection device in an embodiment of the present invention;

[0059] Figure 6 Schematic diagram of the structure of a terminal device in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or their collections. The terms "comprising", "including", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0062] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0063] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0064] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a locking connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0065] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0066] In view of the existing problems, the present invention provides a face deep fake detection method based on voice and expression matching and related equipment.

[0067] like Figure 1-Figure 3 As shown, an embodiment of the present invention provides a method for detecting deep fake faces based on voice and expression matching, comprising:

[0068] Step 1, obtain the video file for training;

[0069] Step 2: Input the video file into the constructed CNN-15-GRU-B model, train the CNN-15-GRU-B model, and obtain a speech sentiment analysis model;

[0070] Step 3, inputting the video file into the improved CNN model, training the improved CNN model, and obtaining a facial expression emotion analysis model;

[0071] Step 4, inputting the video file to be detected into the speech emotion analysis model and the facial expression emotion analysis model for feature extraction, respectively, to obtain speech emotion features and facial expression emotion features;

[0072] Step 5, constructing an emotion similarity matrix through voice emotion features and facial expression emotion features, and analyzing the face deep fake detection results according to the emotion similarity matrix;

[0073] The CNN-15-GRU-B model includes:

[0074] A first conversion module for converting a video file into a voice file, a second conversion module for converting a voice file into a text file, a voice feature extraction module, a BERT module, a splicing module, a GRU network module, and a first output module;

[0075] The first conversion module is the input end of the CNN-15-GRU-B model, the output end of the first conversion module is respectively connected to the input end of the second conversion module and the input end of the speech feature extraction module, the output end of the second conversion module is connected to the input end of the BERT module, the output end of the BERT module and the output end of the speech feature extraction module are both connected to the input end of the splicing module, the output end of the splicing module is connected to the input end of the GRU network module, the output end of the GRU network module is connected to the input end of the first output module, and the output end of the first output module is the output end of the CNN-15-GRU-B model;

[0076] The improved CNN model includes a face detection module for extracting faces, a feature extraction module, a feature integration module, and a second output module which are connected in sequence.

[0077] Specifically, the CNN-15-GRU-B model is used to:

[0078] Inputting the video file into the first conversion module for conversion to obtain a voice file;

[0079] Input the voice file into the second conversion module for conversion to obtain a text file;

[0080] Input the voice file into the voice feature extraction module to extract features and obtain a voice feature matrix;

[0081] Input the text file into the BERT module for multi-head self-attention calculation to obtain the text feature matrix;

[0082] Input the speech feature matrix and the text feature matrix into the splicing module for feature splicing to obtain joint features;

[0083] The joint features are input into the GRU network module to capture dependencies and obtain classification features;

[0084] The classification features are input into the first output module for classification to obtain a variety of speech emotion features.

[0085] Most preferably, after the first conversion module converts the video file into a voice file, it also includes preprocessing the voice file to obtain a preprocessed voice file, and then inputting the preprocessed voice file into the second conversion module and the voice feature extraction module respectively.

[0086] In an embodiment of the present invention, the first conversion module uses Python's FFmpeg library to convert a video file into a voice file. The FFmpeg library is a set of open source computer programs that can be used to record, convert digital audio and video, and convert them into streams. The second conversion module uses Mozilla's DeepSpeech open source ASR technology to convert voice files into text files. The core technology of DeepSpeech open source ASR technology is based on deep learning, and adopts a combined architecture of convolutional neural networks (CNN) and recurrent neural networks (RNN) to capture the timing characteristics in voice signals.

[0087] In an embodiment of the present invention, preprocessing the voice file includes:

[0088] First, the voice file is framed and divided into multiple small segments, with a width of 30ms and a frame shift of 10ms.

[0089] Then, the signal segment is windowed, and each frame of the signal segment is smoothed by applying the Hamming window function to obtain the windowed signal segment;

[0090] Then, a high-pass filtering method is used for each frame signal segment after windowing to enhance the high-frequency part of the signal, and the blocking frequency is set to 300 Hz and the filter order is set to 2 to obtain the enhanced signal segment;

[0091] Finally, the LMS adaptive filtering method is used to perform noise reduction processing to obtain the denoised speech file. The LMS adaptive filtering processing model is shown in the figure below. Figure 4 As shown, the noise reduction process is as follows:

[0092] Assume the input signal vector is X(n),

[0093] X(n)=[x(n)x(n-1)…x(nK)] T

[0094] The weight coefficient vector is W(n),

[0095] W(n)=[w0(n)w1(n-1)…w K (n)] T

[0096] Output

[0097]

[0098] The error e(n) is defined as d(n)-y(n), and the convergence condition of the function is defined as

[0099]

[0100] Define the LMS function: define the filter order J as 25, define the convergence factor as 0.005; use the len function to obtain the length len_x of the input signal X(n); initialize the error signal e(n), the length is the same as the input signal, and the initial value is 0; initialize the weight matrix W, J rows, len_x columns, and the initial value is 0. Start the loop from index 25 until the end of len_x. For each index k, take out the input signal vector x of length J in reverse order, use the weight and x of the previous moment to calculate the filter output y, print the output value y, calculate the error signal en[k-1], and update the current column of the weight matrix W. Initialize the filter output vector yn, all elements are empty. Loop again, starting from index J until the end of len(xn). For each index k, take out the input signal vector x in reverse order, and use the latest weight and x to calculate the filter output yn[k]. Finally, output the denoised speech file yn.

[0101] It should be noted that the speech feature extraction module in the CNN-15-GRU-B model is the branch for processing speech, and the second conversion module and the BERT module constitute the branch for processing text.

[0102] Most preferably, the speech feature extraction module comprises:

[0103] The first convolutional layer, the first maximum pooling layer, the compound convolutional layer, and the second maximum pooling layer are connected in sequence.

[0104] It should be noted that, in the speech branch part, the first convolution layer uses a one-dimensional convolution with a convolution kernel size of 160 to perform a convolution operation on the denoised speech file, and the step size defaults to 4. In the embodiment of the present invention, a batch normalization layer is further provided after the first convolution layer to accelerate the training process and reduce overfitting. A Leaky ReLU activation function is also applied to introduce nonlinearity to enhance the expression ability of the model. The expression of the Leaky ReLU activation function is:

[0105] Leaky ReLU(x)=max(0,x)+α·min(0,x)

[0106] Among them, x represents the input data, α represents the slope of controlling negative input, and the default value is 0.01.

[0107] In the embodiment of the present invention, the first maximum pooling layer and the second maximum pooling layer are both used to downsample the output of the first convolutional layer to reduce the spatial dimension of the features.

[0108] Most preferably, the compound convolutional layer comprises:

[0109] 14 convolutional layers are connected in sequence, where the input of the first convolutional layer is connected to the output of the first maximum pooling layer, and the output of the last convolutional layer is connected to the input of the second maximum pooling layer.

[0110] In the embodiment of the present invention, each convolution layer in the composite convolution layer uses a convolution kernel of size 3, a step size of 1, and an initial output channel number of the composite convolution layer is 18, wherein the output channel number of the first to fourth convolution layers in the composite convolution layer is 18, the output channel number of the fifth to eighth convolution layers is 36, the output channel number of the ninth to eleventh convolution layers is 72, and the output channel number of the twelfth to fourteenth convolution layers is 144. A batch normalization layer and a Leaky ReLU activation function are provided after each convolution layer to accelerate the training process and reduce overfitting, and to introduce nonlinearity to enhance the expression ability of the model. Finally, downsampling is performed through the second maximum pooling layer to obtain a shape of (T, D s ), where T represents the time step of speech, D s Represents the speech feature dimension at each time step.

[0111] Most preferably, in the text branch, the second conversion module uses Mozilla's DeepSpeech open source ASR technology to convert speech into corresponding text, and performs word segmentation and tokenization on the text to obtain text representation, which is then input into the BERT module. The BERT module performs multi-head self-attention calculation on the input text representation to generate a high-dimensional vector containing context information to form a shape of (N, D w ), where represents the number of words in the text, is the dimension of each word embedding, usually 768, and represents the number of text representations, that is, the number of segmentations and words.

[0112] It should be noted that the BERT module is the BERT model. The embodiments of the present invention do not involve improving the structure of the BERT model, so its specific structure is not specifically described.

[0113] Most preferably, the first output module comprises a fully connected layer and a softmax activation function layer connected in sequence;

[0114] The input of the fully connected layer is connected to the output of the GRU network module;

[0115] The output end of the softmax activation function layer is the output end of the CNN-15-GRU-B model, which is used to output a variety of speech emotion features.

[0116] In an embodiment of the present invention, in order to align the features of speech and text, a fixed-length alignment window method is used before inputting the fully connected layer. The time axis of speech features is usually longer than the text sequence, so it is necessary to divide the speech features into multiple windows, each window corresponding to one or more words in the text. Assuming that the text sequence has N words and the speech sequence has T time steps, a fixed window length is set, and the speech sequence is divided into N windows, the size of each window is L=T / N, and the speech feature matrix is ​​divided into N sub-intervals according to the time step, and the length of each sub-interval is L, so that it can correspond to each word in the text feature matrix. For each sub-interval, its features are compressed by an average pooling operation to obtain a feature vector of fixed dimension. This can compress the local features in each window into a vector and ensure alignment with each word of the text. The compression expression is:

[0117] speechfeature i =pooling(x L(i-1)；Li )

[0118] Where pooling(·) represents the pooling operation, i=1, 2, 3,…, N;

[0119] The speech features and text features of the corresponding window are spliced ​​through the splicing module, and the expression is:

[0120] joinfeature i =speechfeature i ;textfeature i

[0121] Among them, joinfeature i Represents joint features, speechfeature i Represents speech features, textfeature i Represents text features;

[0122] The joint feature is of shape (N,D s +D w ), where N represents the number of words, and D s , D w Represent the dimensions of speech features and text features respectively;

[0123] After the feature concatenation is completed, the joint features can be passed as input to the GRU network module. The GRU network module is a GRU network that can capture long-term dependencies, process sequence data, and capture time dependencies. The activation update expression of the GRU network module is:

[0124]

[0125] in, represents candidate activation, z t represents the update gate;

[0126]

[0127] Among them, r represents the reset gate and * represents element-by-element multiplication.

[0128] The output of the last time step of the GRU layer is used as the feature representation for classification. The output of the GRU layer is passed through a fully connected layer to map to the required number of output categories, and then the output of the fully connected layer is classified using a softmax activation function layer. The classification results mainly include four types of emotional states: neutral, happy, angry, and sad;

[0129] The softmax activation function is:

[0130]

[0131] Among them, x i represents the i-th output of the fully connected layer, C represents the number of emotion categories, C=4.

[0132] In this embodiment of the present invention, the CNN-15-GRU-B model uses a cross entropy loss function, and the formula is as follows:

[0133]

[0134] Among them, M represents the number of categories, y o,c represents a binary indicator (0 or 1) that if category c is the correct classification for observation o, then y o,c is 1, otherwise y o,c is 0, indicating that the model predicts the probability that observation o belongs to c.

[0135] Specifically, the improved CNN model is used to:

[0136] Input the video file into the face detection module to extract the facial features and obtain the facial feature map;

[0137] Inputting the partial feature map of the human face into the feature extraction module for fine-grained feature extraction to obtain a fine-grained feature map of the human face;

[0138] Integrate the partial face feature map with the fine-grained face feature map to obtain the global face feature;

[0139] The global facial features are input into the second output module for classification to obtain a variety of facial expression and emotion features.

[0140] In the embodiment of the present invention, the face part is extracted by the face detection module, and then a 48*48 face part feature map is obtained. The extraction process is as follows:

[0141] First, use the VideoCapture method of the cv library to read each frame of the video, scale the read frame image to 800x600 pixels, convert its color space from BGR to RGB and grayscale it;

[0142] Then, the face detector in the face detection module detects the face in the image and returns a list rects containing the face bounding box, the coordinates of the two points in the face bounding box Cp[(c1,y1),(x2,y2)], (x1,y1) is the coordinate of the upper left corner, (x2,y2) is the coordinate of the lower right corner, and initializes an empty list faces to store the detection results of the face key points (full_object_detection), where the face detector is provided by the Dlib library;

[0143] Next, each face bounding box in the list rects is traversed, and the predictor in the face detection module predicts the feature points of the face in each bounding box and adds the prediction results to the faces list;

[0144] Finally, extract the image of each face in the faces list, normalize the pixel value of the grayscale face image to between 0 and 1, and use the linear interpolation method to scale the normalized face image to a size of 48*48 to obtain a 48*48 face feature map.

[0145] In the embodiment of the present invention, before the facial feature map is input into the feature extraction module, the image data is enhanced. The specific process is as follows:

[0146] Initialize an empty list resized_images to store the augmented images;

[0147] Add the original scaled image to the list;

[0148] Crop a sub-image from rows 2 to 45 in the original image and add this sub-image to the list;

[0149] From line 1 to line 47, a sub-image is cropped and added to the list;

[0150] Flip the original image along the horizontal axis and add the flipped image to the list;

[0151] Iterate through each image in the list and scale them to 48*48 again;

[0152] Finally, a new dimension is added to each image, converting it into a 3D tensor and returning the augmented image array resized_images.

[0153] Most preferably, the face part feature map is input into the feature extraction module for fine-grained feature extraction to obtain the face fine-grained feature map, which specifically includes:

[0154] Through the primary convolution layer in the feature extraction module, this layer extracts preliminary features of the augmented image array resized_images through a series of convolution kernels;

[0155] Subsequently, these preliminary features are introduced into nonlinearity through the activation function (ReLU), and the expression of the activation function is:

[0156] Leaky ReLU(x)=max(0,x)

[0157] Next, downsampling is performed through the pooling layer (Pool2) to reduce the spatial dimension of the features while increasing the invariance to image displacement;

[0158] Then, the downsampled features are further abstracted through the intermediate convolutional layers (Conv4, Conv5, and Conv6), capturing more advanced visual patterns through complex feature combinations;

[0159] After the activation function and pooling operation (Pool3) again, a more compact feature representation is formed.

[0160] Subsequently, the network further integrates feature information through multiple fully connected layers (fc1, fc2, and fc3) in the feature integration module;

[0161] Finally, the classification is output through the Softmax activation function. The expression of the Softmax activation function is:

[0162]

[0163] Among them, x i represents the i-th output of the fully connected layer, C is the number of emotion categories, C=4.

[0164] The parameter settings of the improved CNN model are shown in Table 1:

[0165] Table 1

[0166] name Number of convolution kernels Convolution kernel size Step Length filling throw away Feature map size Input 0 0 None None 0 (48,48,1) Conv1 32 1*1 1 0 0 (48,48,32) Conv2 64 3*3 1 1 0 (48,48,64) Conv3 64 5*5 1 2 0 (48,48,64) Pool2 0 2*2 2 0 0 (24,24,64) Conv4 64 3*3 1 1 0 (24,24,64) Conv5 64 5*5 1 2 0 (24,24,64) Conv6 64 5*5 1 2 0 (24,24,64) Pool3 0 2*2 2 0 0 (12,12,64) Fc1 None None None 0 50％ (1,1,2048) Fc2 None None None 0 50％ (1,1,1024) Output None None None 0 0 (1,1,8)

[0167] Among them, the improved CNN model uses the cross entropy function, the formula is as follows:

[0168]

[0169] Among them, C represents the number of categories, y a,b represents a binary indicator (0 or 1) that if class b is the correct classification for observation a, then y a,b is 1, otherwise y a,b is 0, indicating that the model predicts the probability that observation a belongs to b.

[0170] Specifically, the facial expression emotion analysis model predicts the expression emotion of each face image and obtains a prediction result vector. This vector contains the prediction probabilities of different expression emotion categories (4 types). After adding all the prediction result vectors, a total prediction result vector is obtained. The linear weighted expression is:

[0171] W=w1·r1+w2·r2+w3·r3+w4·r4

[0172] Among them, the weight w i All are 1;

[0173] Finally, after weighting, the emotion category with the highest probability is selected as the facial expression emotion feature of this video file.

[0174] Specifically, the face deepfake detection results are analyzed based on the emotion similarity matrix, including:

[0175] When the similarity between the speech emotion features and the facial expression emotion features in the emotion similarity matrix is ​​greater than the preset threshold, the emotions are considered consistent and the face deep fake detection result is no fake phenomenon;

[0176] When the similarity between the speech emotion features and the facial expression emotion features in the emotion similarity matrix is ​​less than the preset threshold, the emotions are considered inconsistent and the face deep fake detection result is that there is a fake phenomenon.

[0177] Specifically, the emotion similarity matrix is ​​shown in Table 2 below, where S(i,j) represents the similarity between speech emotion feature i and facial expression emotion feature j (the value range is 0 to 1, 1 means completely identical, 0 means completely different). Comparing two emotions, if the similarity exceeds the preset threshold of 0.7, it is considered that the emotion consistency indicates that there is no deep fake face phenomenon, otherwise it is judged to be true.

[0178] Table 2

[0179] neutral happy sad angry neutral 1.0 0.6 0.3 0.4 happy 0. 1.0 0.2 0.3 sad 0.3 0.2 1.0 0.5 angry 0.4 0.3 0.5 1.0

[0180] In summary, the embodiment of the present invention respectively inputs the video files used for training into the constructed CNN-15-GRU-B model and the improved CNN model, trains the CNN-15-GRU-B model to obtain a speech emotion analysis model, and trains the improved CNN model to obtain a facial expression emotion analysis model; inputs the video files to be detected into the speech emotion analysis model and the facial expression emotion analysis model for feature extraction, respectively, to obtain speech emotion features and facial expression emotion features; constructs an emotion similarity matrix through the speech emotion features and the facial expression emotion features, and analyzes the facial deep fake detection results according to the emotion similarity matrix; compared with the prior art, the embodiment of the present invention extracts speech emotion features and facial expression emotion features through the speech emotion analysis model and the facial expression emotion analysis model, deeply analyzes the naturalness of the character's expression based on the similarity of the two emotions, and then determines whether there is facial deep fake, while getting rid of the limitation of directly training facial deep fake samples, and improving the detection accuracy.

[0181] like Figure 5 As shown, the embodiment of the present invention also provides a face deep fake detection device 100 based on voice and expression matching, and the video face deep fake detection device 100 includes:

[0182] An acquisition module 101 is used to acquire a video file for training;

[0183] The first training module 102 is used to input the video file into the constructed CNN-15-GRU-B model, train the CNN-15-GRU-B model, and obtain a speech sentiment analysis model;

[0184] The second training module 103 is used to input the video file into the improved CNN model, train the improved CNN model, and obtain a facial expression emotion analysis model;

[0185] Extraction module 104, used for inputting the video file to be detected into the speech emotion analysis model and the facial expression emotion analysis model for feature extraction, respectively, to obtain speech emotion features and facial expression emotion features;

[0186] An analysis module 105 is used to construct an emotion similarity matrix through speech emotion features and facial expression emotion features, and analyze the face deep fake detection results according to the emotion similarity matrix;

[0187] The CNN-15-GRU-B model includes:

[0188] A first conversion module for converting a video file into a voice file, a second conversion module for converting a voice file into a text file, a voice feature extraction module, a BERT module, a splicing module, a GRU network module, and a first output module;

[0189] The first conversion module is the input end of the CNN-15-GRU-B model, the output end of the first conversion module is respectively connected to the input end of the second conversion module and the input end of the speech feature extraction module, the output end of the second conversion module is connected to the input end of the BERT module, the output end of the BERT module and the output end of the speech feature extraction module are both connected to the input end of the splicing module, the output end of the splicing module is connected to the input end of the GRU network module, the output end of the GRU network module is connected to the input end of the first output module, and the output end of the first output module is the output end of the CNN-15-GRU-B model;

[0190] The improved CNN model includes a face detection module for extracting faces, a feature extraction module, a feature integration module, and a second output module which are connected in sequence.

[0191] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0192] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0193] The embodiment of the present invention also provides a terminal device, such as Figure 6 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 6 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100. When the processor D100 executes the computer program D102, the above-mentioned method for detecting deep fake faces based on voice and expression matching is implemented.

[0194] The terminal device D10 may be a computing device such as a desktop computer, a notebook, a PDA, a server, a server cluster, a cloud server, etc. The terminal device may include, but is not limited to, a processor D100 and a memory D101. Those skilled in the art will appreciate that Figure 6 This is only an example of the terminal device D10 and does not constitute a limitation on the terminal device D10. The terminal device D10 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0195] The processor D100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0196] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device D10. Further, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data and other programs, such as the program code of the computer program, etc. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0197] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0198] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0199] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, a method for detecting deep fake faces based on voice and expression matching is implemented.

[0200] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the construction device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0201] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for detecting deep fake faces based on voice and expression matching, characterized in that: include: Step 1, obtain the video file for training; Step 2, inputting the video file into the constructed CNN-15-GRU-B model, training the CNN-15-GRU-B model, and obtaining a speech sentiment analysis model; Step 3, inputting the above video file into the improved CNN model, training the improved CNN model, and obtaining a facial expression emotion analysis model; Step 4, inputting the video file to be detected into the speech emotion analysis model and the facial expression emotion analysis model to perform feature extraction respectively, to obtain speech emotion features and facial expression emotion features; Step 5, constructing an emotion similarity matrix through the voice emotion features and the facial expression emotion features, and analyzing the face deep fake detection result according to the emotion similarity matrix; The CNN-15-GRU-B model includes: A first conversion module for converting a video file into a voice file, a second conversion module for converting a voice file into a text file, a voice feature extraction module, a BERT module, a splicing module, a GRU network module, and a first output module; The first conversion module is the input end of the CNN-15-GRU-B model, the output end of the first conversion module is respectively connected to the input end of the second conversion module and the input end of the speech feature extraction module, the output end of the second conversion module is connected to the input end of the BERT module, the output end of the BERT module and the output end of the speech feature extraction module are both connected to the input end of the splicing module, the output end of the splicing module is connected to the input end of the GRU network module, the output end of the GRU network module is connected to the input end of the first output module, and the output end of the first output module is the output end of the CNN-15-GRU-B model; The improved CNN model includes a face detection module for extracting faces, a feature extraction module, a feature integration module, and a second output module which are connected in sequence.

2. The method for detecting deep fake faces based on voice and expression matching according to claim 1, characterized in that: The CNN-15-GRU-B model is used to: Inputting the video file into the first conversion module for conversion to obtain a voice file; Inputting the voice file into the second conversion module for conversion to obtain a text file; Inputting the voice file into the voice feature extraction module to extract features and obtain a voice feature matrix; Input the text file into the BERT module to perform multi-head self-attention calculation to obtain a text feature matrix; Inputting the speech feature matrix and the text feature matrix into the splicing module to perform feature splicing to obtain joint features; Inputting the joint features into the GRU network module to capture dependencies and obtain classification features; The classification features are input into the first output module for classification to obtain a variety of speech emotion features.

3. The method for detecting deep fake faces based on voice and expression matching according to claim 2 is characterized in that: The speech feature extraction module comprises: The first convolutional layer, the first maximum pooling layer, the compound convolutional layer, and the second maximum pooling layer are connected in sequence.

4. The method for detecting deep fake faces based on voice and expression matching according to claim 3 is characterized in that: The composite convolutional layer comprises: 14 convolutional layers connected in sequence, wherein the input end of the first convolutional layer is connected to the output end of the first maximum pooling layer, and the output end of the last convolutional layer is connected to the input end of the second maximum pooling layer.

5. The method for detecting deep fake faces based on voice and expression matching according to claim 4 is characterized in that: The first output module includes a fully connected layer and a softmax activation function layer connected in sequence; The input end of the fully connected layer is connected to the output end of the GRU network module; The output end of the softmax activation function layer is the output end of the CNN-15-GRU-B model, which is used to output multiple speech emotion features.

6. The method for detecting deep fake faces based on voice and expression matching according to claim 5, characterized in that: The improved CNN model is used to: Input the video file into a face detection module to extract partial facial features, and obtain a partial facial feature map; Inputting the face part feature map into the feature extraction module to perform fine-grained feature extraction to obtain a face fine-grained feature map; Integrate the partial face feature map with the fine-grained face feature map to obtain a global face feature; The global facial features are input into the second output module for classification to obtain a variety of facial expression and emotion features.

7. The method for detecting deep fake faces based on voice and expression matching according to claim 6, characterized in that: Analyzing the face deep fake detection results according to the emotion similarity matrix, including: When the similarity between the speech emotion feature and the facial expression emotion feature in the emotion similarity matrix is ​​greater than a preset threshold, the emotions are considered to be consistent, and the face deep fake detection result is no fake phenomenon; When the similarity between the speech emotion features and the facial expression emotion features in the emotion similarity matrix is ​​less than a preset threshold, it is considered that the emotions are inconsistent and the face deep fake detection result is that there is a fake phenomenon.

8. A facial deep fake detection device based on voice and expression matching, characterized in that: include: An acquisition module, used to acquire video files for training; The first training module is used to input the video file into the constructed CNN-15-GRU-B model, train the CNN-15-GRU-B model, and obtain a speech sentiment analysis model; A second training module is used to input the video file into an improved CNN model, train the improved CNN model, and obtain a facial expression emotion analysis model; An extraction module is used to input the video file to be detected into the speech emotion analysis model and the facial expression emotion analysis model to respectively extract features to obtain speech emotion features and facial expression emotion features; An analysis module, configured to construct an emotion similarity matrix using the speech emotion features and the facial expression emotion features, and analyze the facial deep fake detection result according to the emotion similarity matrix; The CNN-15-GRU-B model includes: A first conversion module for converting a video file into a voice file, a second conversion module for converting a voice file into a text file, a voice feature extraction module, a BERT module, a splicing module, a GRU network module, and a first output module; The first conversion module is the input end of the CNN-15-GRU-B model, the output end of the first conversion module is respectively connected to the input end of the second conversion module and the input end of the speech feature extraction module, the output end of the second conversion module is connected to the input end of the BERT module, the output end of the BERT module and the output end of the speech feature extraction module are both connected to the input end of the splicing module, the output end of the splicing module is connected to the input end of the GRU network module, the output end of the GRU network module is connected to the input end of the first output module, and the output end of the first output module is the output end of the CNN-15-GRU-B model; The improved CNN model includes a face detection module for extracting faces, a feature extraction module, a feature integration module, and a second output module which are connected in sequence.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for detecting deep fake faces based on voice and expression matching as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for detecting deep fake faces based on voice and expression matching as described in any one of claims 1 to 7 is implemented.

Images