A method and system for detecting fake face video based on Mobilenet-GRU

Abstract

The application provides a kind of based on Mobilenet-GRU's fake face video detection method and system, it is related to fake face video detection technical field, specific steps include: by being handled to face image sequence block, obtain motion consistency feature, each frame image is decomposed into low frequency and multiple high frequency regions by wavelet transform, obtain the energy ratio of low frequency and high frequency region, each frame image in face image sequence is input into Mobilenet model, obtain spatial feature vector, set the initial proportion of frame image replacement, spatial feature vector is input into GRU model and carries out time series prediction, iteration optimization the proportion of frame image replacement, using pre-trained discriminant model, the judgment of being detected video is fake.This application sets the initial proportion of frame image replacement and iteration optimizes the proportion, so that spatial feature can dynamically adapt to the change of video content, thereby significantly enhance the capture ability and expression ability of model to video dynamic change feature.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fake face video detection, in particular to a fake face video detection method and system based on Mobilenet-GRU. BACKGROUND

[0002] In the current field of deep fake detection, with the rapid development of generative adversarial networks and other advanced image processing technologies, the quality and authenticity of fake videos are constantly improving, posing great challenges to traditional detection methods.

[0003] In the prior art, a face video deep fake detection method and device provided in CN117542124A includes the following steps: performing face region recognition and cropping on a target video to obtain a face region video containing only the face region; generating M frames of face region thumbnails corresponding to the face region video according to M groups of different consecutive N frames of face region video frames in the face region video, wherein each frame of the face region thumbnail includes consecutive N frames of face region video frames; inputting each frame of the face region thumbnail into a face video deep fake detection model to output a video fake probability corresponding to each frame of the face region thumbnail; and determining that the target video is a fake video if the average value of each video fake probability exceeds a preset first preset threshold.

[0004] However, there are still the following deficiencies. As can be seen from the above statement, the prior art mainly processes video frames by generating thumbnails. This method makes full use of the information of static images, but ignores the dynamic changes in videos. Dynamic features are the core of distinguishing video content, especially when facing fake and tampered operations. The motion consistency between video frames and the relative energy distribution of low-frequency and high-frequency components in images are particularly important for determining the authenticity of videos. Therefore, lack of in-depth analysis of dynamic features can lead to ineffective identification of fake videos.

[0005] The above information disclosed in the background section is only used to enhance the understanding of the background of the present disclosure, and therefore it can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The present application aims to provide a fake face video detection method and system based on Mobilenet-GRU to solve the problems raised in the background.

[0007] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0008] A fake face video detection method based on Mobilenet-GRU, the specific steps include:

[0009] The video to be detected is extracted to obtain an image frame sequence. Face detection and alignment are performed frame by frame in the image frame sequence to standardize the face region image of each frame and generate a face image sequence.

[0010] The face image sequence is divided into multiple non-overlapping blocks frame by frame with a set block size. The pixel change of corresponding blocks in adjacent frames is calculated, and the distribution data of pixel change of all corresponding blocks in adjacent frames is statistically analyzed as a consistency feature of motion between adjacent frames.

[0011] Wavelet transform decomposition is performed on each frame of the face image sequence. Discrete cosine transform is performed on each frequency region after decomposition to obtain the energy ratio of each frame. Wavelet convolution is performed on each frequency region and then the images are concatenated according to the channel dimension to obtain the convolutional frequency domain features of each frame.

[0012] Each frame of the face image sequence is input into the pre-trained MobileNet model to extract spatial feature vectors. Based on the set initial replacement ratio, the spatial feature vectors are input into the GRU model for time series prediction. The frame image replacement ratio is iteratively adjusted to optimize the feature sequence and obtain the optimized spatial features of each frame.

[0013] The consistency feature, energy ratio, and convolutional frequency domain feature are concatenated according to the timestamp to form the shallow feature vector of the video to be detected, and the optimized spatial feature is concatenated according to the timestamp to form the deep feature vector of the video to be detected.

[0014] The shallow physical feature vector and the deep feature vector are concatenated and then input into a pre-trained discrimination model to obtain a forgery probability score and determine the forgery of the video to be detected.

[0015] Furthermore, the steps of performing face detection and alignment frame by frame on the video frame sequence include:

[0016] The video to be detected is decoded into a continuous sequence of video frames. Frames are extracted at fixed frame intervals and sorted by timestamps to obtain an image frame sequence.

[0017] A face detection algorithm is applied to each frame of the image frame sequence to identify the face region and face feature points in the image, and the bounding box of the face region is marked. The face region image is obtained by cropping with the bounding box as the boundary. The face feature points include the coordinates of the eyes, nose and mouth.

[0018] Using the facial feature points in the face region image corresponding to the first image in the image frame sequence as the feature reference, an affine transformation operation is applied to the remaining face region images to align the specific facial points in the remaining face region images with the feature reference.

[0019] The resolution of all the aligned face region images is normalized, i.e., all the normalized face region images are integrated into a face image sequence according to the image frame sequence order.

[0020] Further, the face image sequence is divided into a plurality of non-overlapping blocks in a set block size frame by frame, and the specific logic is as follows:

[0021] The face boundary box is extracted by the face detection algorithm, and the intersection point of the diagonal lines of the boundary box is calculated as the face region center of each frame of face image, and each block represents a sub-block region.

[0022] The face region center is taken as a reference, and the image height and width direction is uniformly divided according to the set block size, to obtain a plurality of non-overlapping blocks.

[0023] According to the size of the face image and the size of each block, the maximum number of blocks in the height direction and the maximum number of blocks in the width direction of each frame of image are calculated, and the total number of blocks of each frame of image is equal to the product of the two.

[0024] The difference between the pixel numbers of the blocks corresponding to adjacent frames is calculated as the pixel change amount of the sub-block region.

[0025] The distribution data of the pixel change amount of the sub-block region, including the mean and variance, is calculated and taken as the consistency feature of the motion of adjacent frames.

[0026] Further, the wavelet transform decomposition is performed on each frame of image of the face image sequence, and the discrete cosine transform is performed on each frequency region after decomposition to obtain the energy ratio of each frame of image, and the specific logic is as follows:

[0027] The low-pass filter and the high-pass filter are used to perform convolution operation on each row of pixels of the image, and the image is divided into two parts along the horizontal direction, i.e., the horizontal low-frequency component and the horizontal high-frequency component.

[0028] The obtained horizontal low-frequency component and horizontal high-frequency component are respectively convolved along the vertical direction with the low-pass filter and the high-pass filter again to obtain the low-frequency region, the first high-frequency region, the second high-frequency region and the third high-frequency region.

[0029] The discrete cosine transform is performed on the low-frequency region, the first high-frequency region, the second high-frequency region and the third high-frequency region respectively to convert the spatial domain information of each region into frequency domain representation to obtain the corresponding frequency coefficient matrix.

[0030] For each frequency coefficient matrix, the low-frequency coefficient matrix and the high-frequency coefficient matrix in the matrix are divided according to the preset frequency threshold.

[0031] Calculate the low-frequency energy and the high-frequency energy of each region respectively: the low-frequency energy is equal to the sum of squares of all elements in the low-frequency coefficient matrix of the corresponding region; the high-frequency energy is equal to the sum of squares of all elements in the high-frequency coefficient matrix of the corresponding region;

[0032] Calculate the ratio of the sum of low-frequency energies of the four regions and the sum of high-frequency energies of the four regions of each frame of image, which is the frequency energy ratio of the frame of image.

[0033] Further, after wavelet convolution of each frequency region, the convolution frequency domain features of each frame of image are obtained by splicing in the channel dimension, and the specific logic is as follows:

[0034] For the low-frequency region, the first high-frequency region, the second high-frequency region and the third high-frequency region decomposed by wavelet transform, a special wavelet convolution kernel suitable for its frequency characteristics is configured respectively;

[0035] The convolution operation is performed on the low-frequency region, the first high-frequency region, the second high-frequency region and the third high-frequency region separately, and the weighted sum of the fixed step sliding convolution kernel and the region pixels is calculated to output the convolution feature map consistent with the input spatial dimension;

[0036] The convolution feature maps of the low-frequency region, the first high-frequency region, the second high-frequency region and the third high-frequency region are spliced in the channel dimension to obtain the convolution frequency domain features of each frame of image.

[0037] Further, the spatial feature vector actual value of each frame of image in the face image sequence is input into the pre-trained Mobilenet model to extract, and the spatial feature includes the nose tip coordinate, the vertical distance from the nose tip to the upper lip, and the horizontal distance between the left and right eye corners. The spatial feature vector actual value of each frame of image is sequentially composed into a feature vector sequence;

[0038] The number of frame images to be replaced is obtained by multiplying the initial replacement ratio and the number of frame images in the video to be detected and then taking the integer part.

[0039] According to the initial replacement ratio, a certain number of frame images are randomly selected from the input sequence, and the corresponding spatial feature vector actual value is replaced by the spatial feature vector predicted value predicted by the GRU model to form a new spatial feature vector. The new spatial feature vector is composed of the predicted value after replacement and the actual value without replacement;

[0040] The new spatial feature vector is input into the GRU model again to calculate the mean square error between the predicted value and the actual value.

[0041] Calculate the change of the mean square error between the current and the previous iteration;

[0042] If the change of the mean square error between the current and the previous iteration is not less than a certain threshold, the replacement ratio is adjusted, and the specific adjustment is as follows:

[0043] If the current mean square error is not greater than the previous mean square error, increase the replacement ratio, and recalculate the frame image number;

[0044] If the current mean square error is greater than the previous mean square error, decrease the replacement ratio, and recalculate the frame image number;

[0045] If the current and previous mean square error change is less than the set threshold, stop iteration, obtain the final frame image replacement ratio and replacement number, use the final frame image replacement ratio and replacement number to replace the actual value of the spatial feature vector, generate the final optimized spatial feature vector, and use the optimized spatial feature vector as the depth feature vector.

[0046] Further, the to-be-detected video is subjected to forgery determination, and the specific logic is as follows:

[0047] If the forgery probability score is less than the preset threshold, the to-be-detected video is determined to be a real video, otherwise, the to-be-detected video is determined to be a forged video.

[0048] To achieve the above purpose, the present application also provides the following technical scheme:

[0049] A forgery face video detection system based on Mobilenet-GRU, the system is used for executing any one of the above-mentioned forgery face video detection methods based on Mobilenet-GRU, comprising:

[0050] A face detection module is used for extracting frames from the to-be-detected video to obtain an image frame sequence, performing face detection and alignment on the image frame sequence frame by frame, standardizing the face region image of each frame image, and generating a face image sequence;

[0051] An image blocking module is used for dividing each frame of the face image sequence into a plurality of non-overlapping blocks with a set block size, calculating the pixel change amount of the corresponding blocks of adjacent frames, and counting the distribution data of the pixel change amount of all corresponding blocks of adjacent frames as the consistency feature of the motion of adjacent frames;

[0052] A feature extraction module is used for wavelet transform decomposition of each frame image in the face image sequence, discrete cosine transform of each frequency region after decomposition to obtain the energy ratio of each frame image, and convolution of each frequency region in the channel dimension to obtain the convolution frequency domain feature of each frame image;

[0053] An iterative optimization module is used for inputting each frame image in the face image sequence into a pre-trained Mobilenet model to extract a spatial feature vector, inputting the spatial feature vector into a GRU model based on a set initial replacement ratio for time series prediction, iteratively adjusting the frame image replacement ratio to optimize the feature sequence, and obtaining the optimized spatial feature of each frame image.

[0054] The splicing module is used for splicing the consistency features, the energy ratios and the convolution frequency domain features according to timestamps respectively, and constructing shallow feature vectors of the video to be detected, and splicing the optimized spatial features according to timestamps to construct deep feature vectors of the video to be detected;

[0055] The determination module is used for inputting the spliced shallow physical feature vectors and deep feature vectors into a pre-trained discrimination model for discrimination, so as to obtain a fake probability score and determine whether the video to be detected is fake.

[0056] Compared with the prior art, the present application has the following advantages:

[0057] The present application detects and aligns the video frame sequence, extracts shallow physical features by combining inter-frame motion consistency, energy ratio and convolution frequency domain features, and introduces a spatial deep feature extraction mechanism based on a pre-trained MobileNet and GRU model, so as to effectively fuse multi-modal features.

[0058] The dynamic optimization process not only improves the accuracy of the deep features, but also effectively preserves the key timing information and avoids the loss of key information caused by improper feature replacement. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 The figure is a schematic diagram of the overall method of the present application.

[0060] Figure 2 The figure is a block diagram of the module of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with specific examples.

[0062] It should be noted that the technical terms or scientific terms used in the present application should be understood as the general meaning understood by those skilled in the art to which the present application belongs, unless otherwise defined. The "first", "second" and similar words used in the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Include" or "contain" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connected" or "connected" and similar words are not limited to physical or mechanical connection, but can include electrical connection, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to represent the relative positional relationship, when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0063] Embodiments:

[0064] Please refer to Figure 1 The present application provides a technical solution:

[0065] A fake face video detection method based on Mobilenet-GRU, the specific steps comprising:

[0066] S1. Extracting frames from the video to be detected to obtain an image frame sequence, performing face detection and alignment on each frame of the image frame sequence to standardize the face region image of each frame of image, and generating a face image sequence;

[0067] On the basis of the above embodiments, the step of performing face detection and alignment on each frame of the video frame sequence comprises:

[0068] Decoding the video to be detected into a continuous video frame sequence, extracting frames according to a fixed frame interval and sorting according to the time stamp to obtain an image frame sequence;

[0069] Applying a face detection algorithm to each frame of image in the image frame sequence, identifying the face region and face feature points appearing in the image, and marking the boundary box of the face region, and cutting the face region image within the boundary box to obtain the face region image, the face feature points include the coordinates of the eyes, nose and mouth;

[0070] Taking the face feature points in the face region image corresponding to the first image in the image frame sequence as a feature reference, performing affine transformation operation on the remaining face region images to align the face feature points in the remaining face region images with the feature reference respectively;

[0071] Standardizing the resolution of all aligned face region images, that is, adjusting them to a standard fixed resolution, and integrating all standardized face region images into a face image sequence according to the image frame sequence order.

[0072] S2. Divide the face image sequence frame by frame into multiple non-overlapping blocks with a set block size, calculate the pixel change of corresponding blocks in adjacent frames, and statistically analyze the distribution data of pixel change of all corresponding blocks in adjacent frames as a consistency feature of motion between adjacent frames.

[0073] Based on the above embodiments, the face image sequence is divided frame by frame into multiple non-overlapping blocks with a set block size. The specific logic is as follows:

[0074] The face bounding box is extracted using a face detection algorithm, and the intersection of the diagonals of the bounding box is calculated as the center of the face region in each frame of the face image.

[0075] Using the center of the face region as a reference, each frame of the face image is divided into blocks according to a set size. Divide into blocks, where, The width of the block. The height of the block;

[0076] The size of the face image is ,in, The width of the face image. If the height of the face image is taken as the base, it is divided into multiple non-overlapping blocks, each representing a segmented region, and the size of each block is [missing information]. Calculate the maximum number of blocks in the height direction of each frame image. Maximum number of blocks in the width direction The formula used is as follows:

[0077] ;

[0078] in, Indicates rounding down;

[0079] Number of blocks for:

[0080] ;

[0081] The formula used to calculate the pixel change in a block of image between adjacent frames is as follows:

[0082] ;

[0083] in, For the first The first frame image The amount of pixel change in each block, For from the first The first frame image extracted from the first frame image Number of pixels per block For from the first The first frame image extracted from the first frame image Number of pixels per block For the Euclidean norm, For the index of the frame image, The number of frames in the video to be detected. For the block index, .

[0084] Based on the above embodiments, the distribution data of pixel changes in the segmented regions are statistically analyzed, including the mean and variance, using the following formula:

[0085] The set of pixel changes for all blocks in each frame of the image is ;

[0086] ;

[0087] in, For the first The mean of pixel changes across all blocks in a frame of an image. For the first The variance of pixel variation across all blocks in a frame image;

[0088] The mean and variance of the pixel changes across all blocks in each frame are used as the inter-frame motion consistency feature for that frame. The inter-frame motion consistency feature vector is: .

[0089] In analyzing temporal images such as facial video sequences, inter-frame motion consistency is an important characteristic used to characterize whether the motion (change) between adjacent frames is natural, coherent, and conforms to real physical laws.

[0090] In real videos, head or facial expression changes are usually holistic and continuous. After segmentation, regardless of the region, the overall motion trend is consistent, and the mean and variance of pixel changes are within a reasonable range. In synthetic or fake videos, splicing boundaries and composite regions often cause abrupt motion changes, resulting in inconsistencies between local and overall changes, thus causing abnormal mean or variance. Therefore, using the mean and variance of pixel changes as inter-frame motion consistency features can effectively capture the naturalness and smoothness of overall and local motion, quickly exposing unnatural inter-frame motion changes such as forgery, synthesis, and tampering. It is a very effective and easily implemented indicator among shallow physical features.

[0091] S3. Perform wavelet transform decomposition on each frame of the face image sequence, perform discrete cosine transform on each frequency region after decomposition to obtain the energy ratio of each frame, and then concatenate each frequency region by channel dimension to obtain the convolutional frequency domain features of each frame.

[0092] Among them, the wavelet is usually selected from the db series wavelet (such as db4, db6).

[0093] db series wavelet: is an orthogonal wavelet, with good time domain localization and frequency domain resolution characteristics, can accurately capture the edge of high frequency area, texture mutation (such as the difference of interframe details of forged video). Among them, db4, db6 is the preferred model of forged face detection, which balances the calculation efficiency and detail extraction accuracy, and is highly compatible with the logic of "dividing high frequency area according to vertical, horizontal and diagonal lines".

[0094] On the basis of the above embodiment, the wavelet transform decomposition is performed on each frame of the face image sequence, and the discrete cosine transform is performed on each frequency area after decomposition to obtain the energy ratio of each frame of image. The specific logic is as follows:

[0095] Convolve each row of pixels of the image with a low-pass filter and a high-pass filter, and split the image into horizontal low-frequency components and horizontal high-frequency components along the horizontal direction;

[0096] The obtained horizontal low-frequency component and horizontal high-frequency component are respectively convolved with a low-pass filter and a high-pass filter along the vertical direction to obtain a low-frequency region, a first high-frequency region, a second high-frequency region and a third high-frequency region;

[0097] Among them, the low-frequency region concentrates the overall outline and low-frequency information of the image, and the pixel value in the region changes gently, which is the embodiment of the core structure of the image;

[0098] The first high-frequency region is specially designed to capture vertical edge details, such as the vertical edge of the nose bridge of the face, the vertical outline of the eye socket, and the vertical change of the upper and lower edges of the lips;

[0099] The second high-frequency region focuses on preserving horizontal edge details, such as the horizontal outline of the eyebrows, the horizontal transition line between the forehead and the cheek, and the horizontal change of the left and right edges of the lips;

[0100] The third high-frequency region mainly carries diagonal details, image noise and fine texture, and is the key area for capturing subtle features of the image;

[0101] Discrete cosine transform is performed on the low-frequency region, the first high-frequency region, the second high-frequency region and the third high-frequency region, respectively, to convert the spatial domain pixel information of each region into a frequency coefficient matrix in the frequency domain, denoted as:

[0102] The frequency coefficient matrix corresponding to the low-frequency region;

[0103] The frequency coefficient matrix corresponding to the first high-frequency region;

[0104] The frequency coefficient matrix corresponding to the second high-frequency region;

[0105] The frequency coefficient matrix corresponding to the third high-frequency region;

[0106] preset frequency threshold The four frequency coefficient matrices are divided into four frequency regions respectively:

[0107] The low frequency coefficient matrix takes the upper left corner of each frequency coefficient matrix Region, the low frequency component in which is concentrated in energy;

[0108] The high frequency coefficient matrix takes the remaining region of each frequency coefficient matrix, that is, the elements starting from the first row or the first column, in which the high frequency component is detailed;

[0109] For each frequency region, the energy is calculated based on its low frequency coefficient matrix and high frequency coefficient matrix: the low frequency energy is equal to the sum of squares of all elements in the low frequency coefficient matrix of the corresponding region, and the high frequency energy is equal to the sum of squares of all elements in the high frequency coefficient matrix of the corresponding region;

[0110] The low frequency energy of the low frequency region is calculated as follows:

[0111] ;

[0112] Wherein, is the low frequency energy of the low frequency region of the first frame image, is the coefficient of the first row and the first column in the low frequency coefficient matrix of the first frame image, , are the row index and the column index respectively, ;

[0113] According to the same calculation method, the low frequency energy of the first high frequency region, the second high frequency region and the third high frequency region is obtained;

[0114] The high frequency energy of the low frequency region is calculated as follows:

[0115] ;

[0116] Wherein, is the high frequency energy of the low frequency region of the first frame image, is the coefficient of the first row and the first column in the high frequency coefficient matrix of the first frame image, , are the total number of rows and the total number of columns of the high frequency coefficient matrix respectively;

[0117] According to the same calculation method, the high frequency energy of the first high frequency region, the second high frequency region and the third high frequency region is obtained;

[0118] The low frequency energy of the low frequency region, the first high frequency region, the second high frequency region and the third high frequency region is added to obtain the total low frequency energy;

[0119] The high frequency energy of the low frequency region, the first high frequency region, the second high frequency region and the third high frequency region is added to obtain the total high frequency energy;

[0120] The ratio of the total low frequency energy to the total high frequency energy is taken as the frequency energy ratio .

[0121] Wherein, the frequency threshold Generally determined according to experience and experiment, in face image forgery detection, when the image region resolution is 128x128, Commonly taken 20-25.

[0122] On the basis of the above, it needs to be explained that:

[0123] Wavelet transform has good time-frequency localization ability, which can capture different frequency components of the signal more carefully, especially in video frames, which can effectively capture the detailed information of the image.

[0124] Wavelet transform can provide more detailed local frequency domain analysis, which is particularly effective for frequently changing forged videos, especially in face video forgery detection, which often involves changes in local details.

[0125] Discrete cosine transform is a transform that converts a finite-length discrete signal into cosine basis function coefficients. Its core idea is to use a set of cosine wave basis functions to decompose the signal into low and high frequency components, obtaining the weight of each frequency component, i.e. frequency coefficient, so as to express the signal as the superposition of different frequency cosine functions.

[0126] Wherein, after multiple processing (synthesis, compression, fusion) of the forged video, the high frequency information is often lost or abnormal, which is manifested as a decrease in high frequency energy; the low frequency component is often complete, as it mainly reflects the overall structure and is easy to be preserved or reproduced.

[0127] Therefore, the frequency energy ratio of the forged video is generally high, i.e. the proportion of low frequency energy to high frequency energy increases.

[0128] The purpose of calculating the frequency energy ratio is to measure the relative energy distribution of low and high frequency components in each image frame, revealing the integrity and authenticity of image details, thereby assisting in identifying the common high frequency detail loss or abnormality in forged videos. As a shallow physical feature, the frequency energy ratio has strong discriminant ability in actual forgery detection tasks.

[0129] On the basis of the above embodiment, the features extracted by wavelet convolution are spliced into convolution frequency domain features for all frequency regions, and the specific logic is as follows:

[0130] For the low-frequency region and the three high-frequency regions (the first high-frequency region, the second high-frequency region, and the third high-frequency region) decomposed by wavelet transform, a dedicated wavelet convolution kernel matching the frequency characteristics thereof is configured respectively:

[0131] A wavelet convolution kernel with low-pass characteristics is used, and the convolution kernel size is set to , which is used to strengthen the extraction of smooth transition features of the overall outline;

[0132] The first high-frequency region: a wavelet convolution kernel sensitive to the vertical direction is used, and the convolution kernel size is set to , which enhances the response to vertical edge details;

[0133] The second high-frequency region: a wavelet convolution kernel sensitive to the horizontal direction is used, and the convolution kernel size is set to , which enhances the response to horizontal edge details;

[0134] The third high-frequency region: a wavelet convolution kernel sensitive to the diagonal direction is used, and the convolution kernel size is set to , which enhances the response to diagonal textures and fine details.

[0135] Wherein, , , , is the convolution kernel size parameter, which is adaptively adjusted according to the image resolution, such as being uniformly set to 3x3 or 5x5;

[0136] Wavelet convolution operation is performed separately for each frequency region:

[0137] Let the low-frequency region of the i-th frame be , after convolution with the corresponding wavelet convolution kernel , the output low-frequency feature map is obtained, and the formula is: ;

[0138] Similarly, the first high-frequency region is convolved with the convolution kernel , and the output vertical high-frequency feature map is obtained, and the formula is: ;

[0139] The second high-frequency region is convolved with the convolution kernel , and the output horizontal high-frequency feature map is obtained, and the formula is: ;​

[0140] Third high-frequency region Convolution kernel After convolution, output the diagonal high-frequency feature map , the formula is: ;

[0141] wherein, represents a convolution operation;

[0142] The weighted sum of the pixels in the region and the convolution kernel is calculated by sliding window, and the feature map consistent with the spatial dimension of the input region is output, and the specific process is as follows:

[0143] First, determine the frequency region after wavelet decomposition and match the characteristics of the convolution kernel (such as 3x3, 5x5). In order to avoid exceeding the image range when the window slides to the edge, fill in zero according to the rule of "convolution kernel size minus 1 and then divided by 2", for example, the edge length of 3x3 convolution kernel is 3, fill in 1 layer of zero (add 1 circle of zero around the input region), and the 5x5 convolution kernel fills in 2 layers of zero. The input region filled in this way can make the window completely cover the original region; then let the sliding window with the same size as the convolution kernel start from the top left corner of the filled input region, slide in the horizontal and vertical directions with a step of 1, and at each position, multiply the weights of the pixels in the window with the corresponding positions of the convolution kernel, then add all the products to get the value of the corresponding position of the output feature map; after the window traverses the entire input region, all the calculation results will form a feature map with the same spatial size as the original input region;

[0144] The convolution feature maps of the four regions are spliced in the channel dimension to form the first Convolution frequency domain feature of frame :

[0145] ;

[0146] wherein, represents a channel splicing operation.

[0147] On the basis of the above, it should be noted that:

[0148] Through wavelet convolution, information of different frequencies can be processed at the same time, the perception ability of the model for different scale features is enhanced, and the detail recovery and global feature fusion in dynamic video analysis are improved.

[0149] S4. Input each frame of image in the face image sequence into the pre-trained Mobilenet model to extract a spatial feature vector, input the spatial feature vector into the GRU model based on the set initial replacement ratio for time sequence prediction, iteratively adjust the frame image replacement ratio to optimize the feature sequence, and obtain the optimized spatial feature of each frame of image;

[0150] Wherein, the Mobilenet model is used to extract the spatial feature vector of the input face image, and the GRU model is used to make time series prediction on the input spatial feature sequence.

[0151] On the basis of the above embodiment, each frame image in the face image sequence is input into the pre-trained Mobilenet model to extract the spatial feature vector actual value, and the spatial feature includes the nose tip coordinate, the vertical distance from the nose tip to the upper lip, and the horizontal distance between the left and right eye corners.

[0152] The initial replacement ratio of the frame image is set as , wherein The first frame image in the sequence is input into the GRU model to predict the spatial feature vector prediction value of the frame image.

[0153] According to the current replacement ratio , the number of frame images to be replaced is calculated, and the formula is as follows:

[0154] ;

[0155] Wherein, is the number of frame images to be replaced, is the rounding up operation;

[0156] According to the current replacement ratio , the frame image is randomly selected from the input sequence , the corresponding spatial feature vector actual value is replaced by the spatial feature vector prediction value predicted by the GRU model, a new spatial feature vector is formed, and the new spatial feature vector is composed of the replaced prediction value and the un-replaced actual value.

[0157] The new spatial feature vector is input into the GRU model again to calculate the mean square error between the prediction value and the actual value.

[0158] The change amount of the mean square error between the current and the previous iteration is calculated.

[0159] If the change amount of the mean square error between the current and the previous iteration is not less than the set threshold, the replacement ratio is adjusted, and the details are as follows:

[0160] If the current mean square error is not greater than the previous mean square error, the replacement ratio is increased, and the number of frame images is recalculated.

[0161] If the current mean square error is greater than the previous mean square error, the replacement ratio is reduced, and the number of frame images is recalculated.

[0162] If the current and previous mean square error change is less than the set threshold, the iteration is stopped, and the final frame image replacement ratio and replacement number are obtained, and the final frame image replacement ratio and replacement number are used for replacement.

[0163] On the basis of the above, it needs to be explained that:

[0164] In the iteration optimization of the frame image replacement ratio, the initial ratio of the frame image replacement is only a hypothetical value, which needs to be gradually adjusted to the optimal state through iteration. The replacement of part of the spatial feature vector is a predicted value, and through repeated iteration optimization, the frame image replacement ratio that can optimize the mean square error can be gradually found. This process not only optimizes the spatial feature sequence, but also improves the model performance.

[0165] By replacing part of the spatial feature vector actual value with the predicted value, the new spatial feature vector contains both the original spatial information and the predicted information generated based on the time sequence dependence. This fusion can improve the expression ability of the spatial feature vector in a dynamic scene. By replacing a certain proportion of the spatial feature vector actual value with the predicted value, a kind of weighted fusion is essentially performed between the spatial feature vector actual value and the spatial feature vector predicted value. This weighted fusion helps to combine the diversity of the spatial feature vector and the potential of the time sequence prediction model, so as to better capture the characteristics of the data. By introducing the predicted value, the model can capture the potential trends and change rules in the time sequence dynamics, improve the capture ability of the data characteristics in a complex environment, and enhance the generalization ability and prediction accuracy of the model. At the same time, by comparing and analyzing the predicted value in the predicted spatial feature vector with the actual observation value, the prediction performance of the GRU model can be effectively detected and evaluated.

[0166] On the basis of the above embodiment, a random selection frame image, after replacing the corresponding spatial feature vector actual value with the spatial feature vector predicted value predicted by the GRU model, the new spatial feature vector includes the reserved spatial feature vector actual value (the part that is not replaced) and the predicted value (the replaced part) generated by the GRU model;

[0167] In the new spatial feature vector, the order of the spatial feature vector remains consistent with the original sequence, that is, the feature vector corresponding to the frame image is still the spatial feature vector of the frame image. However, for some randomly selected frame images, the original actual value is replaced with the predicted value generated by the GRU model, but these replacements occur in the original position and do not change the overall sequence order.

[0168] The final optimized spatial feature vector of each frame image is denoted as , ,in, For the first The first frame image Each optimization space feature These correspond to the coordinates of the tip of the nose, the vertical distance from the tip of the nose to the upper lip, and the horizontal distances of the left and right corners of the eyes, respectively.

[0169] S5. The consistency feature, energy ratio and convolutional frequency domain feature are concatenated according to the timestamp to form the shallow feature vector of the video to be detected, and the optimized spatial feature is concatenated according to the timestamp to form the deep feature vector of the video to be detected.

[0170] Based on the above embodiments, the consistency feature, energy ratio, and convolutional frequency domain feature are concatenated according to the timestamp to form a shallow physical feature vector of the video to be detected:

[0171] ;

[0172] in, For the video to be tested Shallow physical feature vectors of each frame image.

[0173] Shallow physical features refer to objective features extracted from the basic physical features of an image that do not rely on complex semantic understanding; motion consistency and frequency-energy ratio both belong to this category.

[0174] Among them, consistency features and energy ratio reveal the physical authenticity of video frames from the two dimensions of temporal dynamics and spatial frequency, respectively. By splicing these two types of features, a more comprehensive shallow physical feature vector can be obtained.

[0175] Based on the above, it should be noted that:

[0176] By combining the consistency characteristics of dynamic changes, the energy ratio reflecting the energy of low-frequency and high-frequency regions, and the convolutional frequency domain characteristics of multiple frequency bands, a rich shallow physical feature vector is generated.

[0177] Based on the above embodiments, the optimized spatial features are concatenated according to timestamps to form the depth feature vector of the video to be detected:

[0178] ;

[0179] in, is the depth feature vector of the video to be detected.

[0180] S6. After concatenating the shallow physical feature vector and the deep feature vector, input the concatenated data into the pre-trained discriminant model for discrimination to obtain a forgery probability score and determine the forgery of the video to be detected.

[0181] On the basis of the above embodiment, the discriminant model is constituted by a deep learning network based on a multilayer perceptron, the deep neural network of the multilayer perceptron includes an input layer, a first hidden layer, a second hidden layer, a third hidden layer and an output layer, the first hidden layer, the second hidden layer and the third hidden layer each have at least two neurons, and each uses ReLU (Rectified Linear Unit) as an activation function.

[0182] The spliced vector of the shallow physical feature vector and the deep feature vector of the detected video, and the fake probability score are divided into a training set, a validation set and a test set, and a common division ratio is 7:2:1, wherein the training set is used for learning of model parameters; the validation set is used for adjusting hyperparameters in the training process to prevent overfitting; and the test set is used for evaluating the generalization ability of the model after training is completed.

[0183] In the machine learning discriminant model, the input features of the deep learning network of the multilayer perceptron include real and fake video samples.

[0184] The structure of the deep learning network of the multilayer perceptron is as follows:

[0185] The input layer is used for receiving the spliced vector of the shallow physical feature vector and the deep feature vector.

[0186] The first hidden layer has 128 neurons and uses ReLU as an activation function.

[0187] The second hidden layer has 64 neurons and also uses ReLU as an activation function.

[0188] The third hidden layer has 32 neurons and uses ReLU as an activation function.

[0189] The output layer has 1 neuron and is used for outputting the fake probability score.

[0190] The process of training the discriminant model is as follows:

[0191] The discriminant model is trained by taking the spliced vector of the shallow physical feature vector and the deep feature vector of the detected video as input and taking the corresponding fake probability score as label output, using mean square error as a loss function, and stopping training of the discriminant model when the mean square error is in the range of 0-0.1.

[0192] During training of the detected video and discriminant of the to-be-detected video, the input shallow physical feature vector and deep feature vector have consistency: both are sequence-level vectors spliced in the order of timestamps (i.e., the overall vector obtained by integrating the features of video frame sequences in the time dimension), and it is not required that the shallow physical feature vector and the deep feature vector of each frame of image are aligned.

[0193] ​The judgment logic of the video based on the forgery probability score and the set threshold value is as follows:

[0194] If the forgery probability score is less than the preset threshold value, the video to be detected is determined as a real video; otherwise, the video to be detected is determined as a forged video.

[0195] The mean and variance of the forgery probability scores of the forged videos are calculated respectively, and the threshold value is set using the mean and variance, and the specific setting method of the preset threshold value is as follows:

[0196] The spliced vectors of the shallow physical feature vectors and the deep feature vectors of a plurality of groups of detected videos are collected as training samples, the set of forgery probability scores thereof is calculated, the mean and variance thereof are obtained, the threshold value is set as the mean minus the product of the adjustment coefficient and the variance using the statistical characteristics of the forgery probability scores of the forged videos, and the adjustment coefficient is usually set as 1 or 2.

[0197] Referring to Figure 2 , the application also provides a technical solution:

[0198] A forged face video detection system based on Mobilenet-GRU, which is used to execute any one of the above-mentioned forged face video detection methods based on Mobilenet-GRU, comprising:

[0199] A face detection module is used to extract frames from the video to be detected to obtain an image frame sequence, and perform face detection and alignment on the image frame sequence frame by frame to standardize the face region image of each frame of image and generate a face image sequence.

[0200] An image blocking module is used to divide each frame of the face image sequence into a plurality of non-overlapping blocks with a set block size, calculate the pixel change amount of the corresponding blocks of adjacent frames, and statistically obtain the distribution data of the pixel change amount of all corresponding blocks of adjacent frames as the consistency feature of the motion of adjacent frames.

[0201] A feature extraction module is used to perform wavelet transform decomposition on each frame of image in the face image sequence, perform discrete cosine transform on each frequency region after decomposition to obtain the energy ratio of each frame of image, and perform wavelet convolution on each frequency region and then splice according to the channel dimension to obtain the convolution frequency domain feature of each frame of image.

[0202] An iterative optimization module is used to input each frame of image in the face image sequence into a pre-trained Mobilenet model to extract a spatial feature vector, input the spatial feature vector into a GRU model based on a set initial replacement ratio to perform time sequence prediction, iteratively adjust the frame image replacement ratio to optimize the feature sequence, and obtain the optimized spatial feature of each frame of image.

[0203] The splicing module is configured to splice the consistency features, the energy ratio and the convolution frequency domain features according to timestamps respectively, to form a shallow feature vector of the video to be detected, and splice the optimized spatial features according to timestamps to form a deep feature vector of the video to be detected.

[0204] The determination module is configured to input the spliced shallow physical feature vector and deep feature vector into a pre-trained discrimination model for discrimination, to obtain a forgery probability score and perform forgery determination on the video to be detected.

[0205] The above formulas are all dimensionless numerical calculations, and the formulas are obtained by software simulation of a large amount of data to obtain a formula of the nearest real situation, and the preset parameters in the formula are set by the person skilled in the art according to the actual situation.

[0206] The above embodiments can be realized wholly or partially by software, hardware, firmware or any combination thereof. When realized by software, the above embodiments can be realized wholly or partially in the form of a computer program product. Those skilled in the art can realize that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are realized by hardware or software methods depends on the specific application and design constraints of the technical solutions.

[0207] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, which can be located in one place or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0208] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for detecting fake face video based on Mobilenet-GRU, characterized in that, The specific steps include: extracting frames from the video to be detected to obtain an image frame sequence, performing face detection and alignment on the image frame sequence frame by frame to standardize the face region image of each frame of image and generate a face image sequence; Divide each frame of the face image sequence into a plurality of non-overlapping blocks with a set block size, calculate the pixel change amount of the corresponding blocks of adjacent frames, and count the distribution data of the pixel change amount of all corresponding blocks of adjacent frames as the consistency feature of adjacent frames; Perform wavelet transform decomposition on each frame of the face image sequence, perform discrete cosine transform on each frequency region after decomposition to obtain the energy ratio of each frame of image, and concatenate the wavelet convolution of each frequency region by channel dimension to obtain the convolution frequency domain feature of each frame of image; Input each frame of image in the face image sequence into a pre-trained Mobilenet model to extract a spatial feature vector, input the spatial feature vector into a GRU model based on a set initial replacement ratio for time series prediction, iteratively adjust the frame image replacement ratio to optimize the feature sequence, and obtain the optimized spatial feature of each frame of image; Concatenate the consistency feature, energy ratio and convolution frequency domain feature according to the timestamp to form a shallow feature vector of the video to be detected, and concatenate the optimized spatial feature according to the timestamp to form a deep feature vector of the video to be detected; Concatenate the shallow physical feature vector and the deep feature vector, input the pre-trained discriminant model for discrimination to obtain a forgery probability score and perform forgery judgment on the video to be detected.

2. The method of claim 1, wherein the method is based on a MobileNet-GRU. The step of performing face detection and alignment on the video frame sequence frame by frame includes: decode the video to be detected into a continuous video frame sequence, extract frames at a fixed frame interval, and sort them according to the timestamp to obtain an image frame sequence; apply a face detection algorithm to each frame of image in the image frame sequence, identify the face region and face feature points appearing in the image, and mark the boundary box of the face region, and cut the face region image within the boundary box as a boundary; use the face feature points in the first face region image in the image frame sequence as a feature reference, and use affine transformation operation on the remaining face region images to align the face feature points in the remaining face region images with the feature reference respectively; standardize the resolution of all aligned face region images, i.e., adjust them to a standard fixed resolution, and integrate all standardized face region images into a face image sequence according to the image frame sequence order.

3. The method of claim 1, wherein the method is based on a MobileNet-GRU. Divide each frame of the face image sequence into a plurality of non-overlapping blocks with a set block size, the specific logic is as follows: extract the face boundary box through the face detection algorithm, calculate the intersection point of the diagonal line of the boundary box as the face region center of each frame of face image; Divide the image height and width direction uniformly according to the set block size with the face region center as the reference to obtain a plurality of non-overlapping blocks, each block representing a sub-block region; According to the size of the face image and the size of each block, calculate the maximum number of blocks in the height direction and the maximum number of blocks in the width direction of each frame of image, and the total number of blocks of each frame of image is equal to the product of the two; Calculate the difference between the pixel numbers of the corresponding position blocks of adjacent frames as the pixel change amount of the sub-block region; Statistical distribution data of the pixel change amount of the sub-block region, including mean and variance, as the consistency feature of the adjacent frames.

4. The method of claim 1, wherein the method is based on a MobileNet-GRU. Wavelet transform is performed on each frame of the face image sequence, and discrete cosine transform is performed on each frequency region after decomposition to obtain the energy ratio of each frame of image, and the specific logic is as follows: Convolution operation is performed on each row of pixels of the image using a low-pass filter and a high-pass filter, and the image is divided into horizontal low-frequency components and horizontal high-frequency components along the horizontal direction; The obtained horizontal low-frequency components and horizontal high-frequency components are respectively convolved along the vertical direction using a low-pass filter and a high-pass filter again to obtain low-frequency regions, first high-frequency regions, second high-frequency regions and third high-frequency regions; Discrete cosine transform is performed on the low-frequency regions, the first high-frequency regions, the second high-frequency regions and the third high-frequency regions respectively to convert the spatial domain information of each region into frequency domain representation to obtain the corresponding frequency coefficient matrix; For each frequency coefficient matrix, a low-frequency coefficient matrix and a high-frequency coefficient matrix within the matrix are divided according to a preset frequency threshold; The low-frequency energy and the high-frequency energy of each region are calculated respectively: wherein the low-frequency energy is equal to the sum of the squares of all elements in the corresponding low-frequency coefficient matrix; the high-frequency energy is equal to the sum of the squares of all elements in the corresponding high-frequency coefficient matrix; The ratio of the sum of the low-frequency energies of the four regions to the sum of the high-frequency energies of the four regions of each frame of image is calculated, which is the frequency energy ratio of the frame of image.

5. The method of claim 4, wherein the method is based on a MobileNet-GRU. After wavelet convolution, the frequency regions are spliced according to the channel dimension to obtain the convolution frequency domain feature of each frame of image, and the specific logic is as follows: For the low-frequency regions, the first high-frequency regions, the second high-frequency regions and the third high-frequency regions decomposed by wavelet transform, exclusive wavelet convolution kernels suitable for their frequency characteristics are configured respectively; Convolution operation is performed on the low-frequency regions, the first high-frequency regions, the second high-frequency regions and the third high-frequency regions respectively to output convolution feature maps with consistent spatial dimensions by sliding the convolution kernel with fixed step and weighting the sum of the region pixels; The convolution feature maps of the low-frequency regions, the first high-frequency regions, the second high-frequency regions and the third high-frequency regions are spliced according to the channel dimension to obtain the convolution frequency domain feature of each frame of image.

6. The method of claim 1, wherein the method is based on a MobileNet-GRU. The spatial feature vector actual value of each frame of image in the face image sequence is input into the pre-trained Mobilenet model to extract the spatial feature vector actual value, and the spatial feature includes the nose tip coordinate, the vertical distance from the nose tip to the upper lip, and the horizontal distance between the left and right eye corners. The initial replacement ratio is multiplied by the number of frame images in the video to be detected, and the result is rounded up to obtain the number of frame images to be replaced. According to the initial replacement ratio, a certain number of frame images are randomly selected from the input sequence, and the corresponding spatial feature vector actual value is replaced by the spatial feature vector predicted value predicted by the GRU model to form a new spatial feature vector, which is composed of the predicted value after replacement and the actual value without replacement. The new spatial feature vector is input into the GRU model again to calculate the mean square error between the predicted value and the actual value. Calculate the mean square error change of the current and the previous iteration; If the mean square error change of the current and the previous iteration is not less than a set threshold, adjust the replacement ratio, specifically as follows: If the current mean square error is not greater than the previous mean square error, increase the replacement ratio and recalculate the number of frame images; If the current mean square error is greater than the previous mean square error, decrease the replacement ratio and recalculate the number of frame images; If the mean square error change of the current and the previous iteration is less than a set threshold, stop iteration, obtain the final frame image replacement ratio and replacement number, use the final frame image replacement ratio and replacement number to replace the actual value of the spatial feature vector, generate the final optimized spatial feature vector, and use the optimized spatial feature vector as the depth feature vector.

7. The method of claim 6, wherein the method is based on a MobileNet-GRU. Perform forgery judgment on the to-be-detected video, and the specific logic is as follows: If the forgery probability score is less than a preset threshold, the to-be-detected video is determined to be a real video; otherwise, the to-be-detected video is determined to be a fake video. 8.A system for detecting fake face video based on Mobilenet-GRU, the system being configured to perform the method for detecting fake face video based on Mobilenet-GRU according to any one of claims 1-7. It includes: A face detection module is used to extract frames from a to-be-detected video to obtain an image frame sequence, and perform face detection and alignment on the image frame sequence frame by frame to standardize the face region image of each frame of image, and generate a face image sequence; An image block module is used to divide each frame of the face image sequence into a plurality of non-overlapping blocks with a set block size, calculate the pixel change of the corresponding blocks of adjacent frames, and count the distribution data of the pixel changes of all corresponding blocks of adjacent frames as the consistency feature of adjacent frames; A feature extraction module is used to perform wavelet transform decomposition on each frame of image in the face image sequence, perform discrete cosine transform on each frequency region after decomposition to obtain the energy ratio of each frame of image, and splice the wavelet convolution of each frequency region according to the channel dimension to obtain the convolution frequency domain feature of each frame of image; An iterative optimization module is used to input each frame of image in the face image sequence into a pre-trained Mobilenet model to extract a spatial feature vector, input the spatial feature vector into a GRU model based on a set initial replacement ratio for time series prediction, iteratively adjust the frame image replacement ratio to optimize the feature sequence, and obtain the optimized spatial feature of each frame of image; A splicing module is used to splice the consistency feature, energy ratio and convolution frequency domain feature according to the timestamp respectively to form a shallow feature vector of the to-be-detected video, and splice the optimized spatial feature according to the timestamp to form a depth feature vector of the to-be-detected video; A judgment module is used to splice the shallow physical feature vector and the depth feature vector, input the spliced feature vector into a pre-trained discrimination model for discrimination to obtain a forgery probability score and perform forgery judgment on the to-be-detected video.

Images