Facial video tampering detection method based on MesoNet and LSTM

Abstract

The invention discloses a facial video tampering detection method based on MesoNet and LSTM. The method extracts the features of frame images and classifies them through an improved MesoNet network. The extracted features, classification results and preset weights are combined into new parameters. The new parameters of all frame images are input into the LSTM network, so that the LSTM network adaptively weights the frame images, processes the detection influence of frame images with different qualities in a target video on the target video, and improves the reliability of the target video tampering detection accuracy. The tampering detection at the frame image level and the tampering detection at the video level are evaluated separately. The LSTM network strengthens the role of time domain features in video forensics detection, and also avoids the influence of the parameter back-transmission process on the improved MesoNet network parameters at the frame image level. Moreover, the improved MesoNet network has relatively fewer network layers and parameters, can improve the feature learning speed, and can also enhance its facial feature expression ability.

Description

Technical Field

[0001] The present invention relates to the technical field of Deepfake facial video tampering, and in particular to a facial video tampering detection method based on MesoNet and LSTM. Background Art

[0002] Facial video tampering has had a significant impact on the dissemination of multimedia information in the Internet era, especially the deep learning-based automated video tampering technology represented by Deepfake, which has greatly reduced the difficulty of generating tampered videos and expanded the audience of this technology. Video information that is difficult for the public to distinguish between true and false is often tampered by criminals to spread fake news and create financial fraud. In addition, modern people's facial biometric information is often an important way to unlock their smart devices, authenticate their citizenship, and make mobile online payments. Therefore, in the field of computer vision and multimedia forensics, the study of facial video tampering forensics is a very socially valuable topic.

[0003] At present, the research work on facial video tampering based on Deepfake generally starts from the forensic research of video frame images, and measures the authenticity of the video by the authenticity of a large number of frame images. For videos, if they are only identified from the perspective of frame images, the relationship between frame images and videos must be handled well. The current mainstream approach is to partially save and crop the images containing faces in the video, extract and classify features through convolutional neural networks or other machine learning models, and then average the weights of all frame images used in a video during video identification. In fact, this approach is an extremely unfair evaluation method for identifying videos. On the one hand, there will be relatively blurred facial images in the video, and generally, the tampering detection of relatively blurred images is prone to errors; on the other hand, if there are multiple faces of people in a video, or there are multiple facial information of people in a frame, but only part of the facial information is tampered with, the average weight method based on the pictures selected randomly or at intervals cannot be a good way to evaluate the authenticity of a video. Summary of the invention

[0004] The present invention aims to at least solve the technical problems existing in the prior art. To this end, the present invention proposes a facial video tampering detection method based on MesoNet and LSTM, which can improve the accuracy and reliability of target video tampering detection.

[0005] The first aspect of the present invention provides a facial video tampering detection method based on MesoNet and LSTM, comprising the following steps:

[0006] Filter out multiple frame images containing facial regions of human faces from the target video;

[0007] Constructing an improved MesoNet network, wherein the improved MesoNet network includes the first three sub-networks connected in sequence and two fourth-layer sub-networks arranged in parallel, wherein the two fourth-layer sub-networks each include a convolutional layer, a normalization layer, and a pooling layer connected in sequence, wherein the core size of the pooling layer of one of the fourth-layer sub-networks is 2*2, and the core size of the pooling layer of the other fourth-layer sub-network is 4*4; inputting the frame image into the improved MesoNet network, and obtaining a feature vector and a classification result of the frame image output by the improved MesoNet network;

[0008] Calculating a reconstructed vector of the frame image according to a preset adaptive weight, a feature vector of the frame image and a classification result;

[0009] The reorganized vectors of all the frame images are input into the LSTM network, and the LSTM network outputs the tampering detection result of the target video.

[0010] According to the embodiments of the present invention, there are at least the following technical effects:

[0011] 1) This method uses a combination network of an improved MesoNet network and an LSTM network. First, the features of the frame image are extracted through the improved MesoNet network and detection and classification are performed; then, the feature vectors at the frame image level extracted, the classification results and the preset weight parameters are combined into new parameters, and the new parameters of all frame images of the target video are input into the LSTM network. While learning the temporal features through the LSTM network, the frame images are adaptively weighted, and the detection effects of frame images of different qualities in the target video on the target video are reasonably processed (i.e., the temporal features between frame images are automatically learned, and the weights of frame images with higher reliability are automatically adjusted to balance the effects of frame images of different qualities and categories on the detection of the target video), thereby improving the reliability of the accuracy of detection of tampering of the target video;

[0012] 2) This method uses a combination of an improved MesoNet network and an LSTM network to evaluate frame-level tampering detection separately from video-level tampering detection. The improved MesoNet network strengthens the role of time domain features in video forensics detection, while avoiding the influence of the parameter backpropagation process on the LSTM network parameters at the frame-level.

[0013] 3) The improved MesoNet network used in this method is based on the MesoNet network. Compared with the existing CNN or GAN detection network, this network has fewer network layers and parameters, which can improve the feature learning speed of frame images; and compared with the shortcomings of extracting facial features from the micro and macro levels, the improved MesoNet network is used to enhance its facial feature description ability by extracting mesoscopic image features;

[0014] 4) This method takes into account the complexity of frame image classification results when participating in video detection, and uses adaptive adjustment of the weight parameters of a single frame image to replace the traditional averaging weights that may bring possible errors to video tampering detection.

[0015] According to some embodiments of the present invention, the first to third subnetworks in the improved MesoNet network all include sequentially connected convolutional layers, normalization layers, and pooling layers, wherein the convolution kernel of the first subnetwork is 3*3, and the kernel size of the pooling layer is 2*2; the convolution kernel of the second subnetwork is 3*3, and the kernel size of the pooling layer is 2*2; the convolution kernel of the third subnetwork is 5*5, and the kernel size of the pooling layer is 2*2; the convolution kernel of the two fourth subnetworks are both 5*5.

[0016] According to some embodiments of the present invention, the step of calculating the reconstructed vector of the frame image according to the preset adaptive weight, the feature vector of the frame image and the classification result includes:

[0017] X i =w i +l i +v i

[0018] Among them, the X i Represents the reorganized vector of the i-th frame image, the w i represents the preset adaptive weight of the i-th frame image, the v i Represents the feature vector of the i-th frame image, the l i Represents the classification result of the i-th frame image.

[0019] According to some embodiments of the present invention, the i The initial value is generated using random parameters.

[0020] According to some embodiments of the present invention, the LSTM network includes a three-layer bidirectional superimposed recurrent network and a one-way hidden layer with a dimension size of 512.

[0021] According to some embodiments of the present invention, the LSTM network obtains the tampering detection result of the target video through a Sigmoid function, and the Sigmoid function includes:

[0022] According to some embodiments of the present invention, the step of selecting a plurality of frame images containing facial regions of human faces from a target video includes:

[0023] The facial region of the human face in the target video is detected frame by frame by OpenFace, and a frame image containing the facial region of the human face is retained every ten frames, and the retained frame image is cropped.

[0024] The second aspect of the present invention provides a facial video tampering detection system based on MesoNet and LSTM, comprising:

[0025] A frame image acquisition module is used to select multiple frame images containing human face areas from the target video;

[0026] A frame image feature extraction and classification module is used to construct an improved MesoNet network, wherein the improved MesoNet network includes the first three sub-networks connected in sequence and two fourth-layer sub-networks arranged in parallel, wherein the two fourth-layer sub-networks each include a convolutional layer, a normalization layer, and a pooling layer connected in sequence, wherein the core size of the pooling layer of one of the fourth-layer sub-networks is 2*2, and the core size of the pooling layer of the other fourth-layer sub-network is 4*4; the frame image is input into the improved MesoNet network, and the feature vector and classification result of the frame image output by the improved MesoNet network are obtained;

[0027] A vector reorganization module, used for calculating a reorganization vector of the frame image according to a preset adaptive weight, a feature vector of the frame image and a classification result;

[0028] The video tampering detection module is used to input the reorganized vectors of all the frame images into the LSTM network to obtain the tampering detection result of the target video output by the LSTM network.

[0029] According to the embodiments of the present invention, there are at least the following technical effects:

[0030] 1) This system uses a combination network of an improved MesoNet network and an LSTM network. First, the features of the frame images are extracted through the improved MesoNet network and detection and classification are performed; then, the feature vectors and classification results of the extracted frame image level are combined with the preset weight parameters into new parameters, and the new parameters of all frame images of the target video are input into the LSTM network. While learning the temporal features, the LSTM network adaptively weights the frame images, and reasonably handles the detection effects of frame images of different qualities in the target video on the target video (that is, automatically learning the temporal features between frame images, and automatically adjusting the weights of frame images with higher reliability, balancing the effects of frame images of different qualities and categories on the detection of the target video), thereby improving the reliability of the accuracy of detection of tampering of the target video;

[0031] 2) This system uses a combination of an improved MesoNet network and an LSTM network to evaluate frame-level tampering detection separately from video-level tampering detection. The improved MesoNet network strengthens the role of time domain features in video forensics detection, while avoiding the influence of the parameter backpropagation process on the frame-level LSTM network parameters;

[0032] 3) The improved MesoNet network used in this system is based on the MesoNet network. Compared with the existing CNN or GAN detection network, this network has fewer network layers and parameters, which can improve the feature learning speed of frame images; and compared with the shortcomings of extracting facial features from the micro and macro levels, the improved MesoNet network is used to enhance its facial feature description ability by extracting mesoscopic image features;

[0033] 4) This system takes into account the complexity of frame image classification results when participating in video detection, and uses adaptive adjustment of the weight parameters of a single frame image to replace the traditional averaging weights that may bring possible errors to video tampering detection.

[0034] According to a third aspect of the present invention, a facial video tampering detection device based on MesoNet and LSTM is provided, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the facial video tampering detection method based on MesoNet and LSTM as described above.

[0035] A fourth aspect of the present invention is a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the facial video tampering detection method based on MesoNet and LSTM as described above.

[0036] It should be noted that the beneficial effects between the third aspect to the fourth aspect of the present invention and the prior art are the same as the beneficial effects between the above-mentioned facial video tampering detection method based on MesoNet and LSTM and the prior art, and will not be described in detail here.

[0037] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0039] Figure 1 It is a structural diagram of a facial video tampering detection system based on MesoNet and LSTM provided by an embodiment of the present invention;

[0040] Figure 2 It is a flowchart of a facial video tampering detection method based on MesoNet and LSTM provided by an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the MesoNet network structure;

[0042] Figure 4 is a flow chart of a facial video tampering detection method based on MesoNet and LSTM provided by another embodiment of the present invention;

[0043] Figure 5 is a flow chart of a facial video tampering detection method based on MesoNet and LSTM provided by another embodiment of the present invention;

[0044] Figure 6 is a schematic diagram of the structure of an improved MesoNet network provided by an embodiment of the present invention;

[0045] Figure 7 It is a schematic diagram of the structure of an LSTM network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0047] At present, the work of Deepfake-based facial video tampering detection (also known as authenticity identification, authenticity classification, authenticity decision, etc.) generally starts from the forensic research of the video frame images, and measures the authenticity of the video through the authenticity of a large number of frame images. For the video, if it is identified only from the perspective of frame images, the relationship between frame images and videos must be handled well. The current mainstream approach is to partially save and crop the images containing faces in the video, extract and classify features through convolutional neural networks or other machine learning models, and then average the weights of all frame images used in a video during video identification. In fact, this approach is an extremely unfair evaluation method for identifying videos. On the one hand, there will be relatively blurred facial images in the video, and generally the tampering detection of relatively blurred images is prone to errors; on the other hand, if there are multiple faces of people in a video, or there are multiple facial information of people in a frame, but only part of the facial information is tampered with, the average weight method based on the pictures selected randomly or at intervals cannot well evaluate the authenticity of a video.

[0048] In order to solve the above technical defects, refer to Figure 1 to Figure 2 An embodiment of the present invention provides a facial video tampering detection system based on MesoNet and LSTM (Long Short Term Memory), the system comprising: a frame image acquisition module 100, a frame image feature extraction and classification module 200, a vector reconstruction module 300 and a video tampering detection module 400. The system is used to perform a facial video tampering detection method based on MesoNet and LSTM, comprising the following steps:

[0049] Step S101, filter out multiple frame images containing human face regions from the target video. This step is performed by the frame image acquisition module 100. In some embodiments, the frame image acquisition module 100 uses OpenFace to detect and crop facial frame images in the target video. It should be noted that, in this article, facial frame images refer to frame images containing human face regions.

[0050] Step S102, construct an improved MesoNet network, the improved MesoNet network includes the first three layers of cascaded sub-networks and two fourth-layer sub-networks arranged in parallel, the two fourth-layer sub-networks each include a convolutional layer, a normalization layer, and a pooling layer connected in sequence, the pooling layer size of one fourth-layer sub-network is 2*2, and the pooling layer size of the other fourth-layer sub-network is 4*4; input the frame image into the improved MesoNet network, and obtain the feature vector and classification result of the frame image output by the improved MesoNet network. This step is performed by the frame image feature extraction and classification module 200. Figure 3As shown in , the original MesoNet network is a variant network based on CNN (convolutional neural network). Figure 6 As shown, this method is improved on the basis of the original MesoNet network. A branch network is newly added after the third-layer subnetwork of the MesoNet network, that is, two fourth-layer subnetworks are formed in parallel. The two fourth-layer subnetworks include a convolutional layer, a normalization layer, and a pooling layer connected in sequence. The size of the pooling layer of the newly added fourth-layer subnetwork is 2*2, so that the newly added fourth-layer subnetwork is used to output the feature vector of the frame image; the pooling layer of the original fourth-layer subnetwork remains unchanged, with a size of 4*4, and is used to output the classification result of the frame image.

[0051] In a specific example, the first to third subnetworks in the improved MesoNet network all include sequentially connected convolutional layers, normalization layers, and pooling layers, wherein the convolution kernel of the first subnetwork is 3*3, and the kernel size of the pooling layer is 2*2; the convolution kernel of the second subnetwork is 3*3, and the kernel size of the pooling layer is 2*2; the convolution kernel of the third subnetwork is 5*5, and the kernel size of the pooling layer is 2*2; the convolution kernel of the two fourth subnetworks is 5*5. In this embodiment, based on the original MesoNet network, the size of the convolution kernel of the second subnetwork is changed to 3*3, which can prevent the loss of feature information in the initial stage.

[0052] Step S103, calculate the reorganized vector of the frame image according to the preset adaptive weight, the feature vector of the frame image and the classification result. This step is performed by the vector reorganization module 300. In some embodiments, the reorganized vector is represented by the following formula:

[0053] X i =w i +l i +v i

[0054] Among them, X i Represents the reconstructed vector of the i-th frame image, w i Represents the preset adaptive weight of the i-th frame image, v i Represents the feature vector of the i-th frame image, l i represents the classification result of the i-th frame image; w i The initial value is generated using random parameters. In the above formula, the preset adaptive weight, the feature vector of the frame image, and the classification result are added to obtain the recombined vector, and the classification result of the frame image is output in the improved MesoNet network. i On the basis of the frame image, the feature vector v is increased i and an adaptive weight w used to represent the weight of the frame image in evaluating the target video tampering detection classification iBy reorganizing the vector, the LSTM network can adaptively weight the facial frame images and reasonably handle the impact of different quality frame images in the target video on the video detection.

[0055] Step S104: input the reorganized vectors of all frame images into the LSTM network to obtain the classification result of the target video output by the LSTM network. This step is performed by the video tampering detection module 400. The LSTM network is a variant network based on RNN (recurrent neural network). In some embodiments, refer to Figure 7 , the LSTM network includes three layers of bidirectional superimposed recurrent networks and one unidirectional hidden layer with a dimension size of 512. The LSTM network obtains the tampering detection result of the target video through the Sigmoid function: R represents the detection evaluation score of the target video, which is between 0 and 1, where 0 represents that the video is fake and 1 represents that it is real; σ(·) represents the Sigmoid function.

[0056] Compared with the existing technology, this method has the following advantages:

[0057] This method is first improved based on the MesoNet network. A fourth-layer subnetwork is added after the third-layer subnetwork of the MesoNet network, so that one of the fourth-layer subnetworks outputs the feature vector of the image, and the other fourth-layer subnetwork is consistent with the MesoNet network and outputs the classification result of the image; then a corresponding adaptive weight is preset for each frame image to adjust the influence of the subsequent LSTM network on the tampering detection of the target video, and the adaptive weight is recombined with the feature vector of the frame image and the classification result to obtain the recombined vector corresponding to each frame image; finally, the recombined vectors of all frame images are input into the LSTM network as features, and feature fusion and weight adaptive adjustment are realized through the LSTM network until the tampering detection of the target video is completed.

[0058] 1) This method uses a combination network of an improved MesoNet network and an LSTM network. First, the features of facial frame images are extracted through the improved MesoNet network and detection and classification are performed; then, the extracted frame image level feature vectors, classification results and preset weight parameters are combined into new parameters, and the new parameters of all frame images of the target video are input into the LSTM network. While learning the temporal features, the LSTM network adaptively weights the facial frame images, and reasonably handles the detection effects of frame images of different qualities in the target video on the target video (that is, automatically learning the temporal features between frame images, and automatically adjusting the weights of frame images with higher reliability, balancing the effects of frame images of different qualities and categories on the detection of the target video), thereby improving the reliability of the accuracy of target video tampering detection.

[0059] 2) This method uses a combination of an improved MesoNet network and an LSTM network to evaluate frame image level tampering detection separately from video level tampering detection. The LSTM network strengthens the role of time domain features in video forensics detection, while avoiding the influence of the parameter backpropagation process on the improved MesoNet network parameters at the frame image level.

[0060] 3) This method uses an improved MesoNet network based on the MesoNet network. Compared with the existing CNN or GAN detection networks, this network has relatively fewer network layers and parameters, which can improve the feature learning speed of frame images; and compared with the shortcomings of extracting facial features from the micro and macro levels, the improved MesoNet network is used to enhance its facial feature representation ability by extracting mesoscopic level image features.

[0061] 4) This method takes into account the complexity of facial frame image classification results when participating in video detection, and uses adaptive adjustment of the weight parameters of a single frame image to replace the traditional averaging weights that may bring possible errors to video tampering detection.

[0062] 5) This method uses an improved MesoNet network and sets the size of the convolution kernel of the second-layer sub-network to 3*3 to prevent the loss of feature information in the initial stage.

[0063] Reference Figures 4 to 7 An embodiment of the present invention provides a facial video tampering detection method based on MesoNet and LSTM, comprising the following steps:

[0064] Step S201, obtaining training and testing data sets from video sequences; the data sets contain frame images of facial regions of human faces, namely, facial frame images.

[0065] Step S202, preprocessing the data set; using OpenFace to detect and crop facial frame images in the video, and labeling the cropped facial frame images.

[0066] Step S203: Build an improved MesoNet network; Figure 4 and Figure 6 , two improvements were made to the original MesoNet network: on the one hand, the size of the convolution kernel of the second-layer sub-network was changed to 3*3 to prevent the loss of feature information in the initial stage; on the other hand, a new branch network was added after the third-layer sub-network of the original MesoNet network, and the size of the pooling layer of the fourth layer was changed to 2*2. This branch network will output a feature vector v i, used for classification learning features of subsequent target videos, the original MesoNet network continues to output the classification results of whether the input facial frame image is tampered with as a classification model for facial frame images. i The improved MesoNet network has two branches, one outputting the feature vector of the image and the other outputting the classification detection result of the image.

[0067] Step S204, reorganize the feature vector; in the feature vector v i Add l on the basis of i 、w i The two parameters represent the classification result (fake / real) of the facial frame image and the weight of the facial frame image in evaluating the target video tampering detection.

[0068] Step S205: input the reorganized vector as a feature into the RNN-based LSTM network; Figure 4 After training, the LSTM network further learns the parameters v from the facial frame image features generated by the improved MesoNet network i Adaptively adjust and finally detect the authenticity of the video.

[0069] Compared with the existing technology, this method has the following advantages:

[0070] 1) This method uses a combination network of an improved MesoNet network and an LSTM network. First, the features of facial frame images are extracted through the improved MesoNet network and detection and classification are performed; then, the extracted frame image level feature vectors, classification results and preset weight parameters are combined into new parameters, and the new parameters of all frame images of the target video are input into the LSTM network. While learning the temporal features, the LSTM network adaptively weights the facial frame images, and reasonably handles the detection effects of frame images of different qualities in the target video on the target video (that is, automatically learning the temporal features between frame images, and automatically adjusting the weights of frame images with higher reliability, balancing the effects of frame images of different qualities and categories on the detection of the target video), thereby improving the reliability of the accuracy of target video tampering detection.

[0071] 2) This method uses a combination of an improved MesoNet network and an LSTM network to evaluate frame image level tampering detection separately from video level tampering detection. The LSTM network strengthens the role of time domain features in video forensics detection, while avoiding the influence of the parameter backpropagation process on the improved MesoNet network parameters at the frame image level.

[0072] 3) This method uses an improved MesoNet network based on the MesoNet network. Compared with the existing CNN or GAN detection networks, this network has relatively fewer network layers and parameters, which can improve the feature learning speed of frame images; and compared with the shortcomings of extracting facial features from the micro and macro levels, the improved MesoNet network is used to enhance its facial feature representation ability by extracting mesoscopic level image features.

[0073] 4) This method takes into account the complexity of facial frame image classification results when participating in video detection, and uses adaptive adjustment of the weight parameters of a single frame image to replace the traditional averaging weights that may bring possible errors to video tampering detection.

[0074] 5) This method uses an improved MesoNet network and sets the size of the convolution kernel of the second-layer sub-network to 3*3 to prevent the loss of feature information in the initial stage.

[0075] Reference Figures 4 to 7 One embodiment of the present invention provides a facial video tampering detection method based on MesoNet and LSTM, comprising:

[0076] OpenFace detects faces in video frames. It detects faces frame by frame in the input video, retains a facial frame image every ten frames, and crops and saves the face and a small area around the frame.

[0077] Build an improved MesoNet network, and after improvement, realize two functions: one is to perform a facial image f for each input i Output a facial feature representation vector v i , and secondly, output the classification result l of the facial frame image i .

[0078] For each facial frame image, an adaptive weight ω is generated by random parameters i , the adaptive weight ω i , feature vector v i And the classification results i Combine into a recombination vector.

[0079] Build an LSTM network, which reorganizes vectors, performs feature loop learning, and finally makes a decision on the authenticity of the video.

[0080] The following is a detailed description:

[0081] Input the facial frame image of size 256*256*3 into the network to extract facial features;

[0082] The pooling layer kernel size of the third layer sub-network of the improved MesoNet network is 2*2; the fourth layer sub-network includes two branch networks T-1 and T-2;

[0083] The branch network T-1 keeps the parameter structure of the original MesoNet network unchanged, and outputs a frame-level classification result l i ;

[0084] The pooling layer kernel size of branch network T-2 is 2*2, and its output dimension size is 2048 feature vector v i ;

[0085] v i and l i Feature vectors and classification results corresponding to the same facial frame image.

[0086] After the improved MesoNet network extracts features and classifies facial frame images, it is necessary to detect the authenticity of the video through the features of the facial frame images. Ideally, only by integrating as many features of all frame images as possible can the video be detected more reliably.

[0087] Considering the influence of the averaged frame image on the detection of video authenticity, it is likely to obtain a detection result with low credibility. In this embodiment, an adaptive weight method is used to add a weight parameter w to the feature vector of each frame image from the improved MesoNet network. i , used to adjust its impact on video detection;

[0088] Let the reorganized vector X i =w i +l i +v i , where w i The dimension size is 1, l i The dimension size is 1, v i The dimension size is 2048, i represents the i-th frame image in the video;

[0089] The LSTM network consists of three layers of bidirectional stacked recurrent networks and one unidirectional hidden layer with a dimension size of 512.

[0090] All facial frames from the same video are reassembled into a vector X i , (where i = 0, 1, ···n), is input as a feature into an LSTM network with feature fusion and weight adaptive adjustment functions;

[0091] Because w i , l i 、v i There is a very strong correlation between them. The LSTM network will automatically extract useful features and eventually be used in video tampering detection.

[0092] w i Using random parameter initialization and end-to-end training, the LSTM network will give each w involved in feature fusion i Adaptively adjust its size; finally, the tampering detection result of the video is obtained through the Sigmoid function:

[0093] The present invention further provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following is achieved:

[0094] As mentioned above, the facial video tampering detection method based on MesoNet and LSTM.

[0095] The processor and the memory may be connected via a bus or other means.

[0096] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0097] The non-transient software program and instructions required to implement the facial video tampering detection method based on MesoNet and LSTM in the above embodiment are stored in the memory, and when executed by the processor, the facial video tampering detection method based on MesoNet and LSTM in the above embodiment is executed.

[0098] The present application also provides a computer-readable storage medium storing computer-executable instructions for executing:

[0099] As mentioned above, the facial video tampering detection method based on MesoNet and LSTM.

[0100] The computer-readable storage medium stores computer-executable instructions, which are executed by a processor or controller, for example, by a processor in the above-mentioned electronic device embodiment, so that the above-mentioned processor can execute the facial video tampering detection method based on MesoNet and LSTM in the above-mentioned embodiment.

[0101] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing data (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired data and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any data delivery media.

[0102] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0103] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A facial video tampering detection method based on MesoNet and LSTM, characterized in that: The steps include: Filter out multiple frame images containing facial regions of human faces from the target video; Constructing an improved MesoNet network, wherein the improved MesoNet network includes the first three sub-networks connected in sequence and two fourth-layer sub-networks arranged in parallel, wherein the two fourth-layer sub-networks each include a convolutional layer, a normalization layer, and a pooling layer connected in sequence, wherein the core size of the pooling layer of one of the fourth-layer sub-networks is 2*2, and the core size of the pooling layer of the other fourth-layer sub-network is 4*4; inputting the frame image into the improved MesoNet network, and obtaining a feature vector and a classification result of the frame image output by the improved MesoNet network; Calculating a reconstructed vector of the frame image according to a preset adaptive weight, a feature vector of the frame image and a classification result; Inputting the reorganized vectors of all the frame images into an LSTM network, and obtaining a tampering detection result of the target video output by the LSTM network; The first to third subnetworks in the improved MesoNet network all include sequentially connected convolutional layers, normalization layers, and pooling layers, wherein the convolution kernel of the first subnetwork is 3*3, and the kernel size of the pooling layer is 2*2; the convolution kernel of the second subnetwork is 3*3, and the kernel size of the pooling layer is 2*2; the convolution kernel of the third subnetwork is 5*5, and the kernel size of the pooling layer is 2*2; the convolution kernels of the two fourth subnetworks are both 5*5; The step of calculating the reorganized vector of the frame image according to the preset adaptive weight, the feature vector of the frame image and the classification result comprises: Among them, the Indicates The reconstructed vector of the frame image is Indicates the preset The adaptive weight of the frame image, Indicates The feature vector of the frame image, Indicates Classification results of the image frames.

2. The facial video tampering detection method based on MesoNet and LSTM according to claim 1 is characterized in that: Said The initial value is generated using random parameters.

3. The facial video tampering detection method based on MesoNet and LSTM according to claim 1 is characterized in that: The LSTM network includes a three-layer bidirectional stacked recurrent network and a one-layer unidirectional hidden layer with a dimension size of 512.

4. The facial video tampering detection method based on MesoNet and LSTM according to claim 3 is characterized in that: The LSTM network obtains the tampering detection result of the target video through the Sigmoid function, and the Sigmoid function includes: .

5. The facial video tampering detection method based on MesoNet and LSTM according to any one of claims 1 to 4, characterized in that: The step of selecting a plurality of frame images containing human face regions from the target video includes: The facial region of the human face in the target video is detected frame by frame by OpenFace, and a frame image containing the facial region of the human face is retained every ten frames, and the retained frame image is cropped.

6. A facial video tampering detection system based on MesoNet and LSTM, characterized in that: include: A frame image acquisition module is used to select multiple frame images containing human face areas from the target video; A frame image feature extraction and classification module is used to construct an improved MesoNet network, wherein the improved MesoNet network includes the first three sub-networks connected in sequence and two fourth-layer sub-networks arranged in parallel, wherein the two fourth-layer sub-networks each include a convolutional layer, a normalization layer, and a pooling layer connected in sequence, wherein the core size of the pooling layer of one of the fourth-layer sub-networks is 2*2, and the core size of the pooling layer of the other fourth-layer sub-network is 4*4; the frame image is input into the improved MesoNet network, and the feature vector and classification result of the frame image output by the improved MesoNet network are obtained; A vector reorganization module, used for calculating a reorganization vector of the frame image according to a preset adaptive weight, a feature vector of the frame image and a classification result; A video tampering detection module, used for inputting the reorganized vectors of all the frame images into an LSTM network to obtain a tampering detection result of the target video output by the LSTM network; The first to third subnetworks in the improved MesoNet network all include sequentially connected convolutional layers, normalization layers, and pooling layers, wherein the convolution kernel of the first subnetwork is 3*3, and the kernel size of the pooling layer is 2*2; the convolution kernel of the second subnetwork is 3*3, and the kernel size of the pooling layer is 2*2; the convolution kernel of the third subnetwork is 5*5, and the kernel size of the pooling layer is 2*2; the convolution kernels of the two fourth subnetworks are both 5*5; The step of calculating the reorganized vector of the frame image according to the preset adaptive weight, the feature vector of the frame image and the classification result comprises: Among them, the Indicates The reconstructed vector of the frame image is Indicates the preset The adaptive weight of the frame image, Indicates The feature vector of the frame image, Indicates Classification results of the image frames.

7. A facial video tampering detection device based on MesoNet and LSTM, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the facial video tampering detection method based on MesoNet and LSTM as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the facial video tampering detection method based on MesoNet and LSTM as described in any one of claims 1 to 5.

Images