Implicit domain mirror image authentication method in face video and image tampering detection by twin network

Abstract

The invention discloses an implicit domain mirror image authentication method in face video and image tampering detection through a twin network, and belongs to the technical field of computer vision and image analysis. According to the technical scheme, a pre-processing module is used for separating a human face region and a background region in an image to obtain a human face image block and a background image block respectively; the feature extraction module is used for extracting image features by adopting a modified twin network to obtain gray spatial features of the face image blocks and the background image blocks; the feature alignment module is used for splicing the gray spatial features by adopting a feature splicing method, and obtaining a prediction label Y through classification; and correcting the classification by using a voting principle to obtain a result label Yt. The method has the beneficial effects that the classification accuracy is remarkably improved, so that relatively high detection precision is obtained; the reliability of a classification result is enhanced, so that the system can recognize tampered face videos and images more accurately; and the learning process is more stable and robust, so that more satisfactory performance is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and image analysis technology, and particularly focuses on developing a novel, optimized twin network architecture specifically designed for identifying and verifying face tampering in video content. Background Art

[0002] In today's era, the advancement of science and technology has promoted the rapid development of social media and short video platforms, which have become the main way for people to spend their time. People are keen to share their daily lives on these platforms. Whether it is pictures or videos, these contents are rapidly spread on the Internet and have become the main form of information exchange and sharing. However, with the advancement of image tampering technology, especially the abuse of face image tampering technology, the spread of images and videos has brought a lot of negative impacts. Tampered images and videos may be used for improper purposes, such as extortion, violation of personal privacy, dissemination of false information, etc., and may even pose a serious threat to national and social security. In addition, the spread of these tampered contents may also lead to public distrust of the news media, making society more susceptible to interference from false information. Therefore, the development of technology that can detect tampered face images and videos is of extremely important practical significance and value.

[0003] With the rapid development of deep learning and generative adversarial network technology, face tampering technology has become very mature, and common tampering methods include Deepfake, Face2Face, Faceswap and Neural Textures. Although these technologies rely on advanced image processing algorithms, there are now some easy-to-operate software or program frameworks that allow even people without professional knowledge to easily tamper with facial images and videos.

[0004] In order to deal with the harm caused by face image tampering, researchers have proposed a variety of methods to detect tampered face images and videos. At present, the main detection methods are divided into two types: traditional manual feature extraction and neural network feature extraction. Traditional manual features are clearer and more robust in utilizing fine-grained information in images, while neural networks can learn richer image features. The accuracy of detection depends on the selection of features and classifiers. Common classifiers include SVM, CNN, RNN, MLP, etc.

[0005] Although existing methods have achieved good detection accuracy on some public datasets, there are still some problems. First, most methods are not robust enough, especially when faced with highly compressed forged images and videos, the detection accuracy will drop significantly. Second, almost all methods only focus on the features of the face area or the boundary area where the face merges with the background, while ignoring the features of the background. In fact, background features also have great potential in detecting tampered images. Therefore, future research needs to seek breakthroughs in these issues to improve the practicality and reliability of detection technology. Summary of the invention

[0006] In order to improve the accuracy of distinguishing true from false videos, this study proposes a twin network hidden domain mirror authentication method in face video and image tampering detection. The specific technical solution is as follows:

[0007] S1. Pre-processing module: Separate the face area and the background area in the image to obtain face image blocks and background image blocks respectively.

[0008] S2, feature extraction module Feature extraction: The modified twin network is used to extract image features to obtain the grayscale spatial features of the face image block and the background image block.

[0009] S3, feature alignment module Feature alignment: The grayscale space features are spliced ​​using the feature splicing method, and the predicted label Y is obtained through classification.

[0010] S4. Use the voting principle to correct the classification and obtain the result label Yt.

[0011] In the further implementation of step S1, the present invention evaluates the feature similarity between the face area and the background area in the image through the twin network. In the pre-processing module Pre-processing, the present invention crops all the video frame images in the data set to obtain face image blocks and background image blocks of uniform size, and stores them in different folders respectively. At the same time, the present invention stores the image blocks of the same video frame in the same folder. The specific operation steps are as follows:

[0012] S11. For each video in the data set, the present invention uses the software package dlib to detect the face area of ​​each frame in the video, and cuts out a face image block of a fixed size according to the center of the face.

[0013] S12. Cut out n background image blocks of the same size as the face image block from the background area of ​​the image, where n is an odd number, to provide more background information.

[0014] S13, creating a multi-level folder to store the processed face image blocks and background image blocks, so as to facilitate subsequent feature extraction and alignment operations.

[0015] In the further refinement of step S13, the present invention takes the following operations for each sub-data set: create a main folder F to accommodate the relevant data. Inside the main folder F, the present invention constructs two subfolders Fr and Ft, which are respectively used to store the data of the real video and the tampered video. Then, in Fr and Ft, the present invention creates N folders Fk for each real or tampered video, where the value of k ranges from 1 to N, and N represents the total number of real videos or tampered videos. In each Fk folder, the present invention creates two subfolders Ff and Fb, which are respectively used to store the face image blocks and background image blocks after cropping of each video.

[0016] In the training phase, the present invention randomly selects an image block from the Fb and Ff subfolders in each Fk folder to form a pairing of a background image block IB and a face image block IF. Such a pairing mechanism ensures that during the training process, the network can learn the associated features between the face and the background, thereby improving the subsequent detection accuracy.

[0017] In the further implementation of step S2, the present invention uses the twin network to learn the similarity measure between the face image block and the background image block, and applies it to the face verification task. This process is implemented in the twin network through the improved feature extraction module Featureextraction, and its specific steps are as follows:

[0018] S21, the present invention converts the background image block IB and the face image block IF in the image block pair into grayscale images IBG and IFG respectively. The purpose of this conversion is to reduce the influence of the semantic content in the image on feature extraction, so that the network can focus on learning the intrinsic structure of the image.

[0019] S22, then, the present invention inputs the grayscale images IBG and IFG into two Xception networks with shared weights. The two networks are responsible for extracting the grayscale spatial features of the background image block and the face image block, respectively, and outputting two sets of 512-dimensional feature vectors GW (B) and GW (F). This design enables the twin network to learn two types of image features at the same time, and the parameters are shared between the two networks, which enhances the learning ability and generalization ability of the model.

[0020] In this way, the present invention utilizes the improved Siamese network to extract features of face image blocks and background image blocks, which will play a key role in the subsequent feature alignment and classification stages.

[0021] In step S3, the present invention implements a feature alignment strategy through a feature alignment module Feature alignment to measure the similarity between the face image block and the background image block. This strategy involves splicing the extracted features and using a fully connected layer for final classification. The specific implementation method is as follows:

[0022] After obtaining the grayscale spatial features GW(B) and GW(F) of the face image block and the background image block, the present invention calculates the similarity of the two features in the feature alignment module. This is achieved by splicing GW(B) and GW(F) to form a new 1024-dimensional feature vector CW, that is, CW=GW(B)⊕GW(F), where ⊕ represents the splicing operation. This spliced ​​feature vector CW is then input into a network consisting of three fully connected layers, the number of nodes in these layers are 256, 10 and 2 respectively, and finally outputs a binary prediction label Y.

[0023] Such a feature alignment method can not only effectively distinguish real images from tampered images, but also improve the classification accuracy of the model. By measuring similarity at the feature level, the model can better capture the subtle differences between face image patches and background image patches, thereby making more accurate predictions.

[0024] In the detailed description of step S4, the present invention describes a method for correcting the classification results using the voting principle during the test phase. This method involves cutting out an odd number of pairs of image blocks from the same frame image, and counting the classification results of each pair of image blocks to determine the final classification result. The specific steps are as follows:

[0025] S41. In the test phase, the present invention randomly selects a face image block from the folder Ff storing face image blocks. At the same time, n background image blocks are selected from the folder Fb storing background image blocks of the same video frame, where n is an odd number. By copying the face image blocks n times, the present invention forms n groups of image pairs.

[0026] S42, these n groups of image block pairs are input into the trained feature extraction module and feature alignment module to obtain n binary prediction labels Yn. The value range of these labels is 1 to 3, 5 and other odd numbers, because the classification result of each pair of image blocks needs to be based on the majority.

[0027] S43. According to the voting principle of minority obeys majority, the present invention summarizes these prediction labels to obtain the result label Yt. This result label will indicate the classification result of the image to which the face image block and the background image block belong. If the value of Yt is 1, it means that the image block pair comes from a real video and is stored in the folder Fr. If the value of Yt is 0, it means that the image block pair comes from a forged video and is stored in the folder Ft.

[0028] Through this method, the present invention can improve the accuracy and reliability of classification through the consistency of multiple image block pairs, thereby effectively detecting tampered face videos and images.

[0029] The beneficial effects of the present invention mainly include the following points:

[0030] 1. During the testing phase, the classification results were corrected by the defined voting principle, which significantly improved the accuracy of the classification and thus achieved better detection accuracy. This approach enhanced the reliability of the classification results and enabled the system to more accurately identify tampered face videos and images.

[0031] 2. The features after feature alignment retain all the feature information of the image patch pairs, which enables the subsequent fully connected layer to more comprehensively explore the similarities between them. This method makes the learning process more stable and robust, resulting in more satisfactory performance.

[0032] 3. In a real image without tampering, the facial area and the background area are consistent at a certain feature level, and the subsequent processing operations are also completely consistent. However, forged images do not have such characteristics. The present invention not only uses the features of the facial area, but also makes full use of the features of the background area of ​​the image that were ignored by previous methods. The method of detecting forged images by comparing the similarity between the facial area and the background area of ​​the image has very good robust performance. Therefore, finding a feature space in which the facial area and the background area are consistent in the real image is crucial for detecting tampered images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below in combination with the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0034] Figure 1 : Xception network framework diagram of the present invention;

[0035] Figure 2 : Schematic diagram of voting principle of the present invention;

[0036] Figure 3 : Experimental data diagram of the present invention;

[0037] Figure 4 : The flowchart of the present invention for improving the twin network to detect tampered face videos and images;

[0038] Figure 5: Overall structural diagram of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Figure 1-5 The hidden domain mirror authentication method of the twin network in face video and image tampering detection is further explained.

[0040] The present invention introduces a twin network hidden domain image authentication method in face video and image tampering detection. The method includes the following steps:

[0041] 1. Preprocessing module (S1): cropping the face area and background area from the image to obtain the face image block and background image block respectively.

[0042] 2. Feature extraction module (S2): The grayscale spatial features of the image blocks are extracted using the improved Siamese network.

[0043] 3. Feature alignment module (S3): The predicted label Y is obtained through feature concatenation and classification.

[0044] 4. Modify classification based on voting principle (S4): Modify the classification result based on the majority voting principle to obtain the final result label Yt.

[0045] The overall structure diagram is as follows Figure 1-2 shown.

[0046] In the preprocessing module of S1, the present invention uses a twin network to evaluate the similarity of features between the face area and the background area in the image. The specific steps are as follows:

[0047] S11.For each video in the dataset, use the dlib software package to detect the face area of ​​each frame in the video, and crop a fixed-size face image block based on the center of the face.

[0048] S12. Crop n background image blocks of the same size as the face image blocks in the background area of ​​the image, where n is an odd number.

[0049] S13. Create a multi-level folder to store the processed face image blocks and background image blocks, ensuring that the image blocks of the same video frame are stored in the same folder.

[0050] Through these steps, the present invention ensures that in the subsequent feature extraction and analysis, the image blocks of each video frame are traceable and consistent, laying a foundation for subsequent processing.

[0051] In the feature extraction module of S2, the present invention creates a folder F for each sub-dataset to store the corresponding data. In the folder F, the present invention creates two sub-folders Fr and Ft, which are respectively used to store the data of the real video and the tampered video. In each Fr and Ft folder, the present invention creates N folders Fk according to the number N of real videos or tampered videos. In the Fk folder, the present invention creates two sub-folders Ff and Fb, which are respectively used to store the face image blocks and background image blocks after each video is cropped.

[0052] In the training stage, the present invention randomly selects an image block from the Fb and Ff folders of each Fk folder to form a set of image block pairs, where IB represents the background image block and IF represents the face image block. This random selection method helps the network learn the diversity of face image blocks and background image blocks in different video frames, thereby improving the accuracy of detection.

[0053] In the test phase, the present invention modifies the classification results by defining the voting principle, which significantly improves the accuracy of classification and thus obtains better detection accuracy. This method enhances the reliability of the classification results, allowing the system to more accurately identify tampered face videos and images. The features after feature alignment retain all feature information of the image block pairs, which enables the subsequent fully connected layer to more comprehensively mine the similarities between them. This method makes the learning process more stable and robust, thereby obtaining more satisfactory performance.

[0054] In a real image without tampering, the facial area and the background area are consistent at a certain feature level, and the subsequent processing operations are also completely consistent. However, forged images do not have such characteristics. The present invention not only uses the features of the facial area, but also makes full use of the features of the background area of ​​the image that were ignored by previous methods. The forged image is detected by comparing the similarity between the facial area and the background area of ​​the image, and this method has very good robust performance. Therefore, finding the feature space where the facial area and the background area are consistent in the real image is crucial for detecting tampered images.

[0055] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A twin network hidden domain image authentication method in face video and image tampering detection, characterized in that: The steps are as follows: A1, Pre-processing module Pre-processing: The facial area and the background area of ​​the image are cropped out respectively to obtain facial image blocks and background image blocks; Use a Siamese network to measure feature similarity between the facial and background regions of an image; The pre-processing module pre-processes the data set, cuts all video frame images into face image blocks and background image blocks of the same size, stores the face image blocks and background image blocks in different folders, and stores the image blocks of the same video frame in the same folder; A2. Feature extraction module: The improved twin network is used to extract image features to obtain the grayscale spatial features of the face image block and the background image block; The twin network learns similarity measurement and applies it to face verification; the improved feature extraction module Feature extraction in the twin network enables the network to learn the similarity between background image patches and facial image patches; A3, Feature alignment module: Use feature concatenation to concatenate the grayscale spatial features and classify them to obtain the predicted label Y; The feature alignment module uses feature alignment to measure similarity, concatenates the features of the extracted face image blocks and background image blocks, and performs final classification through three fully connected layers; A4. Use the voting principle to correct the classification and obtain the result label Yt; Defining voting principles to correct classification results during the testing phase; Crop n pairs of image blocks from the same frame, where n is an odd number, and statistically analyze the classification results of each pair of image blocks; The classification results are based on the majority results.

2. The method for identifying the hidden domain image in face video and image tampering detection using a twin network according to claim 1 is characterized in that: The specific steps of step A1 are as follows: A11. For each video in the dataset, use the dlib software package to detect the face area in each frame of the video and crop a fixed-size face image block based on the face center. A12, cropping n background image blocks in the background area of ​​the image that have the same size as the face image block, where n is an odd number; A13. Create a multi-level folder to store the processed face image blocks and background image blocks.

3. The method for identifying the hidden domain image in face video and image tampering detection using a twin network according to claim 2 is characterized in that: In step A13, for each sub-dataset, a folder F is created to store the corresponding data, and two folders Fr and Ft are created in the folder F to store the data of the real video and the tampered video respectively; in Fr and Ft, N folders Fk (k = 1, 2, ..., N) are created, where N is the number of real or tampered videos; two folders Ff and Fb are created in Fk to store the cropped face image blocks and background image blocks of each video respectively; in the training stage, an image block is randomly selected from the Fb and Ff folders of the Fk folder to form a set of image block pairs, where IB represents the background image block and IF represents the facial image block.

4. The method for identifying the hidden domain image in face video and image tampering detection using a twin network as claimed in claim 1, characterized in that: The specific steps in step A2 are as follows: A21, converting IB and IF in the image block pair into a grayscale image IBG of the background image block and a grayscale image IFG of the face image block, respectively, to reduce the influence of semantic content on the image; A22, input IBG and IFG into two Xception networks with shared weights, respectively, to obtain two sets of 512-dimensional background image block grayscale spatial features GW(B) and face image block grayscale spatial features GW(F); An improved Siamese network is used to extract features from facial and background image blocks, and two identical Xception networks are used as the backbone of the Siamese network; the convolutional neural network extracts appropriate and effective features from the grayscale space of the image through end-to-end training and supervised learning, extracts features from both facial and background image blocks, and shares parameters between the two networks.

5. The method for identifying the hidden domain image in face video and image tampering detection using a twin network according to claim 1 is characterized in that: A feature alignment method is used to measure the similarity to distinguish whether they are from real images or tampered images; the method is as follows: after obtaining the features in the grayscale space of the image blocks, namely GW(B) and GW(F), their similarity is measured in the feature alignment module, and GW(B) and GW(F) are connected to obtain a 1024-dimensional alignment feature, namely CW, which is defined as: represents the concatenation operation, concatenating GW(B) and GW(F) to obtain CW, so as to obtain the CW input to the fully connected layer; The three fully connected layers have 256 nodes, 10 nodes, and 2 nodes, respectively, to produce a binary prediction label Y.

6. The method for identifying the authenticity of a hidden domain image in face video and image tampering detection using a twin network as claimed in claim 1, characterized in that: In step A4, the steps for selecting data are as follows: A41. When a face image block of a person is randomly selected from the folder Ff storing face image blocks, n background image blocks are selected from the folder Fb storing background image blocks of the same video frame, where n is an odd number, and n face image blocks are copied to form n groups of image pairs; A42, inputting these n pairs of image blocks into the trained feature extraction module and feature alignment module to obtain n binary prediction labels Yn, n = 1, 3, 5; A43. According to the voting principle of minority obeys majority, the result label Yt is obtained, that is, the classification result of the face image block and the background image block belonging to the image; Y=1 means that Fk belongs to the folder Fr, and the folder Fr represents a set of image block pairs from the real video; Y=0 means that Fk is in the folder Ft, indicating that the set of image block pairs comes from the fake video.

Images