Face deep forgery detection method based on StreamLit and multiple attention deep forgery detection network

Abstract

The invention relates to the field of computer vision, in particular to a human face deep forgery detection method based on StreamLit and a multiple attention deep forgery detection network. On the basis of an OfficientNetv2s network in combination with a Haar feature algorithm and a face-recovery method in dlib, real-time face detection can be carried out on embedded equipment through a 3D depth camera, and multi-terminal deployment is supported. The feature extraction capability of the model is enhanced by adopting an attention mechanism, and deep counterfeiting detection is performed through an MAD method, so that an excellent effect is achieved. Experimental results show that the detection performance of the system on data sets such as WIDER FACE, Celeb-DF, DFDC and the like is remarkably superior to that of other existing methods, and meanwhile the system has high robustness and real-time performance. The technology can be effectively applied to face video forgery detection and identity verification, and is suitable for mobile terminal and embedded equipment scenes.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a face deep fake detection method based on StreamLit and a multi-attention deep fake detection network. Background Art

[0002] The rapid advancement of computer technology and Internet technology has greatly improved people's lives. Artificial intelligence is the result of technological progress. Deepfakes is a technology that uses deep learning algorithms to simulate and forge audio and video. Deepfakes mainly synthesize fake videos by exchanging the identity information of the original face and the target face, or by editing the target face attributes.

[0003] The existing facial forgery detection systems and tools on the market mainly rely on deep learning and machine learning algorithms to distinguish between real and forged faces by analyzing subtle features in images or videos.

[0004] Existing technology: 1. FaceForensics++ Introduction: Database and tool for face forgery detection, including images and videos generated by various forgery methods.

[0005] Technical approach: Use deep convolutional neural network (CNN) to detect forged features.

[0006] 2. DeepFaceLab Introduction: Open source face forging software, widely used to generate Deepfake videos.

[0007] Technical approach: Use deep learning and GAN technology to generate fake images and videos.

[0008] 3. Microsoft Video Authenticator Introduction: Microsoft's video verification tool can detect fake faces in videos.

[0009] Technical approach: Use machine learning algorithms to analyze forgery features in video frames.

[0010] Related patent information: 1. Patent number: US20190345678A1 Name: Deep Learning-Based Face Spoofing Detection Introduction: Face forgery detection method based on deep learning.

[0011] Technical method: Use CNN for feature extraction and classification.

[0012] 2. Patent number: CN110643251A Title: Face forgery detection method based on multimodal features Abstract: A face forgery detection method that combines multiple features (such as texture, color, and depth information).

[0013] Technical approach: Use multimodal feature extraction and fusion combined with machine learning algorithms for classification.

[0014] The main problems of existing technologies include: 1. Low training efficiency Problem: The computational complexity of large-size image processing is high, resulting in low training efficiency.

[0015] Reason: Deep learning models require a lot of computation, especially when processing large-size images.

[0016] 2. High hardware requirements Problem: Requires high-performance hardware, which increases costs and technical barriers.

[0017] Reason: When processing high-resolution images and videos, the computational complexity is high and consumes a lot of memory and video memory.

[0018] 3. Poor real-time performance Problem: It is difficult to meet the needs of real-time applications.

[0019] Reason: high computational complexity, slow processing speed, and lack of versatility. Summary of the invention

[0020] To solve the problems of the prior art, the present invention provides a face deep fake detection method based on StreamLit and a multi-attention deep fake detection network.

[0021] In order to achieve the above object, the present invention adopts the following technical solutions: A face deep fake detection method based on StreamLit and multi-attention deep fake detection network, comprising the following steps: S1. Extract facial features based on EfficientNetv2s network; S2. Use the attention mechanism to weight facial features, enhance the importance of the face area, and improve the accuracy of deep fake detection of face videos; S3. Preprocess the experimental dataset to enhance the effect of deep fake detection of face videos; S4. Based on the embedded device-side face detection model of EfficientNetv2s, a real-time face detection system is implemented on the embedded device side; S5. Deep fake detection model for face videos based on attention mechanism; S6. Build a face video forgery alarm system based on MAD method; S7. Realizes face image tampering detection, can be connected to a camera, and expands the detection method; S8. Deploy a local area network to implement system deployment on desktop and mobile terminals.

[0022] The EfficientNetv2s model is compressed and pruned. The distributed computing method is used to disperse the training tasks of large-size images to multiple computing nodes. Appropriate downsampling is performed for preprocessing to reduce the resolution by 10%-20%. Large-size images are processed in blocks. Dividing large images into small blocks for processing can reduce memory usage and computational complexity. A reasonable block size and overlap rate are selected. A mixed precision training method is used, and half-precision floating point numbers are used for calculations.

[0023] The attention mechanism is applied to the key layers in the deep learning network. The attention mechanism is used to dynamically adjust the feature weights and dynamically adjust the model's attention to the features according to different environments and scenarios.

[0024] By converting the model weights from floating point numbers to smaller integer formats, removing redundant neurons and connections, and leveraging the multi-core architecture and NEON hardware acceleration of the Raspberry Pi 4B, parallel computing capabilities are optimized.

[0025] A facial deepfake detection and alarm system was built on the PC side through the StreamLit framework. The system achieved real-time communication and feedback between the PC and mobile terminals in different local area networks, ensuring real-time requirements.

[0026] A facial video forgery alarm system based on the MAD method was built, which supports the connection of cameras. For desktop devices, a computer with at least a quad-core CPU and 8GB RAM is required to ensure smooth processing capabilities; the mobile terminal supports Android and IOS devices with cameras and stable network connections; the system is deployed on desktop and mobile terminals within the local area network.

[0027] Compared with the prior art, the invention has the following beneficial effects: 1. Higher detection accuracy: Advantages: By using the EfficientNetv2s network and attention mechanism, the detection accuracy of deep fake face videos is significantly improved.

[0028] Proof data: Test results on experimental datasets show that the detection model based on EfficientNetv2s is significantly superior to traditional methods in terms of accuracy, precision, and recall.

[0029] 2. Real-time and low cost: Advantages: It realizes real-time face detection on embedded devices (such as Raspberry Pi 4B), reduces the hardware cost of the system, and makes it more market-potential and application-valuable.

[0030] Proof data: Experiments conducted on Raspberry Pi 4B show that the system can achieve real-time detection in a low-power environment and meet practical application requirements.

[0031] 3. Multi-platform deployment and scalability: Advantages: The StreamLit framework enables real-time communication and feedback between PC and mobile terminals, supports multi-platform deployment, and expands the application scenarios of the system.

[0032] Proof data: When tested in different LAN environments, the system can stably achieve real-time communication and data transmission between the PC and mobile terminals, demonstrating good robustness and scalability.

[0033] 4. Enhance the security of the authentication system: Advantages: By building a face video forgery alarm system, the security of the face-based identity authentication system is effectively enhanced to prevent potential deep fake fraud.

[0034] Proof data: The face video forgery alarm system based on the MAD method can accurately identify forged videos and issue alarms in time, thereby improving the security level of the identity authentication system. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Other features, objectives and advantages of the present invention will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0036] Figure 1 It is the Matplotlib visualization data graph of the present invention.

[0037] Figure 2 This is a detection diagram without adding the attention mechanism in the present invention.

[0038] Figure 3 This is a detection diagram in which the present invention adds an attention mechanism.

[0039] Figure 4 This is a picture detection diagram of the face video forgery alarm system of the present invention.

[0040] Figure 5 This is a video detection diagram of the face video forgery alarm system of the present invention.

[0041] Figure 6 It is a mobile terminal detection diagram of the present invention.

[0042] Figure 7 This is a mobile terminal detection diagram of the present invention.

[0043] Figure 8 This is a face recognition graph based on the Opencv Harr factor of the present invention.

[0044] Fig. 9 It is the original learning identity map of the present invention.

[0045] Fig.10 It is the face feature map learned by the present invention.

[0046] Fig.11 It is the identification code diagram of the present invention.

[0047] Fig.12 This is the comparison result of the present invention Figure 1 .

[0048] Fig.13 This is the comparison result of the present invention Figure 2 .

[0049] Fig.14 2 is a diagram of different postures of the present invention.

[0050] Fig.15 It is the multi-target overlapping graph of the present invention. DETAILED DESCRIPTION

[0051] The present invention is further described in detail below by examples, which are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0052] A face deep fake detection method based on StreamLit and multi-attention deep fake detection network, comprising: 1. Extract facial features based on EfficientNetv2s network; 2. Use the attention mechanism to weight facial features, enhance the importance of the facial area, and improve the accuracy of deep fake detection of face videos; use the EfficientNetv2s network to extract facial features, and introduce the attention mechanism to weight facial features. This improves the accuracy of deep fake detection of face videos, enhances the model's attention to the importance of the facial area, and significantly improves the detection effect.

[0053] 3. Preprocess the experimental data set to enhance the effect of deep fake detection of face videos; 4. Based on the face detection model of EfficientNetv2s on embedded devices (Raspberry Pi 4B), a real-time face detection system for embedded devices is implemented; The embedded device-side face detection model based on EfficientNetv2s combines lightweight and high efficiency; it realizes a real-time face detection system on embedded devices, improves the application potential and practical value of mobile devices, and reduces hardware costs.

[0054] 5. Deep fake detection model for face videos based on attention mechanism; 6. Build a face video forgery alarm system based on the MAD method; build a face video forgery alarm system using the MAD method based on the attention mechanism-based face video deep forgery detection model; improve the detection ability of high-quality forged videos, expand the detection methods, and enhance the applicability and detection effect of the system.

[0055] 7. Realize face image tampering detection, can be connected to the camera, and expand the detection method; 8. Deploy a local area network to realize the deployment of the system on the desktop and mobile terminals. Real-time communication and feedback system under the StreamLit framework: Build a face deep fake detection and alarm system on the PC side through the StreamLit framework, and realize real-time communication and feedback between the PC and mobile terminals under different local area networks. Enhance the real-time and interactive nature of the system, so that users can obtain the detection results of fake videos in real time, and improve the practicality of the system.

[0056] Deep fake detection of face videos is challenging due to the continuous advancement of video fake technology and the diversity of fake methods. In particular, fake videos may be masked by complex post-processing or adversarial interference, making it more difficult for detection algorithms to find traces of fakes. In order to effectively detect deep fakes of face videos, the main work of the present invention is arranged as follows: (1) The facial features are extracted through the EfficientNetv2s network, and the attention mechanism is used to weight these features, thereby enhancing the importance of the face area and improving the accuracy of forgery detection. On this basis, the dataset is preprocessed to further improve the detection effect.

[0057] (2) We researched and developed a face detection model for embedded devices (Raspberry Pi 4B) based on EfficientNetv2s. The model is lightweight and efficient, and realizes a real-time face detection system on embedded devices. This innovation significantly improves the application potential and practical value of mobile devices.

[0058] (3) By comparing various deep fake detection methods, we studied the face video deep fake detection model based on the attention mechanism and built a face video fake alarm system based on the MAD method. The system realizes face image tampering detection, can be connected to the camera, expands the detection method, and realizes the deployment of the system on the desktop and mobile terminals in the local area network.

[0059] Finally, a face video forgery alarm system was realized, which helps to enhance the security of face-based identity authentication systems and prevent potential face fraud.

[0060] Specific solutions: 1. Improve the efficiency of large-size image training. The existing system is inefficient when training large-size images, resulting in long training time and excessive resource consumption. By compressing and pruning the EfficientNetv2s model, the model parameters and computational complexity are reduced, thereby improving the training speed. The original model training time is 100 hours, the model parameters are 100M, the computational complexity is 500GFLOPS, and the accuracy is 80%. After compression and pruning, the model training time is 60 hours, the model parameters are 50M, the computational complexity is 300GFLOPS, and the accuracy is 79.5%.

[0061] Distributed training. Using distributed computing methods, the training tasks of large-size images are distributed to multiple computing nodes, significantly shortening the training time. The training time for a single node is 120 hours, and the distributed training time is 40 hours (3 nodes), which is a 66.7% reduction in training time. Data preprocessing optimization. Downsampling large-size images and reducing image resolution can reduce the amount of calculation. The moderate downsampling (reducing the resolution by 10%-20%) in the preprocessing of the present invention usually does not significantly reduce the accuracy, while significantly improving the training efficiency. At the same time, large-size images are processed in blocks. Dividing large images into small blocks for processing can reduce memory usage and computational complexity, and selecting a reasonable block size and overlap rate can achieve higher training efficiency while keeping the edge information intact.

[0062] Mixed precision training. Using mixed precision training methods, smaller data types (such as half-precision floating point numbers) are used for calculations to reduce memory usage and calculation time. Half-precision floating point numbers (FP16) can reduce memory usage by about 50% compared to single-precision floating point numbers (FP32), while increasing calculation speed on modern hardware (such as NVIDIA's Tensor Cores), often increasing overall training speed by about 1.5 times.

[0063] 2. Improve robustness. The existing detection system has poor robustness in different environments and conditions, and it is difficult to stably detect fake videos. This paper enhances the robustness of the system in various complex environments by studying a deep fake detection model for face videos based on an attention mechanism. The attention mechanism can dynamically adjust the feature weights, so that the system can still effectively detect fake videos under different lighting, shooting angles and other conditions.

[0064] In this model, the present invention applies the attention mechanism to the key layers in the deep learning network, such as the attention mechanism module in the convolutional neural network (CNN). By introducing the attention mechanism, the model can automatically learn and focus on key features to improve the recognition ability of forged videos. It includes self-attention mechanism (Self-Attention), channel attention mechanism (Channel Attention) or spatial attention mechanism (Spatial Attention) to improve the model's attention to key information in face videos.

[0065] The attention mechanism is used to dynamically adjust feature weights, dynamically adjusting the model's attention to features according to different environments and scenarios. Experimental results show that in complex environments, dynamically adjusting feature weights can greatly improve detection results. In cases of large lighting changes or complex backgrounds, the model can better identify real and fake videos, improving detection accuracy and robustness.

[0066] 3. Reduce hardware dependence. Existing systems require high-performance hardware support, which increases costs and technical barriers, limiting their widespread application. This paper develops an embedded device-side (Raspberry Pi 4B) face detection model based on EfficientNetv2s, combining the advantages of lightweight and high efficiency to realize a real-time face detection system on the embedded device side. This innovation significantly reduces the dependence on high-performance hardware and improves the universality and application potential of the system.

[0067] When deploying the EfficientNetv2s model on the Raspberry Pi 4B, we faced challenges such as computing resource limitations, memory usage, and processing speed. To address these issues, this solution uses a variety of optimization techniques: 1. Model quantization. By converting the model weights from floating point numbers to smaller integer formats, the model size is reduced by about 30%, memory usage is reduced, and the calculation speed is increased by about 50%.

[0068] 2. Model pruning: Removing redundant neurons and connections reduces computational complexity by about 20% and further improves inference efficiency.

[0069] 3. Multi-core processing and hardware acceleration. Take full advantage of the multi-core architecture and NEON hardware acceleration function of the Raspberry Pi 4B to optimize parallel computing capabilities and improve execution efficiency by about 40%.

[0070] Real-time face detection on embedded devices is a technological breakthrough, especially in low-power and limited hardware resource environments. Our system demonstrates excellent performance: the system's face detection accuracy reaches 95% in different test scenarios. In real-time detection, the system is able to maintain a frame rate of 30 FPS to ensure a smooth user experience. The system response time is shortened to 100 milliseconds, significantly improving the performance of real-time interaction. Through the above optimizations and innovations, this invention demonstrates strong application potential and wide applicability in the field of face detection on embedded devices.

[0071] 4. Achieve real-time performance. Existing systems perform poorly in real-time and it is difficult to effectively detect forged videos in real-time applications.

[0072] The present invention builds a facial deepfake detection and alarm system on the PC side through the StreamLit framework. The system realizes real-time communication and feedback between the PC side and the mobile side in different local area networks, ensuring the real-time requirements.

[0073] The StreamLit framework has significant advantages in interface building and data interaction: StreamLit allows developers to quickly build interactive web applications with concise Python code, greatly shortening development time. Its automated real-time update function ensures that the user interface is synchronized with the background data, improving the user experience. StreamLit supports seamless integration of multiple data formats and visualization tools, enhancing the scalability and flexibility of the system.

[0074] When implementing real-time communication under different LANs, the present invention selects WebSocket and HTTP technologies. WebSocket provides two-way communication capabilities, which is suitable for scenarios that require real-time data updates. Its low latency and high efficiency ensure the rapid response of the system. For non-real-time data transmission, we use the HTTP protocol, which is suitable for data return and logging due to its wide support and reliability. This improves the feasibility and efficiency of actual operations.

[0075] 5. Scalability and multi-platform deployment. The existing system has limitations in scalability and cross-platform deployment, making it difficult to meet diverse application needs.

[0076] The present invention builds a face video forgery alarm system based on the MAD method, supports connecting cameras, and expands the detection method. At the same time, it realizes the deployment of the system on the desktop and mobile terminals in the local area network, improving the scalability and multi-platform application capabilities of the system.

[0077] The MAD (Media Authentication and Detection) method is a technology used to detect the authenticity of video content. Its core is to analyze features and anomalies in the video to identify forgery. The MAD method identifies inconsistent or abnormal parts by analyzing multiple features of the video, such as texture, lighting, color distribution, etc. These anomalies may be signs of forgery. And using a deep learning algorithm, the model is trained to learn the characteristic patterns of normal videos and compare the detected features to identify forged content. By analyzing the continuity and consistency between video frames, the MAD method can detect timing inconsistencies caused by forgery, and detect unnatural edges, shadows, or lighting changes in the video, which may be caused by forgery techniques (such as deep forgery), in order to achieve the purpose of detecting forgery.

[0078] In order to realize the effective deployment of the face video forgery alarm system based on the MAD method on the desktop and mobile terminals to ensure the scalability and multi-platform application capabilities of the system, the present invention requires a computer with at least a quad-core CPU and 8GB RAM for the desktop device to ensure smooth processing capabilities; the mobile terminal supports Android and IOS devices with cameras and stable network connections.

[0079] In order to verify the actual application effect of the system on different platforms, the present invention conducted a multi-platform performance evaluation: 1. Desktop performance. Tested on Windows and Linux platforms, the system can maintain a frame rate of up to 30 FPS in real-time detection, ensuring a smooth user experience.

[0080] 2. Mobile performance. On Android and iOS devices, the system’s detection accuracy reached 95%, and the response time was kept within 200 milliseconds.

[0081] In summary, the present invention solves the defects and shortcomings in the prior art through the following specific methods and structures: 1. Based on EfficientNetv2s network and attention mechanism: Improve the accuracy and robustness of deep fake detection in face videos.

[0082] 2. Real-time detection system on embedded devices: reduce hardware dependence and improve the universality and application potential of the system.

[0083] 3. Real-time communication and feedback: ensure the real-time performance of the system in different LANs to meet real-time application requirements.

[0084] 4. Multi-platform deployment and scalability: The counterfeit alarm system is implemented through the MAD method, supporting multi-platform deployment and expansion, and improving the application scope and flexibility of the system.

[0085] These innovative methods and structures significantly improve the performance and practicality of deep fake detection systems for facial videos, help enhance the security of face-based identity authentication systems, and prevent potential facial fraud.

[0086] Based on the EfficientNetv2s network combined with the Haar feature algorithm and the face-recognition method in dlib, the present invention designs and implements a face video forgery alarm system and an embedded face detection system. The system can perform real-time face detection on embedded devices through a 3D depth camera and supports multi-terminal deployment. In order to improve the detection accuracy and efficiency, the attention mechanism is used to enhance the feature extraction capability of the model, and the MAD method is used for deep forgery detection, achieving excellent results. Experimental results show that the detection performance of the system on data sets such as WIDER FACE, Celeb-DF, and DFDC is significantly better than other existing methods, and it also has high robustness and real-time performance. This technology can be effectively applied to face video forgery detection and identity authentication, and is suitable for mobile and embedded device scenarios.

[0087] This invention mainly studies the face detection model based on EfficientNetv2s and the face video deep fake detection method based on the attention mechanism, as well as the application implementation on embedded devices. First, this invention introduces the basic principles of convolutional neural networks and common classification networks, as well as the related concepts and methods of transfer learning and attention mechanisms, providing a theoretical basis for subsequent model design and experiments. Secondly, based on the EfficientNetv2s network, this invention uses transfer learning technology to construct an efficient face detection model, and conducts experiments on the WIDER FACE dataset to verify the accuracy and robustness of the model. At the same time, a face detection system based on EfficientNetv2s is built on the Raspberry Pi 4B development board, and the real-time face detection function is realized using the camera, demonstrating the feasibility and practicality of the model on embedded devices.

[0088] Finally, based on the attention mechanism, this paper introduces a deep fake detection method for face videos. By introducing channel attention and spatial attention modules, the model's sensitivity and discrimination ability to subtle changes in face videos are enhanced, and experiments are conducted on the Celeb-DF dataset to verify the effectiveness and superiority of the method. In addition, a face video fake alarm system is implemented based on the MAD method, and the functions of image detection and webcam detection are designed using the Streamlit framework, demonstrating the feasibility and practicality of the MAD method.

[0089] The difficulty in detecting deep fakes in face videos lies in the rapid development of deep fake technology, which continuously lowers the technical threshold of face-changing methods, improves the authenticity of face-changing effects, and makes subtle changes in face videos difficult to detect. Therefore, deep fake detection in face videos needs to use advanced methods such as deep learning, multimedia forensics, and attention mechanisms to extract spatial features, spatiotemporal features, and biometric features in face videos, enhance the model's sensitivity and ability to distinguish subtle changes in face videos, and improve detection accuracy and robustness.

[0090] 1. Performance, efficiency and robustness evaluation of the face detection model based on EfficientNetv2s on public datasets In order to evaluate the performance, efficiency and robustness of the face detection model based on EfficientNetv2s, this experiment first selected the WIDER FACE dataset as training and test data, divided it into "Easy", "Medium" and "Hard" according to the difficulty of face detection, and performed data preprocessing on it. EfficientNetv2s was used as the basic network, and a feature pyramid network and two prediction branches were added to its last convolutional layer to extract multi-scale feature maps and predict the category and position of the face. This experiment used the Adam optimizer and appropriate hyperparameters to train the model, and used the validation set to evaluate the performance of the model after each training round. Finally, this experiment used the test set to evaluate the performance of the model, and set different input sizes, confidence thresholds and non-maximum suppression (NMS) thresholds to test the efficiency and robustness of the model. The present invention compares the results of the model with other advanced face detection models, and analyzes the advantages and disadvantages of the model.

[0091] Table 1 Table 1 is a comparison of mAP and FPS of different face detection models on the WIDER FACE dataset, where FPS is the number of image frames processed per second, which reflects the speed of the model. As can be seen from Table 1, the face detection model based on EfficientNetv2s is significantly better than other models in terms of mAP, reaching mAP of 0.95, 0.91 and 0.85 at the three difficulty levels of Easy, Medium and Hard, respectively, while the mAP of other models is below 0.8. This shows that the face detection model based on EfficientNetv2s has high accuracy and can effectively detect faces in different scenes. At the same time, the face detection model based on EfficientNetv2s is also better than other models in FPS, reaching 25.6 FPS, while the FPS of other models are all below 20. This shows that the face detection model based on EfficientNetv2s has high efficiency and can process the dataset quickly.

[0092] Therefore, the results of Experiment 1 show that the face detection model based on EfficientNetv2s has excellent performance on the WIDER FACE dataset, which can improve the speed of face detection while maintaining high accuracy, so as to improve the dataset preprocessing efficiency of deep fake detection of face videos.

[0093] 2. Experimental evaluation and visualization analysis of deep fake detection in face videos based on MAD method In order to evaluate the effect of the experiment of the present invention on the deep fake detection task of face video, the present invention uses the precision index for evaluation and compares it with several other existing methods. Table 2 shows the experimental results on the Celeb-DF and DFDC datasets. Through the analysis of the experimental results data, the MAD (Multi-AttentionalDeepfake Detection) method is better than other methods. Therefore, the present invention uses the MAD method as the detector of deep fake face video.

[0094] Table 2 shows the experimental results on the Celeb-DF and DFDC datasets. It can be seen from Table 2 that the MAD method can effectively use the attention mechanism to extract the trace features left over in the deep fake process, and has good generalization ability and performs well on different datasets.

[0095] Table 2 The present invention also visualizes the experimental data in Table 2 by Matplotlib. Figure 1It can be seen that the MAD method selected by the present invention is superior to other algorithms in terms of Precision index.

[0096] 3. Comparative Analysis of the Effectiveness of the MAD Method Based on Attention Mechanism in Face Deep Fake Detection In order to further analyze the effectiveness of the MAD method, the present invention also compares the MAD method by adding and deleting the attention mechanism. The present invention refers to the MAD method without adding the attention mechanism as the NAFD method. The results show that the MAD method with the added attention mechanism is significantly better than the NAFD method without adding the attention mechanism. The experimental results are shown in Table 3 below (experimental results on the DFDC test set). The experiment uses the DCFC test set and adopts the video frame segmentation method, with every five frames being segmented for deep fake detection. The results show that after adding the attention mechanism, the MAD method can more accurately detect deep fake images of faces. The specific detection effect is as follows: Figure 2 , 3 As shown: Table 3 4. Design and application of multi-terminal face video forgery alarm system based on MAD method Based on the MAD method, this paper uses the StreamLit framework to design a face video forgery alarm system that can be applied to mobile and PC terminals. The system can realize multi-device detection in a local area network environment and realize a face video forgery alarm system, such as Figure 4 , 5 , 6, and 7. This system can lower the threshold for detecting deep fake videos. The system can not only detect face images, but also detect them in real time through cameras. Since the local deployment of the system is on a CPU computing device, some network structures are deleted during the model construction process. The next step is to consider deploying it to the server, adding an attention mechanism or improving the backbone network, further improving the system accuracy, and realizing an online face video forgery alarm system.

[0097] Figure 4 In order to detect deep fake face images using the image detection function of the face video forgery alarm system, the result was determined to be false and further verification was recommended.

[0098] Figure 5 In order to detect the webcam detection function of the forged alarm system using face video, data is acquired through the camera and transmitted to the neural network for detection. The detection result is true and the face has not been tampered with.

[0099] Figure 6 , 7For the image detection of the mobile face video forgery alarm system, the image data is uploaded through the mobile phone and transmitted to the neural network for detection. The detection result is true and the face has not been tampered with.

[0100] 5. Design and implementation of embedded face detection system based on Raspberry Pi 4B and EfficientNetv2s The present invention uses the Raspberry Pi 4B single-board computer development version as an embedded device and a 3D depth camera to implement a face detection system on an embedded device. In order to train a face detection model suitable for the device, this work first preprocesses the training data set and uses Python to batch convert the images in the WIDER FACE data set to 224×224 size. Due to the limitation of the K210 display size, too high a resolution will cause the model to fail to load, so a 224×224 image size is used to ensure the smoothness of the detection effect. After the data set is processed, it is input into the face detection algorithm with EfficientNetv2s as the backbone network for training. After obtaining the model, it is converted into the kmodel format to obtain the embedded device face detection model, which is burned into the embedded device, and the machine Haar algorithm is used as a preprocessing step to preliminarily screen the face area, and the face-recognitioin algorithm is used to recognize and verify the face, and finally an embedded device face detection system based on EfficientNetv2s is realized.

[0101] 6. Face detection method and implementation based on Haar features and EfficientNetv2s The present invention is based on the EfficientNetv2s training model, through the Haar feature algorithm, and trains the classifier to finally achieve face detection. The Haar algorithm is a face detection method based on machine learning, which is usually used to quickly detect faces and is used in combination with EfficientNetv2s as a preprocessing step for preliminary screening of face areas.

[0102] Haar feature is a method for describing local features of an image, and is usually used for object detection. Haar feature describes image features based on the sum or difference of grayscale values ​​in different regions, such as the edges and textures of a face. Integral image is used to accelerate calculations. Through the AdaBoost ensemble learning method, a series of weak classifiers (simple classifiers based on Haar features) are iteratively trained and then combined into a strong classifier. Each weak classifier is used to detect a specific feature in the image. The strong classifier gradually filters out non-face areas by cascading multiple weak classifiers. A fixed-size window is slid across the image to be detected, and then each weak classifier is used to determine whether the window contains a face. If the window passes the detection of all weak classifiers, it is considered that the window contains a face (such as Figure 8 as shown).

[0103] 7. Design and robustness verification of face detection and recognition system based on EfficientNetv2s and dlib The present invention is based on the EfficientNetv2s training model and uses the face-recognition method in dlib to perform face detection, face feature extraction and recognition. First, pre-store the original learning identity image (such as Fig. 9 As shown in Figure 2, we use a convolutional neural network (CNN) to extract features from trained faces and learn facial features (such as Fig.10 As shown in the figure), the next step is feature matching, and finally face recognition is achieved. The recognition code is as follows Fig.11 The feature comparison results are shown in Fig.12 , 13 shown.

[0104] To ensure the robustness of the model, the present invention also selects face recognition under different postures and identity recognition under multi-target overlap. The recognition results are as follows: Fig.14 , 15 shown.

[0105] From the experimental results, it can be seen that the face detection model with EfficientNetv2s as the backbone network can better detect faces in close and long distance images, and has high robustness. It can successfully identify facial information under multi-target overlap and different postures. It can be seen that the face detection model based on EfficientNetv2s is still stable when deployed in the Raspberry Pi 4B embedded device.

[0106] The present invention realizes an efficient and real-time face detection system on embedded devices such as Raspberry Pi 4B through a model combining EfficientNetv2s and Haar features, and is successfully applied to the face video forgery detection task. Experimental results show that the MAD method with attention mechanism performs well in forgery detection and can accurately identify deep forged videos and images on different data sets. Then, the face-recognition method based on dlib further improves the face recognition capability of the system, especially maintaining high robustness under multi-target overlap and different postures. The system has the potential for real-time, convenience and multi-terminal applications. In the future, it can be deployed to the server side by further optimizing the model to achieve larger-scale application scenarios and improve overall detection efficiency and accuracy.

[0107] Although the specific embodiments of the present invention are described above, it should be understood by those skilled in the art that this is only for illustration and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A face deep fake detection method based on StreamLit and multi-attention deep fake detection network, characterized in that: The steps include: S1. Extract facial features based on EfficientNetv2s network; S2. Use the attention mechanism to weight facial features, enhance the importance of the face area, and improve the accuracy of deep fake detection of face videos; S3. Preprocess the experimental dataset to enhance the effect of deep fake detection of face videos; S4. Based on the embedded device-side face detection model of EfficientNetv2s, a real-time face detection system is implemented on the embedded device side; S5. Deep fake detection model for face videos based on attention mechanism; S6. Build a face video forgery alarm system based on MAD method; S7. Realizes face image tampering detection, can be connected to a camera, and expands the detection method; S8. Deploy a local area network to implement system deployment on desktop and mobile terminals.

2. The method for detecting deep fake faces based on StreamLit and multiple attention deep fake detection networks according to claim 1, characterized in that: The EfficientNetv2s model is compressed and pruned. The distributed computing method is used to disperse the training tasks of large-size images to multiple computing nodes. Appropriate downsampling is performed for preprocessing to reduce the resolution by 10%-20%. Large-size images are processed in blocks. Dividing large images into small blocks for processing can reduce memory usage and computational complexity. A reasonable block size and overlap rate are selected. A mixed precision training method is used, and half-precision floating point numbers are used for calculations.

3. The face deep fake detection method based on StreamLit and multiple attention deep fake detection network according to claim 2 is characterized in that: The attention mechanism is applied to the key layers in the deep learning network. The attention mechanism is used to dynamically adjust the feature weights and dynamically adjust the model's attention to the features according to different environments and scenarios.

4. The method for detecting deep fake faces based on StreamLit and multiple attention deep fake detection networks according to claim 3, characterized in that: By converting the model weights from floating point numbers to smaller integer formats, removing redundant neurons and connections, and leveraging the multi-core architecture and NEON hardware acceleration of the Raspberry Pi 4B, parallel computing capabilities are optimized.

5. The method for detecting deep fake faces based on StreamLit and multiple attention deep fake detection networks according to claim 4, characterized in that: A facial deepfake detection and alarm system was built on the PC side through the StreamLit framework; the system realized real-time communication and feedback between the PC and mobile terminals under different local area networks, ensuring real-time requirements.

6. The method for detecting deep fake faces based on StreamLit and multiple attention deep fake detection networks according to claim 5, characterized in that: A face video forgery alarm system based on the MAD method was built. It supports connecting cameras. For desktop devices, a computer with at least a quad-core CPU and 8GB RAM is required. The mobile device supports Android and IOS devices with cameras and stable network connections.

Images