Lightweight real-time face de-recognition privacy protection method and system
Through lightweight real-time face detection and tracking technology, combined with twin networks and StyleGAN, virtual face images are generated, which solves the problem of image quality and privacy protection balance in face data privacy protection in civil aviation travel scenarios, and achieves efficient privacy Protection and enhancement of reality.
Patent Information
- Application Number
- CN202411170508.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-29
AI Technical Summary
In the privacy protection of face data in civil aviation travel scenarios, the occlusion method affects the beauty of the image, the feature point removal method requires high precision, and pixelation affects clarity and recognizability, making it difficult to find between privacy protection and image quality. balance.
Using lightweight real-time face detection, face tracking technology and twin network, face detection and tracking are performed through MediaPipe framework and SSD model, features are extracted using feature map pyramid network and SiameseFC algorithm, and virtual face images are generated in combination with StyleGAN to cover Or replace the original face area.
It improves the comprehensive performance of face recognition and privacy protection, ensures the realism of images and the security of privacy protection, improves the robustness and applicability of the algorithm, and is suitable for large-scale facial data processing.
Smart Images

Figure CN119049111A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure belongs to the field of information security protection, and in particular, relates to a lightweight real-time face de-identification privacy protection method and system. Background Art
[0002] With the development and application of artificial intelligence technology, all civil aviation operation scenarios have been affected. While ensuring the intelligence and convenience of travel, it is crucial to ensure the network security of civil aviation passengers in travel scenarios. In this context, facial data privacy protection technology came into being.
[0003] Facial data privacy protection refers to processing facial images or videos to eliminate or blur personal identity information, thereby protecting personal privacy. This technology is particularly important in situations involving large-scale use of facial data, such as airport security checks and self-service boarding. Common methods for achieving facial data privacy protection include blurring, occlusion, feature point removal, pixelation, etc. The occlusion method can effectively protect personal privacy and prevent faces from being recognized, but it may also affect the beauty and usability of the image. The feature point removal method can effectively protect privacy, but it requires accurate identification and removal of key feature points, so it has high requirements for the accuracy and reliability of the algorithm. Pixelation can protect privacy to a certain extent, but it may affect the clarity and recognizability of the image, so it is necessary to balance privacy protection and image quality. The generative adversarial network (GAN) method can protect personal privacy to a certain extent while maintaining the realism of the image, but it requires high-quality training data and complex algorithm models. Summary of the invention
[0004] To solve the above problems, the present disclosure provides a lightweight real-time face de-identification and privacy protection method and system suitable for civil aviation travel scenarios, which adopts lightweight real-time face detection, face tracking technology, twin network to solve occlusion problems and face de-identification technology, and can improve the comprehensive performance of face recognition and privacy protection.
[0005] In order to achieve the above object, the present invention adopts the following technical scheme.
[0006] A lightweight real-time face de-identification privacy protection method, comprising:
[0007] Perform face detection on the first frame of the video and locate the face area;
[0008] Detect the face area and crop to generate an accurate face area selection box as the target template image;
[0009] Extract key points from the target template image to generate face key points and face existence probability; generate the smallest rectangular selection box based on the coordinates of the face key points;
[0010] Extract features from the target template image and the search area of the current frame, calculate the similarity, and generate a response map;
[0011] By finding the maximum value in the response graph, the face position in the current frame is determined and the face trajectory information is updated for prediction and matching of the next frame.
[0012] Repeat the above method to track the face in each frame until the video ends or the face is lost;
[0013] Based on the tracked face position and key point coordinates, a virtual face image is generated to cover or replace the face area in the original video or image.
[0014] Furthermore, performing face detection on the first frame of the video includes:
[0015] The lightweight face detection algorithm based on the MediaPipe framework performs face detection on the first frame of the video and locates the facial area, including the front and side faces of the person.
[0016] Furthermore, the detecting of the facial area includes:
[0017] The facial region detection model SSD is used to detect the facial region and crop to generate an accurate facial region selection box as the target template image;
[0018] The encoding-decoding feature extractor of the feature graph pyramid network FPN is used to fuse multi-scale face features and enhance the accuracy of face detection.
[0019] Furthermore, a flow limiter is added before the facial area detection, and the flow limiter is used to limit the speed of image input and improve the real-time performance of the detection.
[0020] Furthermore, extracting key points from the target template image includes:
[0021] Use the facial key point extraction model in the MediaPipe framework to extract key points from the target template image and generate facial key points and facial existence probability.
[0022] Furthermore, the facial key points are 468 facial key point coordinates; the coordinates are composed of x, y, and z, where x and y represent pixel coordinates of the key points on the two-dimensional image, and z represents the relative depth of each key point.
[0023] Furthermore, the face existence probability is the probability of a face appearing in the rectangular selection box; if the output face existence probability is lower than a set threshold, the face region detection model SSD is automatically called for re-tracking; if the output face existence probability is higher than the set threshold, the rectangular selection box replaces the face region selection box and is input into the face key point extraction model as the face selection box for the next frame.
[0024] Furthermore, the feature extraction of the target template image and the search area of the current frame includes:
[0025] The SiameseFC algorithm is used to extract features from the target template image and the search area of the current frame.
[0026] Furthermore, generating a virtual face image according to the face position and key point coordinates obtained by tracking includes:
[0027] Based on the tracked face position and key point coordinates, StyleGAN is used to generate a virtual face image.
[0028] A lightweight real-time face de-identification privacy protection system, comprising:
[0029] Face detection module: used to detect faces in the first frame of the video and obtain the face position and key point coordinates;
[0030] Face tracking module: used to track the face of each frame of the video according to the face position and key point coordinates obtained by the face detection module;
[0031] Face de-identification module: used to de-identify faces in videos that have completed detection and tracking.
[0032] Compared with the prior art, the present invention has the following advantages:
[0033] (1) Lightweight real-time face detection: We use lightweight real-time face detection based on the MediaPipe framework and use the single-shot detector model SSD to perform initial position detection on the face area to achieve an efficient face detection process.
[0034] (2) Face tracking technology: By predicting the boundary box from the coordinates of the facial key points in the previous frame as the input of the current frame, the face area detection model is avoided from being reused on each frame, thereby improving the accuracy and efficiency of face tracking.
[0035] (3) Twin network solves the occlusion problem: The twin network based on spatiotemporal context information can effectively solve the recognition problem when the face is occluded, enhancing the robustness and applicability of the algorithm in practical applications.
[0036] (4) Face de-identification technology: StyleGAN is used to generate fake face information, which protects the privacy of real face data while ensuring the security and privacy protection of the system, meeting the requirements of data privacy protection. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0038] Figure 1 A flowchart of a lightweight real-time face de-identification privacy protection method provided by an embodiment of the present invention is shown;
[0039] Figure 2 A schematic diagram of detecting key points of a side face and a front face provided by an embodiment of the present invention is shown;
[0040] Figure 3 A diagram showing the architecture of a face region detection model provided by an embodiment of the present invention is shown;
[0041] Figure 4 A diagram showing the architecture of a facial key point extraction model provided by an embodiment of the present invention is shown;
[0042] Figure 5 A flow chart of face detection provided by an embodiment of the present invention is shown;
[0043] Figure 6 A flow chart of a face tracking algorithm provided by an embodiment of the present invention is shown;
[0044] Figure 7 A schematic diagram of the structure of a lightweight real-time face de-identification privacy protection system provided by an embodiment of the present invention is shown;
[0045] In the figure, 1. Face selection box, 2. Key point coordinates. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0047] In one embodiment of the present invention, a lightweight real-time face de-identification privacy protection method is proposed, such as Figures 1 to 6 As shown, including:
[0048] A lightweight real-time face de-identification privacy protection method according to an embodiment of the present disclosure includes:
[0049] A lightweight face detection algorithm based on the MediaPipe framework performs face detection on the first frame of the video and locates the facial area, including the front and side faces of the person. A flow limiter is added before facial area detection to limit the speed of image input and improve the real-time performance of detection.
[0050] The facial region detection model SSD (Single Shot Detector) is used to detect the facial region and crop to generate an accurate facial region selection box as the target template image.
[0051] The encoding-decoding feature extractor of the Feature Pyramid Networks (FPN) is used to fuse multi-scale facial features and enhance the accuracy of face detection.
[0052] The facial key point extraction model in the MediaPipe framework is used to extract key points from the target template image to generate facial key points and facial existence probabilities; the smallest rectangular selection box is generated based on the key point coordinates; the facial key points are 468 facial key point coordinates; the coordinates are composed of x, y, and z, where x and y represent the pixel coordinates of the key points on the two-dimensional image, and z represents the relative depth of each key point; the facial existence probability is the probability of the face appearing in the rectangular selection box; if the output facial existence probability is lower than the set threshold, the facial region detection model SSD is automatically called for re-tracking; if the output facial existence probability is higher than the set threshold, the rectangular selection box replaces the facial region selection box and is input into the facial key point extraction model as the facial selection box for the next frame.
[0053] The SiameseFC algorithm is used to extract features from the target template image and the search area of the current frame, and the similarity is calculated to generate a response map, which includes:
[0054] The upper branch represents the target template image I ref , the lower branch represents the area I to be searched in the current image frame t ; will I ref and I t Input into two sub-networks respectively for feature extraction; after the same feature mapping operation F, the original image is mapped to the feature space F ref =CNN(I ref ) and F t =CNN(It ), and obtain the feature vector with the same number of channels;
[0055] After the convolution operation*, the response map F is obtained, and the value of each position represents the similarity S between different target candidate images and the target template image. The similarity calculation formula adopts cosine similarity, as shown in formula (1).
[0056]
[0057] By finding the maximum value in the response graph, the face position in the current frame is determined and the face trajectory information is updated for prediction and matching of the next frame. Specifically:
[0058] The maximum similarity S is selected to determine the most similar target candidate area. Assuming the maximum value is (x, y), the center position of the target can be expressed as: (x target ,y target ) = (x + Δx, y + Δy). Where Δx and Δy are the offsets of the maximum value position relative to the center of the feature map.
[0059] In order to train the model, it is necessary to define an appropriate loss function to measure the tracking accuracy. The mean square error loss is used as shown in formula (2), where (x gt ,y gt ) is the actual position of the target in the current frame.
[0060] L=||(x gt -x target ), (y gt -y target )|| 2 (2)
[0061] Repeat the above method to track the face in each frame until the video ends or the face is lost;
[0062] Based on the tracked face position and key point coordinates, StyleGAN is used to generate a virtual face image to cover or replace the face area in the original video or image.
[0063] The embodiment of the present invention also provides a lightweight real-time face de-identification privacy protection system, such as Figure 7 As shown, including:
[0064] Face detection module: used to detect faces in the first frame of the video and obtain the face position and key point coordinates;
[0065] Face tracking module: used to track the face of each frame of the video according to the face position and key point coordinates obtained by the face detection module;
[0066] Face de-identification module: used to de-identify faces in videos that have completed detection and tracking.
[0067] Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.
Claims
1. A lightweight real-time face de-identification privacy protection method, characterized in that: The method comprises: Perform face detection on the first frame of the video and locate the face area; Detect the face area and crop to generate an accurate face area selection box as the target template image; Extract key points from the target template image to generate face key points and face existence probability; generate the smallest rectangular selection box based on the coordinates of the face key points; Extract features from the target template image and the search area of the current frame, calculate the similarity, and generate a response map; By finding the maximum value in the response graph, the face position in the current frame is determined and the face trajectory information is updated for prediction and matching of the next frame. Repeat the above method to track the face in each frame until the video ends or the face is lost; Based on the tracked face position and key point coordinates, a virtual face image is generated to cover or replace the face area in the original video or image.
2. A lightweight real-time face de-identification privacy protection method according to claim 1, characterized in that: The performing face detection on the first frame of the video includes: The lightweight face detection algorithm based on the MediaPipe framework performs face detection on the first frame of the video and locates the facial area, including the front and side faces of the person.
3. A lightweight real-time face de-identification privacy protection method according to claim 1, characterized in that: The detecting of the facial area comprises: The facial region detection model SSD is used to detect the facial region and crop to generate an accurate facial region selection box as the target template image; The encoding-decoding feature extractor of the feature graph pyramid network FPN is used to fuse multi-scale face features and enhance the accuracy of face detection.
4. A lightweight real-time face de-identification privacy protection method according to claim 1, characterized in that: A flow limiter is added before the facial area detection, and the function of the flow limiter is to limit the speed of image input and improve the real-time performance of the detection.
5. A lightweight real-time face de-identification privacy protection method according to claim 1, characterized in that: Extract key points from the target template image, including: Use the facial key point extraction model in the MediaPipe framework to extract key points from the target template image and generate facial key points and facial existence probability.
6. A lightweight real-time face de-identification privacy protection method according to claim 1, characterized in that: The facial key points are 468 facial key point coordinates; the coordinates are composed of x, y, and z, where x and y represent the pixel coordinates of the key points on the two-dimensional image, and z represents the relative depth of each key point.
7. A lightweight real-time face de-identification privacy protection method according to claim 1, characterized in that: The face existence probability is the probability of a face appearing in the rectangular selection box; if the output face existence probability is lower than the set threshold, the face area detection model SSD is automatically called for re-tracking; if the output face existence probability is higher than the set threshold, the rectangular selection box replaces the face area selection box and is input into the face key point extraction model as the face selection box of the next frame.
8. A lightweight real-time face de-identification privacy protection method according to claim 1, characterized in that: The feature extraction of the target template image and the search area of the current frame includes: The SiameseFC algorithm is used to extract features from the target template image and the search area of the current frame.
9. A lightweight real-time face de-identification privacy protection method according to claim 1, characterized in that: The step of generating a virtual face image according to the face position and key point coordinates obtained by tracking includes: Based on the tracked face position and key point coordinates, StyleGAN is used to generate a virtual face image.
10. A lightweight real-time face de-identification privacy protection system, characterized in that: include: Face detection module: used to detect faces in the first frame of the video and obtain the face position and key point coordinates; Face tracking module: used to track the face of each frame of the video according to the face position and key point coordinates obtained by the face detection module; Face de-identification module: used to de-identify faces in videos that have completed detection and tracking.
Citation Information
Patent Citations
Real-time video stream face detection tracking method and detection tracking system
CN110991287A
Monitoring video pedestrian recognition and tracking method and device and storage medium
CN112257502A
In-vehicle video face image desensitization method and device based on generative adversarial network
CN118247824A
