A method and system for facial privacy protection

Abstract

The application discloses a face privacy protection method and system, and relates to the technical field of face detection, comprising: acquiring a patient's medication video and extracting video frames; selecting a mask and marking mask key points; performing face key point detection and selection on each video frame, and performing detailed enhancement on the key point detection and selection through a human body posture detection algorithm to obtain face key points; and performing triangular affine transformation on the face key points and the mask key points to add a mask to the face of the video frame. The application can provide effective privacy protection for patients while not affecting the auditing and management efficiency of the medication video. By combining the FaceMesh and DWPose algorithms, as well as the Delaunay triangulation and triangular affine transformation technologies, it is ensured that the mask can be accurately overlaid on the face, and good privacy protection effect can be maintained even in complex video scenes, ensuring the reliability of privacy protection.

Description

Technical Field

[0001] This invention relates to the field of face detection technology, and more specifically to a method and system for protecting facial privacy. Background Technology

[0002] Tuberculosis (TB) is one of the top ten causes of death worldwide, exhibiting characteristics of both a chronic and infectious disease. The ongoing spread of TB is a major global public health issue. The current public health service model for TB patients is centered on DOTS (Directly Observed Treatment, Short-course), primarily involving patients visiting community health centers or specialized TB institutions, supplemented by home visits and telephone follow-ups by medical staff. While this approach achieved some success in the early stages of TB control, it increased the time, energy, and psychological burden on patients, placing a heavy workload on primary healthcare workers. This approach has gradually revealed a series of drawbacks hindering control effectiveness: extremely low actual DOTS implementation rates, widespread self-medication and irregular medication practices, frequent missed doses, and delayed detection of adverse drug reactions lead to low cure rates, relapses, and even drug resistance. In recent years, to reduce the drawbacks of DOTS and improve management efficiency, Video-Observed Therapy (VOT) has gradually become the mainstream innovative management method for TB patients.

[0003] While the VOT (Video Over-the-Counter) management model has reduced the workload of primary healthcare institutions to some extent, the shortage of human resources directly affects the quality of tuberculosis patient health management, making it difficult to guarantee the timeliness, effectiveness, and even authenticity of the implementation of tuberculosis patient health management. The current VOT management model still falls short of achieving high-quality health management for patients. Furthermore, researchers who implemented VOT management for pulmonary tuberculosis patients for six years, collecting millions of medication videos, found widespread issues of untimely review of medication videos by healthcare staff and unstable quality control. The rapid development of Artificial Intelligence (AI) technology and its widespread application in medical services and health management provide strong support for the intelligent review of medication videos in medical services. Applying AI technology can upgrade the VOT management model, adopting a human-machine collaborative medication review model with intelligent review as the primary method and manual review as a supplement. This is expected to further reduce the workload of primary healthcare staff, improve the quality control of tuberculosis patient medication management, and enhance the timeliness of medication management.

[0004] The VOT (Video Over-the-Counter) management model for tuberculosis patients has proven highly effective, but a significant problem remains: it inevitably requires the direct collection of sensitive personal information from patients, including facial images and the recording environment. If this information is leaked, it can easily be used for illegal activities such as identity theft, hacking financial accounts, and tracking movements, thereby infringing upon patients' personal dignity or endangering their personal and property safety. Consequently, patients' awareness of privacy protection has gradually increased. In the early stages of VOT implementation in 2017, the full upload rate of patient medication videos was 63.4%, while in 2021, the full upload rate dropped to 46.4%.

[0005] Therefore, how to propose a facial privacy protection method and system to avoid the leakage of users' facial and other feature information under the VOT management model and ensure users' privacy and security is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a facial privacy protection method and system to avoid the leakage of users' facial and other feature information under the VOT management mode, and to ensure users' privacy and security. To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for protecting facial privacy includes:

[0008] Obtain videos of patients taking medication and extract video frames;

[0009] Select a mask and mark its key features;

[0010] Facial key points are detected and selected for each video frame, and then the key points are enhanced with details by human pose detection algorithm to obtain facial key points.

[0011] A affine transformation based on facial and mask key points is used to add a mask to the face in the video frame.

[0012] Optionally, the step of detecting and selecting facial landmarks for each video frame includes:

[0013] Facial key points are selected from the eyes, nose, mouth, eyebrows, jawline, and facial contour features.

[0014] Facial landmark detection is performed on each video frame using the FaceMesh algorithm in MediaPipe.

[0015] Optionally, selecting a mask and annotating its key points includes:

[0016] A suitable face mask is selected, and the key points of the mask are fixed. After the mask is selected, the key points are marked manually, and the key point markings are matched with the key points of the selected face.

[0017] Optionally, the step of obtaining facial key points by detecting and selecting key points using a human pose detection algorithm and then performing detail enhancement includes:

[0018] The FaceMesh algorithm is used to detect facial landmarks;

[0019] When the FaceMesh algorithm fails to detect a face, the DWPose pose detection algorithm is activated.

[0020] The DWPose pose detection algorithm is used to estimate the pose of video frames in which no face is detected, and key points are obtained.

[0021] By transforming keypoints, the keypoints obtained by the DWPose pose detection algorithm are converted into keypoints selected by the FaceMesh algorithm.

[0022] Optionally, the step of converting the key points obtained by the DWPose pose detection algorithm into key points selected by the FaceMesh algorithm through key point transformation includes:

[0023] Based on the video frame key points that can be obtained by both the DWPose pose detection algorithm and the FaceMesh algorithm, a mapping relationship is established by comparing the positional relationship of the key points of the two algorithms.

[0024] The key points obtained by the DWpose pose detection algorithm are mapped to the corresponding FaceMesh key points using a mapping relationship.

[0025] The mapping relationship includes coordinate translation or interpolation using multiple DWPose keypoints;

[0026] The key points obtained by the DWPose pose detection algorithm are transformed into key points selected by the FaceMesh algorithm.

[0027] Optionally, it also includes: converting the key points obtained by the DWPose pose detection algorithm into key points selected by the FaceMesh algorithm and then performing key point smoothing correction, and performing smoothing correction on the outermost contour points of the key points.

[0028] Optionally, the smoothing correction of the outermost contour points of the key points includes:

[0029] Each facial contour point is compared with its clockwise and counterclockwise adjacent contour points: Let the target point be A. i The midpoint of the two adjacent contour points is B. iThe distance between the corresponding adjacent contour points is d. i Calculate all contour points A i With B i Distance between points and d i The ratio |A i B i | / d i ;

[0030] If the ratio of contour points exceeds the ratio threshold, then the point A with the largest ratio will be removed. x Move to the corresponding B x The algorithm is iterated over and re-applied until the proportion of all points does not exceed the threshold, or the maximum number of iterations is reached.

[0031] Optionally, the step of applying a mask to the face in the video frame by performing a triangular affine transformation based on facial key points and mask key points includes:

[0032] Obtain facial key points and corresponding mask key points;

[0033] Calculate the Delaunay triangulation for key points of the mask;

[0034] Based on the key point connection method of the mask, the Delaunay triangulation is calculated in the same way on the facial key points of the video frame;

[0035] A affine transformation is used to map the triangles of key points on the mask to the triangles of key points on the face in the video frame, thus covering the face with the mask.

[0036] Optionally, the triangular affine transformation includes: encoding translation, scaling, rotation, and shearing, constructing a 2x3 transformation matrix, with the first two columns encoding rotation, scaling, and shearing, and the last column encoding translation.

[0037] Optionally, a facial privacy protection system includes:

[0038] Acquisition module: Used to acquire videos of patients taking medication and extract video frames;

[0039] Annotation module: Used to select a mask and annotate its key points;

[0040] Detection and selection module: used to detect and select facial key points in each video frame, and to enhance the details of the key point detection and selection through human pose detection algorithm to obtain facial key points;

[0041] Transformation module: Used to perform triangular affine transformation based on facial key points and mask key points to add a mask to the face of the video frame.

[0042] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a facial privacy protection method and system, which has the following beneficial effects:

[0043] This invention proposes a facial privacy protection method, comprising: acquiring a patient's medication video and extracting video frames; selecting a mask and annotating mask key points; detecting and selecting facial key points for each video frame, and enhancing the details of the key point detection and selection using a human pose detection algorithm to obtain facial key points; and applying a mask to the face in the video frame by performing a triangular affine transformation based on the facial key points and mask key points. This invention can provide effective privacy protection for patients without affecting the review and management efficiency of medication videos, and has significant practical application value. By combining FaceMesh and DWPose algorithms, as well as Delaunay triangulation and triangular affine transformation techniques, it ensures that the mask can accurately cover the face, maintaining good privacy protection even in complex video scenes; it can adapt to different video quality and changes in the patient's pose during medication, ensuring the reliability of privacy protection. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of a facial privacy protection method provided by the present invention.

[0046] Figure 2 This is a schematic diagram of a facial privacy protection system provided by the present invention.

[0047] Figure 3 This is a schematic diagram of a facial privacy protection device provided by the present invention.

[0048] Figure 4 This is a schematic diagram of a facial key point detection module provided by the present invention.

[0049] Figure 5 This is a structural schematic diagram provided for the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] This invention discloses a method for protecting facial privacy, such as... Figure 1 As shown, it includes:

[0052] Obtain videos of patients taking medication and extract video frames;

[0053] Select a mask and mark its key features;

[0054] Facial key points are detected and selected for each video frame, and then the key points are enhanced with details by human pose detection algorithm to obtain facial key points.

[0055] A affine transformation based on facial and mask key points is used to add a mask to the face in the video frame.

[0056] Furthermore, the process of detecting and selecting facial landmarks for each video frame includes:

[0057] Facial key points are selected from the eyes, nose, mouth, eyebrows, jawline, and facial contour features.

[0058] Facial landmark detection is performed on each video frame using the FaceMesh algorithm in MediaPipe.

[0059] In a specific embodiment, based on the patient's recorded medication-taking video, video frames are extracted, and facial landmark detection is performed on each video frame using the FaceMesh algorithm from MediaPipe. The FaceMesh algorithm detects 468 facial landmarks, containing rich facial features. However, matching a mask to a face does not require such refined features. The selected landmarks include key facial features such as eyes, nose, mouth, eyebrows, jawline, and facial contours. 75 of the 468 landmarks were selected as the final facial landmarks to be acquired. These 75 landmarks are as follows: Figure 5 As shown. 0-16 is the jawline. 17-21 is the left eyebrow, 22-26 is the right eyebrow, 27-35 is the bridge of the nose and nose, 36-41 is the left eye, 42-47 is the right eye, 48-59 is the outer contour of the mouth, 60-67 is the inner contour of the lips, and 68-74 is the forehead contour.

[0060] Furthermore, the process of selecting a mask and annotating its key points includes:

[0061] A suitable face mask is selected, and the key points of the mask are fixed. After the mask is selected, the key points are manually marked, and these key point markings are matched with the selected facial key points. Specifically, they are matched with the 75 key points determined above.

[0062] Furthermore, the step of obtaining facial key points by detecting and selecting key points using a human pose detection algorithm and then performing detail enhancement includes:

[0063] The FaceMesh algorithm is used to detect facial landmarks;

[0064] When the FaceMesh algorithm fails to detect a face, the DWPose pose detection algorithm is activated.

[0065] The DWPose pose detection algorithm is used to estimate the pose of video frames in which no face is detected, and key points are obtained.

[0066] By transforming keypoints, the keypoints obtained by the DWPose pose detection algorithm are converted into keypoints selected by the FaceMesh algorithm.

[0067] In a specific embodiment, due to the unique nature of medication-taking videos, the act of swallowing involves head and hand movements, as well as facial obstruction. When taking medication, the patient needs to swallow the medicine along with water, inevitably obscuring the face. This can lead to recognition errors and omissions when using the basic FaceMesh algorithm to detect facial landmarks. Therefore, a human pose detection algorithm, utilizing the relative position of the body, can reduce these omissions. The specific operation is as follows:

[0068] S1: Human pose estimation.

[0069] Using the DWPose pose detection algorithm, pose prediction is performed on the input video frames, returning keypoint locations and corresponding confidence scores. Then, the keypoint confidence scores are checked. If the average confidence score is below a threshold, it is considered that no person or pose has been detected, and subsequent calculations and mask application are skipped. Otherwise, subsequent operations continue.

[0070] DWPose is an open-source human pose detection algorithm. For video frames where FaceMesh fails to detect faces due to reasons such as water cup obstruction, DWPose is used to supplement the FaceMesh algorithm when no face is detected. DWPose obtains the detection key points and their corresponding confidence scores. A threshold is predefined. If the mean of all key points does not exceed the threshold, the detection result of DWPose is considered unreliable, and no person or pose is detected. No mask is added to this video frame. If the mean of all key points is greater than the threshold, subsequent key point transformation operations are performed.

[0071] S2: Keypoint Transformation.

[0072] The facial keypoints in DWPose are quite similar to those selected by the FaceMesh algorithm. Therefore, the positions of 1-3 similar DWPose keypoints can be used to approximate the FaceMesh keypoints by considering the positional relationships between the points.

[0073] The locations and number of facial keypoints acquired by DWPose are not entirely identical to those selected by the FaceMesh algorithm, but they share a high degree of similarity. First, based on the keypoints from video frames that can be acquired by both methods, a mapping relationship is established by comparing the positional relationships of the keypoints. Then, this mapping relationship is used to map the DWPose keypoints to their corresponding FaceMesh keypoints. This mapping relationship can involve simple coordinate translation or interpolation using multiple DWPose keypoints. Through this keypoint transformation, the keypoints obtained by DWPose are converted into the keypoints selected by the FaceMesh algorithm, enabling accurate subsequent mask matching.

[0074] S3: Keypoint Smoothing Correction. Furthermore, due to occasional DWPose recognition errors leading to mask deformation observed during the experiment, smoothing correction is required for the outermost contour points of the keypoints after keypoint conversion. In the smoothing correction, each facial contour point is compared with its clockwise and counterclockwise adjacent contour points: Let the target point be A. i The midpoint of the two adjacent contour points is B. i The distance between the corresponding adjacent contour points is d. i Calculate all contour points A i With B i Distance between points and d i The ratio |A i B i | / d i If the ratio exceeds the threshold, then the point A with the largest ratio will be... x Move to the corresponding B xThe algorithm is iterated over and re-applied until the proportion of all points does not exceed the threshold, or the maximum number of iterations is reached.

[0075] Furthermore, the step of applying a mask to the face in the video frame by performing a triangular affine transformation based on facial key points and mask key points includes:

[0076] Obtain facial key points and corresponding mask key points;

[0077] Calculate the Delaunay triangulation for key points of the mask;

[0078] Based on the key point connection method of the mask, the Delaunay triangulation is calculated in the same way on the facial key points of the video frame;

[0079] A affine transformation is used to map the triangles of key points on the mask to the triangles of key points on the face in the video frame, thus covering the face with the mask.

[0080] The triangular affine transformation includes encoding translation, scaling, rotation, and shearing, constructing a 2x3 transformation matrix. The first two columns of the matrix encode rotation, scaling, and shearing, and the last column encodes translation.

[0081] In a specific implementation, the steps for adding a mask to the face in the video frame are as follows:

[0082] After obtaining the facial key points and the corresponding key points of the mask;

[0083] S11: Calculate the Delaunay triangulation for the key points of the mask. For a set of scattered points on a plane, find a way to connect these points to form a series of triangles; this is called triangulation. Among all possible triangulations, the Delaunay triangulation has a special property: for each triangle in the triangulation, its circumcircle does not contain any other points.

[0084] S22: Based on the key point connection method of the mask, calculate the Delaunay triangulation of the facial key points in the video frame in the same way.

[0085] S33: A affine transformation is used to map the triangles of the mask's key points to the triangles of the face's key points in the video frame, thus attaching the mask to the face. The affine transformation encodes translation, scaling, rotation, and shearing. It is a 2x3 matrix; the first two columns encode rotation, scaling, and shearing, and the last column encodes translation.

[0086]

[0087] `a` and `d` typically control scaling and rotation. In pure scaling, `a` and `d` represent the scaling factors in the x and y directions, respectively. In rotational transformations, `a` = cos(θ) and `d` = cos(θ), where θ is the rotation angle. `b` and `c` control tilting and rotation. In pure tilting transformations, `b` and `c` represent the shearing effects of x on y and y on x, respectively. In rotational transformations, `b` = -sin(θ) and `c` = sin(θ). x and t y These are translation parameters, representing the translation amounts in the x and y directions, respectively.

[0088] Specifically, given a point (x, y), it can be moved to the point (x, y) using the transformation matrix S and the equation given below. t ,y t ),

[0089]

[0090] Specifically, practical use requires the eyes and mouth to be visible. The eyes are visible to allow for early warning of drug-induced liver injury through scleral icterus, and the mouth is visible to identify medication use. Therefore, the areas enclosed by the eyes and mouth in the mask image—specifically, the left eye (36-41), the right eye (42-47), and the outer contour of the mouth (48-59)—are set to fully transparent.

[0091] In a specific implementation, a facial privacy protection system, such as Figure 2 As shown, it includes:

[0092] Acquisition module: Used to acquire videos of patients taking medication and extract video frames;

[0093] Annotation module: Used to select a mask and annotate its key points;

[0094] Detection and selection module: used to detect and select facial key points in each video frame, and to enhance the details of the key point detection and selection through human pose detection algorithm to obtain facial key points;

[0095] Transformation module: Used to perform triangular affine transformation based on facial key points and mask key points to add a mask to the face of the video frame.

[0096] In a specific implementation, a facial privacy protection method involves dressing the face in a patient's medication video. This is achieved by acquiring a series of key points for both the face and the mask. However, due to the nature of medication videos, rotation and occlusion can cause omissions and errors in simple facial key point detection. Therefore, a human pose detection algorithm is used to reduce omissions by leveraging the relative position of the body. The facial key points obtained from human pose detection are linearly transformed into predetermined facial key points. Then, Delaunay triangulation is calculated on the mask and video frame key points using the same connection method. Finally, a triangular affine transformation is used to map and superimpose the mask onto the face, completing the mask-wearing operation.

[0097] In specific implementation methods, such as Figure 3 As shown, a facial privacy protection device includes: an input, a facial key point detection module, and a facial key point step connected in sequence. The facial key points are combined with a mask bearing these key points and subjected to a triangular affine transformation to achieve mask-wearing. For acquired patient medication videos, a mask is placed on the face in the video. This is achieved by detecting facial key points in the video and labeling the mask with the same series of key points. Then, Delaunay triangulation is calculated on the mask and facial key points using the same connection method. Finally, the mask is mapped and superimposed onto the face using a triangular affine transformation, thus completing the mask-wearing operation.

[0098] In specific implementation methods, such as Figure 4 As shown, a facial landmark detection module addresses the issue of omissions and errors in acquiring facial landmarks due to the unique characteristics of medication videos, including rotation and occlusion. Simple FaceMesh detection may miss or erroneously capture these landmarks. To mitigate this, the Human Pose Detection (DWPose) algorithm is used. By leveraging the relative position of the body, omissions can be reduced. The facial landmarks obtained through DWPose detection are linearly transformed and smoothed to become predetermined facial landmarks. If DWPose fails to detect a human pose, it is considered that no face exists in the video frame, and the original video frame is output without proceeding with the subsequent mask-wearing process.

[0099] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0100] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for protecting facial privacy, characterized in that, include: Obtain videos of patients taking medication and extract video frames; Select a mask and mark its key features; Facial key points are detected and selected for each video frame, and then the key points are enhanced with details by human pose detection algorithm to obtain facial key points. The process of detecting and selecting facial landmarks for each video frame includes: Facial key points are selected from the eyes, nose, mouth, eyebrows, jawline, and facial contour features. Facial landmark detection is performed on each video frame using the FaceMesh algorithm in MediaPipe. The process of obtaining facial key points through key point detection and selection using a human pose detection algorithm to enhance details includes: The FaceMesh algorithm is used to detect facial landmarks; When the FaceMesh algorithm fails to detect a face, the DWPose pose detection algorithm is activated. The DWPose pose detection algorithm is used to estimate the pose of video frames in which no face is detected, and key points are obtained. By transforming key points, the key points obtained by the DWPose pose detection algorithm are converted into key points selected by the FaceMesh algorithm; The process of converting key points obtained by the DWPose pose detection algorithm into key points selected by the FaceMesh algorithm through key point transformation includes: Based on the video frame key points that can be obtained by both the DWPose pose detection algorithm and the FaceMesh algorithm, a mapping relationship is established by comparing the positional relationship of the key points of the two algorithms. The key points obtained by the DWpose pose detection algorithm are mapped to the corresponding FaceMesh key points using a mapping relationship. The mapping relationship includes coordinate translation or interpolation using multiple DWPose keypoints; The key points obtained by the DWPose pose detection algorithm are transformed into key points selected by the FaceMesh algorithm; A affine transformation based on facial and mask key points is used to add a mask to the face in the video frame.

2. The facial privacy protection method according to claim 1, characterized in that, The process of selecting a mask and marking key points on the mask includes: A suitable face mask is selected, and the key points of the mask are fixed. After the mask is selected, the key points are marked manually, and the key point markings are matched with the key points of the selected face.

3. The facial privacy protection method according to claim 1, characterized in that, Also includes: After converting the key points obtained by the DWPose pose detection algorithm into key points selected by the FaceMesh algorithm, key point smoothing correction is performed, and the contour points of the outermost edge of the key points are smoothed.

4. The facial privacy protection method according to claim 3, characterized in that, The smoothing correction of the outermost contour points of the key points includes: Each facial contour point is compared with its clockwise and counterclockwise adjacent contour points: Let the target point be A. i The midpoint of the two adjacent contour points is B. i The distance between the corresponding adjacent contour points is d. i Calculate all contour points A i With B i Distance between points and d i The ratio |A i B i | / d i ; If the ratio of contour points exceeds the ratio threshold, then the point A with the largest ratio will be removed. x Move to the corresponding B x The algorithm is iterated over and re-applied until the proportion of all points does not exceed the threshold, or the maximum number of iterations is reached.

5. A facial privacy protection method according to claim 1, characterized in that, The process of applying a mask to the face in a video frame using triangular affine transformation based on facial and mask key points includes: Obtain facial key points and corresponding mask key points; Calculate the Delaunay triangulation for key points of the mask; Based on the key point connection method of the mask, the Delaunay triangulation is calculated in the same way on the facial key points of the video frame; A affine transformation is used to map the triangles of key points on the mask to the triangles of key points on the face in the video frame, thus covering the face with the mask.

6. A facial privacy protection method according to claim 5, characterized in that, The triangular affine transformation includes encoding translation, scaling, rotation, and shearing, constructing a 2x3 transformation matrix. The first two columns of the matrix encode rotation, scaling, and shearing, and the last column encodes translation.

7. A facial privacy protection system, characterized in that, include: Acquisition module: Used to acquire videos of patients taking medication and extract video frames; Annotation module: Used to select a mask and annotate its key points; Detection and selection module: used to detect and select facial key points in each video frame, and to enhance the details of the key point detection and selection through human pose detection algorithm to obtain facial key points; The process of detecting and selecting facial landmarks for each video frame includes: Facial key points are selected from the eyes, nose, mouth, eyebrows, jawline, and facial contour features. Facial landmark detection is performed on each video frame using the FaceMesh algorithm in MediaPipe. The process of obtaining facial key points through key point detection and selection using a human pose detection algorithm to enhance details includes: The FaceMesh algorithm is used to detect facial landmarks; When the FaceMesh algorithm fails to detect a face, the DWPose pose detection algorithm is activated. The DWPose pose detection algorithm is used to estimate the pose of video frames in which no face is detected, and key points are obtained. By transforming key points, the key points obtained by the DWPose pose detection algorithm are converted into key points selected by the FaceMesh algorithm; The process of converting key points obtained by the DWPose pose detection algorithm into key points selected by the FaceMesh algorithm through key point transformation includes: Based on the video frame key points that can be obtained by both the DWPose pose detection algorithm and the FaceMesh algorithm, a mapping relationship is established by comparing the positional relationship of the key points of the two algorithms. The key points obtained by the DWpose pose detection algorithm are mapped to the corresponding FaceMesh key points using a mapping relationship. The mapping relationship includes coordinate translation or interpolation using multiple DWPose keypoints; The key points obtained by the DWPose pose detection algorithm are transformed into key points selected by the FaceMesh algorithm; Transformation module: Used to perform triangular affine transformation based on facial key points and mask key points to add a mask to the face of the video frame.

Images