Face posture judgment method and face changing method
By using key point calculation methods for facial pose judgment and pose adaptation, the problems of facial deformation and shaking in facial reconstruction technology have been solved, achieving a more accurate and stable facial fit effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING ZHONGKE RUIJIAN TECH CO LTD
- Filing Date
- 2022-11-08
- Publication Date
- 2026-08-04
AI Technical Summary
In existing face-swapping technologies, face reconstruction schemes have high requirements for expression-driven operations. Large facial rotations cause reconstruction distortion, and small facial key point detection coordinates differ greatly, leading to face-fitting jitter problems.
A face pose determination method is adopted. Euler angles are calculated by affine transformation to determine whether the face pose is large or small. The final face key point coordinates are calculated according to different poses. Euclidean distance or average value is used to handle key point localization under large and small poses. Face fitting is performed by combining affine transformation and Gaussian filtering.
It improves the accuracy of facial key point localization, solves the problems of facial deformation under large poses and facial shaking under small poses, and enhances the accuracy and stability of facial fitting.
Smart Images
Figure CN115937931B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for determining facial pose and a face-swapping method. It is applicable to the field of computer vision technology. Background Technology
[0002] With the arrival of the 5G era, various short videos and live streams have sprung up like mushrooms after rain. Various beauty filters and slimming filters are ubiquitous on various application platforms, especially the face-swapping function (swapping the face area of person A to person B), which has been widely sought after by users.
[0003] Existing face-swapping technologies mainly fall into two categories: face fitting and face reconstruction. Face fitting refers to attaching all or part of the face of person A, such as the mouth, nose, and eyes, to the face of person B. Face reconstruction refers to using the facial expressions and movements of person A to drive the facial expressions and movements of person B, making the head expressions and movements of person A and person B completely consistent.
[0004] Face reconstruction and face-swapping solutions have high requirements for the facial expressions of person A. For example, the video must be unobstructed and there cannot be large facial rotations, otherwise the reconstructed person B will have problems such as deformation and bending. When the face is in a small pose, the coordinates of the same key points detected in each frame of the video are significantly different between consecutive frames, which will cause jitter in the face matching. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to provide a method for judging facial pose and a face-swapping method to address the above-mentioned problems.
[0006] The technical solution adopted in this invention is: a method for determining facial pose, characterized in that:
[0007] Obtain the video containing faces to be judged;
[0008] Facial landmark detection is performed on each frame of the video to be judged, and the location of facial landmarks in each frame is obtained.
[0009] Perform an affine transformation between the facial key point positions of the face in the current frame image and the facial key point positions of the reference face template to obtain the affine transformation matrix between the current frame and the reference face template image.
[0010] The pitch, yaw, and roll angles in the Euler angles of the reference face template are all 0. All subsequent attitude determinations are based on the Euler angles of the reference face template.
[0011] Based on the affine transformation matrix between the current frame and the reference face template image, the Euler angles of the current frame image are obtained;
[0012] The Euler angles of the current frame are used to determine whether the face pose in the current frame image is a large or small pose.
[0013] The step of determining whether the face pose in the current frame image is a large or small pose based on the Euler angles of the current frame includes:
[0014] When the Euler angles satisfy the small pose condition, the face pose is determined to be a small pose; when the Euler angles do not satisfy the small pose condition, the face pose is determined to be a large pose.
[0015] Among them, the small attitude conditions are that the face pitch angle in the Euler angle is in the range of [-15, 15], the yaw angle is in the range of [-30, 30], and the roll angle is in the range of [-15, 15].
[0016] A face-swapping method, characterized by:
[0017] Get video A containing a face and video B containing a face;
[0018] The face pose determination method is used to determine the face pose of the person in the current frame of each of Video A and Video B;
[0019] Based on the face pose in the current frame image, the final facial key point coordinates of the current frame image are calculated.
[0020] Face-swapping is performed based on the final facial landmark coordinates of the current frame images in both Video A and Video B.
[0021] The step of calculating the final facial key point coordinates of the current frame image based on the facial pose of the face in the current frame image includes:
[0022] A. When the face in the current frame image is in a large pose, calculate the Euclidean distance between the same key point in the current frame and the previous frame image.
[0023] If the Euclidean distance is greater than the set threshold (the threshold is determined empirically; if the number of pixels on the longest side of the face detection box is less than 100, the threshold is fixed at 4 pixels; otherwise, the threshold is set to 4% of the number of pixels on the longest side of the face detection box. For example, if the face detection box is 120 long and 100 wide, the threshold is 120 * 4% = 4.8, rounded to 5), then the final face key point coordinates of the key point on the current frame image are the coordinates of the corresponding key point on the current frame image.
[0024] If the Euclidean distance is less than or equal to the set threshold, the final facial key point coordinates of the key point on the current frame image are the coordinates of the corresponding key point on the previous frame image.
[0025] B. When the face in the current frame image is in a small pose, the final facial key point coordinates of each key point in the current frame image are the average of the coordinates of the corresponding key points in the current frame and the previous frame image.
[0026] The face-swapping based on the final facial landmark coordinates of the current frame images in both Video A and Video B includes:
[0027] Based on the final facial key point coordinates of the current frame image in video B, and combined with the affine transformation matrix between the current frame in video A and the reference facial template image, an inverse affine transformation is performed to obtain the facial region mask of the current frame in video B.
[0028] The face region mask of the current frame of video B is fitted to the current frame of video A based on the final facial key point coordinates of the current frame image in video A.
[0029] Gaussian filtering is used to smooth the edges of the image to achieve face-swapping in the current frame of video A.
[0030] A face-swapping device, characterized in that it comprises:
[0031] The video acquisition module is used to acquire video A containing a face and video B containing a face;
[0032] The pose determination module is used to determine the face pose of the face in the current frame image of video A and video B using the face pose determination method.
[0033] The coordinate transformation module is used to calculate the final coordinates of the facial key points in the current frame image based on the facial pose of the face in the current frame image.
[0034] The face fitting module is used to perform face fitting and face swapping based on the final facial key point coordinates of the current frame images in Video A and Video B.
[0035] A storage medium storing a computer program executable by a processor, characterized in that: when the computer program is executed, it implements the steps of the face-swapping method.
[0036] A computer device having a memory and a processor, wherein the memory stores a computer program that can be executed by the processor, characterized in that: when the computer program is executed, it implements the steps of the face-swapping method.
[0037] The beneficial effects of this invention are: by classifying facial poses into two categories, this invention performs corresponding calculations and processing when the face is in a large pose or a small pose, thereby specifically overcoming the problems faced by each of the large and small poses.
[0038] When the face is in a large pose, this invention calculates the Euclidean distance based on the coordinate positions of the same key point in two adjacent frames, and determines the final coordinates of the face key points based on the Euclidean distance, thereby improving the accuracy of face key point localization under large pose and improving the accuracy of face fitting.
[0039] When the face is in a small pose, this invention uses the average value of the coordinates of the same key point in two adjacent frames as the final coordinates of the face key point, which can smooth the coordinates of the same face key point in two consecutive frames and effectively solve the jitter problem. Attached Figure Description
[0040] Figure 1 The flowchart is for an example. Detailed Implementation
[0041] This embodiment describes a face-swapping method that overlays a face from video B onto a face in video A, specifically including the following steps:
[0042] S1. Obtain video A containing only one person's face and video B containing only one person's face. The person in video A and video B can be the same person or different people.
[0043] S2. Use the face pose determination method to determine the face pose of the person in the current frame of video A, and use the face pose determination method to determine the face pose of the person in the current frame of video B.
[0044] The face pose determination method in this embodiment includes the following steps:
[0045] S21. Obtain the video A and video B to be judged;
[0046] S22. Acquire each frame of video A and each frame of video B respectively;
[0047] S23. Perform face detection on each frame of video A and video B, and detect the position coordinates and size of the face on each frame.
[0048] S24. Based on the position coordinates and size of the face in each frame of video A and video B, crop out the face region, perform facial key point detection, and obtain the position coordinates of the face contour, eyes, nose, mouth and eyebrows.
[0049] S25. Perform affine transformation calculations on the facial key point positions of the faces in the current frame images of video A and video B and the facial key point positions of the faces in the reference face template image to obtain the affine transformation matrix between the current frame and the reference face template image.
[0050] The pitch, yaw, and roll angles of the reference face template are all 0. All subsequent attitude determinations are based on the Euler angles of the reference face template. In some embodiments, the reference face template can be the previous frame of the current frame in the same video.
[0051] S26. Based on the affine transformation matrix between the current frame of video A and video B and the reference face template image, obtain the Euler angles of the current frame of video A and video B relative to the reference face template image.
[0052] S27. Based on the Euler angles of the current frame of each of Video A and Video B relative to the reference face template image, determine whether the face pose in the current frame of each of Video A and Video B is a large pose or a small pose.
[0053] When the Euler angles meet the small pose condition, the face pose is determined to be a small pose; when the Euler angles do not meet the small pose condition, the face pose is determined to be a large pose. The small pose condition is that the face pitch angle in the Euler angles is in the range of [-15, 15], the yaw angle is in the range of [-30, 30], and the roll angle is in the range of [-15, 15].
[0054] S3. Based on the facial pose of the face in the current frame image of each of Video A and Video B, calculate the final facial key point coordinates of the current frame image of each of Video A and Video B.
[0055] A. When the face in the current frame is in a large pose, calculate the Euclidean distance between the same key point in the current frame and the previous frame.
[0056] Assume the coordinates of the i-th key point on the face in the current frame are (x... i y i The coordinates of the i-th keypoint of the face in the previous frame are (x, y). i ',y i The Euclidean distance is calculated using the following formula:
[0057]
[0058] Among them, l i This represents the Euclidean distance between the i-th keypoint in the current frame and the i-th keypoint in the previous frame.
[0059] According to l i With the set threshold l threshold Comparison, l i Less than or equal to the set threshold l threshold The final facial keypoint coordinates of the i-th keypoint on the face in the current frame are the original coordinates (x, y, y) of the i-th keypoint on the face in the previous frame. i y i );li Greater than the set threshold l threshold The final facial keypoint coordinates of the i-th keypoint on the face in the current frame are the coordinates (x, y) of the i-th keypoint on the image of the current frame. i ',y i ').
[0060] B. When the face in the current frame is in a small pose, the final facial keypoint coordinates of each keypoint in the current frame are the average of the coordinates of the corresponding keypoints in the current and previous frames. The final facial keypoint (X) of the i-th keypoint on the face in the current frame is... i Y i ) equals the coordinates of the facial landmarks in the current frame (x) i y i (x) and the coordinates of the facial landmarks in the previous frame (x) i ',y i ') Calculate the average value as follows:
[0061]
[0062] S4. Perform face-swapping based on the final facial key point coordinates of the current frame images in Video A and Video B.
[0063] S41. Based on the final facial key point coordinates of the current frame image in video B, and combined with the affine transformation matrix between the current frame in video A and the reference facial template image, perform an inverse affine transformation operation to obtain the facial region mask B1 of the current frame of video B that conforms to the original pose of video A.
[0064] S42. Fit the face region mask B1 of the current frame of video B onto the current frame of video A according to the final facial key point coordinates of the current frame image in video A.
[0065] S43. Smooth the edges of the image by applying Gaussian filtering, save each frame to a specified folder, perform video conversion on the images saved to the specified folder, and generate the final face-swapped video.
[0066] This embodiment also provides a face-swapping device, including: a video acquisition module, a pose determination module, a coordinate transformation module, and a face fitting module. The video acquisition module is used to acquire video A containing a face and video B containing a face. The pose determination module is used to determine the face pose of the face in the current frame image of each of video A and video B using the face pose determination method in this example. The coordinate transformation module is used to calculate the final facial key point coordinates of the current frame image based on the face pose of the face in the current frame image. The face fitting module is used to perform face fitting and face swapping based on the final facial key point coordinates of the current frame images of each of video A and video B.
[0067] This embodiment also provides a storage medium storing a computer program that can be executed by a processor, which, when executed, implements the steps of the face-swapping method in this example.
[0068] This embodiment also provides a computer device having a memory and a processor. The memory stores a computer program that can be executed by the processor. When the computer program is executed, it implements the steps of the face-swapping method in this example.
Claims
1. A face-swapping method, characterized in that: Get video A containing a face and video B containing a face; The face pose determination method is used to determine the face pose of the faces in the current frame of video A and video B respectively; Based on the face pose in the current frame image, the final facial key point coordinates of the current frame image are calculated. Face-swapping is performed based on the final facial landmark coordinates of the current frame images in both Video A and Video B. The face pose determination method includes: Obtain the video containing faces to be judged; Facial landmark detection is performed on each frame of the video to be judged, and the location of facial landmarks in each frame is obtained. Perform an affine transformation between the facial key point positions of the face in the current frame image and the facial key point positions of the reference face template image to obtain the affine transformation matrix between the current frame and the reference face template image. Based on the affine transformation matrix between the current frame and the reference face template image, the Euler angles of the current frame image are obtained; Determine whether the face pose in the current frame image is large or small based on the Euler angles of the current frame; The step of calculating the final facial key point coordinates of the current frame image based on the facial pose of the face in the current frame image includes: When the face in the current frame image is in a large pose, calculate the Euclidean distance between the same key point in the current frame and the previous frame image. If the Euclidean distance is greater than the set threshold, the final facial key point coordinates of the key point on the current frame image are the coordinates of the corresponding key point on the current frame image. If the Euclidean distance is less than or equal to the set threshold, the final facial key point coordinates of the key point on the current frame image are the coordinates of the corresponding key point on the previous frame image. When the face in the current frame image is in a small pose, the final facial key point coordinates of each key point in the current frame image are the average of the coordinates of the corresponding key points in the current frame and the previous frame image. The face-swapping based on the final facial landmark coordinates of the current frame images in both Video A and Video B includes: Based on the final facial key point coordinates of the current frame image in video B, and combined with the affine transformation matrix between the current frame in video A and the reference facial template image, an inverse affine transformation is performed to obtain the facial region mask B1 of the current frame of video B that conforms to the original pose of video A. The face region mask of the current frame of video B is fitted to the current frame of video A based on the final facial key point coordinates of the current frame image in video A. Gaussian filtering is used to smooth the edges of the image to complete the face reconstruction and face swapping of the current frame image in video A.
2. The face swapping method of claim 1, wherein, The step of determining whether the face pose in the current frame image is a large or small pose based on the Euler angles of the current frame includes: When the Euler angles satisfy the small pose condition, the face pose is determined to be a small pose; when the Euler angles do not satisfy the small pose condition, the face pose is determined to be a large pose. Among them, the small attitude conditions are that the face pitch angle in the Euler angle is in the range of [-15, 15], the yaw angle is in the range of [-30, 30], and the roll angle is in the range of [-15, 15].
3. A face-swapping device, characterized in that, include: The video acquisition module is used to acquire video A containing a face and video B containing a face; The pose determination module is used to determine the face pose of the face in the current frame of video A and video B using face pose determination methods. The coordinate transformation module is used to calculate the final coordinates of the facial key points in the current frame image based on the facial pose of the face in the current frame image. The face fitting module is used to perform face fitting and face swapping based on the final facial key point coordinates of the current frame images in video A and video B. The face pose determination method includes: Obtain the video containing faces to be judged; Facial landmark detection is performed on each frame of the video to be judged, and the location of facial landmarks in each frame is obtained. Perform an affine transformation between the facial key point positions of the face in the current frame image and the facial key point positions of the reference face template image to obtain the affine transformation matrix between the current frame and the reference face template image. Based on the affine transformation matrix between the current frame and the reference face template image, the Euler angles of the current frame image are obtained; Determine whether the face pose in the current frame image is large or small based on the Euler angles of the current frame; The step of calculating the final facial key point coordinates of the current frame image based on the facial pose of the face in the current frame image includes: When the face in the current frame image is in a large pose, calculate the Euclidean distance between the same key point in the current frame and the previous frame image. If the Euclidean distance is greater than the set threshold, the final facial key point coordinates of the key point on the current frame image are the coordinates of the corresponding key point on the current frame image. If the Euclidean distance is less than or equal to the set threshold, the final facial key point coordinates of the key point on the current frame image are the coordinates of the corresponding key point on the previous frame image. When the face in the current frame image is in a small pose, the final facial key point coordinates of each key point in the current frame image are the average of the coordinates of the corresponding key points in the current frame and the previous frame image. The face-swapping based on the final facial landmark coordinates of the current frame images in both Video A and Video B includes: Based on the final facial key point coordinates of the current frame image in video B, and combined with the affine transformation matrix between the current frame in video A and the reference facial template image, an inverse affine transformation is performed to obtain the facial region mask B1 of the current frame of video B that conforms to the original pose of video A. The face region mask of the current frame of video B is fitted to the current frame of video A based on the final facial key point coordinates of the current frame image in video A. Gaussian filtering is used to smooth the edges of the image to complete the face reconstruction and face swapping of the current frame image in video A.
4. A storage medium storing a computer program executable by a processor, characterized in that: When the computer program is executed, it implements the steps of the face-swapping method according to any one of claims 1 to 2.
5. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable by the processor, characterized in that: When the computer program is executed, it implements the steps of the face-swapping method according to any one of claims 1 to 2.