User-oriented video face changing method
By separating the front and side face objects in the video analysis module and using the target tracking algorithm and different face swap models to process, the problem of additional operations in the existing technology is solved, and the efficiency and accuracy of video face swap is improved.
Patent Information
- Application Number
- CN202411557723.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-04
- Publication Date
- 2025-02-21
AI Technical Summary
The existing video face-changing method for users is required to handle the situation where the face rotation amplitude is too large, and the user requires additional operations on specific frames, resulting in increased user workload, reduced efficiency and inaccurate face-changing results.
By separating the front face and side face objects in the video analysis module, and using the target tracking algorithm to determine the face position of the user specified target person, use different face swap models to deal with the possible positive side face situations in the target video, and generate high-quality Face change result video.
It effectively reduces the additional operations of users in the video face change work, improves the accuracy and efficiency of face change results, and ensures high-quality output during the video face change process.
Smart Images
Figure CN119494768A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of AI image processing, and in particular to a user-oriented video face-changing method. Background Art
[0002] The existing user-oriented video face-changing methods are roughly as follows:
[0003] 1. Divide the video into individual frames and perform face detection on each frame.
[0004] 2. Select a target face from the detected faces according to the rules specified by the user. The rules specified by the user may be the relative position of the face in the image, or the face of a specific person.
[0005] 3. Based on the facial data obtained from the target face detection in step 1, the target face is replaced with the source face specified by the user.
[0006] 4. Merge all video frames to generate the face-changing result video.
[0007] Existing solutions have achieved good results in the field of user-oriented face replacement. However, in the actual application of face replacement systems, we found that the above technical solutions have the following problems:
[0008] The position and angle of the face in the video are very complex. Existing solutions often have better face-changing effects on target faces in the front. For side faces or faces that have rotated, commonly used face detection models cannot generate the facial feature points required to support the normal reasoning of face-changing models. At this time, users are often required to perform additional processing on the corresponding video frames to make the final face-changing result video meet the requirements. Summary of the invention
[0009] The purpose of the present invention is to provide a user-oriented video face-changing method, which solves the problem that in the current user-oriented video face-changing solutions, when the face rotation range is too large, the user needs to perform additional operations on specific frames, thereby reducing the user's workload and improving the user's efficiency in video face-changing work and the accuracy of the face-changing results.
[0010] To achieve the above object, the present invention provides the following technical solutions:
[0011] A user-oriented video face-changing method comprises the following steps:
[0012] S1: Front and side face input module: the user inputs the front face source face image and the face video of the target user;
[0013] S2: Video analysis module: receives the target video input by the user and outputs the target front and side faces of all video frames;
[0014] S3: front and side face separation module: separates the front face object and side face object inputted previously, and outputs the separated target front face object and side face object;
[0015] S4: front and side face swap module: receives the target face and the source face, and outputs the face object after the face swap;
[0016] S5: Post-processing module.
[0017] Preferably, the video analysis module will output the object containing the target face position and feature information in the target video by dividing it into the front face and the side face.
[0018] Preferably, if the output includes a side face object, the user is required to input a side face source face, and the side face swap module receives the side face source face and the side face target face position and feature object, and outputs a side face swap result object after the face swap is completed.
[0019] Preferably, the front and side face face swapping module receives the front face source face and the front face target face position and feature objects, and outputs a front face face swapping result object after the face swapping is completed.
[0020] Preferably, the post-processing module integrates the face-changing result objects into a high-quality face-changing result video.
[0021] Preferably, the video analysis module includes a video preprocessing module, a video storyboard module and a task target tracking module.
[0022] Preferably, if the user specifies the face-changing rule as a relative position, the face position and feature information object of the corresponding position will be input into the front and side face separation module; if the user specifies one or more characters in the target video as the face-changing rule, the target video will be input into the video storyboard module, the video storyboard module will segment the target video, each segment does not contain the part of the switching shot, the segmented video as well as the face position and feature information object will be input into the character target tracking module, the character target tracking module will extract the face position and feature information objects of all user-specified characters, and pass them into the front and side face separation module.
[0023] Preferably, the front and side face separation module separates the front face object and the side face object inputted previously, and outputs the separated target front face object and side face object.
[0024] In summary, due to the adoption of the above technology, the beneficial effects of the present invention are:
[0025] 1. By analyzing the target video, separating the front face and side face objects, using the target tracking algorithm to determine the facial position of the target person specified by the user, and using different face-changing models to deal with the front and side face situations that may appear in the target video, the present invention effectively solves the problem that the current user-oriented video face-changing solutions require users to perform additional operations on specific frames, reduces the user's workload, and improves the user's efficiency in video face-changing work and the accuracy of the face-changing results.
[0026] 2. The present invention proposes Figure 2 The video analysis module technical solution shown in the figure avoids the problem of accuracy of face detection model for sideways and inverted faces by processing video frames in four different directions; through the video splitting and character target tracking modules, it reduces the probability of face recognition omissions and recognition errors, and improves the accuracy and versatility of target face extraction.
[0027] 3. The present invention proposes Figure 1 The user-oriented video face-changing technology solution shown processes the front and side faces of the target face separately, avoiding the problem that the original face-changing model that tends to favor the front face becomes invalid in side faces and other situations due to the reduction in the number of feature points. It reduces the extra work of users in using video face-changing products and improves the efficiency and quality of face-changing.
[0028] 4. Different from the existing schemes that directly perform face detection and target tracking on the original video, a video preprocessing scheme and a video storyboard processing scheme are proposed to improve the accuracy of target face detection, which is creative. In view of the problem that the face-changing model fails to work for the side face target face in the existing face-changing schemes, it is proposed to use different face-changing models to process the front face and side face target faces respectively, which is creative.
[0029] 5. High fidelity and natural effects: The video after face replacement can maintain high definition, eliminate the jitter of key points, colors, etc., and has very high stability, making the face replacement effect natural and realistic. Easy operation: users only need to provide a picture to achieve video-level image replacement. The operation is simple and easy to understand, and the results can be obtained quickly. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention, so that other features, purposes and advantages of the present invention become more obvious. The accompanying drawings of the exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:
[0031] Figure 1 This is the overall flow chart of the user-oriented video face-changing system of the present invention;
[0032] Figure 2This is the overall flow chart of the video analysis module of the present invention;
[0033] Figure 3 A hardware scene graph is used for the present technique. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.
[0035] In the description of the present invention, it is necessary to understand that the terms indicating orientation or positional relationship are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0036] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances of the specification.
[0037] The present invention provides Figure 1 A user-oriented video face-changing method is shown, comprising the following steps:
[0038] The overall process of the program is as follows Figure 1 As shown:
[0039] 1. The user inputs a frontal source face image and a target video, and the target video is passed to the video analysis module. The video analysis module will output the objects in the target video that contain the target face position and feature information as frontal and side faces. If the output does not contain side face objects, the output will be passed to the front face swap module;
[0040] 2. If the output contains a side face object, the user needs to input a side face source face, and the side face swap module receives the side face source face and the side face target face position and feature object, and outputs the side face swap result object after the face swap is completed;
[0041] 3. The frontal face swap module receives the frontal source face and the frontal target face position and feature objects, and outputs the frontal face swap result object after the face swap is completed;
[0042] 4. The face-changing front and side face result objects will be passed to the post-processing module, which will integrate the face-changing result objects into high-quality face-changing result videos.
[0043] The processing flow of the video analysis module is as follows: Figure 2 As shown;
[0044] The target video input by the user is the input of the video analysis module, and the user needs to specify the face-changing rules for the target video;
[0045] The target video enters the video preprocessing module, and the output is the face position and feature information objects of all video frames;
[0046] If the user specifies the face swap rule as relative position, the face position and feature information object of the corresponding position are input into the front and side face separation module;
[0047] If the user specifies one or more characters in the target video as face-swapping rules, the target video is input into the video storyboard module. The video storyboard module segments the target video, and each segment does not contain the part where the shot is switched. The segmented video and the face position and feature information objects are input into the character target tracking module. The character target tracking module extracts the face position and feature information objects of all the user-specified characters and inputs them into the front and side face separation module;
[0048] The front and side face separation module separates the input front face object and side face object, and outputs the separated target front face object and side face object.
[0049] The present invention proposes an innovative user-oriented video face-swapping technology solution, which analyzes the target video, separates the frontal face and side face objects, uses a target tracking algorithm to determine the face position of the user-specified target person, and uses different face-swapping models to deal with the frontal and side face situations that may appear in the target video. The present invention effectively solves the problem that the current user-oriented video face-swapping solution requires the user to perform additional operations on specific frames, reduces the user's workload, and improves the user's efficiency in video face-swapping work and the accuracy of the face-swapping results. Specific embodiment:
[0051] 1. The user provides the target video and source face image, and provides an image of the specified target face.
[0052] 2. If Figure 1 As shown, the target video enters the video analysis module, such as Figure 2 As shown in the figure, the video preprocessing module is entered. The video will be divided into single frames and rotated by 90 degrees, 180 degrees and 270 degrees respectively to obtain four-way video frames. After resizing and color normalization, it is input into the face detection model. Face detection models include yoloface, scrfd, etc. Finally, the face position and feature information objects of all video frames are obtained.
[0053] 3. If the user specifies the rule as a target task, the original target video will be divided into video clips without camera switching through the video storyboard model. The video clips will be combined with the face position and feature information objects of the video frame obtained in 2 and input into the human target tracking module.
[0054] 4. The target person tracking module finds the face position information of all corresponding target persons in the video clip based on the target face image provided by the user. The specific method includes comparing the facial feature vectors of the two. At this time, there are still objects in the target clip that are not within the feature vector threshold distance and still belong to the target face. Then, the target tracking model will be used to infer the objects that are still the target face that have not been found in the frames before and after the found target task face.
[0055] 5. All target face objects will be input into the front and side face separation module. A specific step of the front and side face separation module is to use a face feature point extraction model, such as the Face Landmark 68 model, to extract important feature points and normal vectors of face objects. If the absolute value of the angle between the normal vector of the face object and the normal vector of the photographic plane is greater than a threshold, it will be judged as a side face object, otherwise it is a front face object. Finally, the front face and side face target face objects are obtained respectively, where each object contains the position information, angle information and feature point information of the target face.
[0056] 6. As Figure 1 As shown in FIG. 5 , if the output in 5 contains a side face object, the side face source face image provided by the user is required and the side face swap module is called. The side face swap module is very similar to the front face swap module, the only difference being the data used to train the face swap model. Taking the side face swap module as an example, the source face image is preprocessed by scaling and normalization, and the source face object is obtained through the face detection and feature point extraction models mentioned in 2 and 5. For all input target side face objects, a facial mask is generated, for example, a mask is generated using the dfl xseg model, and then a face swap model is used to swap the face, such as GHOSt[1]. Finally, the face object after face swap is obtained.
[0057] 7. The face object after face swapping is input into the post-processing module. This module performs image enhancement on the face object after face swapping to obtain a face object with higher quality. Facial image enhancement models such as GFPGAN, etc. Then the post-processing module merges the final face object with the original video frame and merges all video frames to obtain the final face swapping result video.
[0058] Beneficial effects:
[0059] 1. In step 2 of the embodiment, video frames in four directions are processed to avoid the problem that the face detection model cannot detect faces due to direction problems. Compared with training a model that can detect faces in all directions, the requirements for training data and training time are reduced, thereby reducing the cost and difficulty of model training.
[0060] 2. In step 3 of the embodiment, the video is divided according to the storyboard, which avoids the difficulty of target tracking due to sudden change of the target position and improves the accuracy of the target tracking result.
[0061] 3. The use of the target tracking model in step 4 of the embodiment avoids the problem of the distance from the source face feature vector being too far due to excessive facial movements, thereby reducing the risk of failure of the face recognition model and the probability of face swapping failure due to failure to recognize the target face.
[0062] 4. In step 5 of the embodiment, the front face object and the side face object are separated, which facilitates the subsequent use of different face-changing models to change the front face and the side face. Compared with using the same face-changing model, the requirements for training data and training time are reduced, the training cost and training difficulty of the model are reduced, and the accuracy of the face-changing results is improved.
[0063] The present invention proposes Figure 2 The video analysis module technical solution shown in the figure avoids the problem of accuracy of face detection model for sideways and inverted faces by processing video frames in four different directions; through the video splitting and character target tracking modules, it reduces the probability of face recognition omissions and recognition errors, and improves the accuracy and versatility of target face extraction.
[0064] The present invention proposes Figure 1 The user-oriented video face-changing technology solution shown processes the front and side faces of the target face separately, avoiding the problem that the original face-changing model that tends to favor the front face becomes invalid in side faces and other situations due to the reduction in the number of feature points. It reduces the extra work of users in using video face-changing products and improves the efficiency and quality of face-changing.
Claims
1. A user-oriented video face-changing method, characterized in that: The following steps are involved: S1: Front and side face input module: the user inputs the front face source face image and the face video of the target user; S2: Video analysis module: receives the target video input by the user and outputs the target front and side faces of all video frames; S3: front and side face separation module: separates the front face object and side face object inputted previously, and outputs the separated target front face object and side face object; S4: front and side face swap module: receives the target face and the source face, and outputs the face object after the face swap; S5: Post-processing module.
2. A user-oriented video face-changing method according to claim 1, characterized in that: The video analysis module will output the objects in the target video that contain the target face position and feature information as the front face and the side face.
3. A user-oriented video face-changing method according to claim 1, characterized in that: If the output includes a side face object, the user needs to input a side face source face, and the side face swap module receives the side face source face and the side face target face position and feature object, and outputs a side face swap result object after the face swap is completed.
4. A user-oriented video face-swapping method according to claim 1, characterized in that: The front and side face swap module receives the front face source face and the front face target face position and feature objects, and outputs a front face swap result object after the face swap is completed.
5. A user-oriented video face-changing method according to claim 1, characterized in that: The post-processing module integrates the face-changing result objects into a high-quality face-changing result video.
6. A user-oriented video face-changing method according to claim 1, characterized in that: The video analysis module includes a video preprocessing module, a video splitting module and a task target tracking module.
7. A user-oriented video face-changing method according to claim 6, characterized in that: If the user specifies the face-changing rule as a relative position, the face position and feature information object of the corresponding position will be input into the front and side face separation module; if the user specifies one or more characters in the target video as the face-changing rule, the target video will be input into the video storyboard module, and the video storyboard module will segment the target video, and each segment does not contain the part of the switching shot. The segmented video and the face position and feature information object are input into the character target tracking module, and the character target tracking module will extract the face position and feature information objects of all user-specified characters, and pass them into the front and side face separation module.
8. A user-oriented video face-changing method according to claim 1, characterized in that: The front and side face separation module separates the front face object and the side face object inputted previously, and outputs the separated target front face object and side face object.