Sketch-based deep face video editing method and system
By using sketch editing methods in videos, editing vectors are generated and time-dependent and expression-driven propagation is performed, combined with area perception fusion, the problem of low face details editing in videos is solved, and high-quality video editing effect is achieved.
Patent Information
- Application Number
- CN202210698610.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-20
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-06-20
AI Technical Summary
The prior art is difficult to edit the detailed content of the face in a video, and sketch editing cannot be effectively propagated to the entire video, resulting in low video editing efficiency and inability to achieve simple and fast video detail editing.
A sketch-based face video editing method is designed. By aligning and encoding frames in face videos, hidden codes are generated, and sketch generation branches are added using StyleGAN generation network, reversely optimize image hidden codes, and edit vectors are generated to realize Timing-independent and expression-driven editing propagation, and editing operations of different frames are fused through a region-aware fusion method.
It realizes the effective propagation of sketch editing to the entire video, improves the efficiency and quality of video editing, and can quickly and accurately edit the details of face videos, generating high-reality editing results.
Smart Images

Figure CN115278106B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical fields of computer graphics and computer vision, and in particular to a method and system for synthesizing and sketching facial videos of human faces. Background Art
[0002] Video editing is a very challenging scientific research problem. With the development of deep learning, the work of video editing and modification is increasing. Most of the existing video editing methods modify the global properties of the video, convert black and white videos into color videos, or stylize the video to generate artistic video editing results. For face videos, the existing technology mainly completes editing tasks such as face swapping, and also only modifies the global attribute of identity. Some methods can edit the detail area of the video, but they need to use professional software such as PS and PR, which requires a high cost of time and energy. Sketch is an efficient and accurate interactive tool with high user-friendliness. It is widely used for image generation and editing problems. However, the existing technology cannot extend sketch editing from images to videos, and it is difficult to handle the propagation and synthesis problems of editing operations. Video editing has broad application prospects and has high value in cultural fields such as filmmaking and new media communication. However, the existing technology cannot complete the task of video detail editing simply and quickly.
[0003] Regarding the problem of video editing, the existing technology can realize automatic colorization of videos, but its function is relatively simple and can only change the color information of the video. Or the video is stylized, which can not only change the color characteristics of the video, but also artistically transform the content of the video to generate an artistic video. However, the above work can only edit the global features of the video and cannot modify the details of the video. Even if the work represents the video in the atlas space, after editing the image through software such as PS, the relevant detail editing operations will be extended to the video clips. However, the above method requires professional software to operate, and the editing and generation of videos are very time-consuming. Sketches are a more friendly interactive tool, and users' editing operations are simpler and more accurate. The existing technology can realize the operation of editing face images with sketch images, but it cannot propagate the editing operations to the entire video. Summary of the invention
[0004] In order to solve the problem that the above-mentioned prior art is difficult to edit the details of video content and the sketch editing cannot be propagated to the video, the present invention is based on the face image generation network StyleGAN, abstractly represents the sketch editing as a latent vector, and designs an innovative propagation and fusion mechanism to edit face videos. The present invention proposes a sketch-based face video editing method and system, which can select any one or more frames, use sketches to edit face details and propagate them to the entire video in a specified manner.
[0005] Specifically, the present invention proposes a sketch-based deep face video editing method, which includes:
[0006] Step 1: Align and crop the faces in the original video, and encode the faces into the latent space to obtain the latent codes of all frames in the face video;
[0007] Step 2: Add a sketch generation branch To StyleGAN to generate the network, reversely optimize the image hidden code and generate the edit vector δ edit ;
[0008] Step 3: Edit vector δ edit The hidden code superimposed on all frames completes the propagation of timing-independent edits;
[0009] Step 4: Use the weighted superposition edit vector δ of the piecewise linear function edit , complete the editing and dissemination of actions or expressions;
[0010] Step 5: Calculate the weighted superposition edit vector δ according to the similarity between the expression parameters of the current frame and the edited frame edit , so that edits correspond to specific expressions, completing expression-driven edit propagation;
[0011] Step 6: Use the region-aware fusion method to fuse different types of edits added to different frames, and fuse the face into the original video to obtain the sketch-based face video editing result.
[0012] The sketch-based deep face video editing method, wherein step 1 comprises: detecting the face key points of the face video, and aligning and cropping the face after smoothing using a time window, to generate a video frame sequence f1, f2, ..., f N , where N is the number of frames of the face video; project the frame sequence to the latent space W + , generate hidden code sequence w1,w2,…,w N .
[0013] The sketch-based deep face video editing method, wherein step 2 comprises:
[0014] Get the StyleGAN original generative network G and build a generative network for modeling the joint probability distribution of real face images and sketches Generative Network include and Two branches, is the original generative network of G, used to generate realistic face images, Used to generate the corresponding sketch image; given the hidden code w of the image, Generate feature maps F1, F2, ..., F14 , where F1 is used as The initial feature map of the branch; The feature map of the branch is upsampled and compared with the feature map F i The residual images after convolution are added to generate the sketch image corresponding to the hidden code w;
[0015] Use the dataset of image and sketch matching to train the sketch generation network S, which takes the face image as input and generates the corresponding sketch for training the sketch generation branch Randomly sample the hidden code w and input it Generate high-fidelity face images and the corresponding sketch Constructing the loss function Training sketch generation branch
[0016]
[0017] L VGG is the perceptual loss function, using the VGG19 model to measure visual similarity, L L2 is the pixel L2 loss, α1 and α2 are both preset weights;
[0018] After modeling the distribution of real images and sketches, according to the input face image x, the sketch s drawn edit and selected area m edit ; Project the face image x to W + space, get the hidden code w edit , the generated sketch The generated image is the same as the input sketch in the editing area. The non-edited area is the same as the original image, where w is obtained by the following loss function edit :
[0019] L editing (w edit )=β1L sketch +β2L rgb ,
[0020] L sketch The constraint editing area has the same structure as the sketch result. rgb The non-editable region is constrained to remain unchanged, β1 and β2 are hyperparameters, and the network is generated by fixing The weight of w edit ;
[0021] The final edit vector δ edit =w edit -w,δ edit represents the edit of the sketch and propagates it to the entire face video; for each frame fi , generate the corresponding edit vector:
[0022] δ i =δ edit ,i=1,2,…,N
[0023] Step 3 includes converting each frame f i The corresponding δ edit Propagate to the entire face video to generate an edited frame sequence.
[0024] The sketch-based deep face video editing method, wherein step 4 comprises:
[0025] Add a blink or smile action at a specific time in the face video, and at a specific frame f t Add edit vector δ edit , input duration h and change time l, for each frame f i , the present invention uses a piecewise linear function to generate a smooth propagation edit vector δ i , and obtain the new edit vector δ i :
[0026] δ i =γ·δ edit ,i=1,2,…,M
[0027]
[0028] t1=th / 2-1, t2=th / 2, t3=t+h / 2, t4=t+h / 2+1, t is the edit frame f t
[0029] The corresponding time;
[0030] These new edit vectors δ i Used to synthesize realistic face images;
[0031] This step 5 includes:
[0032] Given multiple key frames in the face video Extract facial expression parameters using 3D reconstruction and the corresponding edit vector M is the number of keyframes, and the expression guide edit is propagated as follows:
[0033]
[0034] e i is the input frame f i The expression parameter, C is the normalization term And edit vector Targeting the same area;
[0035] This step 6 includes:
[0036] Given a series of frame sequences f1,f2,…,f N , the user selects M key frames k 1 ,k 2 ,…,k M Edit different regions, corresponding to M drawn marked regions m 1 ,m 2 ,…,m M ; For each frame f i , generate M edit vectors
[0037] For each frame to be predicted f i Generate a deformation field, deform the input marked area to generate M new marked areas is m j After the deformation of action and expression
[0038] Domain; replace the local area of the feature map of the original frame with the new feature map:
[0039]
[0040] in, The initial feature map is G is the generative network of StyleGAN;
[0041] Downsampling Make it and have the same resolution; feature map For each of the M editing operations, we update them M times in total. We update the middle 5 feature maps of StyleGAN with a resolution from 32×32 to 128×128. The high resolution is obtained by converting the original hidden code w i Adjust the StyleGAN algorithm; apply the above fusion operation to all frames f i ,i=1,2,..,N, generate the edited and fused aligned face video;
[0042] Using the face segmentation method, the face marker areas of the input frame and the edited frame are generated and merged, and a smooth edge is generated for the merged marker area, which is further used as a fusion weight to fuse the faces before and after editing and de-align the fused face image to the original video to synthesize the face video editing result.
[0043] The present invention also proposes a sketch-based deep face video editing system, which includes:
[0044] Module 1 is used to align and crop the faces in the original video, and encode the faces into the latent space to obtain the latent codes of all frames in the face video;
[0045] Module 2, for adding sketch generation branches To StyleGAN to generate the network, reversely optimize the image hidden code and generate the edit vector δ edit ;
[0046] Module 3 is used to convert the edit vector δ edit The hidden code superimposed on all frames completes the propagation of timing-independent edits;
[0047] Module 4, for weighted superposition of edit vectors δ using piecewise linear functions edit , complete the editing and dissemination of actions or expressions;
[0048] Module 5 is used to calculate the weighted superposition editing vector δ according to the similarity of the expression parameters between the current frame and the edited frame. edit , so that edits correspond to specific expressions, completing expression-driven edit propagation;
[0049] Module 6 is used to use the region-aware fusion method to fuse different types of edits added to different frames, and fuse the face into the original video to obtain a sketch-based face video editing result.
[0050] The sketch-based deep face video editing system, wherein the module 1 is used to detect the face key points of the face video, and after smoothing with a time window, align and crop the face to generate a video frame sequence f1, f2, ..., f N , where N is the number of frames of the face video; project the frame sequence to the latent space W + , generate hidden code sequence w1,w2,…,w N .
[0051] The sketch-based deep face video editing system, wherein the module 2 is used to obtain the StyleGAN original generation network G and construct a generation network for modeling the joint probability distribution of real face images and sketches Generative Network include and Two branches, is the original generative network of G, used to generate realistic face images, Used to generate the corresponding sketch image; given the hidden code w of the image, Generate feature maps F1, F2, ..., F 14 , where F1 is used as The initial feature map of the branch; The feature map of the branch is upsampled and compared with the feature map Fi The residual images after convolution are added to generate the sketch image corresponding to the hidden code w;
[0052] Use the dataset of image and sketch matching to train the sketch generation network S, which takes the face image as input and generates the corresponding sketch for training the sketch generation branch Randomly sample the hidden code w and input it Generate high-fidelity face images and the corresponding sketch Constructing the loss function Training sketch generation branch
[0053]
[0054] L VGG is the perceptual loss function, using the VGG19 model to measure visual similarity, L L2 is the pixel L2 loss, α1 and α2 are both preset weights;
[0055] After modeling the distribution of real images and sketches, according to the input face image x, the sketch s drawn edit and selected area m edit ; Project the face image x to W + space, get the hidden code w edit , the generated sketch The generated image is the same as the input sketch in the editing area. The non-edited area is the same as the original image, where w is obtained by the following loss function edit :
[0056] L editing (w edit )=β1L sketch +β2L rgb ,
[0057] L sketch The constraint editing area has the same structure as the sketch result. rgb The non-editable region is constrained to remain unchanged, β1 and β2 are hyperparameters, and the network is generated by fixing The weight of w edit ;
[0058] The final edit vector δ edit =w edit -w,δ edit represents the edit of the sketch and propagates it to the entire face video; for each frame f i , generate the corresponding edit vector:
[0059] δ i =δedit ,i=1,2,…,N
[0060] The module 3 includes converting each frame f i The corresponding δ edit Propagate to the entire face video to generate an edited frame sequence.
[0061] The sketch-based deep face video editing system, wherein the module 4 is used to add a blink or smile action at a specific time in the face video, and at a specific frame f t Add edit vector δ edit , input duration h and change time l, for each frame f i , the present invention uses a piecewise linear function to generate a smooth propagation edit vector δ i , and obtain the new edit vector δ i :
[0062] δ i =γ·δ edit ,i=1,2,…,M
[0063]
[0064] t1=th / 2-1, t2=th / 2, t3=t+h / 2, t4=t+h / 2+1, t is the edit frame f t
[0065] The corresponding time;
[0066] These new edit vectors δ i Used to synthesize realistic face images;
[0067] This module 5 includes:
[0068] Given multiple key frames in the face video Extract facial expression parameters using 3D reconstruction and the corresponding edit vector M is the number of keyframes, and the expression guide edit is propagated as follows:
[0069]
[0070] e i is the input frame f i The expression parameter, C is the normalization term And edit vector Targeting the same area;
[0071] The module 6 is used to give a series of frame sequences f1, f2, ..., f N , the user selects M key frames k 1 ,k2 ,…,k M Edit different regions, corresponding to M drawn marked regions m 1 ,m 2 ,…,m M ; For each frame f i , generate M edit vectors
[0072] For each frame to be predicted f i Generate a deformation field, deform the input marked area to generate M new marked areas is m j The area after the deformation of the action and expression; replace the local area of the feature map of the original frame with the new feature map:
[0073]
[0074] in, The initial feature map is G is the generative network of StyleGAN;
[0075] Downsampling Make it and have the same resolution; feature map For each of the M editing operations, we update them M times in total. We update the middle 5 feature maps of StyleGAN with a resolution from 32×32 to 128×128. The high resolution is obtained by converting the original hidden code w i Adjust the StyleGAN algorithm; apply the above fusion operation to all frames f i ,i=1,2,..,N, generate the edited and fused aligned face video;
[0076] Using the face segmentation method, the face marker areas of the input frame and the edited frame are generated and merged, and a smooth edge is generated for the merged marker area, which is further used as a fusion weight to fuse the faces before and after editing and de-align the fused face image to the original video to synthesize the face video editing result.
[0077] The present invention also proposes a storage medium for storing a program for executing any one of the sketch-based deep face video editing methods.
[0078] The present invention also proposes a client for use in any one of the sketch-based deep face video editing systems.
[0079] It can be seen from the above scheme that the advantages of the present invention are:
[0080] The system designed by the present invention can select one or more editing frames, and the user draws a sketch and a corresponding editing area mask, and specifies the editing propagation mode to realize the editing and propagation operations of the video. BRIEF DESCRIPTION OF THE DRAWINGS
[0081] Figure 1 It is a schematic diagram of the system flow of the present invention;
[0082] Figure 2 Optimize schematics for sketches;
[0083] Figure 3 Result graphs for timing-independent editing and time-windowed editing;
[0084] Figure 4 Result graphs for timing-independent editing and expression-driven editing;
[0085] Figure 5 Draw result graphs for different drawing styles;
[0086] Figure 6 Edit the result image for the rotated face;
[0087] Figure 7 After editing the vectors to optimize the sketch, the resulting graphs are fused using different methods;
[0088] Figure 8 Fusion result graph for different edits;
[0089] Fig. 9 This is a schematic diagram of the intermediate results of face video editing;
[0090] Fig.10 Smooth the resulting map for keypoints. DETAILED DESCRIPTION
[0091] The defects in the existing technology are caused by the fact that sketch editing does not take into account the problem of video propagation. The reason is that the faces in the video have changes in expressions and movements, and the input sketch editing operations are difficult to directly act on other frames. At the same time, the sketch can change the identity characteristics of the face (such as the shape of the facial features), as well as the expressions and movements of the face (how to add a smile). It is very difficult to distinguish them and propagate them reasonably. Video editing also needs to ensure the stability of the timing. The existing methods do not consider the flicker problem of video generation, and the generated results are of poor quality.
[0092] The inventors found through sketch editing of images and videos that this defect can be solved by designing a reasonable video encoding method and a sketch editing propagation and fusion method. After inputting the face video, the face area is first cropped and aligned after key point detection. Further, we use the image coding network to encode the face images of all frames into the latent space of the StyleGAN generation network. For the sketch editing input by the user, we designed an optimization strategy to abstractly represent the editing operation as an editing vector. In the process of editing propagation, the editing operations are divided into two categories, timing-independent editing and timing-dependent editing. The timing-dependent editing is further divided into time window editing and expression-driven editing. The specific type of editing is specified by the user and propagated in different ways. At the same time, we designed a region-aware fusion strategy to fuse different editing operations input in different frames to generate an edited face video. Finally, the generated face video is anti-aligned to the input original video, and the face area is fused to generate the sketch video editing result.
[0093] The core invention points of the present invention include:
[0094] Key point 1, video encoding module based on StyleGAN. After inputting a face video, use dlib's face key point detection technology to crop and align the face image, and use a time window for smoothing. Based on the E4E face to StyleGAN latent space encoding technology, encode the input frame sequence into a latent sequence. According to the input frame sequence and latent sequence, based on the PTI reconstruction technology, fine-tune the weights of the StyleGAN generation network so that it can perfectly reconstruct the original video, complete the video encoding task, and serve the subsequent video editing;
[0095] Key point 2, sketch editing optimization module. Based on the pre-trained sketch generation network, the original StyleGAN is extended to add a sketch generation branch. Furthermore, an optimization strategy is designed. The user inputs the sketch and the edit area mask. The edit area constraint is the same as the sketch, while the non-edit area is the same as the original image. The original latent code is iteratively optimized. The optimized latent code is subtracted from the input latent code to obtain the edit vector, which abstractly represents the sketch editing operation.
[0096] Key point 3: Time-independent propagation technology. Some editing operations have the feature of being time-independent, that is, the editing should be applied evenly to the frame sequence of the entire video, regardless of expressions and actions, such as editing the face shape of a person. This type of editing directly adds the edit vector to the hidden code of the input frame to achieve the propagation effect of the edit.
[0097] Key point 4: Time window propagation technology. Some editing operations represent specific movements of the face, including smiling, closing eyes, etc. Such movements are divided into three stages: the start stage, the continuation stage, and the end stage. The edit vectors are superimposed with linearly changing weights in the start and end stages, and fixed weights are superimposed in the continuation stage to achieve the whole process of the start, continuation, and end of the movement.
[0098] Key point 5: Expression-driven communication technology. Some editing operations are related to expressions, such as closing eyes when smiling. For this type of editing, 3D reconstruction is used to extract facial expression parameters, and the weights are calculated based on the cosine similarity of the expression parameters of the edited frame and the predicted frame, and the edit vectors are superimposed.
[0099] Key point 6: Region-aware fusion module. During video editing, users often select multiple frames and edit different regions at the same time. This module uses deformation operations to predict the deformation field according to the changes in facial movements and expressions, and deforms the drawn mask. Furthermore, according to the deformed mask area, the feature map of a specific position of the generated network is replaced. Finally, the generated face area is fused and back-projected to the original frame to complete the fusion operation of different edits.
[0100] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.
[0101] System flow chart as follows Figure 1 As shown, the system includes multiple technologies such as edit vector generation, timing-independent edit propagation, time window edit propagation, expression-driven edit propagation, and region-aware fusion.
[0102] Sketch optimization flow chart as follows Figure 2 As shown in Figure 1, the original StyleGAN generative network is expanded into two branches, one branch generates sketch images and the other branch generates images with high realism. The optimization process includes two loss terms, L_sketch and L_RGB, which respectively constrain the edited area to correspond to the sketch edit and other areas to remain unchanged.
[0103] like Figure 1 As shown, a high-quality face video editing method and system based on sketch interaction includes:
[0104] S1: After inputting the video, align and crop the face, and encode the face into the latent space;
[0105] S2: Extending the StyleGAN generation network and adding a sketch generation branch Reverse optimization of image hidden code to generate edit vector δ edit ;
[0106] S3: Timing-independent editing, edit vector δ edit Hidden codes directly superimposed on all frames complete the propagation of timing-independent edits;
[0107] S4: Time window editing, using piecewise linear function weighted superposition edit vector δ edit , complete the editing and dissemination of actions or expressions;
[0108] S5: Expression-driven editing, calculate the weighted superposition editing vector δ according to the similarity of the expression parameters between the current frame and the edited frame edit , so that the edit corresponds to a specific expression;
[0109] S6: Use region-aware fusion to fuse different types of edits added to different frames and merge the face into the original video;
[0110] Wherein, the method of S1 comprises:
[0111] Given an input video, use dlib to detect facial key points, use a time window to smooth the coordinates of the key points, align and crop the faces, and generate a video frame sequence f1,f2,…,f N , where N is the number of frames. The present invention uses E4E to project the frame sequence onto W + space, generating hidden code sequence w1,w2,…,w N The edit vector generated later will be superimposed on the hidden code sequence. Smoothing is to smooth the coordinates of the facial key points in the sequence. The face detection method is a single-frame detection, so there will be some jitter between frames. The purpose of smoothing is to eliminate the impact of jitter.
[0112] Wherein, the method of S2 is as follows Figure 2 As shown, including:
[0113] S21: Given the StyleGAN original generative network G, the present invention designs a new generative network Model the joint probability distribution of real face images and sketches. It consists of two branches: Generate highly realistic face images, which is the original generation network of G. Generate the corresponding sketch image. Given the hidden code w of the image, Generate feature maps F1, F2, ..., F 14 , where F1 is used as The initial feature map of the branch. The intermediate feature maps of the branches are repeatedly upsampled, and F iThe residual images after convolution are added. After completing the above operations for i=2~14, the sketch image corresponding to the hidden code w is finally generated. The original generation network of StyleGAN3 has an expansion of 10 pixels. The present invention crops the intermediate feature map and only uses the cropped pixel content.
[0114] S22: Generate branches for training sketches We first use the image-sketch matching dataset to train a sketch generation network S based on the Pix2PixHD network. The sketch generation network takes real face images as input and generates corresponding sketches for training the sketch generation branch. Then, the present invention randomly samples the hidden code w and inputs it into Generate high-fidelity face images and the corresponding sketch Afterwards, the present invention uses the following loss function to train the sketch generation branch:
[0115]
[0116] L VGG is the perceptual loss function, using the VGG19 model to measure visual similarity, L L2 is the pixel L2 loss, α1 and α2 are both preset weights, and α1=α2=1 in this example.
[0117] S23: After modeling the distribution of real images and sketches, the present invention designs an optimization technique to optimize the real image x input by the user and the sketch s drawn by the user. edit and mark area m edit , generate the edit vector δ edit First, the real image x is projected onto W + space, generating the initial hidden code w. Then, the present invention optimizes and obtains the new hidden code w edit , the generated sketch The generated image is the same as the input sketch in the editing area. The non-edited area is the same as the original image. In order to optimize w edit , the present invention uses the following loss function:
[0118]
[0119] Among them, L LPIPS is the LPIPS distance, ⊙ is the matrix dot product. L sketch The constraint editing area has the same structure as the sketch result. rgb The non-editing area is constrained to remain unchanged. The final optimization loss function is:
[0120] L editing (w edit)=β1L sketch +β2L rgb ,
[0121] β1 and β2 are hyperparameters. During the optimization process, the weight of the network is fixed and the only optimized parameter is w edit .
[0122] S24: The final edit vector is:
[0123] δ edit =w edit -w
[0124] δ edit The edits of the sketch are abstracted and propagated to the entire video.
[0125] Wherein, the method of S3 includes:
[0126] Some editing operations have a significant impact on the entire video and are less relevant to expressions and actions. These editing operations mainly change the basic shape of the face, such as the shape of the face and the shape of the facial features. edit It has decoupling and semantic characteristics, and can be directly applied to the entire video frame sequence. i , generate the corresponding edit vector:
[0127] δ i =δ edit ,i=1,2,…,N
[0128] These edit vectors propagate the edits throughout the video, generating an edited frame sequence.
[0129] Wherein, the method of S4 includes:
[0130] Unlike single-frame editing, videos have expressions and movements that change over time. Users often edit facial movements in a timed manner, such as adding a blink or smile at a specific time. t Add edit vector δ edit , the user also needs to input the duration h and the change time l. Then, for each frame f i , the present invention uses a piecewise linear function to generate a smooth propagation edit vector δ i :
[0131] δ i =γ·δ edit ,i=1,2,…,M
[0132]
[0133] t1=th / 2-1, t2=th / 2, t3=t+h / 2, t4=t+h / 2+1, t is the edit frame f t
[0134] The corresponding time. These new edit vectors δ i It will be used to synthesize highly realistic facial images. Using the above editing method, the present invention can not only generate the editing effect in a specific time window, but also form a smooth transition, that is, the appearance and disappearance of the editing, for example, from a natural expression to a smiling expression, and then from a smiling expression to a natural expression.
[0135] Wherein, the method of S5 includes:
[0136] In some scenarios, users only want to add some edits under certain expressions, while keeping the original state or adding new edits under other expressions. Such editing operations include expression-driven wrinkles (such as nasolabial folds, forehead wrinkles, etc.), and some shape edits that only appear under certain expressions (such as making the eyes smaller when smiling). In order to propagate this type of expression-driven editing, the present invention uses 3D reconstruction to extract the expression parameters of the face. More specifically, given the keyframes of the expression edit, M is the number of key frames, and the present invention extracts expression parameters and the corresponding edit vector
[0137] Some key frames may not have any editing operation, and serve as key reference frames, indicating that no editing operation has been performed on a certain expression. For these frames, the editing vector is a zero vector. The present invention uses the following method to propagate expression-guided editing:
[0138]
[0139] e i is the input frame f i The expression parameter, C is the normalization term In the present invention, the editing vector Targeting the same area.
[0140] Wherein, the method of S6 includes:
[0141] S6.1: The present invention supports using sketches to edit any number of frames and merge the editing effects. After editing multiple frames, multiple editing vectors are generated. A simple method is to directly add the editing vectors. However, Figure 7 As shown, this method will generate defects, and the present invention designs a region-aware fusion method.
[0142] S6.2: Given a series of frame sequences f1,f2,…,f N, the user selects M key frames k 1 ,k 2 ,…,k M Edit different areas, corresponding to M edit mark areas m 1 ,m 2 ,…,m M Using the above-mentioned editing propagation method, for each frame f i , generate M edit vectors Represents different editing operations.
[0143] For each frame to be predicted f i The present invention uses the First-order method to generate a deformation field, deforming the input marked area to generate M new areas The region m drawn with the input j Mark similar regions, but consider frame f i Edit keyframes j In order to integrate different editing operations, the present invention replaces the local area of the feature map of the original frame with a new feature map:
[0144]
[0145] in, The initial feature map is G is the generative network of StyleGAN. Make it and The same resolution. For M editing operations, the above formula is iterated M times, j = 1 ~ M, to complete the editing and fusion of multiple local areas. The present invention updates the middle 5 feature maps of the generated network. These feature maps mainly control the facial structure information, and the resolution ranges from 32×32 to 128×128. The high resolution is obtained by the original hidden code w i , using the StyleGAN network algorithm for adjustment. The present invention applies the above fusion operation to all frames f i ,i=1,2,..,N, generate the edited and fused aligned face video.
[0146] S6.3: The present invention fuses the synthesized face into the original video to synthesize the final edited video. First, a face segmentation method is used to generate face region markers for the input frame and the edited frame, and the union of the face regions is calculated. The merged region is further expanded to make the edges transition smoothly. The smoothed face region marker map is converted into a fusion weight, where the weight of the marked region is 1, the weight of the unmarked region is 0, and the transition edge weight is between 0 and 1. The faces before and after editing are fused based on the weight. Finally, the face image is anti-aligned to the original video to synthesize the final edited video.
[0147] like Figure 3 As shown, the result of the fusion of the timing-independent editing and the time window editing of the present invention is shown. The timing-independent editing on the left is to add hair and beard to the edited character, and the time window editing is to add the eyebrow-raising action to the edited character. The first row on the right is the original video, and the second row on the right is the edited video result. The edited video has more hair and beard, and also has the eyebrow-raising action.
[0148] like Figure 4 As shown in FIG. 1 , the result of the fusion of the timing-independent editing and the time window editing of the present invention is shown. The timing-independent editing on the left side makes the nose of the face smaller, and the expression-driven editing makes the eyes smaller when the mouth is opened wide, and keeps the original shape when the mouth is closed. The right side shows the video frames before and after editing, and the editing operations are well propagated to the entire video.
[0149] like Figure 5 As shown in the figure, the results of editing faces using sketches of different drawing styles are shown. The first column of images shows the drawn sketches and selected areas, and the second column of images shows the results of single-frame editing. The first row on the right shows the original video frame, and the subsequent rows show the results of edit propagation. For sketches of different drawing styles, the present invention generates higher quality results with better robustness.
[0150] like Figure 6 As shown in FIG. 1 , the result of face editing with angle changes is shown. The first row of images is the original video frame sequence, the second row of images is the edited video frame sequence, and the left side is the sketch drawn by the user and the selected area. Even if the input face video has rotation and angle changes, the present invention can generate high-quality editing results.
[0151] like Figure 7 As shown in the figure, the results of two editing fusion methods are shown. The user added two time-independent edits, changing the face shape and hair, and added a time window edit. The first row shows the editing results of the edited sketch, the second row shows the original video, the third row shows the results of the fusion method of directly adding multiple edit vectors, and the fourth row shows the results of the region-aware fusion method. The quality of the results generated by the region-aware fusion method is higher than that of the direct addition of edit vectors, which proves the effectiveness of the region-aware fusion module in the present invention.
[0152] like Figure 8 As shown in Figure 1, the fusion results of different edits are shown. The first row is the original video, the second row is the result of time-independent editing, modifying the hair area of the face, the third row is the time window editing, adding a smile to the face, and the last row is the result of the fusion of the two edits.
[0153] like Fig. 9As shown in the figure, the intermediate results of face video editing are shown. The second row shows the real video, and the third row shows the aligned result. The fourth row shows the result of drawing the mask after deformation according to the expression and action, and the fifth row shows the edited result of the aligned face. The sixth row shows the result of face region segmentation, and the last row shows the final face edited result without alignment.
[0154] like Fig.10 The figure shows the results of key point smoothing. The first three rows are the results without key point smoothing. The face after cropping and alignment has a very large shake. The last three rows are the results with key point smoothing. The face after cropping and alignment has no shake problem.
[0155] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.
[0156] The present invention also proposes a sketch-based deep face video editing system, which includes:
[0157] Module 1 is used to align and crop the faces in the original video, and encode the faces into the latent space to obtain the latent codes of all frames in the face video;
[0158] Module 2, for adding sketch generation branches To StyleGAN to generate the network, reversely optimize the image hidden code and generate the edit vector δ edit ;
[0159] Module 3 is used to convert the edit vector δ edit The hidden code superimposed on all frames completes the propagation of timing-independent edits;
[0160] Module 4, for weighted superposition of edit vectors δ using piecewise linear functions edit , complete the editing and dissemination of actions or expressions;
[0161] Module 5 is used to calculate the weighted superposition editing vector δ according to the similarity of the expression parameters between the current frame and the edited frame. edit , so that edits correspond to specific expressions, completing expression-driven edit propagation;
[0162] Module 6 is used to use the region-aware fusion method to fuse different types of edits added to different frames, and fuse the face into the original video to obtain a sketch-based face video editing result.
[0163] The sketch-based deep face video editing system, wherein the module 1 is used to detect the face key points of the face video, and after smoothing with a time window, align and crop the face to generate a video frame sequence f1, f2, ..., f N , where N is the number of frames of the face video; project the frame sequence to the latent space W + , generate hidden code sequence w1,w2,…,w N .
[0164] The sketch-based deep face video editing system, wherein the module 2 is used to obtain the StyleGAN original generation network G and construct a generation network for modeling the joint probability distribution of real face images and sketches Generative Network include and Two branches, is the original generative network of G, used to generate realistic face images, Used to generate the corresponding sketch image; given the hidden code w of the image, Generate feature maps F1, F2, ..., F 14 , where F1 is used as The initial feature map of the branch; The feature map of the branch is upsampled and compared with the feature map F i The residual images after convolution are added to generate the sketch image corresponding to the hidden code w;
[0165] Use the dataset of image and sketch matching to train the sketch generation network S, which takes the face image as input and generates the corresponding sketch for training the sketch generation branch Randomly sample the hidden code w and input it Generate high-fidelity face images and the corresponding sketch Constructing the loss function Training sketch generation branch
[0166]
[0167] L VGG is the perceptual loss function, using the VGG19 model to measure visual similarity, L L2 is the pixel L2 loss, α1 and α2 are both preset weights;
[0168] After modeling the distribution of real images and sketches, according to the input face image x, the sketch s drawn edit and selected area m edit ; Project the face image x to W + space, get the hidden code w edit, the generated sketch The generated image is the same as the input sketch in the editing area. The non-edited area is the same as the original image, where w is obtained by the following loss function edit :
[0169] L editing (w edit )=β1L sketch +β2L rgb ,
[0170] L sketch The constraint editing area has the same structure as the sketch result. rgb The non-editable region is constrained to remain unchanged, β1 and β2 are hyperparameters, and the network is generated by fixing The weight of w edit ;
[0171] The final edit vector δ edit =w edit -w,δ edit represents the edit of the sketch and propagates it to the entire face video; for each frame f i , generate the corresponding edit vector:
[0172] δ i =δ edit ,i=1,2,…,N
[0173] The module 3 includes converting each frame f i The corresponding δ edit Propagate to the entire face video to generate an edited frame sequence.
[0174] The sketch-based deep face video editing system, wherein the module 4 is used to add a blink or smile action at a specific time in the face video, and at a specific frame f t Add edit vector δ edit , input duration h and change time l, for each frame f i , the present invention uses a piecewise linear function to generate a smooth propagation edit vector δ i , and obtain the new edit vector δ i :
[0175] δ i =γ·δ edit ,i=1,2,…,M
[0176]
[0177] t1=th / 2-1, t2=th / 2, t3=t+h / 2, t4=t+h / 2+1, t is the edit frame f t
[0178] The corresponding time;
[0179] These new edit vectors δ i Used to synthesize realistic face images;
[0180] This module 5 includes:
[0181] Given multiple key frames in the face video Extract facial expression parameters using 3D reconstruction and the corresponding edit vector M is the number of keyframes, and the expression guide edit is propagated as follows:
[0182]
[0183] e i is the input frame f i The expression parameter, C is the normalization term And edit vector Targeting the same area;
[0184] The module 6 is used to give a series of frame sequences f1, f2, ..., f N , the user selects M key frames k 1 ,k 2 ,…,k M Edit different regions, corresponding to M drawn marked regions m 1 ,m 2 ,…,m M ; For each frame f i , generate M edit vectors
[0185] For each frame to be predicted f i Generate a deformation field, deform the input marked area to generate M new marked areas is m j The area after the deformation of the action and expression; replace the local area of the feature map of the original frame with the new feature map:
[0186]
[0187] in, The initial feature map is G is the generative network of StyleGAN;
[0188] Downsampling Make it and have the same resolution; feature map For each of the M editing operations, we update them M times in total. We update the middle 5 feature maps of StyleGAN with a resolution from 32×32 to 128×128. The high resolution is obtained by converting the original hidden code w i Adjust the StyleGAN algorithm; apply the above fusion operation to all frames f i ,i=1,2,..,N, generate the edited and fused aligned face video;
[0189] Using the face segmentation method, the face marker areas of the input frame and the edited frame are generated and merged, and a smooth edge is generated for the merged marker area, which is further used as a fusion weight to fuse the faces before and after editing and de-align the fused face image to the original video to synthesize the face video editing result.
[0190] The present invention also proposes a storage medium for storing a program for executing any one of the sketch-based deep face video editing methods.
[0191] The present invention also proposes a client for use in any one of the sketch-based deep face video editing systems.
Claims
1. A sketch-based deep face video editing method, characterized in that: include: Step 1: Align and crop the faces in the original video, and encode the faces into the latent space to obtain the latent codes of all frames in the face video; Step 2: Add a sketch generation branch To StyleGAN to generate the network, reversely optimize the image hidden code and generate the edit vector δ edit ; where the sketch generates branches The processing object is a sketch drawn by a user, and the sketch is drawn on a marked area, and the marked area covers an area of the real image; Step 3: Edit vector δ edit Hidden codes applied to all frames, completing the propagation of timing-independent edits; Step 4: Use the weights and edit vector δ of the piecewise linear function edit Multiply to complete the editing and spreading of actions or expressions; Step 5: Calculate the weighted superposition edit vector δ according to the similarity between the expression parameters of the current frame and the edited frame edit , so that the edit vector δ edit Corresponding to specific expressions, complete expression-driven editing and dissemination; Step 6: Use the region-aware fusion method to fuse different types of edits added to different frames, and fuse the face into the original video to obtain the sketch-based face video editing result; The facial key points of the face video are detected, and the face is aligned and cropped after smoothing using a time window to generate a video frame sequence f1,f2,…,f N , where N is the number of frames of the face video; project the frame sequence to the latent space W + , generate hidden code sequence w1,w2,…,w N ; This step 2 includes: Get the StyleGAN original generative network G and build a generative network for modeling the joint probability distribution of real face images and sketches Generative Network include and Two branches, is the original generative network of G, used to generate realistic face images, Used to generate the corresponding sketch image; given the hidden code w of the image, Generate feature maps F1, F2, ..., F 14 , where F1 is used as The initial feature map of the branch; The feature map of the branch is upsampled and compared with the feature map F i The residual images after convolution are added to generate the sketch image corresponding to the hidden code w; Use the image and sketch matching dataset to train the sketch generation network S, which takes the face image as input and generates the corresponding sketch for training the sketch generation branch Randomly sample the hidden code w and input it Generate high-fidelity face images and the corresponding sketch Constructing the loss function Training sketch generation branch L VGG is the perceptual loss function, using the VGG19 model to measure visual similarity, L L2 is the pixel L2 loss, α1 and α2 are both preset weights; After modeling the distribution of real images and sketches, according to the input face image x, the sketch s drawn edit and selected area m edit ; Project the face image x to W + space, get the hidden code w edit , the generated sketch The generated image is the same as the input sketch in the editing area. The non-edited area is the same as the original image, where w is obtained by the following loss function edit : L editing (w edit )=β1L sketch +β2L rgb , L sketch The constraint editing area has the same structure as the sketch result. rgb The non-editable region is constrained to remain unchanged, β1 and β2 are hyperparameters, and the network is generated by fixing The weight of w edit ; The final edit vector δ edit =w edit -w,δ edit represents the edit of the sketch and propagates it to the entire face video; for each frame f i , generate the corresponding edit vector: d i =d edit ,i=1,2,…,N Step 3 includes converting each frame f i The corresponding δ edit Propagate to the entire face video and generate an edited frame sequence.
2. The sketch-based deep face video editing method according to claim 1, characterized in that: This step 4 includes: Add a blink or smile action at a specific time in the face video, and at a specific frame f t Add edit vector δ edit , input duration h and change time l, for each frame f i , the present invention uses a piecewise linear function to generate a smooth propagation edit vector δ i , and obtain the new edit vector δ i : d i =c·d edit ,i=1,2,…,M t1=th / 2-1, t2=th / 2, t3=t+h / 2, t4=t+h / 2+1, t is the edit frame f t The corresponding time; These new edit vectors δ i Used to synthesize realistic face images; This step 5 includes: Given multiple key frames in the face video Extract facial expression parameters using 3D reconstruction and the corresponding edit vector M is the number of keyframes, and the expression guide edit is propagated as follows: e i is the input frame f i The expression parameter, C is the normalization term And edit vector Targeting the same area; This step 6 includes: Given a series of frame sequences f1,f2,…,f N , the user selects M key frames k 1 ,k 2 ,…,k M Edit different regions, corresponding to M drawn marked regions m 1 ,m 2 ,…,m M ; For each frame f i , generate M edit vectors For each frame to be predicted f i Generate a deformation field, deform the input marked area to generate M new marked areas is m j The area after the deformation of the action and expression; replace the local area of the feature map of the original frame with the new feature map: in, The initial feature map is G is the generative network of StyleGAN; Downsampling Make it and have the same resolution; feature map For each of the M editing operations, we update them M times in total. We update the middle 5 feature maps of StyleGAN with a resolution from 32×32 to 128×128. The high resolution is obtained by converting the original hidden code w i Adjust the StyleGAN algorithm; apply the above fusion operation to all frames f i ,i=1,2,..,N, generate the edited and fused aligned face video; Using the face segmentation method, the face marker areas of the input frame and the edited frame are generated and merged, and a smooth edge is generated for the merged marker area, which is further used as a fusion weight to fuse the faces before and after editing and de-align the fused face image to the original video to synthesize the face video editing result.
3. A sketch-based deep face video editing system, characterized in that: include: Module 1 is used to align and crop the faces in the original video, and encode the faces into the latent space to obtain the latent codes of all frames in the face video; Module 2, for adding sketch generation branches To StyleGAN to generate the network, reversely optimize the image hidden code and generate the edit vector δ edit ; where the sketch generates branches The processing object is a sketch drawn by a user, and the sketch is drawn on a marked area, and the marked area covers an area of the real image; Module 3 is used to convert the edit vector δ edit Hidden codes applied to all frames, completing the propagation of timing-independent edits; Module 4, for using piecewise linear function weights and edit vectors δ edit Multiply to complete the editing and spreading of actions or expressions; Module 5 is used to calculate the weighted superposition editing vector δ according to the similarity of the expression parameters between the current frame and the edited frame. edit , so that the edit vector δ edit Corresponding to specific expressions, complete expression-driven editing and dissemination; Module 6 is used to use a region-aware fusion method to fuse different types of edits added to different frames and fuse the face into the original video to obtain a sketch-based face video editing result; The module 1 is used to detect the key points of the face in the face video, and align and crop the face after smoothing with the time window to generate a video frame sequence f1, f2, ..., f N , where N is the number of frames of the face video; project the frame sequence to the latent space W + , generate hidden code sequence w1,w2,…,w N ; This module 2 is used to obtain the StyleGAN original generation network G and construct a generation network for modeling the joint probability distribution of real face images and sketches. Generative Network include and Two branches, is the original generative network of G, used to generate realistic face images, Used to generate the corresponding sketch image; given the hidden code w of the image, Generate feature maps F1, F2, ..., F 14 , where F1 is used as The initial feature map of the branch; The feature map of the branch is upsampled and compared with the feature map F i The residual images after convolution are added to generate the sketch image corresponding to the hidden code w; Use the image and sketch matching dataset to train the sketch generation network S, which takes the face image as input and generates the corresponding sketch for training the sketch generation branch Randomly sample the hidden code w and input it Generate high-fidelity face images and the corresponding sketch Constructing the loss function Training sketch generation branch L VGG is the perceptual loss function, using the VGG19 model to measure visual similarity, L L2 is the pixel L2 loss, α1 and α2 are both preset weights; After modeling the distribution of real images and sketches, according to the input face image x, the sketch s drawn edit and selected area m edit ; Project the face image x to W + space, get the hidden code w edit , the generated sketch The generated image is the same as the input sketch in the editing area. The non-edited area is the same as the original image, where w is obtained by the following loss function edit : L editing (w edit )=β1L sketch +β2L rgb , L sketch The constraint editing area has the same structure as the sketch result. rgb The non-editable region is constrained to remain unchanged, β1 and β2 are hyperparameters, and the network is generated by fixing The weight of w edit ; The final edit vector δ edit =w edit -w,δ edit represents the edit of the sketch and propagates it to the entire face video; for each frame f i , generate the corresponding edit vector: d i =d edit ,i=1,2,…,N The module 3 includes converting each frame f i The corresponding δ edit Propagate to the entire face video and generate an edited frame sequence.
4. The sketch-based deep face video editing system as claimed in claim 3, characterized in that: The module 4 is used to add a blink or smile action at a specific time in the face video, and at a specific frame f t Add edit vector δ edit , input duration h and change time l, for each frame f i , the present invention uses a piecewise linear function to generate a smooth propagation edit vector δ i , and obtain the new edit vector δ i : d i =c·d edit ,i=1,2,…,M t1=th / 2-1, t2=th / 2, t3=t+h / 2, t4=t+h / 2+1, t is the edit frame f t The corresponding time; These new edit vectors δ i Used to synthesize realistic face images; This module 5 includes: Given multiple key frames in the face video Extract facial expression parameters using 3D reconstruction and the corresponding edit vector M is the number of keyframes, and the expression guide edit is propagated as follows: e i is the input frame f i The expression parameter, C is the normalization term And edit vector Targeting the same area; The module 6 is used to give a series of frame sequences f1, f2, ..., f N , the user selects M key frames k 1 ,k 2 ,…,k M Edit different regions, corresponding to M drawn marked regions m 1 ,m 2 ,…,m M ; For each frame f i , generate M edit vectors For each frame to be predicted f i Generate a deformation field, deform the input marked area to generate M new marked areas is m j The area after the deformation of the action and expression; replace the local area of the feature map of the original frame with the new feature map: in, The initial feature map is G is the generative network of StyleGAN; Downsampling Make it and have the same resolution; feature map For each of the M editing operations, we update them M times in total. We update the middle 5 feature maps of StyleGAN with a resolution from 32×32 to 128×128. The high resolution is obtained by converting the original hidden code w i Adjust the StyleGAN algorithm; apply the above fusion operation to all frames f i ,i=1,2,..,N, generate the edited and fused aligned face video; Using the face segmentation method, the face marker areas of the input frame and the edited frame are generated and merged, and a smooth edge is generated for the merged marker area, which is further used as a fusion weight to fuse the faces before and after editing and de-align the fused face image to the original video to synthesize the face video editing result.
Citation Information
Patent Citations
Face sketch synthesis system based on conditional enhancement generative adversarial network
CN111489405A
Face image generation method and device, equipment and medium
CN111652828A
Hand-drawn sketch guided image editing method based on hidden space search
CN113112572A
Video generation method and device, server and storage medium
CN113901894A
Video generation method and device, electronic equipment and storage medium
CN114255496A
