Video portrait consistency editing method based on optical flow guiding and text driving

Abstract

The invention discloses a video portrait consistency editing method based on optical flow guidance and text driving, and the method comprises the steps: firstly, employing a face key point extraction method and a depth estimation method to extract a face key point and a depth map of each frame of an input video, and guaranteeing the consistency of the facial expression of an edited portrait and a source image; a diffusion control network is used to edit and generate the face depth map, the key points and the user input text description as conditions, and the consistency of multi-view editing is realized through the potential code alignment of an attention mechanism and the guidance of optical flow change; thirdly, constructing a diffusion model with consistent time by utilizing the inter-frame optical flow change and the depth map change; through the above steps, the problems of multi-view and time consistency in video editing are solved, and high-quality video portrait editing is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of portrait generation in computer vision, and in particular to a video portrait consistency editing method based on optical flow guidance and text drive. Background Art

[0002] The importance of video portrait editing has increased significantly in virtual reality and related applications, such as avatar editing and stylization. However, this field is still in its early stages and faces significant challenges. Early portrait editing methods have made significant progress in manipulating the latent space of 3DGAN models. However, they are limited to editing preset attributes and cannot process natural language input. To address these issues, several advanced text-supported image editing methods have emerged, such as AnyFace and InstructPix2Pix. However, these methods are limited to the 2D domain and cannot directly generate 3D results, resulting in a lack of consistency under multiple viewpoints.

[0003] Recent approaches transform the challenge of 3D editing into a more manageable 2D editing task. For example, InstructNeRF2NeRF (IN2N) uses InstructPix2Pix to incrementally modify images rendered from NeRF (neural radiance fields) while updating the underlying NeRF reconstruction. IN2N edits one image at a time, optimizes the 3D NeRF, and produces an average result. Although this approach introduces some geometric consistency, it converges slowly and requires multiple full NeRF optimizations. Therefore, the problem of maintaining consistency between rendered views remains. To address the issue of multi-view consistency, GaussCtrl combines image editing with depth maps by using ControlNet to ensure geometric consistency of the generated view. In addition, they introduce an attention-based latent code alignment module, select multiple reference views, and use cross-view attention to enforce consistency in appearance during editing. However, these operations may cause geometric inconsistencies in some frames if the selected view is too similar to the current view.

[0004] Since volume rendering usually involves a manual approximation of the real-world imaging process, the rendered images may suffer from a loss of details. In addition, diffusion model-based methods have difficulty separating temporal information from the 3D representation. This limitation leads to temporal inconsistencies in the avatar-driven output, which is a critical issue in applications involving video portrait editing.

[0005] In summary, in order to better develop downstream applications of virtual reality, human-computer interaction, and video portrait editing, a video portrait editing method that can achieve multi-perspective consistency and temporal consistency is urgently needed. Summary of the invention

[0006] The present invention aims to provide a video portrait consistency editing method based on optical flow guidance and text drive. First, the facial key points and depth map of each frame of the video are extracted by a facial key point extraction method and a depth estimation method to ensure the consistency of facial expressions after editing; then, a diffusion control network is used to combine the depth map, key points and user text for editing, and the attention mechanism and optical flow changes are used to achieve multi-view consistency; then, a time-consistent diffusion model is constructed in combination with optical flow and depth map changes, thereby solving the multi-view and time consistency problems in video editing and achieving high-quality video portrait editing.

[0007] The technical solution to achieve the purpose of the present invention is: a video portrait consistency editing method based on optical flow guidance and text drive, comprising the following steps:

[0008] (1) Obtain a video of a person speaking, a portrait depth map corresponding to each video frame, and a portrait key point map corresponding to each video frame;

[0009] (2) The portrait depth map, portrait key point map, image to be edited, and source text prompt are used to calculate their latent codes through the variational encoder of the control network; the denoising diffusion implicit model inversion method is used to iteratively convert the codes into corresponding Gaussian noise;

[0010] (3) Calculate the optical flow changes between other viewpoint video frames and the frame to be edited, and select the first two frames with the largest optical flow changes as candidate frames for the attention alignment module; perform self-attention operation on the frame to be edited, and perform cross-attention operation on the frame to be edited and the candidate frame, and perform weighted sum of the two attentions to achieve attention alignment;

[0011] (4) By controlling the network to receive the aligned attention features in step (3) and the user edited text description, the noise is decoded to obtain a multi-view consistent edited portrait;

[0012] (5) Construct a temporally consistent diffusion model conditioned on the temporal information of the source video. The temporal information of the source video is jointly represented by the optical flow change prior between frames, the depth change prior, and the portrait contour constraint prior. The edited portrait in step (4) and the above prior information are input into the temporally consistent diffusion model to obtain an edited portrait that is consistent in multiple perspectives and in temporal order.

[0013] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned video portrait consistency editing method based on optical flow guidance and text drive is implemented.

[0014] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned video portrait consistency editing method based on optical flow guidance and text drive.

[0015] A computer program product includes a computer program, which, when executed by a processor, implements the above-mentioned video portrait consistency editing method based on optical flow guidance and text drive.

[0016] Compared with the prior art, the significant progress of the present invention lies in: (1) the present invention proposes a ControlNet-based neural renderer, which is designed to edit speaker avatars with consistency across multiple viewpoints and time series; (2) the optical flow changes in the source video frames are used to select more valuable reference viewpoints for attention alignment, thereby achieving better multi-view consistency; (3) the optical flow changes and depth map changes in the source video frames are used to train the diffusion model and refine the edited avatars, thereby enhancing temporal consistency.

[0017] In order to more clearly illustrate the functional characteristics and structural parameters of the present invention, further description is given below in conjunction with the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of the video portrait consistency editing method based on optical flow guidance and text drive of the present invention.

[0019] Figure 2 This is the overall framework diagram of the video portrait consistency editing method proposed by the present invention.

[0020] Figure 3 This is a graph showing the experimental results of the portrait editing of the present invention.

[0021] Figure 4 This is a graph showing the experimental comparison results of the present invention and other methods. DETAILED DESCRIPTION

[0022] like Figure 1 As shown, the present invention proposes a video portrait consistency editing method based on optical flow guidance and text driving, comprising the following steps:

[0023] Step 1: Get a video of a person speaking , the portrait depth map corresponding to each video frame And the portrait key point map corresponding to each video frame ,in is the number of video frames;

[0024] Portrait Depth Map And portrait key point map The acquisition method is: use the face key point extraction method EMOCA and the depth estimation method DepthFM to extract the facial key points and depth map of each frame of the input video respectively:

[0025] ;

[0026] and They are the key point extraction function and the depth estimation function respectively.

[0027] Step 2: Transform the portrait depth map , Portrait key point map , the image to be edited And source text hint , through the control network The variational encoder computes its latent code ; Using the denoising diffusion implicit model inversion method, the code is converted into the corresponding Gaussian noise in an iterative manner ; The specific inversion method of denoising diffusion implicit model (DDIM) is:

[0028]

[0029] ,

[0030] in , represents the time step of the diffusion process, represents the total step length of diffusion; Refers to the The noise of the moment, is text describing the content of the video. It is Gaussian noise at time, Unet network representing the diffusion model, Represents a vector concatenation operation, Representative The diffusion weight coefficient at the moment When , we get the Gaussian noise of step 2 .

[0031] Step 3: Calculate the optical flow changes between other viewpoint video frames and the frame to be edited, and select the first two frames with the largest optical flow changes as candidate frames for the attention alignment module; perform self-attention operation on the frame to be edited, and perform cross-attention operation on the frame to be edited and the candidate frame, and perform weighted sum of the two attentions to achieve attention alignment; the details are as follows:

[0032] Calculate other perspective video frames The frame to be edited Optical flow changes:

[0033] = ;

[0034] is the optical flow function, The frame to be edited Video frames from his perspective The optical flow change map, where the brightness and color of each pixel represent the intensity and direction of the optical flow, select the first two frames with the largest intensity as candidate frames for the attention alignment module ;

[0035] The frame to be edited and candidate frames Perform a cross-attention operation to weight the two attentions and align the attentions. Specifically:

[0036]

[0037]

[0038] , and is a linear projection of the attention operation query, key, and value, is the scale factor. It is Frame and Cross-attention operation across frames. Refers to Self-attention operation on frames. It is Attention alignment operation for frames. , =0.6 represents the weight coefficient.

[0039] Step 4: Control the network Receive the aligned attention features from step 3 and the user edited text Describe to decode the noise to get a multi-view consistent edited portrait , specifically:

[0040]

[0041]

[0042] Refers to the noise of the reference view frame at time t, is the noise of the reference view frame at time t.

[0043] Step 5, construct a time-consistent diffusion model based on the time sequence information of the source video. The time sequence information of the source video is jointly represented by the optical flow change prior between frames, the depth change prior, and the portrait contour constraint prior. The edited portrait in step 4 and the above prior information are jointly input into the time-consistent diffusion model to obtain an edited portrait that is consistent in multiple perspectives and time sequence.

[0044] A temporal consistency diffusion model based on the source video temporal information is constructed, specifically:

[0045]

[0046] is the noise of the edited frame at time t, is the noise vector of the edited frame at time t; The frame to be edited Optical flow changes compared to frame 1 , optical flow changes with the previous frame , and the depth change from the first frame , the depth change from the previous frame , the contour constraints of the current frame The final edited portrait result with consistent multi-view and time sequence is obtained by decoding the noise. .

[0047] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0048] Example

[0049] like Figure 2 As shown, the present invention includes two stages, which are implemented based on a multi-view consistency module and a temporal consistency module. The specific steps of the optical flow-based selection strategy (OFS) in the multi-view consistency module are as follows:

[0050] (1) Calculate the frame to be edited With other video frames The optical flow change is calculated using the calcOpticalFlowFarneback function in OpenCV:

[0051] = ,

[0052] is the optical flow function, The frame to be edited Video frames from his perspective The optical flow change map, the brightness and color of each pixel represents the intensity and direction of the optical flow.

[0053] (2) According to the intensity of optical flow change, a descending sorting algorithm is used to select the first ranked video frame as the first reference view candidate frame:

[0054]

[0055] in , represents the set in descending order according to the magnitude of optical flow change, is a sorting algorithm, Represents the total number of pixels, Represents the amplitude information of the optical flow change map, is the index of the optical flow change map, is the pixel index.

[0056] (3) In order to select the second video candidate frame, we do not select the second ranked video frame as the result, which may make the two candidate frames too similar and lose the value of multi-view reference. Therefore, under the premise of maximizing the optical flow change, the direction of the optical flow change is used as the basis for selecting the candidate frame:

[0057]

[0058]

[0059]

[0060] in is the threshold of the change in optical flow direction (unit: °), For the The second reference perspective frame of the edited frame. To get the specified value The index function of .

[0061] represents the direction of optical flow change, , The number of video frames.

[0062] like Figure 2 As shown, the first stage of the method of the present invention is training-free, and the second stage requires training, wherein the purpose of optimizing the model is achieved by minimizing the RGB loss, and the loss function here is the loss calculation performed on the final output result.

[0063]

[0064]

[0065] in is the total number of pixels, refers to the optical flow loss function, For the The number of pixels between the first prediction edit frame and the second prediction edit frame is The optical flow RGB value of For the The pixel difference between the first source video frame and the second source video frame is The optical flow RGB value of . Represents the RGB color of the optical flow map, Refers to the set of pixels. The final loss function is expressed as follows:

[0066]

[0067] in is the total loss function of network training. The training process is iteratively optimized according to the calculation results until the maximum number of training iterations is reached and the entire training process is completed.

[0068] After completing the entire training process, the corresponding editing results can be generated based on the model trained in the second stage. The generated results are as follows: Figure 3 In addition, the comparison results of the present invention and other methods are shown in Figure 4 As shown, the present invention can be more accurately aligned with the user instruction description and has better 3D consistency.

[0069] The above is a detailed description of the specific implementation examples of the present invention. It should be noted that the present invention is not limited to the above specific implementation examples, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essential content of the present invention.

Claims

1. A video portrait consistency editing method based on optical flow guidance and text drive, characterized in that: The following steps are involved: (1) Obtain a video of a person speaking, a portrait depth map corresponding to each video frame, and a portrait key point map corresponding to each video frame; (2) The portrait depth map, portrait key point map, image to be edited, and source text prompt are used to calculate their latent codes through the variational encoder of the control network; the denoising diffusion implicit model inversion method is used to iteratively convert the codes into corresponding Gaussian noise; (3) Calculate the optical flow changes between other viewpoint video frames and the frame to be edited, and select the first two frames with the largest optical flow change as the candidate frame for the attention alignment module; Perform self-attention operation on the frame to be edited, and perform cross-attention operation on the frame to be edited and the candidate frame, and perform weighted sum of the two attentions to achieve attention alignment; (4) By controlling the network to receive the aligned attention features in step (3) and the user edited text description, the noise is decoded to obtain a multi-view consistent edited portrait; (5) Construct a temporally consistent diffusion model conditioned on the temporal information of the source video. The temporal information of the source video is jointly represented by the optical flow change prior between frames, the depth change prior, and the portrait contour constraint prior. The edited portrait in step (4) and the above prior information are input into the temporally consistent diffusion model to obtain an edited portrait that is consistent in multiple perspectives and in temporal order.

2. The method for consistent video portrait editing based on optical flow guidance and text drive according to claim 1, characterized in that: In step (1), obtain a video of a person speaking , the portrait depth map corresponding to each video frame And the portrait key point map corresponding to each video frame ,in is the number of video frames; portrait depth map And portrait key point map The acquisition method is: use the face key point extraction method and the depth estimation method to extract the facial key points and depth map of each frame of the input video respectively: ； in, and They are the key point extraction function and the depth estimation function respectively.

3. The method for consistent video portrait editing based on optical flow guidance and text drive according to claim 1, characterized in that: In step (2), the portrait depth map , Portrait key point map , the image to be edited And source text hint , through the control network The variational encoder computes its latent code ; Using the denoising diffusion implicit model inversion method, the code is converted into the corresponding Gaussian noise in an iterative manner ; The denoising diffusion implicit model inversion method is specifically as follows: ； ； in, , represents the time step of the diffusion process, represents the total step length of diffusion; For the The noise of the moment, For the The noise vector at time , Represents a vector concatenation operation, Representative The diffusion weight coefficient at the moment When , we get the Gaussian noise of step (2) .

4. The method for consistent video portrait editing based on optical flow guidance and text drive according to claim 3, characterized in that: In step (3), calculate the video frames of other perspectives The frame to be edited Optical flow changes: = ； in, is the optical flow function, The frame to be edited Video frames from his perspective The optical flow change map, where the brightness and color of each pixel represent the intensity and direction of the optical flow, select the first two frames with the largest intensity as candidate frames for the attention alignment module .

5. The method for consistent video portrait editing based on optical flow guidance and text drive according to claim 4, characterized in that: In step (3), the frame to be edited and candidate frames Perform a cross-attention operation to weight the two attentions and align the attentions. Specifically: ； ； in, , and is a linear projection of the attention operation query, key, and value, is the scale factor; It is Frame and Cross-attention operation on frames; It is Self-attention operation on frames; It is Frame attention alignment operation; , =0.6 represents the weight coefficient.

6. The method for consistent video portrait editing based on optical flow guidance and text drive according to claim 5, characterized in that: In step (4), by controlling the network Receive the aligned attention features from step (3) and the user edited text Describe to decode the noise to get a multi-view consistent edited portrait , specifically: ； ； in, is the noise vector of the reference view frame at time t.

7. The method for consistent video portrait editing based on optical flow guidance and text drive according to claim 6, characterized in that: In step (5), a temporal consistency diffusion model based on the temporal information of the source video is constructed, specifically: ； in, is the noise of the edited frame at time t, is the noise vector of the edited frame at time t; The frame to be edited Optical flow changes compared to frame 1 , optical flow changes with the previous frame , and the depth change from the first frame , the depth change from the previous frame , the contour constraints of the current frame Vector splicing; by decoding the noise, the final multi-view consistent and time-series consistent edited portrait result is obtained .

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images