A multi-modal portrait video editing method, electronic device, and storage medium

Abstract

The application provides a multi-modal portrait video editing method, which can be applied to the technical field of video editing. The method comprises the following steps: given a portrait video, pre-processing to obtain camera parameters, human identity, expression, posture coefficient, human semantic segmentation graph and two-dimensional portrait mask; based on the neural Gaussian texture mechanism, embedding the learnable three-dimensional Gaussian feature into the parameterized human body geometric surface, using the neural renderer to convert the three-dimensional Gaussian splashing feature map into an image, optimizing the reconstruction of the three-dimensional portrait through the RGB and segmentation graph information of the video; using the iterative dataset update technology, distilling the knowledge of the multi-modal two-dimensional image generation model into the three-dimensional portrait editing, using the expression similarity guide and facial perception portrait editing model, and improving the editing quality. The method improves the two-dimensional editing task to three-dimensional, and has good three-dimensional consistency and time domain consistency. Through the knowledge of the multi-modal generation model, the high-quality portrait video editing function can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to the technical field of portrait video editing, and more particularly to a multi-modal portrait video editing method, an electronic device and a storage medium. BACKGROUND

[0002] Portrait video editing has a wide range of applications in the fields of movies, arts and augmented reality / virtual reality. How to ensure the structural similarity and temporal consistency of the edited video while achieving high-quality, multi-modal editing effects has always been an important problem in the field of image processing technology.

[0003] Early two-dimensional portrait editing methods are based on generative adversarial networks (GANs). However, two-dimensional portrait editing methods based on GANs are limited by the representation ability of GAN models, and the generated portraits often have low quality. Other two-dimensional portrait editing methods based on denoising diffusion models (Denoising Diffusion Probabilistic Models) are superior to two-dimensional portrait editing methods based on GANs in terms of performance, but the pictures generated by the above two-dimensional portrait editing methods based on denoising diffusion models often have temporal inconsistency problems between video frames when applied to video editing.

[0004] In order to solve the problems of low quality of generated portraits and inter-frame temporal inconsistency of the above two-dimensional portrait editing methods, the skilled person in the art uses a pre-trained image diffusion model for training-free video editing. However, due to the lack of 3D priori and facial or body priori, the method of using a pre-trained image diffusion model for training-free video editing may not be able to generate satisfactory video results in terms of quality and temporal consistency. At the same time, due to the complex denoising process of the denoising diffusion model, this method needs to spend several minutes of computing time to generate only 1 second of video segment. SUMMARY

[0005] In view of the above problems, the present application provides a multi-modal portrait video editing method, an electronic device and a storage medium.

[0006] According to a first aspect of the present application, a multi-modal portrait video editing method is provided, comprising:

[0007] The frame images of the portrait video are subjected to a dimensionality increasing operation by embedding a Gaussian point in a three-dimensional Gaussian field into a two-dimensional texture space of a parameterized human body geometry representation, to obtain a three-dimensional dynamic Gaussian feature field;

[0008] The three-dimensional Gaussian field is subjected to three-dimensional Gaussian splash to obtain three-dimensional Gaussian rendering features, and the three-dimensional Gaussian rendering features are converted by using a convolutional neural renderer to obtain a neural rendering RGB image;

[0009] The target person in the portrait video is reconstructed in three dimensions by using the three-dimensional dynamic Gaussian feature field, and the three-dimensional portrait reconstruction process is optimized by using the neural rendering RGB image and the original RGB image of the portrait video through the loss function in the reconstruction stage.

[0010] The editing information of the neural rendering RGB image is distilled to the optimized three-dimensional portrait reconstruction result to complete the updating of the three-dimensional portrait, and the edited neural rendering RGB image is used to replace the original frame image of the portrait video to complete the editing of the video frame dataset.

[0011] The editing process of the video frame dataset and the updating process of the three-dimensional portrait are alternately iterated, and the edited neural rendering RGB image and the original frame image of the portrait video are mapped to the same expression space.

[0012] The convolutional neural renderer is optimized by using the expression similarity loss function and the loss function in the reconstruction stage, and the three-dimensional dynamic Gaussian feature field is optimized again until the editing of all video frame images in the portrait video is completed.

[0013] According to the embodiment of the present application, the above-mentioned dimension increasing operation on the frame image of the portrait video by embedding the Gaussian points in the three-dimensional Gaussian field into the two-dimensional texture space of the parameterized human body geometry representation includes:

[0014] An initial parameterized human body geometry representation is constructed by using the portrait identity coefficient, the portrait posture coefficient and the expression coefficient of the target person obtained from each frame image of the portrait video.

[0015] The difference between the initial parameterized human body geometry representation and the real human body of the target person is corrected by using the learnable vertex offset to obtain a corrected parameterized human body geometry representation.

[0016] Based on the neural Gaussian texture mechanism, the Gaussian points in the three-dimensional Gaussian field mapped in the two-dimensional texture space are uniformly embedded on the surface of the corrected parameterized human body geometry representation to obtain a three-dimensional dynamic Gaussian feature field.

[0017] According to the embodiment of the present application, the above-mentioned construction of the initial parameterized human body geometry representation by using the portrait identity coefficient, the portrait posture coefficient and the expression coefficient of the target person obtained from each frame image of the portrait video includes:

[0018] The image identity coefficient, the image posture coefficient and the expression coefficient of the target person in each frame image in the portrait video are obtained by portrait tracking on each frame image in the portrait video.

[0019] Based on the image identity coefficient, the image posture coefficient and the expression coefficient of the target person, an initial parameterized human body geometry representation is constructed by using a predefined skin function and skin weight, a predefined joint matrix and human body geometry in a T-shaped posture.

[0020] According to the embodiment of the present application, the three-dimensional Gaussian rendering features are obtained by performing three-dimensional Gaussian splashing on the three-dimensional dynamic Gaussian feature field, and the neural rendering RGB image is obtained by converting the three-dimensional Gaussian rendering features by using the convolutional neural renderer.

[0021] The three-dimensional Gaussian rendering features are obtained by performing three-dimensional Gaussian splashing on the three-dimensional dynamic Gaussian feature field by using the camera intrinsic parameters and the camera pose of each frame image in the portrait video and the attribute information of the three-dimensional dynamic Gaussian feature field.

[0022] The convolutional neural renderer including an input layer, a convolutional layer, a nonlinear layer, a pooling layer, a fully connected layer and a loss layer is constructed, and the neural rendering RGB image is obtained by converting the three-dimensional Gaussian rendering features by using the convolutional neural renderer.

[0023] According to the embodiment of the present application, the attribute information of the three-dimensional dynamic Gaussian feature field includes spatial position information of Gaussian points, quaternions representing three-dimensional orientation of Gaussian field, stretch vectors, opacity and learnable feature vectors.

[0024] According to the embodiment of the present application, the three-dimensional reconstruction of the target person in the portrait video by using the three-dimensional dynamic Gaussian feature field, and the optimization of the three-dimensional portrait reconstruction process by using the neural rendering RGB image and the original RGB image of the portrait video by the loss function in the reconstruction stage include:

[0025] The two-dimensional portrait mask of each frame image in the portrait video is obtained by using a predefined image processing model to pre-process the portrait video.

[0026] The loss value in the reconstruction stage is obtained by processing the neural rendering RGB image, the original RGB image and the two-dimensional portrait mask by using the reconstruction loss function, the mask loss function and the stability loss function.

[0027] The three-dimensional portrait reconstruction process is supervised and trained based on the loss value in the reconstruction stage, and the optimized three-dimensional portrait reconstruction result is obtained.

[0028] According to the embodiment of the present application, the editing information of the neural rendering RGB image is distilled to the optimized three-dimensional portrait reconstruction result to complete the update of the three-dimensional portrait, which includes:

[0029] The portrait video is preprocessed by using a predefined image processing model to obtain a human body semantic segmentation map of each frame image in the portrait video, wherein the human body semantic segmentation map includes a head region and a torso part of a target person;

[0030] Based on the multi-modal editing instruction input by the user, a two-dimensional multi-modal generation module of the pre-trained face-aware portrait editing model is used to edit the head region and the torso part in the neural rendering RGB image respectively;

[0031] The edited head region and the edited torso part are synthesized by using the human body semantic segmentation map of each frame image, and the synthesized image is inserted into the video frame data set to further update the video frame data set;

[0032] The optimization of the three-dimensional dynamic high-strength characteristic field is completed to optimize the three-dimensional portrait reconstruction result, and the editing information is distilled to the optimized three-dimensional portrait reconstruction result by the face-aware portrait editing model to further update the three-dimensional portrait;

[0033] The updating operation of the three-dimensional portrait and the updating operation of the video frame data set are alternately iterated to continuously optimize the three-dimensional dynamic high-strength characteristic field.

[0034] According to the embodiments of the present application, the two-dimensional multi-modal generation module includes a two-dimensional multi-modal generation module based on text editing, a two-dimensional multi-modal generation module based on image editing, and a two-dimensional multi-modal image generation model of portrait relighting.

[0035] The second aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the multi-modal portrait video editing method.

[0036] The third aspect of the present application further provides a computer readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions are executed by a processor to implement the steps of the multi-modal portrait video editing method.

[0037] The multi-modal portrait video editing method provided by the application upgrades the portrait in the video, uses a three-dimensional dynamic Gaussian feature field to represent the person in the video by means of parameterized human geometry, and effectively ensures the three-dimensional consistency and time consistency of the portrait in the video; unlike the original three-dimensional Gaussian splash represented by spherical harmonic coefficients, the feature is represented by self-adaptive learning, so that the model capacity is higher; the convolutional neural renderer can effectively combine the three-dimensional Gaussian splash feature map in depth, further improve the expression ability of the portrait representation, make the edited result more consistent with the editing instruction, and at the same time show higher quality, especially good at fitting non-three-dimensional visual style effect; the face-aware portrait editing model can make the editing process pay more attention to the face area, and significantly enhance the editing quality of the face area; at the same time, the expression similarity guide term can ensure the correctness of the expression of the three-dimensional portrait, and avoid the expression degradation problem. BRIEF DESCRIPTION OF DRAWINGS

[0038] The above and other objects, features and advantages of the present application will become more apparent from the following description of embodiments of the present application, taken in conjunction with the accompanying drawings, in which:

[0039] Figure 1 is a flowchart of a multi-modal portrait video editing method according to an embodiment of the application;

[0040] Figure 2 is a pipeline diagram of a portrait video editing method based on a three-dimensional dynamic Gaussian feature field and a multi-modal generative prior according to an embodiment of the application;

[0041] Figure 3 is a visual result comparison schematic diagram based on text editing according to an embodiment of the application;

[0042] Figure 4 is a visual result comparison schematic diagram based on picture editing according to an embodiment of the application;

[0043] Figure 5 is a visual result comparison schematic diagram based on relighting according to an embodiment of the application;

[0044] Figure 6 The block diagram of the electronic device suitable for implementing the multi-modal portrait video editing method according to the embodiment of the application is schematically shown. DETAILED DESCRIPTION

[0045] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It should be understood, however, that the description is merely exemplary of the present application, and is not intended to limit the scope of the present application. In the following detailed description of the embodiments of the present application, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present application.

[0046] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so forth, shall be read expansively and without limitation. The terms "comprising," "including," and "having," and tautological expressions thereof (e.g., "comprises," "comprising," "containing," "contains," "includes," "including," "have," etc.) serve to specify the presence of stated features, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0047] All terms used herein (including technical and scientific terms) have the meanings commonly understood by one of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted as having a meaning that is consistent with the context of the specification, and not to be interpreted in an idealized or overly formal way.

[0048] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted to include any one of A, B, or C, to include any combination of A, B, and C, and to include all the possibilities of including each of A or B and C.

[0049] In the technical solutions of the present application, the user information (including but not limited to user personal information, user image information, user equipment information, such as location information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved are information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, public order and good customs are not violated, and corresponding operation entrances are provided for users to choose authorization or refusal.

[0050] Portrait video editing has a wide range of application scenarios and is the focus of research by those skilled in the art. Early two-dimensional portrait editing technical solutions mainly use generative adversarial networks (GAN) for portrait editing or portrait animation generation, and specify the style required for editing by giving a style label or a reference picture. A contrastive language-image pre-training (CLIP) model builds a bridge between text and images, and under the guidance of CLIP similarity, a GAN model can generate pictures that meet the text description. However, such work is limited by the representation ability of the GAN model, and the quality of the generated portraits is often not high. Recently, denoising diffusion probabilistic models (DDPM) have shown better performance than GANs in terms of generation ability. Based on this paradigm, a series of high-quality stylized portrait image generation models, fine-tuning methods, and adapters have emerged. However, directly using these image generation models for video editing often makes it difficult to maintain temporal consistency between frames.

[0051] To further improve the continuity of the edited video, some works choose to use pre-trained image diffusion models for training-free video editing. They constrain the motion of the edited video by using dense correspondences, or enhance the perception of the structural information of the input video through cross-frame attention or feature injection. Some works also attempt to connect frames in the time dimension and train a temporal attention module to ensure temporal or multi-view consistency. However, due to the lack of 3D priors and face / body priors, they may not be able to generate satisfactory video results in terms of quality and temporal consistency. At the same time, due to the complex denoising process of denoising diffusion models, these methods require several minutes of computation time to generate only 1 second of video footage.

[0052] To solve the technical problems existing in the prior art, the present application provides a high-efficiency and high-quality multi-modal portrait video editing method, which raises the portrait video editing problem to three dimensions, uses a three-dimensional dynamic Gaussian field to ensure three-dimensional consistency and temporal consistency of the video, distills the knowledge of existing two-dimensional multi-modal generation modules to three-dimensional portraits to achieve multi-modal and high-quality portrait editing, and uses three-dimensional Gaussian splashing technology to render the three-dimensional Gaussian portrait to obtain the final video editing result. With the help of three-dimensional Gaussian field construction and the efficiency of three-dimensional Gaussian splashing, the technical solution provided by the present application can achieve efficient portrait video editing and high-frame-rate portrait video rendering.

[0053] Figure 1 is a flowchart of a multi-modal portrait video editing method according to an embodiment of the present application.

[0054] As shown in Figure 1 , the multi-modal portrait video editing method includes operations S110-S160.

[0055] In operation S110, a frame image of a portrait video is subjected to a dimensionality increasing operation by embedding a Gaussian point in a three-dimensional Gaussian field into a two-dimensional texture space of a parameterized human body geometry, to obtain a three-dimensional dynamic Gaussian feature field.

[0056] The dimensionality of a portrait in a portrait video (or video, the same below) is increased, and a three-dimensional dynamic Gaussian feature field is used to represent a person in the video with the help of a parameterized human body geometry, the present application proposes a mechanism of neural Gaussian texture, which embeds a Gaussian point into a parameterized human body surface , and the Gaussian attributes include: Gaussian point position , quaternion representing Gaussian three-dimensional orientation , stretch vector , opacity , and a learnable feature . A uniform neural Gaussian field exists in the UV space (i.e., U-VEEZ, a two-dimensional texture coordinate with vertex component information of a polygon and a subdivision surface mesh) of the parameterized human body geometry, and based on the neural Gaussian field and the UV space of the human body geometry , a three-dimensional Gaussian feature field is obtained.

[0057] In operation S120, a three-dimensional Gaussian splashing is performed on the three-dimensional dynamic Gaussian feature field to obtain a three-dimensional Gaussian rendering feature, and a convolutional neural renderer is used to convert the three-dimensional Gaussian rendering feature to obtain a neural rendering RGB image.

[0058] Given the camera intrinsic , camera pose , and the above three-dimensional Gaussian feature field, a feature map is rendered. Then a convolutional neural network U is used as a neural renderer to convert it to an RGB image.

[0059] In operation S130, a three-dimensional reconstruction is performed on a target person in the portrait video using the three-dimensional dynamic Gaussian feature field, and the neural rendering RGB image and the original RGB image of the portrait video are used to optimize the three-dimensional portrait reconstruction process through the loss function in the reconstruction stage.

[0060] The three-dimensional reconstruction of the dynamic portrait in the video is given, and the camera intrinsic and the portrait identity coefficient simultaneously obtaining the camera pose of each frame , portrait pose , expression coefficient , and body semantic segmentation map and two-dimensional portrait mask. Given the body parameters and camera parameters and of each frame, render a three-dimensional portrait RGB picture, and optimize the reconstruction of the three-dimensional portrait by the RGB and segmentation map information of the video.

[0061] In operation S140, the editing information of the neural rendering RGB image is distilled to the optimized three-dimensional portrait reconstruction result to complete the update of the three-dimensional portrait, and the edited neural rendering RGB image is used to replace the original frame image of the portrait video to complete the editing of the video frame dataset.

[0062] In operation S150, the editing process of the video frame dataset and the updating process of the three-dimensional portrait are alternately iterated, and the edited neural rendering RGB image and the original frame image of the portrait video are mapped to the same expression space.

[0063] Using the iterative dataset updating technique, the dataset video frame editing and three-dimensional portrait updating are alternately performed, and the editing of the two-dimensional multi-modal generation module is distilled to three dimensions. Specifically, the three-dimensional portrait is optimized every few steps, and the two-dimensional multi-modal generation module is used to edit a frame in the dataset, and then the edited frame is replaced by the original frame. With the progressive updating of the three-dimensional portrait and the dataset, the knowledge of the two-dimensional multi-modal generation module can finally be distilled to the three-dimensional portrait.

[0064] In operation S160, the convolutional neural renderer is optimized using the expression similarity loss function and the loss function in the reconstruction stage, and the three-dimensional dynamic high-frequency feature field is optimized again until the editing of all video frame images in the portrait video is completed.

[0065] The above operation S150 and operation S160 also involve editing the head (or face, the same below) region and the torso part of the target person in the portrait video. Specifically, when editing an upper body image, if the face occupies a small proportion, the editing may not be fine enough to handle the face structure. The present application further proposes a face-aware portrait editing model which does not need to be trained on the two-dimensional multi-modal generation module and can significantly enhance the editing quality of the face region. First, the face region is cropped, then the face part and the portrait part are edited by the image editing model, and the edited two parts are synthesized into the final edited image by the head-torso semantic segmentation map.

[0066] To further avoid the problem of facial expression degradation caused by the accumulation of editing errors during iterative dataset updates, this invention uses a pre-trained facial expression recognition network to map the rendered frames and original frames into the same facial expression space and calculate their facial expression similarity.

[0067] The multimodal portrait video editing method provided by this invention enhances the dimensionality of the portraits in the video. By leveraging parametric human geometry and using a three-dimensional dynamic Gaussian feature field to represent the people in the video, it effectively ensures the three-dimensional consistency and temporal consistency of the portraits in the video. Unlike the original three-dimensional Gaussian splashing which uses spherical harmonic coefficients to represent color, this method utilizes adaptively learnable features, resulting in a higher model capacity. The convolutional neural renderer of this invention can effectively perform deep combination of the three-dimensional Gaussian splashing feature map, further improving the expressive power of the portrait representation, making the edited result more consistent with the editing instructions, and exhibiting higher quality, especially adept at fitting non-three-dimensional visual style effects. The face-aware portrait editing model of this invention allows the editing process to focus more on the facial area, significantly enhancing the quality of facial area editing. At the same time, the expression similarity guidance term of this invention can ensure the accuracy of the expressions of the three-dimensional portraits and avoid expression degradation problems.

[0068] The following specific embodiments and appendices Figure 2 The multimodal portrait video editing method provided by the present invention will be described in further detail below.

[0069] Figure 2 This is a pipeline diagram of a portrait video editing method based on a three-dimensional dynamic Gaussian feature field and a multimodal generative prior, according to an embodiment of the present invention.

[0070] like Figure 2 As shown, the multimodal portrait editing provided by the present invention can be roughly divided into three-dimensional portrait reconstruction, video frame dataset updating, and three-dimensional portrait editing stages.

[0071] According to an embodiment of the present invention, the above-described method of performing a dimensionality-up operation on frame images of a portrait video by embedding Gaussian points in a three-dimensional Gaussian field into a two-dimensional texture space of a parameterized human body geometry representation to obtain a three-dimensional dynamic Gaussian feature field includes: constructing an initial parameterized human body geometry representation using the portrait identity coefficient, portrait pose coefficient, and expression coefficient of the target person obtained from each frame image of the portrait video; correcting the difference between the initial parameterized human body geometry representation and the real human body of the target person using learnable vertex offsets to obtain a corrected parameterized human body geometry representation; and uniformly embedding Gaussian points mapped from the three-dimensional Gaussian field in the two-dimensional texture space onto the surface of the corrected parameterized human body geometry representation based on a neural Gaussian texture mechanism to obtain a three-dimensional dynamic Gaussian feature field.

[0072] According to an embodiment of the present invention, the above-mentioned construction of an initial parametric human geometric representation using the portrait identity coefficient, portrait pose coefficient, and expression coefficient of the target person obtained from each frame of the portrait video includes: performing portrait tracking on each frame of the portrait video to obtain the portrait identity coefficient, portrait pose coefficient, and expression coefficient of the target person in each frame; and constructing an initial parametric human geometric representation based on the portrait identity coefficient, portrait pose coefficient, and expression coefficient of the target person, using predefined skinning functions and skinning weights, predefined joint matrix, and human geometry under T-pose.

[0073] According to an embodiment of the present invention, the above-described method of performing three-dimensional Gaussian splashing on a three-dimensional dynamic Gaussian feature field to obtain three-dimensional Gaussian rendering features, and then using a convolutional neural renderer to transform the three-dimensional Gaussian rendering features to obtain a neurally rendered RGB image, includes: performing three-dimensional Gaussian splashing on the three-dimensional dynamic Gaussian feature field using the camera intrinsics and camera pose of each frame in the portrait video, as well as the attribute information of the three-dimensional dynamic Gaussian feature field, to obtain three-dimensional Gaussian rendering features; constructing a convolutional neural renderer including an input layer, a convolutional layer, a nonlinear layer, a pooling layer, a fully connected layer, and a loss layer, and using the convolutional neural renderer to transform the three-dimensional Gaussian rendering features to obtain a neurally rendered RGB image.

[0074] According to an embodiment of the present invention, the attribute information of the above-mentioned three-dimensional dynamic Gaussian feature field includes the spatial location information of Gaussian points, quaternions representing the three-dimensional orientation of the Gaussian field, scaling vector, opacity, and learnable feature vectors.

[0075] The following specific embodiments, in conjunction with the appendix, demonstrate this process. Figure 2 The process of acquiring and correcting parametric human geometric representation, the principle of neural Gaussian texture mechanism, and the acquisition process of neurally rendered RGB images are explained in further detail.

[0076] This invention elevates the problem of portrait video editing to three dimensions, utilizing a three-dimensional dynamic Gaussian feature field to ensure the three-dimensional and temporal consistency of portrait videos. This invention constructs a three-dimensional dynamic Gaussian feature field based on a parameterized human geometric representation, such as... Figure 2 The three-dimensional Gaussian features are shown. The three-dimensional human portrait representation is obtained using the following operations.

[0077] Parametric human geometry based on human identity coefficients Human pose and expression coefficient Decision, expressed as As shown in formula (1):

[0078] (1),

[0079] in is a skin function, is a skin weight, is a joint. Human geometry in T-pose From formula (2):

[0080] (2).

[0081] The present application uses a learnable vertex offset to further correct the difference between the parameterized human geometry and the real human, and obtains the corrected human geometry , as formula (3):

[0082] (3).

[0083] The present application proposes a mechanism of neural Gaussian texture, which embeds Gaussian points on the surface of the parameterized human The Gaussian attributes include: Gaussian point position , quaternion representing Gaussian three-dimensional orientation , stretch vector , opacity , a learnable feature , as shown in Figure 2 The present application exists in the UV space of the parameterized human geometry , based on the human geometry and the neural Gaussian field of the UV space , a three-dimensional Gaussian feature field is obtained, as shown in formula (4):

[0084] (4),

[0085] Wherein, is UV mapping.

[0086] Given the camera intrinsic , camera pose , and the above three-dimensional Gaussian feature field, a feature map is rendered, as shown in formula (5):

[0087] (5),

[0088] Then, a convolutional neural network is used as a neural renderer , which is converted into an RGB image, as shown in formula (6):

[0089] (6).

[0090] According to the embodiment of the present application, the above-mentioned three-dimensional reconstruction of the target person in the portrait video by using the three-dimensional dynamic high-strength characteristic field, and optimization of the three-dimensional portrait reconstruction process by using the loss function in the reconstruction stage, the RGB image of the neural rendering, and the original RGB image of the portrait video includes: pre-processing the portrait video by using a pre-defined image processing model to obtain a two-dimensional portrait mask of each image in the portrait video; processing the neural rendering RGB image, the original RGB image and the two-dimensional portrait mask by using the reconstruction loss function, the mask loss function and the stability loss function to obtain the loss value in the reconstruction stage; and supervising and training the three-dimensional portrait reconstruction process based on the loss value in the reconstruction stage to obtain the optimized three-dimensional portrait reconstruction result.

[0091] According to the embodiment of the present application, the above-mentioned updating of the three-dimensional portrait by distilling the editing information of the neural rendering RGB image to the optimized three-dimensional portrait reconstruction result includes: pre-processing the portrait video by using a pre-defined image processing model to obtain a human body semantic segmentation map of each image in the portrait video, wherein the human body semantic segmentation map includes a head region and a torso part of the target person; based on the multi-modal editing instruction input by the user, a two-dimensional multi-modal generation module of a pre-trained face-aware portrait editing model is used to edit the head region and the torso part in the neural rendering RGB image respectively; the edited head region and the edited torso part are synthesized by using the human body semantic segmentation map of each image, and the synthesized image is inserted into the video frame data set to update the video frame data set; the three-dimensional portrait reconstruction result is optimized by optimizing the three-dimensional dynamic high-strength characteristic field, and the editing information is distilled to the optimized three-dimensional portrait reconstruction result by the face-aware portrait editing model to update the three-dimensional portrait; the updating operation of the three-dimensional portrait and the updating operation of the video frame data set are alternately iterated to continuously optimize the three-dimensional dynamic high-strength characteristic field.

[0092] According to the embodiment of the present application, the above-mentioned two-dimensional multi-modal generation module includes a two-dimensional multi-modal generation module based on text editing, a two-dimensional multi-modal generation module based on image editing, and a two-dimensional multi-modal image generation model of portrait relighting.

[0093] The above-mentioned three-dimensional portrait reconstruction, optimization and updating process and the editing (or updating) process of the video frame data set provided by the present application will be further described in detail below through specific embodiments and in combination with the accompanying drawings. Figure 2 The above-mentioned three-dimensional portrait reconstruction, optimization and updating process and the editing (or updating) process of the video frame data set provided by the present application will be further described in detail below through specific embodiments and in combination with the accompanying drawings.

[0094] Given a portrait video, first, the original video is reconstructed into a three-dimensional portrait, and then the editing knowledge of the two-dimensional multi-modal generation module is distilled to three dimensions.

[0095] In the data preprocessing stage, given a portrait video, the camera intrinsic parameters are obtained by tracking the portrait and human portrait identity coefficient , while obtaining the camera pose of each frame , human posture , expression coefficient , human body semantic segmentation map and two-dimensional human portrait mask.

[0096] In the reconstruction phase, the present application uses a reconstruction loss function, a mask loss function and a stability loss function.

[0097] Among them, the reconstruction loss function ( is the rendered RGB image, is the frame image of the original video) as shown in formula (7):

[0098] (7).

[0099] Among them, the mask loss function ( is the alpha value rendered, is the two-dimensional human portrait mask) as shown in formula (8):

[0100] (8).

[0101] Among them, the stability loss function ( is the first three channels of ) as shown in formula (9):

[0102] (9).

[0103] In addition, in order to improve the details, the present application uses VGG network (Visual Geometry Group Network) as backbone to calculate the perception loss function.

[0104] After the reconstruction is completed, the present application uses iterative dataset update technology to alternately perform dataset video frame editing and three-dimensional human portrait updating, and distills the editing of the two-dimensional multi-modal generation module to three dimensions. Specifically, the present application uses the two-dimensional multi-modal generation module to edit a frame in the dataset every few steps of optimization of the three-dimensional human portrait, and then replaces the original frame with the edited frame. With the progressive updating of the three-dimensional human portrait and the dataset, the knowledge of the two-dimensional multi-modal generation module can finally be distilled to the three-dimensional human portrait.

[0105] In addition to the loss functions used in the reconstruction, the present application additionally uses an expression similarity loss function to further avoid the expression degradation problem caused by the accumulation of editing errors in the iterative dataset update process. The present application uses a pre-trained expression recognition network to map the rendered frame and the original frame into the same expression space and calculate the expression similarity, as shown in formula (10):

[0106] (10)

[0107] in It is a network that encodes facial expressions in images. It renders video frames. These are the original video frames.

[0108] When editing an upper-body image, if the face occupies a small proportion, the editing may not be sufficient to finely process the facial structure. This invention further proposes a face-aware portrait editing model (which includes the aforementioned two-dimensional multimodal generation module), which significantly enhances the editing quality of the facial region without requiring training of the two-dimensional multimodal generation module. Figure 2 As shown, the facial region is first cropped. Then, both the facial and portrait parts are edited using a facial perception portrait editing model. The two edited parts are then synthesized using a head-to-thigh semantic segmentation map to create the final edited image.

[0109] To support the multimodal portrait editing needs in different application scenarios, this invention uses pre-trained image generation models based on text editing, image editing, and portrait relighting. The above methods are also applicable to other structure-preserving image generation models.

[0110] This invention provides a portrait video editing method based on a 3D dynamic Gaussian feature field and multimodal generative priors. Given a portrait video, it first preprocesses the video to obtain camera parameters, human identity, expression, pose coefficients, human semantic segmentation map, and a 2D portrait mask. Based on a neural Gaussian texture mechanism, learnable 3D Gaussian features are embedded into a parameterized human geometric surface. A neural renderer converts the 3D Gaussian splash feature map into an image. The 3D portrait is then optimized and reconstructed using the video's RGB and segmentation map information. An iterative dataset update technique is used to distill the knowledge of the multimodal 2D image generation model into the 3D portrait editing. During this process, expression similarity guidance and a facial perception portrait editing model are used to further improve editing quality. This method elevates the 2D editing task to 3D, and by leveraging 3D human priors and 3D Gaussian splashing technology, the edited video exhibits excellent 3D and temporal consistency. Utilizing the knowledge of the multimodal generative model, high-quality portrait video editing functions such as text editing, image editing, and portrait relighting can be achieved.

[0111] The following comparative experiments, combined with appendices, demonstrate this process. Figures 3-5 To verify the advantages and effectiveness of the method provided by the present invention. Figure 3 This is a schematic diagram comparing visual results based on text editing according to an embodiment of the present invention. Figure 4 This is a schematic diagram comparing visual results based on image editing according to an embodiment of the present invention. Figure 5is a comparison diagram of visual results based on re-lighting according to an embodiment of the present application.

[0112] The methods to be compared are: (1) TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR 2024); (2) Rerender A Video (RAV): Zero-Shot Text-Guided Video-to-Video Translation (SIGGRAPH Asia 2023 Conference Proceedings); (3) CoDeF: Content Deformation Fields for Temporally Consistent Video Processing (CVPR 2024 Highlight); (4) AnyV2V: A Plug-and-Play Framework For Any Video-to-Video Editing Tasks (arXiv preprint).

[0113] It should be particularly noted that: Figures 3-5 The image data involved herein is only used for illustrative purposes and will not be used for commercial purposes; wherein, Figures 3-5 The images of public figures involved come from research institutions for scientific research purposes, and the image data of the above-mentioned public figures provided by them has obtained the authorization of the relevant public figures, is only used for scientific research purposes, and is an open data set facing the public; Figures 3-5 The image data of non-public figures in the above-mentioned public figures has obtained the authorization of the relevant parties, and the processing of the portrait is only used to illustrate the advantages and effectiveness of the method provided by the present application. In addition, all the above-mentioned image data has taken effective security measures, and the process of processing also strictly follows the requirements of relevant laws and regulations.

[0114] In terms of editing video quality comparison, as Figures 3-5 shown, the multi-modal portrait video editing method provided by the present application is superior to other methods in terms of the degree of fit with the editing instruction, the structure preservation degree, the time consistency, and the overall video quality.

[0115] In the video editing efficiency comparison method, the editing efficiency of the method provided by the present application is further verified by analyzing the number of frames processed per minute of each method, as shown in Table 1. Different methods use different inference modes. In order to compare fairly, the present application calculates the time cost including reconstruction and editing, and averages on the number of processed frames. It can be seen that the method of the present application exceeds the previous video editing methods in efficiency.

[0116] Table 1 Comparison of editing efficiency of various methods

[0117]

[0118] Figure 6 A block diagram of an electronic device suitable for implementing the multi-modal portrait video editing method according to an embodiment of the present application is schematically shown.

[0119] As shown in Figure 6 , the electronic device 600 according to an embodiment of the present application includes a processor 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or loaded into a random access memory (RAM) 603 from a storage portion 608. The processor 601 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), and the like. The processor 601 can also include an on-board memory for cache use. The processor 601 can include a single processing unit or multiple processing units for performing different actions of the method processes according to embodiments of the present application.

[0120] In the RAM 603, various programs and data required for the operation of the electronic device 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The processor 601 performs various operations of the method processes according to embodiments of the present application by executing programs in the ROM 602 and / or the RAM 603. It should be noted that the programs can also be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 can also perform various operations of the method processes according to embodiments of the present application by executing programs stored in the one or more memories.

[0121] According to an embodiment of the present application, the electronic device 600 can further include an input / output (I / O) interface 605 that is also connected to the bus 604. The electronic device 600 can further include one or more of the following components connected to the input / output (I / O) interface 605: an input part 606 including a keyboard, a mouse, etc.; an output part 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 608 including a hard disk, etc.; and a communication part 609 including a network interface card such as a LAN card, a modem, etc. The communication part 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as necessary. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 610 as necessary, so that a computer program read out therefrom is installed in the storage part 608 as necessary.

[0122] The present application also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present application.

[0123] According to an embodiment of the present application, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, the computer readable storage medium can include one or more memories such as the ROM 602 and / or the RAM 603 described above and / or one or more memory other than the ROM 602 and the RAM 603.

[0124] The computer program product of the present application can be a computer program product comprising a computer readable storage medium and a computer program mechanism embedded in the computer readable storage medium. Such computer program product can further include a computer readable storage medium and program means for causing a processor or other programmable processing apparatus to function in a particular manner, such that the computer program mechanism embedded in the computer readable storage medium can be used to actually effect the apparatus functions.

[0125] Those skilled in the art will appreciate that the features recited in the various embodiments of the present application can be combined and / or integrated in a variety of ways, even if such combinations or integrations are not expressly noted in the present application. In particular, the features recited in the various embodiments of the present application can be combined and / or integrated in a variety of ways without departing from the spirit and scope of the present application. All such combinations and / or integrations are within the scope of the present application.

[0126] The above-described embodiments of the application are merely descriptive of its application and not restrictive. Although the above embodiments have been described in some detail, those skilled in the art will appreciate that various modifications and substitutions can be made of the embodiments without departing from the spirit and scope of the application as defined by the appended claims.

Claims

1. A multimodal portrait video editing method, characterized in that, The method includes: By embedding Gaussian points in a three-dimensional Gaussian field into a two-dimensional texture space representing parameterized human geometry, the frame images of the portrait video are subjected to dimensionality upscaling to obtain a three-dimensional dynamic Gaussian feature field. The three-dimensional dynamic Gaussian feature field is subjected to three-dimensional Gaussian splashing to obtain three-dimensional Gaussian rendering features, and the three-dimensional Gaussian rendering features are transformed using a convolutional neural renderer to obtain a neurally rendered RGB image. The target person in the portrait video is reconstructed in three dimensions using the three-dimensional dynamic Gaussian feature field, and the three-dimensional portrait reconstruction process is optimized by using the RGB image of the neural rendering and the original RGB image of the portrait video through the loss function of the reconstruction stage. The editing information of the neurally rendered RGB image is distilled onto the optimized 3D human portrait reconstruction result to complete the 3D human portrait update, and the original frame image of the human portrait video is replaced with the edited neurally rendered RGB image to complete the editing of the video frame dataset. The video frame dataset editing process and the 3D human portrait update process are performed alternately and iteratively, and the edited neurally rendered RGB image is mapped to the same expression space as the original frame image of the human portrait video; The convolutional neural renderer is optimized using the facial expression similarity loss function and the loss function of the reconstruction stage, and the three-dimensional dynamic Gaussian feature field is optimized again until the editing of all video frame images in the portrait video is completed.

2. The method according to claim 1, characterized in that, By embedding Gaussian points in a three-dimensional Gaussian field into a two-dimensional texture space representing parameterized human geometry, the frame images of the portrait video are subjected to dimensionality upscaling to obtain a three-dimensional dynamic Gaussian feature field, including: Using the facial identity coefficient, facial pose coefficient, and facial expression coefficient of the target person obtained from each frame of the facial video, an initial parameterized human geometric representation is constructed. The difference between the initial parametric human geometry representation and the real human body of the target person is corrected by using learnable vertex offsets, resulting in a corrected parametric human geometry representation. Based on the neural Gaussian texture mechanism, the three-dimensional Gaussian field is mapped onto Gaussian points in the two-dimensional texture space and uniformly embedded into the surface of the corrected parameterized human geometry representation to obtain the three-dimensional dynamic Gaussian feature field.

3. The method according to claim 2, characterized in that, Constructing an initial parametric human geometric representation using the target person's facial identity coefficient, facial pose coefficient, and facial expression coefficient obtained from each frame of the facial video includes: By performing portrait tracking on each frame of the portrait video, the portrait identity coefficient, portrait pose coefficient, and expression coefficient of the target person in each frame are obtained; Based on the target person's facial identity coefficient, facial pose coefficient, and facial expression coefficient, the initial parameterized human geometric representation is constructed using predefined skinning functions and skinning weights, predefined joint matrix, and human geometry under T-pose.

4. The method according to claim 1, characterized in that, The three-dimensional dynamic Gaussian feature field is subjected to three-dimensional Gaussian splashing to obtain three-dimensional Gaussian rendering features. These features are then transformed using a convolutional neural renderer to obtain a neurally rendered RGB image, including: Using the camera intrinsics and camera pose of each frame in the portrait video, as well as the attribute information of the three-dimensional dynamic Gaussian feature field, a three-dimensional Gaussian splash is performed on the three-dimensional dynamic Gaussian feature field to obtain the three-dimensional Gaussian rendering features. A convolutional neural renderer is constructed, comprising an input layer, a convolutional layer, a nonlinear layer, a pooling layer, a fully connected layer, and a loss layer. The convolutional neural renderer is then used to transform the 3D Gaussian rendering features to obtain a neurally rendered RGB image.

5. The method according to claim 4, characterized in that, The attribute information of the three-dimensional dynamic Gaussian feature field includes the spatial location information of the Gaussian points, the quaternion representing the three-dimensional orientation of the Gaussian field, the scaling vector, the opacity, and the learnable feature vector.

6. The method according to claim 1, characterized in that, The target person in the portrait video is reconstructed in three dimensions using the three-dimensional dynamic Gaussian feature field. The three-dimensional portrait reconstruction process is optimized using the RGB image rendered by the neural network and the original RGB image of the portrait video through a loss function in the reconstruction stage. The portrait video is preprocessed using a predefined image processing model to obtain a two-dimensional portrait mask for each frame of the portrait video; The neurally rendered RGB image, the original RGB image, and the two-dimensional human face mask are processed using the reconstruction loss function, the masking loss function, and the stabilization loss function to obtain the loss value in the reconstruction stage. The 3D human portrait reconstruction process is supervised and trained based on the loss value of the reconstruction stage to obtain the optimized 3D human portrait reconstruction result.

7. The method according to claim 1, characterized in that, Distilling the editing information from the neurally rendered RGB image onto the optimized 3D portrait reconstruction result to update the 3D portrait includes: The portrait video is preprocessed using a predefined image processing model to obtain a human semantic segmentation map for each frame of the portrait video, wherein the human semantic segmentation map includes the head region and torso of the target person. Based on user-input multimodal editing commands, the two-dimensional multimodal generation module of the pre-trained facial perception portrait editing model is used to edit the head region and the torso in the neurally rendered RGB image respectively; Using the human semantic segmentation map of each frame image, the edited head region and the edited torso are synthesized, and the synthesized image is inserted into the video frame dataset to complete the update of the video frame dataset; The 3D portrait reconstruction result is optimized by optimizing the 3D dynamic Gaussian feature field, and the editing information is distilled onto the optimized 3D portrait reconstruction result through the face perception portrait editing model to complete the 3D portrait update. The three-dimensional dynamic Gaussian feature field is continuously optimized by iteratively updating the three-dimensional human image and the video frame dataset.

8. The method according to claim 7, characterized in that, The two-dimensional multimodal generation module includes a two-dimensional multimodal generation module based on text editing, a two-dimensional multimodal generation module based on image editing, and a two-dimensional multimodal image generation model for portrait relighting.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, It stores a computer program or instructions thereon, characterized in that, when the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images