Multi-mode portrait video editing method, electronic equipment and storage medium

Abstract

The invention provides a multi-mode portrait video editing method which can be applied to the technical field of video editing. The method comprises the steps that a portrait video is given, and camera parameters, human body identities, expressions, posture coefficients, a human body semantic segmentation map and a two-dimensional portrait mask are obtained through preprocessing; based on a neural Gaussian texture mechanism, three-dimensional Gaussian features capable of being learned are embedded into a parameterized human body geometric surface, a neural renderer is used for converting a three-dimensional Gaussian splashing feature graph into an image, and a three-dimensional portrait is optimized and reconstructed through RGB and segmentation graph information of a video; an iterative data set updating technology is used, knowledge of a multi-modal two-dimensional image generation model is distilled to three-dimensional portrait editing, expression similarity is used for guiding a facial perception portrait editing model, and the editing quality is improved. According to the method, a two-dimensional editing task is improved to a three-dimensional editing task, and good three-dimensional consistency and time domain consistency are achieved. And a high-quality portrait video editing function can be realized through knowledge of a multi-modal generation model.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, specifically to the field of portrait video editing technology, and more specifically to a multimodal portrait video editing method, electronic device and storage medium. Background Art

[0002] Portrait video editing has wide applications in film, art, and augmented reality / virtual reality. How to ensure the structural similarity and temporal consistency of the edited videos while achieving high-quality, multimodal editing effects has always been an important issue in the field of image processing technology.

[0003] Early 2D portrait editing methods were based on generative adversarial networks (GANs). However, due to the limited representation capabilities of the GAN model, the quality of the portraits generated by GAN-based 2D portrait editing methods is often not high. Although other 2D portrait editing methods based on denoising diffusion probabilistic models are better than GAN-based 2D portrait editing methods in performance, the images generated by the above-mentioned denoising diffusion probabilistic models often have the problem of temporal inconsistency between video frames when applied to video editing.

[0004] In order to solve the problems of low quality of generated portraits and time inconsistency between frames in the above-mentioned two-dimensional portrait editing methods, technicians in this field have used pre-trained image diffusion models for untrained video editing. However, due to the lack of 3D priors and face or body priors, the method of untrained video editing using pre-trained image diffusion models may not be able to generate satisfactory video results in terms of quality and time consistency. At the same time, due to the complex denoising process of the denoising diffusion model, such methods require several minutes of computing time to generate only 1 second of video clips. Summary of the invention

[0005] In view of the above problems, the present invention provides a multimodal portrait video editing method, an electronic device and a storage medium.

[0006] According to a first aspect of the present invention, a multimodal portrait video editing method is provided, comprising:

[0007] By embedding Gaussian points in the three-dimensional Gaussian field into the two-dimensional texture space of parameterized human body geometry representation, the frame image of the portrait video is upgraded to obtain a three-dimensional dynamic Gaussian feature field;

[0008] Perform 3D Gaussian splashing on the 3D dynamic Gaussian feature field to obtain 3D Gaussian rendering features, and use the convolutional neural renderer to transform the 3D Gaussian rendering features to obtain a neural rendered RGB image;

[0009] The target person in the portrait video is reconstructed in 3D using the 3D dynamic Gaussian feature field, and the 3D portrait reconstruction process is optimized using the neural rendered RGB image and the original RGB image of the portrait video through the loss function of the reconstruction stage;

[0010] The editing information of the neural rendered RGB image is distilled to the optimized 3D portrait reconstruction result to complete the update of the 3D portrait, and the edited neural rendered RGB image is used to replace the original frame image of the portrait video to complete the editing of the video frame dataset;

[0011] The video frame dataset editing process and the 3D portrait updating process are performed alternately and iteratively, and the edited neural-rendered RGB image and the original frame image of the portrait video are mapped to the same expression space;

[0012] The convolutional neural renderer is optimized using the expression similarity loss function and the loss function of the reconstruction stage, and the dynamic three-dimensional Gaussian feature field is optimized again until the editing of all video frame images in the portrait video is completed.

[0013] According to an embodiment of the present invention, the above-mentioned dimensionality-enhancing operation on the frame image of the portrait video by embedding the Gaussian points in the three-dimensional Gaussian field into the two-dimensional texture space represented by the parameterized human body geometry to obtain the three-dimensional dynamic Gaussian feature field includes:

[0014] The initial parameterized human body geometry representation is constructed by using the target person's portrait identity coefficient, portrait posture coefficient and expression coefficient obtained from each frame of the portrait video;

[0015] The difference between the initial parameterized human body geometry representation and the real human body of the target person is corrected by using the learnable vertex offset to obtain the corrected parameterized human body geometry representation;

[0016] Based on the neural Gaussian texture mechanism, the Gaussian points mapped in the 3D Gaussian field in the 2D texture space are uniformly embedded into the surface of the rectified parameterized human body geometry representation to obtain a 3D dynamic Gaussian feature field.

[0017] According to an embodiment of the present invention, the above-mentioned construction of an initial parameterized human body geometry representation using the portrait identity coefficient, portrait posture coefficient and expression coefficient of the target person obtained from each frame image of the portrait video includes:

[0018] By tracking each frame of the portrait video, the portrait identity coefficient, portrait posture coefficient and expression coefficient of the target person in each frame are obtained;

[0019] Based on the target person's portrait identity coefficient, portrait posture coefficient and expression coefficient, an initial parameterized human body geometry representation is constructed using predefined skinning functions and skinning weights, predefined joint point matrix and human body geometry in T-pose.

[0020] According to an embodiment of the present invention, the three-dimensional dynamic Gaussian feature field is subjected to three-dimensional Gaussian splashing to obtain three-dimensional Gaussian rendering features, and the three-dimensional Gaussian rendering features are transformed using a convolutional neural renderer to obtain a neural rendered RGB image, including:

[0021] The camera intrinsic parameters and camera pose of each frame in the portrait video and the attribute information of the 3D dynamic Gaussian feature field are used to perform 3D Gaussian splashing on the 3D dynamic Gaussian feature field to obtain the 3D Gaussian rendering feature;

[0022] A convolutional neural renderer including input layer, convolution layer, nonlinear layer, pooling layer, fully connected layer and loss layer is constructed, and the convolutional neural renderer is used to transform the three-dimensional Gaussian rendering features to obtain a neural rendered RGB image.

[0023] According to an embodiment of the present invention, the attribute information of the three-dimensional dynamic Gaussian feature field includes spatial position information of Gaussian points, a quaternion representing the three-dimensional orientation of the Gaussian field, a scaling vector, opacity, and a learnable feature vector.

[0024] According to an embodiment of the present invention, the above-mentioned three-dimensional reconstruction of the target person in the portrait video using the three-dimensional dynamic Gaussian feature field, and optimizing the three-dimensional portrait reconstruction process using the neural rendered RGB image and the original RGB image of the portrait video through the loss function of the reconstruction stage includes:

[0025] Preprocess the portrait video using a predefined image processing model to obtain a two-dimensional portrait mask for each frame of the portrait video;

[0026] The reconstruction loss function, mask loss function and stability loss function are used to process the neural rendered RGB image, the original RGB image and the 2D portrait mask to obtain the loss value of the reconstruction stage;

[0027] The 3D portrait reconstruction process is supervised and trained based on the loss value in the reconstruction stage to obtain the optimized 3D portrait reconstruction result.

[0028] According to an embodiment of the present invention, the above-mentioned distillation of the editing information of the neural rendered RGB image to the optimized 3D portrait reconstruction result to complete the update of the 3D portrait includes:

[0029] Preprocess the portrait video using a predefined image processing model to obtain a human body semantic segmentation map for each frame of the portrait video, wherein the human body semantic segmentation map includes the head area and torso part of the target person;

[0030] Based on the multimodal editing instructions input by the user, the 2D multimodal generation module of the pre-trained face-aware portrait editing model is used to edit the head area and torso part in the neural rendered RGB image respectively;

[0031] Using the human body semantic segmentation map of each frame image, the edited head area and the edited torso part are synthesized, and the synthesized image is inserted into the video frame data set to complete the update of the video frame data set;

[0032] The 3D portrait reconstruction result is optimized by optimizing the 3D dynamic Gaussian feature field, and the editing information is distilled to the optimized 3D portrait reconstruction result through the face-aware portrait editing model to complete the 3D portrait update.

[0033] The three-dimensional dynamic Gaussian feature field is continuously optimized by alternately and iteratively updating the three-dimensional portrait and the video frame data set.

[0034] According to an embodiment of the present invention, the above-mentioned two-dimensional multimodal generation module includes a two-dimensional multimodal generation module based on text editing, a two-dimensional multimodal generation module based on image editing, and a two-dimensional multimodal image generation model for portrait relighting.

[0035] A second aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the multimodal portrait video editing method.

[0036] The third aspect of the present invention also provides a computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the steps of the multimodal portrait video editing method are implemented.

[0037] The above-mentioned multimodal portrait video editing method provided by the present invention upgrades the dimension of the portrait in the video, uses parameterized human body geometry, and uses dynamic three-dimensional Gaussian feature fields to characterize the people in the video, effectively ensuring the three-dimensional consistency and time consistency of the portrait in the video; unlike the original three-dimensional Gaussian splashing that uses spherical harmonic coefficients to characterize color, it uses adaptively learnable features to characterize, so that the model capacity is higher; the convolutional neural renderer of the present invention can effectively deeply combine the three-dimensional Gaussian splashing feature map, further improve the expressive ability of the portrait representation, so that the edited result is more in line with the editing instructions, and at the same time shows higher quality, especially good at fitting non-three-dimensional visual style effects; the facial perception portrait editing model of the present invention can make the editing process pay more attention to the facial area, significantly enhancing the editing quality of the facial area; at the same time, the expression similarity guidance item of the present invention can ensure the correctness of the expression of the three-dimensional portrait and avoid the problem of expression degradation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0039] Figure 1 is a flowchart of a multimodal portrait video editing method according to an embodiment of the present invention;

[0040] Figure 2 is a pipeline diagram of a portrait video editing method based on a three-dimensional dynamic Gaussian feature field and multimodal generative priors according to an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of visual result comparison based on text editing according to an embodiment of the present invention;

[0042] Figure 4 is a schematic diagram of visual result comparison based on picture editing according to an embodiment of the present invention;

[0043] Figure 5 is a schematic diagram of visual result comparison based on relighting according to an embodiment of the present invention;

[0044] Figure 6 A block diagram of an electronic device suitable for implementing a multimodal portrait video editing method according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0045] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0046] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0047] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0048] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0049] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0050] Portrait video editing has a wide range of application scenarios and is a key research method for technicians in this field. Early two-dimensional portrait editing technology solutions mainly used generative adversarial networks for portrait editing or portrait animation generation, specifying the style required for editing by giving a style label or a reference image. The Contrastive Language-Image Pre-Training (CLIP) model builds a bridge between text and images. Under the guidance of CLIP similarity, the GAN model can generate images that match the text description. However, this type of work is limited by the representation ability of the GAN model, and the quality of the generated portraits is often not high. Recently, the Denoising Diffusion Probabilistic Models has demonstrated superior performance to GAN in terms of generation capabilities. Based on this paradigm, a series of image generation models, fine-tuning methods, and adapters for high-quality stylized portraits have emerged. However, it is often difficult to maintain temporal consistency between frames when these image generation models are directly used for video editing.

[0051] In order to further improve the continuity of the edited video, some works choose to use pre-trained image diffusion models for untrained video editing. They constrain the motion of the edited video by using dense correspondences, or enhance the perception of the structural information of the input video by means of cross-frame attention or feature injection. Some works also try to connect frames in the temporal dimension and train the temporal attention module to ensure temporal or multi-view consistency. However, due to the lack of 3D priors and face / body priors, they may not be able to generate satisfactory video results in terms of quality and temporal consistency. At the same time, due to the complex denoising process of the denoising diffusion model, these methods take several minutes of computing time to generate only 1 second of video clips.

[0052] In order to solve the technical problems existing in the prior art, the present invention provides an efficient and high-quality multimodal portrait video editing method, which elevates the portrait video editing problem to three dimensions, uses a dynamic three-dimensional Gaussian feature field to ensure the three-dimensional consistency and time consistency of the video; distills the knowledge of the existing two-dimensional multimodal generation module to the three-dimensional portrait to achieve multimodal and high-quality portrait editing; and uses the three-dimensional Gaussian splashing technology to render the three-dimensional Gaussian portrait to obtain the final video editing result. With the help of the three-dimensional Gaussian field construction and the high efficiency of the three-dimensional Gaussian splashing, the technical solution provided by the present invention can achieve efficient portrait video editing and high-frame rate portrait video rendering.

[0053] Figure 1 4 is a flowchart of a multimodal portrait video editing method according to an embodiment of the present invention.

[0054] like Figure 1 As shown, the above-mentioned multimodal portrait video editing method includes operations S110 to S160.

[0055] In operation S110, a dimensionality increase operation is performed on the frame image of the portrait video by embedding Gaussian points in the three-dimensional Gaussian field into a two-dimensional texture space represented by parameterized human body geometry to obtain a three-dimensional dynamic Gaussian feature field.

[0056] The human figures in the portrait video (or video, the same below) are upgraded in dimension. With the help of parameterized human body geometry, a dynamic three-dimensional Gaussian feature field is used to characterize the human figures in the video. The present invention proposes a neural Gaussian texture mechanism to embed Gaussian points into the parameterized human body surface. The Gaussian properties include: Gaussian point position , quaternion representing Gaussian three-dimensional orientation , Stretch Vector , Opacity , a learnable feature . The uniform neural Gaussian field In the parametric human body geometry UV space (i.e. U-VEEZ, which is a two-dimensional texture coordinate with vertex component information of polygons and subdivision surface meshes), based on the human body geometry and the neural Gaussian field in UV space , and obtain the three-dimensional Gaussian feature field.

[0057] In operation S120, three-dimensional Gaussian splashing is performed on the three-dimensional dynamic Gaussian feature field to obtain three-dimensional Gaussian rendering features, and the three-dimensional Gaussian rendering features are transformed using a convolutional neural renderer to obtain a neural-rendered RGB image.

[0058] Given camera intrinsics , camera pose , and the above three-dimensional Gaussian feature field, rendering to obtain the feature map . Then the convolutional neural network U is used as a neural renderer to convert it into an RGB image.

[0059] In operation S130, a target person in the portrait video is 3D reconstructed using a 3D dynamic Gaussian feature field, and the 3D portrait reconstruction process is optimized using a neural rendered RGB image and an original RGB image of the portrait video through a loss function in the reconstruction phase.

[0060] Perform 3D reconstruction of dynamic portraits in the video. Given a portrait video, track the portrait to obtain the camera internal parameters. and portrait identity coefficient , and get the camera pose of each frame , Portrait pose , expression coefficient As well as the human body semantic segmentation map and 2D portrait mask. Given the human body parameters of each frame And camera parameters and , render 3D portrait RGB images, and optimize and reconstruct 3D portraits through the RGB and segmentation map information of the video.

[0061] In operation S140, the editing information of the neural-rendered RGB image is distilled onto the optimized 3D portrait reconstruction result to complete the updating of the 3D portrait, and the edited neural-rendered RGB image is used to replace the original frame image of the portrait video to complete the editing of the video frame data set.

[0062] In operation S150, the editing process of the video frame data set and the updating process of the three-dimensional portrait are alternately and iteratively performed, and the edited neural-rendered RGB image and the original frame image of the portrait video are mapped to the same expression space.

[0063] By using iterative dataset update technology, dataset video frame editing and 3D portrait updating are performed alternately, and the editing of the 2D multimodal generation module is distilled to the 3D. Specifically, after every optimization of the 3D portrait, the present invention uses the 2D multimodal generation module to edit a frame in the dataset, and then replaces the original frame with the edited frame. With the progressive update of the 3D portrait and the dataset, the knowledge of the 2D multimodal generation module can be finally distilled to the 3D portrait.

[0064] In operation S160, the convolutional neural renderer is optimized using the expression similarity loss function and the loss function of the reconstruction phase, and the dynamic three-dimensional Gaussian feature field is optimized again until the editing of all video frame images in the portrait video is completed.

[0065] The above-mentioned operations S150 and S160 also involve editing the head (or face, the same below) area and torso part of the target person in the portrait video. Specifically, when editing an upper body image, if the face occupies a small proportion, the editing may not be sufficient to finely process the facial structure. The present invention further proposes a face-aware portrait editing model, which can significantly enhance the editing quality of the facial area without the need to train a two-dimensional multimodal generation module. First, the facial area is cropped, and then both the facial part and the portrait part are edited through the image editing model, and the edited two parts are synthesized into the final edited image through the head-torso semantic segmentation map.

[0066] In order to further avoid the expression degradation problem caused by the accumulation of editing errors during the iterative dataset update process, the present invention uses a pre-trained expression recognition network to map the rendered frame and the original frame into the same expression space and calculate their expression similarity.

[0067] The above-mentioned multimodal portrait video editing method provided by the present invention upgrades the dimension of the portrait in the video, uses parameterized human body geometry, and uses dynamic three-dimensional Gaussian feature fields to characterize the people in the video, effectively ensuring the three-dimensional consistency and time consistency of the portrait in the video; unlike the original three-dimensional Gaussian splashing that uses spherical harmonic coefficients to characterize color, it uses adaptively learnable features to characterize, so that the model capacity is higher; the convolutional neural renderer of the present invention can effectively deeply combine the three-dimensional Gaussian splashing feature map, further improve the expressive ability of the portrait representation, so that the edited result is more in line with the editing instructions, and at the same time shows higher quality, especially good at fitting non-three-dimensional visual style effects; the facial perception portrait editing model of the present invention can make the editing process pay more attention to the facial area, significantly enhancing the editing quality of the facial area; at the same time, the expression similarity guidance item of the present invention can ensure the correctness of the expression of the three-dimensional portrait and avoid the problem of expression degradation.

[0068] The following is a detailed description of the embodiments and the attached Figure 2 The multimodal portrait video editing method provided by the present invention is further described in detail.

[0069] Figure 2 It is a pipeline diagram of a portrait video editing method based on a three-dimensional dynamic Gaussian feature field and multimodal generative priors according to an embodiment of the present invention.

[0070] like Figure 2 As shown, the above-mentioned multimodal portrait editing provided by the present invention is roughly divided into three-dimensional portrait reconstruction, video frame data set update and three-dimensional portrait editing stages.

[0071] According to an embodiment of the present invention, the above-mentioned dimensionality increase operation is performed on the frame image of the portrait video by embedding the Gaussian points in the three-dimensional Gaussian field into the two-dimensional texture space of the parameterized human body geometry representation to obtain the three-dimensional dynamic Gaussian feature field, including: using the portrait identity coefficient, portrait posture coefficient and expression coefficient of the target person obtained from each frame image of the portrait video to construct an initial parameterized human body geometry representation; using a learnable vertex offset to correct the difference between the initial parameterized human body geometry representation and the real human body of the target person to obtain a corrected parameterized human body geometry representation; based on the neural Gaussian texture mechanism, the Gaussian points mapped in the three-dimensional Gaussian field in the two-dimensional texture space are uniformly embedded into the surface of the corrected parameterized human body geometry representation to obtain a three-dimensional dynamic Gaussian feature field.

[0072] According to an embodiment of the present invention, the above-mentioned construction of an initial parameterized human body geometry representation using the portrait identity coefficient, portrait posture coefficient and expression coefficient of the target person obtained from each frame image of the portrait video includes: obtaining the portrait identity coefficient, portrait posture coefficient and expression coefficient of the target person in each frame image by performing portrait tracking on each frame image in the portrait video; based on the portrait identity coefficient, portrait posture coefficient and expression coefficient of the target person, using a predefined skinning function and skinning weight, a predefined joint point matrix and the human body geometry in T-type posture, constructing an initial parameterized human body geometry representation.

[0073] According to an embodiment of the present invention, the above-mentioned three-dimensional Gaussian splashing of the three-dimensional dynamic Gaussian feature field to obtain three-dimensional Gaussian rendering features, and using a convolutional neural renderer to transform the three-dimensional Gaussian rendering features to obtain a neural-rendered RGB image includes: using the camera intrinsic parameters and camera posture of each frame image in the portrait video and the attribute information of the three-dimensional dynamic Gaussian feature field to perform three-dimensional Gaussian splashing on the three-dimensional dynamic Gaussian feature field to obtain three-dimensional Gaussian rendering features; constructing a convolutional neural renderer including an input layer, a convolutional layer, a nonlinear layer, a pooling layer, a fully connected layer and a loss layer, and using the convolutional neural renderer to transform the three-dimensional Gaussian rendering features to obtain a neural-rendered RGB image.

[0074] According to an embodiment of the present invention, the attribute information of the three-dimensional dynamic Gaussian feature field includes spatial position information of Gaussian points, a quaternion representing the three-dimensional orientation of the Gaussian field, a scaling vector, opacity, and a learnable feature vector.

[0075] The following is a specific embodiment and combined with the attached Figure 2 The acquisition and correction process of the parametric human body geometry representation, the principle of the neural Gaussian texture mechanism, and the acquisition process of the neural rendered RGB image are further explained in detail.

[0076] The present invention elevates the portrait video editing problem to three dimensions, and uses a three-dimensional dynamic Gaussian feature field to ensure the three-dimensional consistency and time consistency of the portrait video. The present invention constructs a three-dimensional dynamic Gaussian feature field based on parameterized human body geometry representation, such as Figure 2 The three-dimensional Gaussian features are shown in FIG. Among them, the three-dimensional portrait representation is obtained by the following operation.

[0077] Parameterized human geometry by portrait identity coefficients , portrait pose and expression coefficient Decide, expressed as , as shown in formula (1):

[0078] (1),

[0079] in is the skinning function, is the skin weight, The human body geometry in T-stance From formula (2), we get:

[0080] (2).

[0081] The present invention uses learnable vertex offsets to further correct the difference between parameterized human body geometry and real human body, and obtains the corrected human body geometry. , as shown in formula (3):

[0082] (3).

[0083] This paper proposes a neural Gaussian texture mechanism to embed Gaussian points onto the parametric human body surface. , Gaussian properties include: Gaussian point position , quaternion representing Gaussian three-dimensional orientation , Stretch Vector , Opacity , a learnable feature ,like Figure 2 As shown. The present invention converts the uniform neural Gaussian field In the UV space of parametric human body geometry, based on human body geometry and the neural Gaussian field in UV space , we get the three-dimensional Gaussian feature field, as shown in formula (4):

[0084] (4),

[0085] in, It is UV mapping.

[0086] Given camera intrinsics , camera pose , and the above three-dimensional Gaussian feature field, rendering to obtain the feature map , as shown in formula (5):

[0087] (5),

[0088] Then use convolutional neural network as neural renderer , convert it into an RGB image, as shown in formula (6):

[0089] (6).

[0090] According to an embodiment of the present invention, the above-mentioned use of a three-dimensional dynamic Gaussian feature field to perform three-dimensional reconstruction of a target person in a portrait video, and optimizing the three-dimensional portrait reconstruction process using a neural-rendered RGB image and an original RGB image of the portrait video through a loss function in the reconstruction stage includes: preprocessing the portrait video using a predefined image processing model to obtain a two-dimensional portrait mask for each frame of the portrait video; processing the neural-rendered RGB image, the original RGB image and the two-dimensional portrait mask using a reconstruction loss function, a mask loss function and a stable loss function to obtain a loss value in the reconstruction stage; and supervising the three-dimensional portrait reconstruction process based on the loss value in the reconstruction stage to obtain an optimized three-dimensional portrait reconstruction result.

[0091] According to an embodiment of the present invention, the above-mentioned distillation of the editing information of the neural rendered RGB image to the optimized three-dimensional portrait reconstruction result to complete the update of the three-dimensional portrait includes: using a predefined image processing model to preprocess the portrait video to obtain a human body semantic segmentation map of each frame image in the portrait video, wherein the human body semantic segmentation map includes the head area and torso part of the target person; based on the multimodal editing instructions input by the user, using the two-dimensional multimodal generation module of the pre-trained facial perception portrait editing model to edit the head area and torso part in the neural rendered RGB image respectively; using the human body semantic segmentation map of each frame image, synthesizing the edited head area and the edited torso part, and inserting the synthesized image into the video frame data set to complete the update of the video frame data set; completing the optimization of the three-dimensional portrait reconstruction result by optimizing the three-dimensional dynamic Gaussian feature field, and distilling the editing information to the optimized three-dimensional portrait reconstruction result through the facial perception portrait editing model to complete the update of the three-dimensional portrait; alternating and iteratively performing the update operation of the three-dimensional portrait and the update operation of the video frame data set to continuously optimize the three-dimensional dynamic Gaussian feature field.

[0092] According to an embodiment of the present invention, the above-mentioned two-dimensional multimodal generation module includes a two-dimensional multimodal generation module based on text editing, a two-dimensional multimodal generation module based on image editing, and a two-dimensional multimodal image generation model for portrait relighting.

[0093] The following is a specific embodiment and combined with the attached Figure 2 The reconstruction, optimization and updating process of the above-mentioned three-dimensional portrait and the editing (or updating) process of the video frame data set provided by the present invention are further described in detail.

[0094] Given a portrait video, we first reconstruct the 3D portrait of the original video, and then distill the editing knowledge of the 2D multimodal generation module to 3D.

[0095] In the data preprocessing stage, given a portrait video, the portrait is tracked to obtain the camera internal parameters. and portrait identity coefficient , and get the camera pose of each frame , Portrait pose , expression coefficient , human semantic segmentation map and 2D portrait mask.

[0096] In the reconstruction stage, the present invention uses a reconstruction loss function, a mask loss function and a stabilization loss function.

[0097] Among them, the reconstruction loss function ( is the rendered RGB image, is the frame image of the original video) as shown in formula (7):

[0098] (7).

[0099] Among them, the mask loss function ( is the rendered alpha value, is a two-dimensional portrait mask) as shown in formula (8):

[0100] (8).

[0101] Among them, the stable loss function ( yes The first three channels of ) are shown in formula (9):

[0102] (9).

[0103] In addition, in order to enhance the details, the present invention uses the VGG network (Visual Geometry Group Network) as the backbone to calculate the perceptual loss function.

[0104] After the reconstruction is completed, the present invention uses the iterative dataset update technology to alternately perform dataset video frame editing and 3D portrait updating, and distill the editing of the 2D multimodal generation module to the 3D. Specifically, after every optimization step of the 3D portrait, the present invention uses the 2D multimodal generation module to edit a frame in the dataset, and then replaces the original frame with the edited frame. With the progressive update of the 3D portrait and the dataset, the knowledge of the 2D multimodal generation module can be finally distilled to the 3D portrait.

[0105] In addition to the loss function used during reconstruction, the present invention uses an additional expression similarity loss function to further avoid the expression degradation problem caused by the accumulation of editing errors during the iterative dataset update process. The present invention uses a pre-trained expression recognition network to map the rendered frame and the original frame into the same expression space and calculate their expression similarity, as shown in formula (10):

[0106] (10),

[0107] in It is a network that encodes expression in pictures. is to render the video frame, is the original video frame.

[0108] When editing an upper body image, if the face occupies a small proportion, the editing may not be sufficient to finely process the facial structure. The present invention further proposes a face-aware portrait editing model (the model includes the above-mentioned two-dimensional multimodal generation module), which can significantly enhance the editing quality of the facial area without the need to train the two-dimensional multimodal generation module. Figure 2 As shown, the facial area is first cropped, and then both the facial part and the portrait part are edited through the face-aware portrait editing model. The edited two parts are then synthesized into the final edited image through the head-torso semantic segmentation map.

[0109] In order to support multimodal portrait editing needs in different application scenarios, the present invention uses pre-trained image generation models based on text editing, picture editing, and portrait re-lighting. The above method is also applicable to other structure-preserving image generation models.

[0110] The present invention provides the above-mentioned portrait video editing method based on three-dimensional dynamic Gaussian field and multimodal generative prior. After a given portrait video, the given portrait video is first preprocessed to obtain camera parameters, human identity, expression, posture coefficient, human semantic segmentation map and two-dimensional portrait mask; based on the neural Gaussian texture mechanism, the learnable three-dimensional Gaussian features are embedded into the parameterized human body geometric surface, and the three-dimensional Gaussian splash feature map is converted into an image using a neural renderer. The three-dimensional portrait is optimized and reconstructed through the RGB and segmentation map information of the video; the knowledge of the multimodal two-dimensional image generation model is distilled into the three-dimensional portrait editing using iterative data set update technology, and the expression similarity guidance and facial perception portrait editing model are used in this process to further improve the editing quality. This method upgrades the two-dimensional editing task to three-dimensional, and with the help of three-dimensional human prior and three-dimensional Gaussian splash technology, the edited video has good three-dimensional consistency and time domain consistency. With the help of the knowledge of the multimodal generative model, high-quality portrait video editing functions such as text-based editing, image-based editing, and portrait relighting can be realized.

[0111] The following comparative experiments and combined with the attached Figures 3-5 To verify the advantages and effectiveness of the above method provided by the present invention. Figure 3 is a schematic diagram of visual result comparison based on text editing according to an embodiment of the present invention, Figure 4 is a schematic diagram of visual result comparison based on picture editing according to an embodiment of the present invention, Figure 5It is a schematic diagram of comparing visual results based on re-lighting according to an embodiment of the present invention.

[0112] The methods to be compared are: (1) TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR 2024); (2) Rerender A Video (RAV): Zero-Shot Text-Guided Video-to-Video Translation (SIGGRAPH Asia 2023 Conference Proceedings); (3) CoDeF: Content Deformation Fields for Temporally Consistent VideoProcessing (CVPR 2024 Highlight); (4) AnyV2V: A Plug-and-Play Framework For AnyVideo-to-Video Editing Tasks (arXiv preprint).

[0113] It is important to note that: Figure 3~Figure 5 The image data involved is only used for illustrative purposes and will not be used for commercial purposes; Figure 3~Figure 5 The images of public figures involved come from research institutions for scientific research purposes. The image data of the public figures provided by them have been authorized by the relevant public figures and are only used for scientific research purposes. They are open data sets for the public; Figure 3~Figure 5 The image data of non-public figures in the above-mentioned images have been authorized by the relevant parties, and the processing of their portraits is only used to illustrate the advantages and effectiveness of the method provided by the present invention. In addition, all the above-mentioned image data have been effectively kept confidential, and the processing process also strictly complies with the requirements of relevant laws and regulations.

[0114] In terms of editing video quality comparison, Figures 3-5 As shown, the multimodal portrait video editing method provided by the present invention is superior to other methods in terms of the degree of fit with editing instructions, structure preservation, time consistency and overall video quality.

[0115] In the video editing efficiency comparison method, the editing efficiency of the method provided by the present invention is further verified by analyzing the number of frames processed per minute by each method, as shown in Table 1. Different methods use different inference modes. For a fair comparison, the present invention calculates the time cost including reconstruction and editing, and averages it on the number of frames processed. It can be seen that the method of the present invention exceeds the previous video editing method in terms of efficiency.

[0116] Table 1 Comparison of editing efficiency of various methods

[0117]

[0118] Figure 6 A block diagram of an electronic device suitable for implementing a multimodal portrait video editing method according to an embodiment of the present invention is schematically shown.

[0119] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0120] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0121] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0122] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0123] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0124] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0126] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A multimodal portrait video editing method, characterized in that: The method comprises: By embedding Gaussian points in the three-dimensional Gaussian field into a two-dimensional texture space represented by parameterized human body geometry, a dimensionality-enhancing operation is performed on the frame image of the portrait video to obtain a three-dimensional dynamic Gaussian feature field; Performing three-dimensional Gaussian splashing on the three-dimensional dynamic Gaussian feature field to obtain three-dimensional Gaussian rendering features, and transforming the three-dimensional Gaussian rendering features using a convolutional neural renderer to obtain a neural-rendered RGB image; Performing three-dimensional reconstruction of the target person in the portrait video using the three-dimensional dynamic Gaussian feature field, and optimizing the three-dimensional portrait reconstruction process using the neural rendered RGB image and the original RGB image of the portrait video through a loss function in the reconstruction stage; Distilling the editing information of the neural-rendered RGB image to the optimized three-dimensional portrait reconstruction result to complete the update of the three-dimensional portrait, and using the edited neural-rendered RGB image to replace the original frame image of the portrait video to complete the editing of the video frame data set; Iteratively and alternately performing the editing process of the video frame data set and the updating process of the three-dimensional portrait, and mapping the edited neural-rendered RGB image and the original frame image of the portrait video to the same expression space; The convolutional neural renderer is optimized using the expression similarity loss function and the loss function of the reconstruction stage, and the dynamic three-dimensional Gaussian feature field is optimized again until the editing of all video frame images in the portrait video is completed.

2. The method according to claim 1, characterized in that By embedding Gaussian points in the three-dimensional Gaussian field into the two-dimensional texture space of the parameterized human body geometry representation, the frame image of the portrait video is subjected to a dimensionality increase operation, and the three-dimensional dynamic Gaussian feature field is obtained, including: Using the portrait identity coefficient, portrait posture coefficient and expression coefficient of the target person obtained from each frame image of the portrait video, an initial parameterized human body geometry representation is constructed; Correcting the difference between the initial parameterized human body geometry representation and the real human body of the target person by using learnable vertex offsets to obtain a corrected parameterized human body geometry representation; Based on the neural Gaussian texture mechanism, the Gaussian points mapped in the two-dimensional texture space of the three-dimensional Gaussian field are uniformly embedded into the surface of the corrected parameterized human body geometry representation to obtain the three-dimensional dynamic Gaussian feature field.

3. The method according to claim 2, characterized in that Using the portrait identity coefficient, portrait posture coefficient and expression coefficient of the target person obtained from each frame image of the portrait video, constructing an initial parameterized human body geometry representation includes: By performing portrait tracking on each frame of the portrait video, a portrait identity coefficient, a portrait posture coefficient and an expression coefficient of the target person in each frame of the portrait video are obtained; Based on the portrait identity coefficient, portrait posture coefficient and expression coefficient of the target person, the initial parameterized human body geometry representation is constructed using a predefined skinning function and skinning weight, a predefined joint point matrix and human body geometry in a T-pose.

4. The method according to claim 1, characterized in that: The three-dimensional dynamic Gaussian feature field is subjected to three-dimensional Gaussian splashing to obtain three-dimensional Gaussian rendering features, and the three-dimensional Gaussian rendering features are transformed using a convolutional neural renderer to obtain a neural-rendered RGB image including: Performing three-dimensional Gaussian splashing on the three-dimensional dynamic Gaussian feature field using the camera intrinsic parameters and camera pose of each frame image in the portrait video and the attribute information of the three-dimensional dynamic Gaussian feature field to obtain a three-dimensional Gaussian rendering feature; A convolutional neural renderer including an input layer, a convolutional layer, a nonlinear layer, a pooling layer, a fully connected layer and a loss layer is constructed, and the three-dimensional Gaussian rendering features are transformed by using the convolutional neural renderer to obtain a neural-rendered RGB image.

5. The method according to claim 4, characterized in that The attribute information of the three-dimensional dynamic Gaussian feature field includes spatial position information of Gaussian points, a quaternion representing the three-dimensional orientation of the Gaussian field, a scaling vector, opacity, and a learnable feature vector.

6. The method according to claim 1, characterized in that The target person in the portrait video is reconstructed in three dimensions using the three-dimensional dynamic Gaussian feature field, and the three-dimensional portrait reconstruction process is optimized using the neural rendered RGB image and the original RGB image of the portrait video through the loss function of the reconstruction stage, including: Preprocessing the portrait video using a predefined image processing model to obtain a two-dimensional portrait mask for each frame of the portrait video; Processing the neural-rendered RGB image, the original RGB image, and the two-dimensional portrait mask using a reconstruction loss function, a mask loss function, and a stabilization loss function to obtain a loss value in a reconstruction phase; The three-dimensional portrait reconstruction process is supervised and trained based on the loss value of the reconstruction stage to obtain an optimized three-dimensional portrait reconstruction result.

7. The method according to claim 1, characterized in that Distilling the edit information of the neural rendered RGB image to the optimized 3D portrait reconstruction result to complete the update of the 3D portrait includes: Preprocessing the portrait video using a predefined image processing model to obtain a human body semantic segmentation map of each frame image in the portrait video, wherein the human body semantic segmentation map includes a head area and a torso part of the target person; Based on the multimodal editing instructions input by the user, the head region and the torso portion in the neural rendered RGB image are edited respectively using a two-dimensional multimodal generation module of a pre-trained face-aware portrait editing model; Using the human body semantic segmentation map of each frame image, synthesizing the edited head region and the edited torso part, and inserting the synthesized image into the video frame data set to complete the update of the video frame data set; The three-dimensional portrait reconstruction result is optimized by optimizing the three-dimensional dynamic Gaussian feature field, and the editing information is distilled to the optimized three-dimensional portrait reconstruction result through the face-aware portrait editing model to complete the update of the three-dimensional portrait; The three-dimensional portrait updating operation and the video frame data set updating operation are alternately and iteratively performed to continuously optimize the three-dimensional dynamic Gaussian feature field.

8. The method according to claim 7, characterized in that The two-dimensional multimodal generation module includes a two-dimensional multimodal generation module based on text editing, a two-dimensional multimodal generation module based on image editing, and a two-dimensional multimodal image generation model for portrait relighting.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A computer program or instruction is stored thereon, wherein the computer program or instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images