Three-dimensional dynamic portrait editing method and system based on fine-tuning three-dimensional generative adversarial network

Abstract

The application provides a three-dimensional dynamic portrait editing method and system based on fine-tuning of a three-dimensional generative adversarial network, comprising: completing joint three-dimensional generative adversarial network inverse mapping, fine-tuning the three-dimensional generative adversarial network based on aligned images, fine-tuning the three-dimensional generative adversarial network based on original images, rendering identity neural radiation field, and fine-tuning the three-dimensional generative adversarial network after editing and rendering images using a diffusion model; the joint three-dimensional generative adversarial network inverse mapping effectively decouples identity and motion, and camera information; the three-dimensional generative adversarial network based on aligned images is fine-tuned, so that the mapping accuracy of the three-dimensional generative adversarial network for identity information of input video is improved; the three-dimensional generative adversarial network based on aligned head images is expanded to image rendering of half bodies by fine-tuning the three-dimensional generative adversarial network based on original images; the identity neural radiation field is rendered, the three-dimensional generative adversarial network is fine-tuned after editing and rendering images using a diffusion model, and stylized editing is effectively realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a three-dimensional dynamic portrait editing method and system based on fine-tuning a three-dimensional generative adversarial network, and more particularly to a three-dimensional dynamic portrait editing method and system based on fine-tuning a three-dimensional generative adversarial network using a diffusion model. BACKGROUND

[0002] With the rapid progress of digital technology, the technology of generating three-dimensional stylized Avatars from video data can provide users with an unprecedented immersive experience. Generative adversarial networks (GANs), especially its variants such as StyleGAN, have set a new benchmark in the field of image and video synthesis with their outstanding generation ability. StyleGAN, through its unique hierarchical style control mechanism, can finely regulate the details of the generated images, making it possible to extract and generate high-fidelity three-dimensional portrait Avatars from a single video in real time. This technological breakthrough not only preserves the identity features and expression details in the original video, but also greatly enriches the possibilities of personalized expression based on model improvements. In addition to the application of GANs, optimization-based methods also provide another path for the generation of three-dimensional stylized Avatars. These methods first use advanced three-dimensional reconstruction technology to recover the real three-dimensional face model from video or image. Subsequently, through complex mesh geometry deformation and texture migration algorithms, the final generated virtual Avatar can integrate the features of the target style while maintaining the identity features of the original face. Although this method has a high computational complexity, it can achieve a more fine and controllable stylization effect.

[0003] However, despite the significant progress made by existing technologies, there are still many challenges in the stylized editing of three-dimensional dynamic portraits. For example, the stylization process is often difficult to control accurately, resulting in inconsistency in the generated results; at the same time, how to maintain the consistency of three-dimensional data while achieving efficient and natural dynamic changes is also a difficult problem in current research. In view of the above problems, the present application proposes an innovative solution - a three-dimensional dynamic portrait editing method based on fine-tuning a three-dimensional generative adversarial network using a diffusion model. This method combines the flexibility of diffusion models in data generation and the advantages of 3D GANs in image quality, and through fine-tuning the 3D GAN network, it realizes precise control of the stylization of three-dimensional dynamic portraits. SUMMARY

[0004] In view of the defects in the prior art, the purpose of the present application is to provide a three-dimensional dynamic portrait editing method and system based on fine-tuning a three-dimensional generative adversarial network.

[0005] According to the three-dimensional dynamic portrait editing method based on fine-tuning a three-dimensional generative adversarial network provided by the present application, the method comprises:

[0006] Step S1: obtaining each portrait image in a portrait video, and pre-processing the obtained portrait image, using a pre-trained generative adversarial network, based on the pre-processed portrait image and a corresponding loss function, obtaining inverse mapping parameters corresponding to the portrait image;

[0007] Step S2: obtaining a portrait face image based on the pre-processed portrait image, constructing a personalized three-dimensional generative adversarial network, fine-tuning the personalized three-dimensional generative adversarial network based on the portrait face image and the corresponding inverse mapping parameters, and obtaining a fine-tuned personalized three-dimensional generative adversarial network;

[0008] Step S3: second fine-tuning the fine-tuned personalized three-dimensional generative adversarial network based on each portrait image in the portrait video and the corresponding inverse mapping parameters, and obtaining a second fine-tuned personalized three-dimensional generative adversarial network;

[0009] Step S4: rendering an identity portrait multi-view image based on the inverse mapping identity information parameters through the second fine-tuned personalized three-dimensional generative adversarial network, editing the obtained portrait image using a diffusion model and a corresponding text prompt to obtain an edited portrait image, training the second fine-tuned personalized three-dimensional generative adversarial network based on the edited portrait image and the corresponding inverse mapping parameters, and obtaining an optimized stylized three-dimensional generative adversarial network.

[0010] Preferably, the step S1 includes pre-processing the obtained portrait image, including cropping and aligning the face of the obtained portrait image, and extracting camera parameters.

[0011] Preferably, the loss function in the step S1 includes:

[0012]

[0013] wherein w id represents the identity information of the portrait; o n represents the action information of the portrait; c′ n represents the camera parameters corresponding to the pre-processed image, G raw (w id +o n ,c n ) represents the generated sample image, I′ n represents the pre-processed input image, L lpips and λ lpips represent the learnable perceptual image block similarity loss and the corresponding weight, L L2 represents the Euclidean distance loss, λ L2 and λ regrespectively, are weights corresponding to the balanced Euclidean distance loss and the action information penalty loss.

[0014] Preferably, the step S2 comprises fine-tuning the personalized three-dimensional generative adversarial network based on the portrait face image and the corresponding inverse mapping parameter, and the loss function comprises:

[0015]

[0016] wherein G′ raw represents the personalized three-dimensional generative adversarial network.

[0017] Preferably, the step S3 comprises second fine-tuning the fine-tuned personalized three-dimensional generative adversarial network based on each frame of portrait image in the portrait video and the corresponding inverse mapping parameter, and the loss function comprises:

[0018]

[0019] wherein c n corresponds to the camera parameter of the image without preprocessing, I n represents each frame of portrait image in the portrait video, G′ raw represents the personalized three-dimensional generative adversarial network.

[0020] Preferably, in the step S4, the identity portrait multi-view image is rendered by the personalized three-dimensional generative adversarial network based on the inverse mapping identity information parameter, and the obtained portrait image is edited using a diffusion model and a corresponding text prompt to obtain an edited portrait image, comprising:

[0021] Step S4.1: based on the obtained portrait identity information parameter w id rendering the identity neural radiance field by the second fine-tuned personalized three-dimensional generative adversarial network to obtain a multi-view portrait image;

[0022] Step S4.2: editing the front portrait using a diffusion model, and editing the multi-view portrait image using a video stylization method;

[0023]

[0024] wherein Diff represents a pre-trained text-guided image editing diffusion model, G edit represents the fine-tuned three-dimensional generative adversarial network to be edited, c m_front represents the camera parameter corresponding to the front view; represents the front view corresponds to the unedited portrait image of the half body, represents the edited portrait image of the half body rendered by the front view.

[0025]

[0026] wherein, represents the edited image after diffusion model and video stylization editing, G edit (w id ,c m ) represents the image rendered based on the identity latent variable, and Video_style represents the video stylization algorithm. m

[0027] Preferably, the personalized three-dimensional generative adversarial network after the second fine-tuning is trained based on the edited portrait image and the corresponding inverse mapping parameter, and the loss function comprises:

[0028]

[0029] wherein, c m represents the preset surround camera parameter, G edit (w id ,c m ) represents the image rendered based on the identity latent variable, represents the edited image after diffusion model and video stylization editing.

[0030] According to the three-dimensional dynamic portrait editing system based on the fine-tuned three-dimensional generative adversarial network provided by the application, the three-dimensional dynamic portrait editing system based on the fine-tuned three-dimensional generative adversarial network comprises:

[0031] Module M1: obtaining each frame of portrait image in a portrait video, and pre-processing the obtained portrait image, using a pre-trained generative adversarial network, based on the pre-processed portrait image and the corresponding loss function, obtaining the inverse mapping parameter corresponding to the portrait image;

[0032] Module M2: obtaining a portrait face image based on the pre-processed portrait image, constructing a personalized three-dimensional generative adversarial network, fine-tuning the personalized three-dimensional generative adversarial network based on the portrait face image and the corresponding inverse mapping parameter, and obtaining the fine-tuned personalized three-dimensional generative adversarial network;

[0033] Module M3: fine-tuning the fine-tuned personalized three-dimensional generative adversarial network based on each frame of portrait image in the portrait video and the corresponding inverse mapping parameter, and obtaining the second fine-tuned personalized three-dimensional generative adversarial network;

[0034] ​Module M4: rendering the identity portrait multi-view image based on the personalized three-dimensional generative adversarial network after the second fine-tuning based on the inverse mapping identity information parameter, editing the obtained portrait image using the diffusion model and the corresponding text prompt to obtain the edited portrait image, training the personalized three-dimensional generative adversarial network after the second fine-tuning based on the edited portrait image and the corresponding inverse mapping parameter, and obtaining the optimized stylized three-dimensional generative adversarial network.

[0035] Preferably, the module M1 includes: pre-processing the obtained portrait image, including: cropping and aligning the face of the obtained portrait image, and extracting the camera parameter;

[0036] The loss function in the module M1 includes:

[0037]

[0038] wherein w id represents the identity information of the portrait; o n represents the action information of the portrait; c′ n represents the camera parameter corresponding to the pre-processed image, G raw (w id +o n ,c n ) represents the generated sample image, I′ n represents the pre-processed input image, L lpips and λ lpips represent the learnable perceptual image block similarity loss and the corresponding weight, L L2 represents the Euclidean distance loss, λ L2 and λ reg are the weights corresponding to balancing the Euclidean distance loss and the action information penalty loss.

[0039] Preferably, the module M2 includes: fine-tuning the personalized three-dimensional generative adversarial network based on the portrait face image and the corresponding inverse mapping parameter, and the loss function includes:

[0040] (2)

[0041] wherein, represents the personalized three-dimensional generative adversarial network;

[0042] The module M3 includes: second fine-tuning the fine-tuned personalized three-dimensional generative adversarial network based on each frame of the portrait image in the portrait video and the corresponding inverse mapping parameter, and the loss function includes:

[0043] (3)

[0044] wherein, camera parameters corresponding to the image without pre-processing, representing each frame of the portrait image in the portrait video, representing the personalized half-body three-dimensional generative adversarial network;

[0045] The module M4 renders the identity portrait multi-view image based on the inverse mapping identity information parameter through the personalized three-dimensional generative adversarial network, and edits the obtained portrait image using the diffusion model and the corresponding text prompt to obtain the edited portrait image, including:

[0046] Module M4.1: based on the obtained portrait identity information parameter render the identity neural radiance field through the second fine-tuned personalized three-dimensional generative adversarial network to obtain a multi-view portrait image;

[0047] Module M4.2: edit the front portrait using the diffusion model, and edit the multi-view portrait image using the video stylization method;

[0048]

[0049] where Diff represents a pre-trained text-guided image editing diffusion model, G edit represents the three-dimensional generative adversarial network to be fine-tuned, c m_front represents the camera parameters corresponding to the front view; represents the front view corresponding to the unedited half-body portrait image, represents the edited half-body portrait image rendered by the front view;

[0050]

[0051] where, represents the edited image edited by the diffusion model and the video stylization, G edit (w id ,c m ) represents the image of the corresponding view rendered based on the identity latent variable, c m , and Video_style represents the video stylization algorithm;

[0052] The second fine-tuned personalized three-dimensional generative adversarial network is trained based on the edited portrait image and the corresponding inverse mapping parameter, and the loss function includes:

[0053]

[0054] where c m represents a preset surround camera parameter, G edit (wid ,c m ) represents an image rendered based on an identity latent variable, represents an edited image after diffusion model and video stylization editing.

[0055] Compared with the prior art, the present application has the following beneficial effects:

[0056] 1. The present application proposes a three-dimensional dynamic portrait editing method based on fine-tuning three-dimensional generative adversarial networks using a diffusion model, which effectively solves the problems of uncontrollable stylization and low consistency in generating three-dimensional stylized Avatars from videos by applying technologies such as joint three-dimensional generative adversarial network inverse mapping, fine-tuning three-dimensional generative adversarial networks based on aligned images, fine-tuning three-dimensional generative adversarial networks based on original images, rendering identity neural radiance fields, using diffusion models and video stylization to edit rendered images and fine-tuning three-dimensional generative adversarial networks, and improving the quality of three-dimensional dynamic portrait editing;

[0057] 2. The present application effectively decouples identity and motion, camera information by joint three-dimensional generative adversarial network inverse mapping, and obtains inverse mapping parameters that can represent three-dimensional dynamic portraits;

[0058] 3. The present application realizes high matching degree of three-dimensional generative adversarial networks with given portraits by fine-tuning based on aligned images and original images respectively;

[0059] 4. The present application realizes editing of dynamic portraits by fine-tuning identity neural radiance fields through the representativeness of identity radiance fields for dynamic portraits, and realizes multi-view consistent image editing through the consistency of multiple frames in video stylization. BRIEF DESCRIPTION OF DRAWINGS

[0060] Other features, objects and advantages of the present application will become more apparent after reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0061] Figure 1 is a flowchart of a three-dimensional dynamic portrait editing method based on fine-tuning three-dimensional generative adversarial networks.

[0062] Figure 2 is a schematic diagram of the generation effect of twice fine-tuning and editing.

[0063] Figure 3 is a schematic diagram of the generation effect of 360 degrees before and after editing.

[0064] Figure 4 is a schematic diagram of the representative effect of a three-dimensional dynamic portrait editing method based on fine-tuning three-dimensional generative adversarial networks. DETAILED DESCRIPTION

[0065] The application will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the application. These are within the scope of protection of the application.

[0066] Example 1

[0067] According to the three-dimensional dynamic portrait editing method based on fine-tuning three-dimensional generative adversarial network provided by the application, as shown in Figure 1 , comprising:

[0068] Step S1: obtaining each frame of portrait image in the portrait video, and pre-processing the obtained portrait image, using a pre-trained generative adversarial network, based on the pre-processed portrait image and the corresponding loss function, obtaining the inverse mapping parameter corresponding to the portrait image;

[0069] In this embodiment, the training of the generative adversarial network requires that the data have the same structural attributes, which requires that the input image be aligned. In order to complete the preprocessing of the portrait video, the face key points are extracted and the image is cropped and aligned to make it meet the data needs of the generative adversarial network. The pose parameters of the face are extracted as the pose input of the three-dimensional generative adversarial network.

[0070] The inverse mapping of the generative adversarial network is used to inverse map the video portrait to the latent variable of the generative adversarial network. For the generative adversarial network, it can generate a real image from the latent variable, and vice versa. Given a real image, the corresponding latent variable can be solved, which is called inverse mapping. Given a video, to realize the inverse mapping of the identity-consistent latent variable, the identity and action information decoupled latent variable w id +o n represents the latent variable corresponding to each frame, wherein w id represents the identity information of the portrait, o n represents the action information of the portrait. The joint inverse mapping uses equation (1) to express the loss function:

[0071]

[0072] In the formula, c′ n corresponds to the camera parameters of the pre-processed image, G raw (w id +o n , c n ) represents the generated sample image, I′ n represents the pre-processed input image, L lpips and λ lpipsrepresents a learned perceptual image patch similarity (LPIPS) loss and a corresponding weight, L L2 represents a Euclidean distance loss, λ L2 and λ reg are weights corresponding to balancing the Euclidean distance loss and the action information penalty loss, respectively, and are empirically determined.

[0073] Step S2: obtaining a portrait face image based on the preprocessed portrait image, constructing a personalized three-dimensional generative adversarial network, and fine-tuning the personalized three-dimensional generative adversarial network based on the portrait face image and the corresponding inverse mapping parameter to obtain a fine-tuned personalized three-dimensional generative adversarial network.

[0074] In this embodiment, after completing the joint latent variable inverse mapping, the generated image is constrained by a loss function with respect to the preprocessed image, and the personalized three-dimensional generative adversarial network is optimized to obtain the personalized three-dimensional generative adversarial network; wherein the fine-tuning of the three-dimensional generative adversarial network is expressed by the loss function of formula (2):

[0075] (2)

[0076] In the formula, c corresponds to the camera parameter of the preprocessed image, represents the preprocessed input image, represents the personalized three-dimensional generative adversarial network.

[0077] Step S3: fine-tuning the fine-tuned personalized three-dimensional generative adversarial network based on each frame of the portrait image in the portrait video and the corresponding inverse mapping parameter to obtain a second fine-tuned personalized three-dimensional generative adversarial network.

[0078] Based on the personalized three-dimensional generative adversarial network optimized on the cropped and aligned image, the second fine-tuning is performed to make it consistent with the attributes of the original half-body image; the loss function of the second fine-tuning of the three-dimensional generative adversarial network is expressed by formula (3):

[0079]

[0080] In the formula, c n corresponds to the camera parameter of the image without preprocessing, I n represents the input image without preprocessing, G″ raw represents the personalized half-body three-dimensional generative adversarial network.

[0081] Step S4: rendering the identity portrait multi-view image based on the second fine-tuned personalized three-dimensional generative adversarial network, and editing the obtained portrait image using a diffusion model and a corresponding text prompt to obtain an edited portrait image, training the second fine-tuned personalized three-dimensional generative adversarial network based on the edited portrait image and the corresponding inverse mapping parameter to obtain an optimized stylized three-dimensional generative adversarial network.

[0082] In the embodiment, the identity neural radiance field is rendered to obtain a multi-view portrait image, the diffusion model is used to edit the front portrait, and the video stylization method is used to edit the multi-view portrait. edit id m represents an image rendered based on the identity latent variable, and the front image G edit id m_front can be obtained by editing using the diffusion model and the text control, Based on the edited front image and the unedited side image, the video stylization is used to complete the multi-view image editing:

[0083]

[0084] wherein Diff represents a pre-trained text-guided image editing diffusion model, including but not limited to a stable diffusion model (SD, Stable Diffusion); G edit represents a three-dimensional generative adversarial network to be fine-tuned, i.e., the second fine-tuned personalized three-dimensional generative adversarial network, (G edit is initialized as G″ raw ); c m_front represents the camera parameter corresponding to the front view; represents the front view corresponding to the unedited half-body portrait image, represents the edited half-body portrait image rendered by the front view;

[0085] The edited personalized three-dimensional generative adversarial network is obtained by optimizing the generated image and the edited image through the loss function. The editing of the three-dimensional generative network is expressed by the loss function of formula (4):

[0086]

[0087] wherein c m represents a preset surround camera parameter, G edit (w id ,c m ) represents an image rendered based on the identity latent variable, ​​​​An edited image that is subjected to a diffusion model and video stylization editing.

[0088] The application also provides a three-dimensional dynamic portrait editing system based on fine-tuning a three-dimensional generative adversarial network, which can be realized by performing the process steps of the three-dimensional dynamic portrait editing method based on fine-tuning a three-dimensional generative adversarial network, that is, the three-dimensional dynamic portrait editing method based on fine-tuning a three-dimensional generative adversarial network can be understood by those skilled in the art as a preferred embodiment of the three-dimensional dynamic portrait editing system based on fine-tuning a three-dimensional generative adversarial network.

[0089] The application realizes personalized generator fine-tuning through twice fine-tuning. Specifically, two cases are selected to further illustrate the effectiveness of twice fine-tuning; as shown in Figure 2 For each case, the results of direct inverse mapping rendering of the half body and head, fine-tuning the three-dimensional generative adversarial network on the head image alone, fine-tuning the three-dimensional generative adversarial network on the half body image alone, and the twice fine-tuning proposed by the application are respectively shown. The twice fine-tuning proposed by the application can better capture the whole body and expression attributes, and generate rendering results consistent with the original video.

[0090] The application proposes to synthesize and edit dynamic three-dimensional portraits based on a three-dimensional generative adversarial network. Specifically, a case is selected to discuss the three-dimensional capability of the method; as shown in Figure 3 The first row shows the 360-degree display effect of the two poses after inverse mapping. The second row shows the 360-degree display effect of the edited pose. It can be seen that the method of the application can ensure high-quality reconstruction and rendering of 360 degrees.

[0091] Further, a portrait video sequence is selected for discussion; as shown in Figure 4 The first row shows a half-body video sequence of 7 poses with different poses and expressions, and the next three rows show the synthesized editing results with high consistency with the text command under different text commands. The results of the last row also show multi-view rendering results. The application can realize free and high-quality editing results.

[0092] The computer hardware conditions used in the experiments in the application are: an Nvidia RTX 3090 graphics card (24G of video memory), an i9-10980XE processor. The input image resolution is 512x512.

[0093] The fine-tuning pre-trained 3D GAN, joint latent variable inversion, fine-tuning based on aligned images, fine-tuning based on original images, fine-tuning based on edited images are iterated 1000 times respectively. For a ten-second (300-frame) video, the joint latent variable inverse mapping takes about 80 minutes, and the three fine-tuning takes 10 minutes respectively. The time of using the diffusion model and editing the image of the video style takes only about ten seconds.

[0094] Those skilled in the art know that, in addition to implementing the system provided by the present application and each device, module, unit thereof in the form of pure computer readable program code, the system provided by the present application and each device, module, unit thereof can also be realized in the form of logic gate, switch, application specific integrated circuit, programmable logic controller and embedded microcontroller, etc. by logically programming the method steps to achieve the same function. Therefore, the system provided by the present application and each device, module, unit thereof can be considered as a hardware component, and the devices, modules, units included therein for realizing various functions can also be considered as structures within the hardware component; the devices, modules, units for realizing various functions can also be considered as both software modules realizing the method and structures within the hardware component.

[0095] The specific embodiments of the present application are described above. It needs to be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the present application. The embodiments of the present application and the features in the embodiments can be arbitrarily combined with each other without conflict.

Claims

1. A three-dimensional dynamic portrait editing method based on fine-tuning a three-dimensional generative adversarial network, characterized in that, Comprising: Step S1: obtaining each frame of portrait image in the portrait video, and pre-processing the obtained portrait image, using a pre-trained generative adversarial network, based on the pre-processed portrait image and the corresponding loss function, obtaining the inverse mapping parameter corresponding to the portrait image; Step S2: obtaining a portrait face image based on the pre-processed portrait image, constructing a personalized three-dimensional generative adversarial network, and fine-tuning the personalized three-dimensional generative adversarial network based on the portrait face image and the corresponding inverse mapping parameter, to obtain a fine-tuned personalized three-dimensional generative adversarial network; Step S3: based on each frame of portrait image in the portrait video and the corresponding inverse mapping parameter, the fine-tuned personalized three-dimensional generative adversarial network is fine-tuned for the second time, to obtain a second fine-tuned personalized three-dimensional generative adversarial network; Step S4: based on the inverse mapping identity information parameter, the second fine-tuned personalized three-dimensional generative adversarial network is used to render an identity portrait multi-view image, and a diffusion model and a corresponding text prompt are used to edit the obtained portrait image to obtain an edited portrait image, and the second fine-tuned personalized three-dimensional generative adversarial network is trained based on the edited portrait image and the corresponding inverse mapping parameter, to obtain an optimized stylized three-dimensional generative adversarial network; The loss function in step S1 includes: wherein w id represents the identity information of the portrait; o n represents the action information of the portrait; c′ n represents the camera parameters corresponding to the preprocessed image, G raw (w id + o n , c′ n ) represents the generated sample image, I′ n represents the preprocessed input image, L lpips and λ lpips represent the learnable perceptual image block similarity loss and the corresponding weight, L L2 represents the Euclidean distance loss, λ L2 and λ reg are the weights corresponding to balancing the Euclidean distance loss and the action information penalty loss, respectively; The step S2 includes: fine-tuning the personalized three-dimensional generative adversarial network based on the portrait face image and the corresponding inverse mapping parameter, and the loss function includes: Among them, G′ raw A personalized 3D generative adversarial network; The step S3 includes: based on each frame of portrait image in the portrait video and the corresponding inverse mapping parameter, the fine-tuned personalized three-dimensional generative adversarial network is fine-tuned for the second time, and the loss function includes: wherein c n corresponding to the camera parameters of the image which has not been preprocessed, I n denotes each frame portrait image in the portrait video, G" raw denotes the personalized half-body three-dimensional generative adversarial network; In the step S4, based on the inverse mapping identity information parameter, the personalized three-dimensional generative adversarial network is used to render an identity portrait multi-view image, and a diffusion model and a corresponding text prompt are used to edit the obtained portrait image to obtain an edited portrait image, including: Step S4.1: based on the acquired portrait identity information parameter w id The personalized three-dimensional generative adversarial network is fine-tuned again, and an identity neural radiation field is rendered to obtain a multi-view portrait image. Step S4.2: using a diffusion model to edit a front portrait, and using a video stylization method to edit a multi-view portrait image; wherein Diff denotes a pre-trained text-guided image editing diffusion model, G edit denotes a three-dimensional generative adversarial network to be edited and fine-tuned, c m_front denotes camera parameters corresponding to the front view; denotes a front view corresponding unedited half-length portrait image, denotes an edited front view rendered half-length portrait image; wherein, denotes the edited image after diffusion model and video stylization editing, G edit (w id , c m ) denotes the image of the corresponding view c m rendered based on the identity latent variable; Video_style denotes the video stylization algorithm; The loss function for training the second fine-tuned personalized three-dimensional generative adversarial network based on the edited portrait image and the corresponding inverse mapping parameter includes: where G edit (w id , c m ) represents the image of the corresponding view c m based on the identity hidden variable rendering; represents the edited image after diffusion model and video stylization editing.

2. The method of claim 1, wherein the fine-tuning three-dimensional generative adversarial network is based on a three-dimensional dynamic portrait editing method, characterized in that, The step S1 includes: pre-processing the obtained portrait image, including: cropping and aligning the face of the obtained portrait image, and extracting camera parameters.

3. A three-dimensional dynamic portrait editing system based on fine-tuning a three-dimensional generative adversarial network, characterized in that, Comprising: Module M1: obtaining each frame of portrait image in the portrait video, and pre-processing the obtained portrait image, using a pre-trained generative adversarial network, based on the pre-processed portrait image and the corresponding loss function, obtaining the inverse mapping parameter corresponding to the portrait image; Module M2: obtaining a portrait face image based on the pre-processed portrait image, constructing a personalized three-dimensional generative adversarial network, and fine-tuning the personalized three-dimensional generative adversarial network based on the portrait face image and the corresponding inverse mapping parameter, to obtain a fine-tuned personalized three-dimensional generative adversarial network; Module M3: based on each frame of portrait image in the portrait video and the corresponding inverse mapping parameter, the fine-tuned personalized three-dimensional generative adversarial network is fine-tuned for the second time, and the second fine-tuned personalized three-dimensional generative adversarial network is obtained; Module M4: based on the inverse mapping identity information parameter, the identity portrait multi-view image is rendered through the second fine-tuned personalized three-dimensional generative adversarial network, and the obtained portrait image is edited using the diffusion model and the corresponding text prompt to obtain the edited portrait image, and based on the edited portrait image and the corresponding inverse mapping parameter, the second fine-tuned personalized three-dimensional generative adversarial network is trained to obtain the optimized stylized three-dimensional generative adversarial network; The module M1 comprises: the obtained portrait image is preprocessed, including: the obtained portrait image is cropped and aligned with the face, and the camera parameter is extracted; The loss function in the module M1 comprises: wherein w id represents the identity information of the portrait; o n represents the action information of the portrait; c′ n represents the camera parameters corresponding to the preprocessed image, G raw (w id + o n , c′ n ) represents the generated sample image, I′ n represents the preprocessed input image, L lpips and λ lpips represent the learnable perceptual image block similarity loss and the corresponding weight, L L2 represents the Euclidean distance loss, λ L2 and λ reg are the weights corresponding to balancing the Euclidean distance loss and the action information penalty loss, respectively; The module M2 comprises: based on the portrait face image and the corresponding inverse mapping parameter, the personalized three-dimensional generative adversarial network is fine-tuned, and the loss function comprises: G' = G - G raw denotes a personalized three-dimensional generative adversarial network; The module M3 comprises: based on each frame of portrait image in the portrait video and the corresponding inverse mapping parameter, the fine-tuned personalized three-dimensional generative adversarial network is fine-tuned for the second time, and the second fine-tuned personalized three-dimensional generative adversarial network is obtained; wherein c n corresponding to the camera parameters of the image not subjected to preprocessing, I n denotes each frame portrait image in the portrait video, G" raw denotes the personalized half-body three-dimensional generative adversarial network; The module M4 comprises: based on the inverse mapping identity information parameter, the identity portrait multi-view image is rendered through the personalized three-dimensional generative adversarial network, and the obtained portrait image is edited using the diffusion model and the corresponding text prompt to obtain the edited portrait image, comprising: Module M4.1: based on the acquired portrait identity information parameter w id The personalized three-dimensional generative adversarial network is fine-tuned again, and an identity neural radiation field is rendered to obtain a multi-view portrait image. Module M4.2: edit the front portrait using the diffusion model, and edit the multi-view portrait image using the video stylization method; wherein Diff denotes a pre-trained text-guided image editing diffusion model, G edit denotes a three-dimensional generative adversarial network to be edited and fine-tuned, c m_front denotes camera parameters corresponding to the front view; denotes a front view corresponding unedited half-length portrait image, denotes an edited half-length portrait image rendered from the front view; wherein, represents the edited image after diffusion model and video stylization editing, G edit (w id , c m ) represents the image of the corresponding view c m rendered based on the identity latent variable; Video_style represents the video stylization algorithm; Based on the edited portrait image and the corresponding inverse mapping parameter, the second fine-tuned personalized three-dimensional generative adversarial network is trained, and the loss function comprises: where G edit (w id , c m ) denotes the image of the corresponding view c m based on the identity latent variable rendering, denotes the edited image after diffusion model and video stylization editing.

Images