A video switching method and device, electronic equipment and storage medium

Abstract

The application provides a video head replacement method, which can be applied to the technical fields of computer vision, graphics and deep learning. The method comprises the following steps: obtaining a pretreated source character image by separating the background and clothes in a source character image, and performing head region redrawing on a target video by using a fine-tuned head generation model to obtain a head replacement dataset; generating a 4D neural high-frequency feature field based on a statistical extended human body three-dimensional model, and training the 4D neural high-frequency feature field by using the head replacement dataset to obtain a trained 4D neural high-frequency feature field; processing a foreground head region of the target video to obtain a repaired target video, and performing head replacement on the repaired target video by using the trained 4D neural high-frequency feature field to obtain a head replacement result of the repaired target video. The application also provides a video head replacement device, an electronic device and a storage medium.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, graphics and deep learning, in particular to a video head changing method and device, an electronic device and a storage medium. BACKGROUND

[0002] The head changing technology has a very wide application prospect in many fields such as film production, artistic creation, augmented reality (AR) and virtual reality (VR). With the continuous progress of technology, especially the innovation in the field of computer vision and deep learning, the head changing technology is no longer limited to traditional static image processing, and more and more researches begin to explore how to realize the head changing effect in dynamic video. However, how to seamlessly transfer the identity information of the source person to the target video without changing other attributes (such as posture, expression, background, etc.) of the target video is still a challenging research problem. It is not only a technical problem, but also a complex issue involving multiple disciplines such as deep learning, computer graphics and human-computer interaction.

[0003] Early video head changing technology solutions mainly use generative adversarial network (GAN) models, although some good results have been achieved, but when facing high-difficulty head changing tasks, there are still problems such as difficulty in guaranteeing the high fidelity of identity features. Some other head changing technology solutions, such as head changing solutions based on diffusion models, often cannot guarantee the consistency of time, resulting in not smooth face switching in the video. In addition, other video head changing solutions in the prior art also have problems such as poor face imaging effect, unnatural head changing effect, and easy to appear broken feeling. SUMMARY

[0004] In view of the above problems, the present application provides a video head changing method and device, an electronic device and a storage medium for improving the video head changing effect.

[0005] According to a first aspect of the present application, a video head changing method is provided, comprising:

[0006] By separating the background and clothes in the source person image, a preprocessed source person image is obtained, and a fine-tuned pre-trained head generation model is used to redraw the head region of the target video to obtain a head exchange data set;

[0007] A 4D neural high-frequency feature field is generated based on a statistical extended human three-dimensional model, and the 4D neural high-frequency feature field is trained using the head exchange data set to obtain a trained 4D neural high-frequency feature field, wherein the attributes of each pixel in the 4D neural high-frequency feature field include neural features, opacity, scale and rotation degree;

[0008] The foreground head region of the target video is processed to obtain a repaired target video, and based on a foreground and background fusion re-rendering technology of a video image, a 4D neural high frequency signature field trained is used to perform a head replacement operation on the repaired target video to obtain a head replacement result of the repaired target video.

[0009] According to an embodiment of the present application, the above processing of the foreground head region of the target video to obtain the repaired target video comprises:

[0010] The foreground head region of the target video is removed to obtain a target video with the foreground head region removed;

[0011] The target video with the foreground head region removed is repaired using a predefined image repair tool to obtain a repaired target video, wherein the repaired target video comprises a background region and a torso region of a target person.

[0012] According to an embodiment of the present application, the above predefined image repair tool comprises an image segmentation model, an image inpainting model and an artificial intelligence-based content generation model.

[0013] According to an embodiment of the present application, the above foreground and background fusion re-rendering technology of a video image uses a 4D neural high frequency signature field trained to perform a head replacement operation on the repaired target video to obtain a head replacement result of the repaired target video.

[0014] The repaired target video is converted into a feature domain using a 2D background encoder to obtain a target video feature domain;

[0015] The 4D neural high frequency signature field trained is two-dimensionally splashed to obtain a foreground head region;

[0016] The 4D neural high frequency signature field trained is processed to obtain a rendered alpha channel;

[0017] During the head reproduction process of the 4D neural high frequency signature field trained, a plurality of key points of a person's face are deformed to obtain a head region mask;

[0018] Based on the overlapping region between the rendered alpha channel and the head region mask, the foreground head region and the background region are fused in the target video feature domain to obtain a fusion feature map;

[0019] The fusion feature map is converted into an RGB domain using a 2D neural re-renderer to obtain a head replacement result of the repaired target video.

[0020] According to an embodiment of the present application, the above-mentioned pre-processing of the source person image by separating the background and clothes in the source person image, and re-drawing the head region of the target video by using the fine-tuned pre-trained head generation model to obtain the head exchange data set include:

[0021] The background and clothes in the source person image are separated by cropping the foreground person region in the source person image to obtain a cropped source person image;

[0022] The cropped source person image is repaired using a pre-trained head generation model to obtain a source person image without clothing information;

[0023] Based on a pre-defined model fine-tuning method, the pre-trained head generation model is fine-tuned using the source person image without clothing information to obtain a fine-tuned pre-trained head generation model;

[0024] Based on a pre-defined conditional control neural network key point information injection method, the fine-tuned pre-trained head generation model is used to control the generation of the pose and expression of the head to repair the head region of the target video to obtain the head exchange data set.

[0025] According to an embodiment of the present application, the above-mentioned generation of a 4D neural Gaussian signature field based on a statistical extended human three-dimensional model includes:

[0026] A unified human three-dimensional surface is obtained using the statistical extended human three-dimensional model, and a 3D neural Gaussian field is constructed in the two-dimensional texture coordinate space of the unified human three-dimensional surface;

[0027] A learnable vertex offset is introduced into the 3D neural Gaussian field to obtain a 4D neural Gaussian signature field.

[0028] According to an embodiment of the present application, the above-mentioned training of the 4D neural Gaussian signature field using the head exchange data set includes:

[0029] The 4D neural Gaussian signature field is parameter tracked and adjusted using the head exchange data set, and based on dynamic scene modeling technology, the time consistency of the 4D neural Gaussian signature field in the process of generating images is enhanced by introducing time condition features to obtain a trained 4D neural Gaussian signature field.

[0030] According to a second aspect of the present application, a video head changing device is provided, comprising:

[0031] The image preprocessing and data generation module is used to separate the background and clothing in the source person image to obtain the preprocessed source person image, and use the fine-tuned pre-trained head generation model to redraw the head area of ​​the target video to obtain the head swap dataset;

[0032] The feature field generation and model training module is used to generate a 4D neural Gaussian feature field using a statistically based extended human 3D model and train the 4D neural Gaussian feature field using a head-swapped dataset to obtain a trained 4D neural Gaussian feature field. The attributes of each pixel in the 4D neural Gaussian feature field include neural signature, opacity, scale, and rotation.

[0033] The video restoration and video head-changing module is used to process the foreground head area of ​​the target video to obtain the restored target video. Based on the foreground and background fusion re-rendering technology of the video image, the trained 4D neural Gaussian feature field is used to perform a head-changing operation on the restored target video to obtain the head-changing result of the restored target video.

[0034] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0035] The fourth aspect of the present invention further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0036] The video head-changing method provided by the present invention utilizes a fine-tuned pre-trained head generation model, so that the generated exchange data can have better identity similarity and skin color consistency, greatly improving the adaptability and accuracy of the video head-changing technology, and can more accurately adapt to the characteristics of the source head domain; at the same time, the present invention can make full use of geometric relationships through the generated 4D neural Gaussian feature field, and upgrade the traditional 2D portrait video to a 4D neural Gaussian feature field, so that the generated video head-changing result has better geometric consistency and inter-frame consistency, and can more naturally handle the connection between the head and neck, thereby making the head-changing effect more realistic and stable; in addition, the present invention is based on the foreground and background fusion re-rendering technology, so that the video head-changing structure has a more natural head-to-body relationship and a seamless foreground and background splicing relationship, which solves the problem of foreground and background incoordination that may occur during the video head-changing process, thereby improving the overall smoothness and naturalness of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0038] Figure 1 2. This is a diagram showing an application scenario of a video head-changing method according to an embodiment of the present invention;

[0039] Figure 2 is a flowchart of a video head-changing method according to an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of a fine-tuning process of a pre-trained head generation model and a process of generating an exchange data set according to an embodiment of the present invention;

[0041] Figure 4 is a framework diagram of a video head-changing method according to an embodiment of the present invention;

[0042] Figure 5 is a structural block diagram of a video head-changing device according to an embodiment of the present invention;

[0043] Figure 6 4 is a block diagram of an electronic device suitable for implementing a video head-changing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0045] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0046] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0047] In cases where expressions such as "at least one of A, B, or C" are used, it is generally intended that the inclusion of A and B, A and C, B and C, or A and B and C is also (i.e., these expressions are treated as open-ended) unless otherwise indicated.

[0048] In the task of face swapping, the transfer of facial features between source and target videos is often not a simple matter. With the deepening of research in the field of face swapping, face swapping technology has gradually become a hot topic in computer vision. Early methods mainly adopted generative adversarial network (GAN) models, which fused the identity features of the source face with other features of the target face through adversarial training to ensure the visual authenticity of the generated face. Although these methods have achieved good results in many tasks, they also have some inherent limitations. For example, the representation ability of GAN models is limited, especially in complex facial details and expression changes, often failing to achieve ideal results. This makes early face swapping methods often inadequate when faced with high-difficulty face swapping tasks, making it difficult to ensure high fidelity of identity features.

[0049] In recent years, with the emergence of diffusion models, face swapping technology has been further improved. Some new research has begun to use diffusion models for face swapping. Compared with traditional GAN models, diffusion models can capture more detailed image details during the generation process, with better results. However, although diffusion models perform very well on single images, they still face great challenges when dealing with dynamic videos. Due to the randomness of the diffusion process, these methods often cannot maintain temporal consistency in video sequences, resulting in unsmooth face switching in videos, and even abrupt changes may occur. Therefore, how to achieve high-quality face swapping in video sequences is still a major problem faced by diffusion models.

[0050] In addition, some research has also attempted to use 3D deformable models (3DMM) for face swapping, aiming to enhance the model's expression ability by modeling the three-dimensional geometry of the face. Some methods use 3DMM to generate more realistic face swapping results, especially when dealing with complex facial expressions and angle changes. These methods can better handle changes in facial geometry. However, 3DMM is limited to modeling the face region, so when the task requires full-face face swapping, the application of 3DMM is inadequate. Changes in other regions of the head, such as hair and neck structures, are still problems that current 3DMM technology cannot effectively solve.

[0051] Compared with face swapping tasks, head swapping tasks are more difficult because they not only require high visual consistency in facial features and expressions in the target video, but also need to capture structural details of the source character's head, hair, neck, and other regions. Especially in the area matching problem between the head and the background, traditional methods often struggle to cope. This is because there is often a certain spatial structure difference between the head and the background, and how to seamlessly connect becomes a core problem in head swapping tasks. In response to these challenges, the latest previous method proposes a way to fuse the reenacted source head with the segmentation results of the target video to achieve head switching between the source head and the target video. Although this method has solved the matching problem between the head and the background to some extent, it still has poor results when there is a large change in posture, and due to the lack of 3D prior knowledge, it causes problems in temporal consistency. This makes the head swapping effect in dynamic videos not satisfactory, and it is easy to appear broken and unnatural.

[0052] In addition, due to the limited capacity of GAN models, although the initial head swapping effect is achieved through this method, the generated results often have certain limitations. Specifically, due to the complexity of facial and head structures, and the high variability in the target video, existing GAN models often struggle to maintain consistency between the head and the target video, resulting in a lack of naturalness and details in the generated results. This makes high-quality head swapping tasks still a great challenge in computer vision.

[0053] Therefore, although existing methods have promoted the progress of head swapping technology to varying degrees, how to achieve high-quality, natural, and consistent head swapping in dynamic videos is still an important problem in this field. Future research may need to combine more advanced 3D modeling techniques, extended spatiotemporal consistency models, and more efficient generation models to cope with various complex situations in head swapping tasks, thereby promoting the widespread application of this technology in movies, VR, AR, and other fields.

[0054] It should be particularly noted that in the technical solutions of the present application, the user information (including but not limited to user personal information, user image information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure, and application of related data comply with relevant laws, regulations, and standards, take necessary security measures, do not violate public order and good customs, and provide corresponding operation portals for users to choose authorization or refusal.

[0055] In the scenario of making automated decisions by using personal information, the method, device and system provided by the embodiment of the present application all provide corresponding operation entrances for the user to select to agree or reject the automated decision result; if the user selects to reject, the expert decision process is entered. The expression "automated decision" here refers to the activity of making decisions by automatically analyzing and evaluating the behavior habits, interests and hobbies or economic, health and credit conditions of a person by a computer program. The expression "expert decision" here refers to the activity of making decisions by a person who is engaged in a certain field, has special experience, knowledge and skills and reaches a certain professional level.

[0056] It should be noted that in the embodiment of the present application, some software, components, models and the like of the prior art may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the scheme.

[0057] The embodiment of the present application provides a video head changing method, which is used to solve the technical problems of poor image quality, inconsistent head changing result, poor 3D consistency and identity consistency and the like of the prior art video head changing method.

[0058] Figure 1 is an application scenario diagram of the video head changing method according to the embodiment of the present application.

[0059] As shown in Figure 1 The application scenario 100 according to the embodiment can include the technical fields of computer vision, graphics, artificial intelligence and deep forgery, and the like. The network 104 is used to provide a communication link medium between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, and the like.

[0060] The user can use the first terminal device 101, the second terminal device 102 and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages and the like. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102 and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software and the like (only as examples).

[0061] The first terminal device 101, the second terminal device 102 and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers and the like.

[0062] The server 105 can be a server providing various services, such as a background management server (only as an example) providing support for a website browsed by a user using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server can analyze and process received user requests and the like, and feed back the processing results (such as a webpage, information, or data generated or obtained according to a user request) to the terminal device.

[0063] It should be noted that the video head changing method provided by the embodiments of the present application can generally be executed by the server 105. Accordingly, the video head changing device provided by the embodiments of the present application can generally be arranged in the server 105. The video head changing method provided by the embodiments of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Accordingly, the video head changing device provided by the embodiments of the present application can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0064] It should be understood that Figure 1 the number of terminal devices, networks, and servers in the above-mentioned scenario is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks, and servers.

[0065] The video head changing method of the disclosed embodiments will be described in detail below based on the scenario described above. Figure 1 Figures 2-4 The video head changing method of the disclosed embodiments will be described in detail below based on the scenario described above.

[0066] Figure 2 is a flowchart of the video head changing method according to the embodiments of the present application.

[0067] As shown in Figure 2 , the video head changing of this embodiment includes operations S210-S230.

[0068] In operation S210, by separating the background and the clothes in the source person image, a pre-processed source person image is obtained, and a fine-tuned pre-trained head generation model is used to redraw the head region of the target video to obtain a head exchange data set.

[0069] The source person image includes a source person portrait for replacing the head of the target person in the target video. After the source person authorizes, the source person image is processed, and the processing of the source person image is only used for the video head changing method provided by the present application. The relevant processing process strictly complies with the regulations of laws and regulations, and strict security measures and anti-abuse measures are taken. ​

[0070] In addition, before operation S210 is performed, a clear and explicit prompt message needs to be given to inform the owner of the source person image or the source person that operation S210 is to be performed for video face swapping and related image processing operations, and operation S210 is performed after authorization or permission of the owner of the source person image or the source person is obtained. If the owner of the source person image or the source person refuses to provide authorization or permission, operation S210 is not performed.

[0071] Before the head generation model is used, the head generation model needs to be pre-trained, and the pre-trained head generation model is parameter fine-tuned to obtain a fine-tuned pre-trained head generation model, and the fine-tuned model is used to generate a head exchange dataset.

[0072] The head exchange dataset refers to a head sample dataset for training, verification, and inference.

[0073] Using the fine-tuned pre-trained head generation model to redraw the head region of the target video can make the obtained head exchange dataset focus on the head and face regions.

[0074] Operation S210 adapts the pre-trained 2D head generation model to the source head image field by using a fine-tuning strategy. The fine-tuning strategy finely tunes the pre-trained 2D head generation model to better capture the features of the source person's head. Subsequently, the present application generates a set of high-quality exchange datasets using diffusion repair technology. These exchange datasets provide rich source images for model training and ensure that the present application can perform the video face swapping task with high accuracy.

[0075] In operation S220, a 4D neural high-frequency feature field is generated based on a statistical extended human three-dimensional model, and the 4D neural high-frequency feature field is trained using the head exchange dataset to obtain a trained 4D neural high-frequency feature field, wherein the attributes of each pixel in the 4D neural high-frequency feature field include neural features, opacity, scale, and rotation.

[0076] The statistical extended human three-dimensional model includes SMPL-X (Skinned Multi-Person Linear Model-X) and can be used to reconstruct 3D poses and shapes from 2D images.

[0077] In order to ensure 3D consistency and temporal consistency during face swapping and more naturally model the relationship between the head and the neck, the present application introduces an intrinsic 3D high-frequency feature field and embeds it into a complete human SMPL-X surface model. Through this innovative technical means, the present application can accurately capture the spatial structure of the source person's head and maintain the coordination of the head and the body and the background.

[0078] In operation S230, the foreground head region of the target video is processed to obtain a repaired target video, and based on a foreground and background fusion re-rendering technology of a video image, the repaired target video is processed using the trained 4D neural high frequency field to obtain a head replacement result of the repaired target video.

[0079] The target video includes a portrait of a target person whose head is replaced, and the target video is processed after the target person, and the processing of the target video is only used for the video head replacement method provided by the application, and the related processing process strictly complies with the regulations of laws and regulations, and strict security measures and abuse prevention measures are taken.

[0080] In addition, before operation S230, obvious and clear prompt information needs to be given to inform the owner of the target video or the target person in the target video that operation S230 is a video head replacement operation and related image processing, and operation S230 is performed after obtaining the authorization or permission of the owner of the target video or the target person in the target video. If the owner of the target video or the target person in the target video refuses to provide authorization or permission, operation S230 is not performed.

[0081] The foreground feature map generated by Gaussian scattering is fused with the background feature map, and this process can effectively enhance the visual consistency between the head and the background. On this basis, the application uses a neural re-rendering technology to seamlessly fuse the two feature maps, thereby realizing a more natural and smooth head replacement effect.

[0082] As can be seen from the above operations S210 to S230, the application can maintain high identity similarity with the source image by using the expression ability of the diffusion model (i.e. the pre-trained head generation model, the same below); at the same time, the portrait representation is improved to 4D space, so that the result has better 3D consistency and temporal consistency; in addition, using the re-rendering module on the hidden space feature map, the natural head and trunk relationship can be obtained, and the mismatched area can be naturally filled.

[0083] The video head changing method provided by the application can make the generated exchange data have better identity similarity and skin color consistency by using the fine-tuned pre-trained head generation model, greatly improves the adaptability and precision of the video head changing technology, and can more accurately adapt to the characteristics of the source head domain; at the same time, the generated 4D neural high frequency characteristic field can fully utilize the geometric relationship, improve the traditional 2D portrait video to the 4D neural high frequency characteristic field, so that the generated video head changing result has better geometric consistency and interframe consistency, and can more naturally process the connection between the head and the neck, so that the head changing effect is more real and stable; in addition, the video head changing structure has more natural head-body relationship and seamless foreground-background splicing relationship based on the foreground-background fusion and re-rendering technology, solves the problem of incoordination between the foreground and the background in the video head changing process, and improves the overall fluency and naturalness of the video.

[0084] The operations S210-S230 will be further described in detail below by means of specific embodiments and in combination with the accompanying drawings.

[0085] The application provides a real video head changing technical solution, which mainly solves various defects of the previous video head changing algorithm, for example, the previous research on head changing by 3D deformable model (3DMM) can process face geometry deformation, but is limited to the face area and cannot effectively process the changes of other areas of the head such as hair and neck. Compared with face changing, the head changing task is more challenging, in addition to keeping the face features consistent, it also needs to capture details such as head, hair and neck, and solve the matching problem of the head and the background. Other existing technologies such as using GAN to fuse the segmentation results of the reenacted source head and the target video can solve the background matching to some extent, but still have deficiencies in pose change and time consistency. In addition, the representation ability of the existing GAN model is limited, and the generated result often lacks naturalness and details, making it difficult to cope with high-variation dynamic video. Therefore, the application provides a video head changing method which has obvious advantages in the quality, naturalness, 3D consistency and identity reservation of the generated result.

[0086] According to the embodiment of the present application, the above-mentioned head exchange data set is obtained by separating the background and clothes in the source person image, obtaining the pre-processed source person image, and repairing the head region of the pre-processed source person image using the fine-tuned pre-trained head generation model, which includes: separating the background and clothes in the source person image by cropping the foreground person region in the source person image to obtain the cropped source person image; repairing the cropped source person image using the pre-trained head generation model to obtain the source person image without clothing information; based on the pre-defined model fine-tuning method, the pre-trained head generation model is fine-tuned based on the source person image without clothing information to obtain the fine-tuned pre-trained head generation model; based on the pre-defined conditional control neural network key point information injection method, the fine-tuned pre-trained head generation model is used to control the generation of the posture and expression of the head, and the head region of the target video is repaired to obtain the head exchange data set.

[0087] The above-mentioned pre-trained head generation model includes Arc2Face model; the pre-defined model fine-tuning method includes Dreambooth model adjustment method based on generative adversarial network; and the pre-defined conditional control neural network includes ControlNet.

[0088] The above-mentioned Arc2Face is an identity-conditioned face base model, which can generate diversified, photo-realistic images according to the ArcFace embedding of a person, and the images are far superior to existing models in face similarity. The core advantage of Arc2Face lies in its ability to generate highly consistent face images with input ArcFace embedding, which is particularly important for application scenarios that require high identity consistency.

[0089] The above-mentioned Dreambooth is a personalized generation model technology, mainly used for fine-tuning a pre-trained generation model through a small amount of pictures, so as to generate images with specific attributes or styles. The core idea of Dreambooth is to retrain part of the generation model using a small amount of personalized data, so that it can generate images highly similar to the input data.

[0090] The above-mentioned ControlNet is a plug-in based on Stable Diffusion function, which aims to guide image generation by adding additional control conditions, so as to improve the controllability and accuracy of AI image generation. It can use edge features, depth features or skeletal features of human posture in input pictures to accurately guide the image generation process.

[0091] Figure 3 is the schematic diagram of the fine-tuning process of the pre-trained head generation model and the exchange data set generation process according to the embodiment of the present application.

[0092] As shown in Figure 3 The present application adopts a 2D head generation model (i.e., head generation model, same below) to repair the head region in the target video, thereby constructing a data set for head replacement. Although existing human head generation models perform well in the head generation task of maintaining identity consistency, they often lack in preserving the details and personalized features of the source character head. To solve this problem, the present application fine-tunes the pre-trained head generation model: Arc2Face, and uses a small amount of source images to enhance identity similarity.

[0093] Before fine-tuning, the present application first needs to separate the background and clothing information from the source image. Specifically, the present application crops the foreground character region in the source image and superimposes it onto the background of the target video. At the same time, the present application uses a diffusion model to repair the cropped image to remove clothing information. These steps are very important because they can effectively avoid the leakage of irrelevant information (such as background and clothing details), thereby ensuring that the fine-tuning process focuses on the face and head region. The present application fine-tunes the portrait generation model according to the method of Dreambooth, and injects key point information with the help of ControlNet to accurately control the pose and expression of the head. Finally, the present application uses the fine-tuned model to repair the head region in the target video to obtain a head exchange data set. The mask of the head region is obtained by deforming the head mask in the rough reproduction result based on five facial key points.

[0094] According to the embodiment of the present application, the above-mentioned generation of a 4D neural Gaussian signature field based on a statistical extended human body three-dimensional model comprises: obtaining a unified human body three-dimensional surface based on a statistical extended human body three-dimensional model, and constructing a 3D neural Gaussian field in a two-dimensional texture coordinate space of the unified human body three-dimensional surface; introducing a learnable vertex offset into the 3D neural Gaussian field to obtain a 4D neural Gaussian signature field.

[0095] According to the embodiment of the present application, the above-mentioned training of the 4D neural Gaussian signature field based on the head exchange data set comprises: parameter tracking and adjustment of the 4D neural Gaussian signature field based on the head exchange data set, and based on dynamic scene modeling technology, introducing time condition features to enhance the time consistency of the 4D neural Gaussian signature field in the process of generating images, to obtain a trained 4D neural Gaussian signature field.

[0096] Since the poses and expressions in the training data generated by the fine-tuned diffusion model (i.e., the fine-tuned pre-trained head generation model) are slightly different from the target video in some cases, showing a certain randomness, this may lead to slight inconsistencies in temporal consistency. To solve this problem, the present application re-extracts the SMPL-X parameters in the training data and introduces time-based conditional features.

[0097] Figure 4 is a framework diagram of a video head replacement method according to an embodiment of the present application.

[0098] The present application uses the generated data frames to supervise the 4D neural Gaussian portrait model (i.e., 4D neural Gaussian feature field, same below). In order to deal with the inconsistency of the generated training data frames, the present application adopts the SMPL-X re-tracking technology. In addition, the present application adds time conditional features in the neural Gaussian texture mechanism to effectively deal with the inconsistency between different frames.

[0099] Due to the randomness of the diffusion model itself, the generated head replacement frames often exhibit inconsistency, especially in facial expressions and head poses, which may lead to a large difference between the generated head replacement image and the target video. Especially in the process of dynamic video generation, each frame generated is affected by the randomness of the model, which may lead to unstable performance between different frames, thereby affecting the coherence and naturalness of the entire video sequence. In order to effectively solve this problem, the present application draws lessons from the previous 3D Gaussian portrait modeling method and proposes an innovative solution by converting the knowledge of the fine-tuned diffusion model into an efficient 3D representation method, thereby significantly enhancing the consistency and stability in the generation process.

[0100] Specifically, as shown in Figure 4 , the present application constructs a 3D Gaussian field (i.e., 3D neural Gaussian field, same below) in the UV space of the SMPL-X surface, and introduces a learnable vertex offset to dynamically adjust and deform the Gaussian body according to the deformation of the underlying mesh in the input video. With this method, the present application can more accurately align the morphology of the Gaussian body with the poses and expressions of the characters in the target video, thereby achieving a more natural and realistic head replacement effect. In addition, by embedding the 3D Gaussian field into the surface, the present application can use the shape, pose and expression parameters of the SMPL-X model to efficiently transform the Gaussian field, ensuring that the generated image is consistent with the dynamics of the target video in terms of shape, expression, pose and other dimensions.

[0101] In this process, the present application not only realizes the deformation of Gaussian body, but also stores the learnable features of each Gaussian body, so as to continuously optimize these features in the generation process, thereby improving the quality and naturalness of the generated image. In order to further refine this process, the present application establishes a neural Gaussian field in the UV space, where each pixel is characterized by four key attributes: neural features, opacity, scale and rotation. These attributes enable each pixel to maintain consistency under different transformations while precisely controlling details. Through UV mapping technology, the present application can accurately convert the neural Gaussian body from two-dimensional UV space to three-dimensional space, thereby completing the spatial transformation in the entire head replacement process and ensuring that the generated image can be naturally embedded in the target scene in three-dimensional space.

[0102] Although the present application has used ControlNet to inject key point information into the generation process to help better adjust the pose and expression in the head replacement image, in some cases there is still a certain expression and pose misalignment problem between the generated image and the target video. This problem mainly manifests itself in the subtle differences in facial expression, head pose, etc. between the generated head replacement image and the character in the target video, resulting in imperfect generation effect. In order to effectively alleviate this problem, the present application re-tracks and adjusts the SMPL-X parameters of the generated image during the training phase. By accurately tracking and optimizing the SMPL-X parameters of the generated image, the present application can better control the pose and expression in the generated image, thereby reducing the inconsistency problem caused by pose or expression differences. In the inference phase, the present application uses the SMPL-X parameters tracked from the original target video to further optimize the pose and expression of the head during the generation process, so that the generated head replacement image can be more closely matched with the dynamics of the target video.

[0103] In order to further solve the inconsistency problem between different frames in the generated data set, the present application draws on the latest dynamic scene modeling technology by introducing time conditional features to enhance the temporal consistency of the generated image. Specifically, during the training phase, the present application stores a set of learnable temporal features and broadcasts these features to each Gaussian body in all frames, thereby ensuring that the generation of each frame maintains temporal coherence. This method effectively avoids sudden changes or incoordination between different frames, making the generated head replacement sequence consistent in time and improving the smoothness and naturalness of the video. In the inference phase, the present application uses the features to maintain the consistency of all frames, further enhancing the temporal coherence and ensuring that the generated video exhibits consistent head pose and expression in multiple consecutive frames, thereby significantly improving the stability and naturalness of the generated sequence.

[0104] According to an embodiment of the present application, the processing of the foreground head region of the target video to obtain the repaired target video includes: removing the foreground head region of the target video to obtain a target video with the foreground head region removed; and repairing the target video with the foreground head region removed by using a predefined image repairing tool to obtain the repaired target video, wherein the repaired target video includes a background region and a torso region of the target person.

[0105] According to an embodiment of the present application, the predefined image repairing tool includes an image segmentation model, an image inpainting model, and an artificial intelligence-based content generation model.

[0106] According to an embodiment of the present application, the foreground and background fusion re-rendering technology based on a video image uses a trained 4D neural high-strength feature field to perform a head replacement operation on the repaired target video to obtain a head replacement result of the repaired target video, including: converting the repaired target video into a feature domain by using a 2D background encoder to obtain a target video feature domain; performing two-dimensional splashing on the trained 4D neural high-strength feature field to obtain a foreground head region; processing the trained 4D neural high-strength feature field to obtain a rendered alpha channel; deforming a plurality of key points of a face of a person during a head reproduction process of the trained 4D neural high-strength feature field to obtain a head region mask; fusing the foreground head region and the background region in the target video feature domain based on an overlapping region between the rendered alpha channel and the head region mask to obtain a fusion feature map; and converting the fusion feature map into an RGB domain by using a 2D neural re-renderer to obtain the head replacement result of the repaired target video.

[0107] After obtaining the Gaussian rendered feature map, the neural re-renderer of the present application is used to seamlessly fuse the foreground scatter portrait features and the background features, thereby completing a natural head replacement effect.

[0108] The 4D portrait representation method of the present application can effectively model the head after head replacement, but it still faces challenges to reintegrate this 3D head into the original video. In particular, due to differences in head shape and hairstyle, the head region between the foreground and the target image often exhibits significant mismatches. Therefore, the neural re-rendering module is designed to solve this problem.

[0109] Firstly, the foreground head region in the target image is removed and repaired using Inpaint-Anything. Then, the repaired image (only containing the background and torso) is processed by a 2D background encoder to be converted into a feature domain. Based on the overlapping area between the rendered alpha channel and the head mask, the foreground head and the background are fused in the feature domain. Subsequently, the fused feature map is processed by a 2D neural re-renderer to be converted into the RGB domain, and finally the natural fusion of the head is completed.

[0110] Compared with various video face / head replacement technical solutions in the prior art, the present application has the following advantages: the present application proposes to use source image personalized fine-tuning of a pre-trained head diffusion model, so that the generated training data can have better identity similarity and skin color consistency. The present application proposes to use a generated data re-tracking method and a learnable temporal conditional feature to realize reconstruction of any consistent three-dimensional representation from inconsistent training data, which is embodied as expression consistency with the target image in the video face / head replacement task. The foreground and background fusion re-rendering technology based on feature maps proposed by the present application makes the face / head replacement result have a more natural head and body relationship and a seamless foreground and background splicing relationship. The four-dimensional Gaussian model proposed by the present application solves the face / head replacement task by fully utilizing the geometric relationship, so that the present application has better geometric consistency and inter-frame consistency. Due to the network structure design proposed by the present application, the learnable features of the foreground and background and the convolutional neural network are added to fully extract the high-frequency information of the training data, so that the present application has high clarity.

[0111] Based on the above-mentioned video face / head replacement method, the present application further provides a video face / head replacement device. The following will be described in detail Figure 5 with reference to the device.

[0112] Figure 5 is a structural block diagram of the video face / head replacement device according to the embodiment of the present application.

[0113] As Figure 5 shown, the video face / head replacement device 500 of the embodiment includes an image preprocessing and data generation module 510, a feature field generation and model training module 520, and a video repair and video face / head replacement module 530.

[0114] The image preprocessing and data generation module 510 is configured to separate the background and the clothes in the source person image to obtain a preprocessed source person image, and use the fine-tuned pre-trained head generation model to redraw the head region of the target video to obtain a head exchange data set. In an embodiment, the image preprocessing and data generation module 510 can be configured to perform the operation S210 described above, and details are not repeated here.

[0115] The feature field generation and model training module 520 is configured to generate a 4D neural high-dimensional feature field based on a statistical extended human body three-dimensional model, and train the 4D neural high-dimensional feature field based on a head swap dataset, to obtain a trained 4D neural high-dimensional feature field. In an embodiment, the feature field generation and model training module 520 can be configured to perform the operation S220 described above, and details are not repeated here.

[0116] The video repair and video head swap module 530 is configured to process a foreground head region of a target video to obtain a repaired target video, and perform a head swap operation on the repaired target video based on a foreground and background fusion re-rendering technology of a video image and the trained 4D neural high-dimensional feature field, to obtain a head swap result of the repaired target video. In an embodiment, the video repair and video head swap module 530 can be configured to perform the operation S230 described above, and details are not repeated here.

[0117] According to an embodiment of the present application, any one or more of the image preprocessing and data generation module 510, the feature field generation and model training module 520, and the video repair and video head swap module 530 can be combined in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of other modules, and implemented in one module. According to an embodiment of the present application, at least one of the image preprocessing and data generation module 510, the feature field generation and model training module 520, and the video repair and video head swap module 530 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. hardware or firmware, or any one of software, hardware and firmware or any appropriate combination of several of them. Alternatively, at least one of the image preprocessing and data generation module 510, the feature field generation and model training module 520, and the video repair and video head swap module 530 can be at least partially implemented as a computer program module which can perform corresponding functions when running.

[0118] The application provides a continuous and realistic video head changing device based on a 4D neural Gaussian portrait prior. The application embeds an intrinsic 3D Gaussian feature field into a full-body SMPL-X surface, upgrading traditional 2D portrait videos to 4D neural Gaussian fields. This effectively ensures high consistency of 3D structures and can naturally handle the connection between the head and the neck, making the head changing effect more realistic and stable. When creating the training data set required for reconstruction, the application uses a small amount of source images to fine-tune the pre-trained 2D portrait generation model, enabling it to accurately adapt to the characteristics of the source domain. This process greatly improves the adaptability and accuracy of the model. At the same time, the application introduces an advanced neural re-rendering strategy that can achieve seamless fusion of foreground and background. This strategy effectively eliminates the problem of inconsistent foreground and background that may occur during head changing, thereby improving the overall smoothness and naturalness of the video. Experimental results show that, compared with existing head changing methods, the device of the application has significant advantages in image quality, naturalness, 3D consistency and identity preservation, and can generate more realistic, stable and highly consistent head changing effects.

[0119] Figure 6 is a block diagram of an electronic device suitable for implementing a video head changing method according to an embodiment of the application.

[0120] As shown in Figure 6 , the electronic device 600 according to an embodiment of the application includes a processor 601 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), and / or the like. The processor 601 can also include an on-board memory for cache use. The processor 601 can include a single processing unit or multiple processing units for performing different actions of the method processes according to embodiments of the application.

[0121] In the RAM 603, various programs and data required for the operation of the electronic device 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. The processor 601 performs various operations of the method processes according to embodiments of the application by executing programs in the ROM 602 and / or the RAM 603. It should be noted that the programs can also be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 can also perform various operations of the method processes according to embodiments of the application by executing programs stored in the one or more memories.

[0122] According to an embodiment of the present application, the electronic device 600 can further include an input / output (I / O) interface 605 that is also connected to the bus 604. The electronic device 600 can further include one or more of the following components connected to the input / output (I / O) interface 605: an input part 606 including a keyboard, a mouse, etc.; an output part 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 608 including a hard disk, etc.; and a communication part 609 including a network interface card such as a LAN card, a modem, etc. The communication part 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as necessary. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 610 as necessary, so that a computer program read out therefrom is installed in the storage part 608 as necessary.

[0123] The present application also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, when the one or more programs are executed, the method according to the embodiments of the present application is implemented.

[0124] According to an embodiment of the present application, the computer readable storage medium can be a non-volatile computer readable storage medium, for example, can include but not limited to: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, the computer readable storage medium can include one or more memories of the above-described ROM 602 and / or RAM 603 and / or one or more memories other than the ROM 602 and the RAM 603.

[0125] The computer program product of the present application can be a computer program product comprising a computer readable storage medium and a computer program mechanism embedded in the computer readable storage medium. Such computer program product can further include a computer readable storage medium and program means for causing a processor or other programmable processing apparatus to function in a particular manner, such that the computer program mechanism that can be executed by such a processor or processing apparatus causes the processor or processing apparatus to implement the functions of the system, method or computer program product. The computer program product can be intended for use in one or more embodiments described herein.

[0126] Those skilled in the art will appreciate that the features recited in the various embodiments of the present application can be combined and / or interchanged, even if this is not explicitly stated in the present application. In particular, the features recited in the various embodiments of the present application can be combined and / or interchanged, without departing from the spirit and teachings of the present application. All such combinations and / or interchanges are to be considered within the scope of the present application.

[0127] The embodiments of the present application described above are merely intended for illustration and not intended to limit the scope of the present application. Although the embodiments are described separately above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various alternatives and modifications can be made to the embodiments of the present application by those skilled in the art without departing from the scope of the present application, and all such alternatives and modifications are to be considered within the scope of the present application.

Claims

1. A video head-changing method, characterized in that: The method comprises: By separating the background and clothing in the source person image, we obtain the preprocessed source person image, and use the fine-tuned pre-trained head generation model to redraw the head area of ​​the target video to obtain the head swap dataset; generating a 4D neural Gaussian feature field using a statistically based extended human three-dimensional model, and training the 4D neural Gaussian feature field using the head-swapped dataset to obtain a trained 4D neural Gaussian feature field, wherein the attributes of each pixel in the 4D neural Gaussian feature field include neural features, opacity, scale, and rotation; Processing the foreground head region of the target video to obtain a restored target video, and performing a head swap operation on the restored target video using the trained 4D neural Gaussian feature field based on a foreground and background fusion re-rendering technique of a video image to obtain a head swap result of the restored target video; Among them, the generation of 4D neural Gaussian feature fields using the statistically based extended human 3D model includes: A unified three-dimensional human body surface is obtained using a statistically extended three-dimensional human body model, and a 3D neural Gaussian field is constructed in the two-dimensional texture coordinate space of the unified three-dimensional human body surface. A learnable vertex offset is introduced into the 3D neural Gaussian field to obtain the 4D neural Gaussian feature field.

2. The method according to claim 1, characterized in that The foreground header area of ​​the target video is processed, and the repaired target video includes: Removing the foreground header region of the target video to obtain the target video with the foreground header region removed; The target video with the foreground head area removed is repaired using a predefined image repair tool to obtain a repaired target video, wherein the repaired target video includes a background area and a torso area of ​​the target person.

3. The method according to claim 2, characterized in that The predefined image restoration tools include an image segmentation model, an image patching model, and an artificial intelligence-based content generation model.

4. The method according to claim 2, characterized in that Based on the foreground and background fusion re-rendering technology of the video image, the trained 4D neural Gaussian feature field is used to perform a head-changing operation on the repaired target video, and the head-changing result of the repaired target video is obtained, including: Converting the restored target video into a feature domain using a 2D background encoder to obtain a target video feature domain; Performing two-dimensional splattering on the trained 4D neural Gaussian feature field to obtain a foreground head region; Processing the trained 4D neural Gaussian feature field to obtain a rendered alpha channel; In the process of reproducing the head using the trained 4D neural Gaussian feature field, multiple key points of the face of the character are deformed to obtain a head region mask; Based on an overlapping area between the rendered alpha channel and the head region mask, fusing the foreground head region and the background region within the target video feature domain to obtain a fused feature map; The fused feature map is converted into an RGB domain using a 2D neural re-renderer to obtain a head-changing result of the repaired target video.

5. The method according to claim 1, wherein By separating the background and clothing in the source person image, a preprocessed source person image is obtained, and the head area of ​​the target video is redrawn using the fine-tuned pre-trained head generation model. The head swap dataset includes: Separating the background and clothing in the source person image by cropping the foreground person region in the source person image to obtain a cropped source person image; Restoring the cropped source person image using the pre-trained head generation model to obtain the source person image without clothing information; Based on a predefined model fine-tuning method, fine-tuning parameters of the pre-trained head generation model using the source person image with the clothing information removed, to obtain a fine-tuned pre-trained head generation model; Based on predefined conditions, the key point information injection method of the neural network is controlled, the fine-tuned pre-trained head generation model is used to control the generation of head posture and expression, and the head area of ​​the target video is repaired to obtain the head exchange dataset.

6. The method according to claim 1, characterized in that The 4D neural Gaussian feature field is trained using the head exchange dataset to obtain a trained 4D neural Gaussian feature field including: The head swap dataset is used to track and adjust the parameters of the 4D neural Gaussian feature field, and based on dynamic scene modeling technology, time condition features are introduced to enhance the temporal consistency of the 4D neural Gaussian feature field in the image generation process, thereby obtaining the trained 4D neural Gaussian feature field.

7. A video head-changing device, characterized in that: The device comprises: The image preprocessing and data generation module is used to separate the background and clothing in the source person image to obtain the preprocessed source person image, and use the fine-tuned pre-trained head generation model to redraw the head area of ​​the target video to obtain the head swap dataset; a feature field generation and model training module, configured to generate a 4D neural Gaussian feature field using a statistically based extended human 3D model, and train the 4D neural Gaussian feature field using the head-swapped dataset to obtain a trained 4D neural Gaussian feature field, wherein the attributes of each pixel in the 4D neural Gaussian feature field include neural features, opacity, scale, and rotation; The video restoration and video head-changing module is used to process the foreground head area of ​​the target video to obtain a restored target video, and based on the foreground and background fusion re-rendering technology of the video image, use the trained 4D neural Gaussian feature field to perform a head-changing operation on the restored target video to obtain the head-changing result of the restored target video; Among them, the generation of 4D neural Gaussian feature fields using the statistically based extended human 3D model includes: A unified three-dimensional human body surface is obtained using a statistically extended three-dimensional human body model, and a 3D neural Gaussian field is constructed in the two-dimensional texture coordinate space of the unified three-dimensional human body surface. A learnable vertex offset is introduced into the 3D neural Gaussian field to obtain the 4D neural Gaussian feature field.

8. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images