Video face swapping method, device and storage medium

Abstract

The application discloses a video face replacement method and device and a storage medium, relates to the technical field of video face replacement, and comprises the following steps: acquiring an original video to be replaced and an appearance image of a target person, inputting the original video and the appearance image into a preset video face replacement model, and replacing a person in the original video with the target person, wherein the video face replacement model comprises a high-frequency refinement module, the high-frequency refinement module extracts high-frequency latent variables of the appearance image, modulates appearance features of the appearance image based on the high-frequency latent variables, obtains modulated appearance features after high-frequency modulation, and appearance of the target person in the target video is determined based on the modulated appearance features. By modulating the appearance features, the model can better learn high-frequency details in the high-frequency latent variables, so that high-frequency details of the person in the output target video are better depicted, and the realism of the generated face replacement video is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video face swapping, and in particular to a video face swapping method, device and storage medium. BACKGROUND

[0002] The video face swapping task is a task of replacing the appearance of one or more specific characters in a video with the appearance of another target character while keeping the original video action and scene background as much as possible.

[0003] The current video face swapping task usually extracts the appearance features of the target task, the action features of the original video characters and the background information, receives the extracted appearance features, action features and background information through the generated denoising manner, and gradually generates a video with character replacement effect.

[0004] However, the existing character appearance feature extraction method mainly depends on the spatial domain image features, and it is difficult to capture high-frequency detail features such as facial features and clothing textures, so that the generated result loses a lot of high-frequency details, thereby causing the generated face swapping video to have low realism. SUMMARY

[0005] The main purpose of the present application is to provide a video face swapping method, device and storage medium, which aims to solve the technical problem of low realism of the generated face swapping video.

[0006] To achieve the above purpose, the present application provides a video face swapping method, which comprises:

[0007] Obtaining an original video and an appearance image of a target character that need to be replaced;

[0008] Inputting the original video and the appearance image into a preset video face swapping model to replace the original video characters in the original video with the target character to obtain a target video, wherein the video face swapping model extracts high-frequency latent variables of the appearance image, modulates the appearance features of the appearance image based on the high-frequency latent variables to obtain modulated appearance features after high-frequency modulation, and the appearance of the target character in the target video is determined based on the modulated appearance features.

[0009] In an embodiment, before the step of inputting the original video and the appearance image into a preset video face swapping model to replace the original video characters in the original video with the target character to obtain a target video, the method further comprises:

[0010] Extracting the original video characters and the background of the original video to obtain a character mask of the original video characters, a character skeleton graph of the original video characters and a video background of the original video;

[0011] extracting features of the appearance image to obtain the modulation appearance feature and the semantic feature of the target person;

[0012] training a preset to-be-trained model based on the preset Gaussian noise, the person mask, the person skeleton map, the video background, the modulation appearance feature, the semantic feature, and the appearance image, to obtain the video face swapping model.

[0013] In an embodiment, the step of extracting features of the appearance image to obtain the modulation appearance feature and the semantic feature of the target person comprises:

[0014] extracting semantic features of the appearance image to obtain the semantic feature;

[0015] mapping the appearance image to a latent space based on a preset encoder to obtain a latent vector of the appearance image;

[0016] extracting the appearance feature of the appearance image based on the latent vector;

[0017] high-pass filtering and mapping the latent vector in the frequency domain to obtain modulation parameters corresponding to the appearance feature;

[0018] modulating the appearance feature based on the modulation parameters to obtain the modulation appearance feature.

[0019] In an embodiment, the step of high-pass filtering and mapping the latent vector in the frequency domain to obtain modulation parameters corresponding to the appearance feature comprises:

[0020] performing Fourier transform on the latent vector to obtain a latent vector spectrum of the latent vector;

[0021] moving a part with a frequency of zero in the latent vector spectrum from an original position to a spectrum center of the latent vector spectrum to obtain a target spectrum;

[0022] high-pass filtering the target spectrum to obtain a high-frequency spectrum;

[0023] moving a part with a frequency of zero in the high-frequency spectrum to the original position, and performing inverse Fourier transform on the high-frequency spectrum after the frequency is moved to obtain a high-frequency latent vector;

[0024] determining the modulation parameters based on the high-frequency latent vector and a preset high-frequency refinement mapping method.

[0025] In an embodiment, the modulation parameters include scaling parameters and bias parameters, and the step of determining the modulation parameters based on the high-frequency latent vector and a preset high-frequency refinement mapping method comprises:

[0026] convolve the high-frequency latent vector through a preset first mapping convolutional layer to obtain a first target latent vector in a first target shape, wherein the first target shape comprises a first target number of channels;

[0027] convolve the first target latent vector through a preset second mapping convolutional layer to obtain a second target latent vector in a second target shape, wherein the second target shape comprises a second target number of channels, and the second target number of channels is twice the first target number of channels;

[0028] split the second target latent vector based on the first target number of channels to obtain the scaling parameter with the first target number of channels and the bias parameter with the second target number of channels.

[0029] In an embodiment, the step of high-pass filtering the target spectrum to obtain a high-frequency spectrum comprises:

[0030] determining distances between each frequency in the target spectrum and a spectrum center of the target spectrum to obtain a radius set of the target spectrum;

[0031] cumulatively summing amplitude values of frequencies corresponding to each radius in the radius set, and sorting the summed amplitude values to obtain a cumulative amplitude distribution of the target spectrum, wherein the cumulative amplitude distribution comprises a cumulative amplitude of each distribution and a frequency corresponding to the cumulative amplitude;

[0032] determining a target cumulative amplitude based on the cumulative amplitude distribution and a preset amplitude threshold, and taking a frequency corresponding to the target cumulative amplitude as a cutoff frequency;

[0033] determining a high-frequency frequency higher than the cutoff frequency in the target spectrum to obtain the high-frequency spectrum.

[0034] In an embodiment, the step of training a preset to-be-trained model based on the preset Gaussian noise, the person mask, the person skeleton map, the video background, the modulated appearance feature, the semantic feature, and the appearance image to obtain the video face-swapping model comprises:

[0035] extracting an action feature of the target person based on a preset action convolutional module and the person skeleton map;

[0036] generating an initial face-swapped video through a preset to-be-trained model based on the Gaussian noise, the person mask, the video background, the modulated appearance feature, the semantic feature, and the action feature;

[0037] Adjust parameters of the to-be-trained model based on the person mask, the appearance image, and the initial video face swapping video to obtain the video face swapping model.

[0038] In an embodiment, the step of adjusting parameters of the to-be-trained model based on the person mask, the appearance image, and the initial video face swapping video to obtain the video face swapping model comprises:

[0039] Increase pixel points of the person mask based on a preset dilation operation to obtain an expansion region.

[0040] Reduce pixel points of the person mask based on a preset erosion operation to obtain a contraction region.

[0041] Calculate a difference between the expansion region and the contraction region to obtain a ring region.

[0042] Calculate a similarity loss of the appearance image and the initial video face swapping video in the ring region.

[0043] Adjust parameters of the to-be-trained model based on the similarity loss to obtain the video face swapping model.

[0044] In addition, to achieve the above-mentioned purposes, the present application also proposes a video face swapping method and device, the device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the video face swapping method as described above.

[0045] In addition, to achieve the above-mentioned purposes, the present application also proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the video face swapping method as described above.

[0046] The one or more technical solutions proposed in the present application have at least the following technical effects:

[0047] Obtain an original video and an appearance image of a target person that need to be replaced, input the original video and the appearance image into a preset video face swapping model, replace a person in the original video with the target person, wherein the video face swapping model comprises a high-frequency refinement module, the high-frequency refinement module extracts a high-frequency latent variable of the appearance image, modulates appearance features of the appearance image based on the high-frequency latent variable to obtain modulated appearance features after high-frequency modulation, and appearance of the target person in the target video is determined based on the modulated appearance features.

[0048] In view of the problem that the existing method is difficult to capture high-frequency detail features such as facial features and clothing texture, so that the generated result loses a lot of high-frequency details, resulting in low realism of the generated person-swapped video, the appearance features of the appearance image are modulated based on the high-frequency latent variable. Since the appearance of the target person in the target video is determined based on the modulated appearance features, and the appearance features are obtained based on the modulation of the high-frequency latent variable, the model can better learn the high-frequency details in the high-frequency latent variable by modulating the appearance features, so that the high-frequency details of the person in the output target video are better depicted, and the realism of the generated person-swapped video is improved. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings required to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0051] Figure 1 The flowchart provided for the video person-swapping method embodiment one of the present application;

[0052] Figure 2 The overall flowchart provided for the video person-swapping method embodiment one of the present application;

[0053] Figure 3 The module architecture diagram provided for the video person-swapping method embodiment one of the present application;

[0054] Figure 4 The feature modulation overall flowchart provided for the video person-swapping method embodiment one of the present application;

[0055] Figure 5 The feature modulation detailed flowchart provided for the video person-swapping method embodiment one of the present application;

[0056] Figure 6 The flowchart provided for the video person-swapping method embodiment two of the present application;

[0057] Figure 7 The device structure diagram of the hardware running environment involved in the video person-swapping method in the embodiments of the present application;

[0058] Figure 8 The data acquisition consent diagram involved in the video person-swapping method in the embodiments of the present application.

[0059] The objectives, functional features and advantages of the present application will be further illustrated in conjunction with the embodiments, with reference to the accompanying drawings. DETAILED DESCRIPTION

[0060] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0061] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0062] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a video swapping method device, etc. The present embodiment and the following embodiments will be described below by taking the video swapping method device as an example.

[0063] The video swapping task refers to taking a person reference image and a target video frame as input, and after model processing, the original person in the output video sequence is replaced by the person corresponding to the reference image. The generation result not only needs to be consistent in appearance, but also should be naturally integrated with the original video scene background, and the overall effect is coherent and realistic.

[0064] In the existing video swapping method, the person appearance feature extraction method mainly relies on spatial domain image features, which is difficult to effectively capture high-frequency texture information in the reference image, especially in the face area. In the generated video, the texture and structure of the person's face are prone to abnormalities, such as blurred skin details, slight deviation of facial proportions, and even distortion or deformation, resulting in distorted generation results, which are difficult to maintain the same realistic feeling as the original video. At the same time, the high-frequency details such as wrinkles, shadows and small texture changes of clothes often cannot be effectively restored, and after replacement, a smooth effect similar to ironing is presented, making the person's appearance lack of level and dynamic texture. Overall, these problems make the generated video appear stiff and unnatural, resulting in a low realistic feeling of the generated video.

[0065] In addition, in the existing method, the person and the background are usually synthesized through the image level, and the person edge is prone to appear bright edge (an unnatural light outline appears between the person edge pixels and the background) or flicker (the hair of the person in a certain frame is normally transitioned with the background, but the next frame suddenly appears a bright line or obvious color block artifact at the edge, and the next frame disappears). The appearance of bright edges and artifacts will make the viewer feel that the person is spliced by post-editing, rather than being originally integrated with the scene, resulting in a low realistic feeling of the generated video.

[0066] Based on this, the embodiment of the present application provides a video face replacement method, referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the video face replacement method of the present application.

[0067] In this embodiment, the video face replacement method includes steps S10-S20:

[0068] Step S10, obtaining an original video and an appearance image of a target person that need to be replaced;

[0069] It should be noted that the original video refers to an input video sequence containing a person to be replaced, and the person in the video will be replaced by the target person, but the scene background, lighting conditions and action poses will be retained for generating a new video. The appearance image of the target person refers to a static image provided by a user, which is used to define the appearance features of the replaced person, including facial features, skin color, hairstyle, clothing texture and other visual attributes. In this embodiment, the image is used as a reference image or a person reference image to provide appearance consistency constraints for the generated model.

[0070] It can be understood that in this embodiment, the system receives an original video containing a person to be replaced and a static image representing the appearance of a target person. The original video is used to extract background information, person position mask and action pose sequence, while the appearance image of the target person is used to extract high-fidelity appearance features to ensure that the identity of the person in the generated result is consistent with the image.

[0071] Step S20, inputting the original video and the appearance image into a preset video face replacement model to replace the original video person in the original video with the target person to obtain a target video, wherein the video face replacement model extracts high-frequency latent variables of the appearance image, modulates the appearance features of the appearance image based on the high-frequency latent variables to obtain modulated appearance features after high-frequency modulation, and the appearance of the target person in the target video is determined based on the modulated appearance features.

[0072] It should be noted that the preset video face replacement model is a generation system based on a diffusion model framework constructed in this embodiment. The video face replacement model of this embodiment includes an appearance feature extraction module (ReferenceNet), an action control module (Motion Control Module), a high-frequency reference feature refinement module (High-Frequency Reference Refinement, HF-Refine), a temporal modeling module (Temporal Layer) and a denoising generation module (DenoisingNet), and integrates a mask narrowband constraint mechanism for realizing high-quality person replacement.

[0073] The high-frequency latent variable refers to a high-frequency component extracted by performing high-pass filtering in a frequency domain on a latent space representation of a vector after appearance images are encoded by a VAE (Variational Autoencoder), and mainly includes frequency domain features corresponding to details such as facial features, hair edge, and clothing texture of a person. The modulated appearance feature refers to an enhanced feature obtained by adaptively adjusting the output features of each sampling module of the appearance feature extraction module based on the scaling and bias parameters generated by introducing the high-frequency latent variable into the spatial domain appearance features extracted by the appearance feature extraction module, and the modulated appearance feature carries more rich high-frequency detail information. The target video refers to a video sequence output after the video face swapping model is processed, in which the person in the original video is replaced by the target person, and the new person is highly consistent with the reference image in appearance, is synchronized with the original video in action, and is natural and has no artifacts in boundary fusion. The overall process of video face swapping in this embodiment can be referred to as Figure 2 .

[0074] It can be understood that the traditional video face swapping method usually only relies on spatial domain features for appearance modeling, and it is difficult to capture high-frequency texture information in the reference image, resulting in problems such as blurred face, slight facial feature offset, and clothing texture loss in the generated result. And the present application actively separates the high-frequency component by performing frequency domain analysis on the latent variable of the reference image, and converts it into learnable scale and offset parameters, realizing fine enhancement of the appearance feature. Based on the modulated appearance feature, the feature expression of key areas such as facial feature edge, hair detail, and clothing wrinkle can be accurately enhanced, thereby improving the visual realism of the generated face swapping video.

[0075] In a feasible implementation manner, before the specific implementation manner of inputting the original video and the appearance image into a preset video face swapping model and replacing the original video person in the original video with the target person to obtain a target video, the following implementation manners can also be used:

[0076] The original video person and the background of the original video are extracted to obtain a person mask of the original video person, a person skeleton graph of the original video person, and a video background of the original video. The appearance image is subjected to feature extraction to obtain the modulated appearance feature and the semantic feature of the target person. Based on the preset Gaussian noise, the person mask, the person skeleton graph, the video background, the modulated appearance feature, the semantic feature, and the appearance image, a preset to-be-trained model is trained to obtain the video face swapping model.

[0077] It should be noted that the character mask refers to a binary image representing the area where the character is located extracted from the original video, which is used to identify the position of the character to be replaced in each frame and to perform local repair and boundary constraint in the subsequent generation process. The character skeleton map refers to a human key point connection map extracted from the original video character through pose estimation technology, which reflects the limb structure and action pose of the character, and is used to provide action consistency constraint for the generated result. The video background refers to the scene part retained after the area where the character is located in the original video is blocked or removed according to the character mask, which is used to keep the background unchanged in the generation process to ensure the scene consistency of the replaced video.

[0078] The semantic feature refers to high-level semantic information extracted from the appearance image through a visual language model, which is used to describe the overall visual semantics of the character and provide consistency guidance at the semantic level for the generation process. In the embodiment, the extraction of visual semantic features can be performed through CLIP (Contrastive Language-Image Pre-training). The preset Gaussian noise is a random noise tensor used as initial input in the diffusion model training process. In the embodiment, the preset Gaussian noise is sampled from a standard Gaussian distribution and used as the starting point for the diffusion model to gradually denoise the target image. The model to be trained refers to the initial version of the video face replacement model that has not been trained, which includes basic structures such as Denoising Net, ReferenceNet, and action control module, and needs to learn how to fuse multi-modal conditions and generate high-quality replacement videos through the training process of the embodiment.

[0079] It can be understood that before performing video character replacement through the model, the training of the video face replacement model needs to be completed first. Specifically, the embodiment first processes the original video to extract the character mask and character skeleton map of each frame, and removes the character area in the original video based on the character mask to obtain a pure video background. At the same time, the embodiment performs feature extraction on the appearance image of the target character: on the one hand, it generates modulation appearance features through VAE encoding and high-frequency latent variables; on the other hand, it extracts semantic features through a visual language model to capture high-level semantic information of the image.

[0080] After obtaining the necessary information and features, the embodiment splices the noise, character mask, and background information to obtain the spliced features as the input of the denoising generation module. In the embodiment, the size of the noise is 4xh xw, the size of the character mask is 1xh xw, and the size of the background information is 4xh xw. In the dimension size representation of the noise, the character mask, and the background information, h is the height, w is the width, and the remaining dimensions are the number of channels. The noise, character mask, and background information are spliced in the channel dimension, and finally an input feature with a size of 9xh xw is obtained and input into the denoising generation module.

[0081] After obtaining the input features, the embodiment inputs the input features with a size of 9xh xw into the denoising generation module, and in the diffusion denoising process of the denoising generation module, the appearance features, the semantic features and the action features obtained based on the human skeleton graph are fused and modulated to predict the noise to be removed, and the image is denoised based on the predicted noise to gradually generate the latent space representation conforming to the target action and appearance features.

[0082] Further, the embodiment calculates the loss value between the predicted noise and the real noise according to the preset noise prediction loss function, determines the gradient of the model parameter adjustment based on the loss value, and adjusts the parameters based on the determined gradient. Through multiple rounds of iterative training, the model gradually learns how to start from the Gaussian noise and gradually generate video frames consistent with the target human appearance, synchronized with the action, and naturally fused with the background, under the given conditions. Finally, the trained video face replacement model is obtained.

[0083] It can be understood that the embodiment enables the to-be-trained model to fully fuse multi-dimensional information such as human appearance, action, background and semantic information in the learning stage through the above steps, thereby improving its comprehensive performance in actual replacement tasks. Since the human mask and the video background are added in the training process, the model can learn to generate content in the specified area, avoid modifying the background, ensure the stability of the scene, and by adding the human skeleton graph, the model follows the real human motion law in the generation process, improving the naturalness and coherence of the action.

[0084] Further, the embodiment generates high-frequency enhanced appearance representation in the training stage by fusing and modulating the appearance features, thereby improving the restoration ability of high-frequency information such as facial details, hair edges and clothing textures. The semantic features provide high-level semantic guidance for the model, enhance the understanding ability of the overall image of the human, and avoid overall distortion caused by local feature overfitting. The embodiment trains the model by fusing the Gaussian noise and the above-mentioned various features, so that the model not only learns the denoising path from noise to clear image, but also adjusts the relationship and weight between the above-mentioned different features, and finally generates a target video that is visually real, consistent in identity and natural in action.

[0085] In a feasible implementation manner, the specific implementation manner of the feature extraction on the appearance image to obtain the modulated appearance features and the semantic features of the target human can also be:

[0086] The appearance image is subjected to semantic extraction to obtain the semantic feature, the appearance image is mapped to a latent space based on a preset encoder to obtain a latent vector of the appearance image, the appearance feature of the appearance image is extracted based on the latent vector, the latent vector is subjected to high-pass filtering and mapping in a frequency domain to obtain a modulation parameter corresponding to the appearance feature, and the appearance feature is modulated based on the modulation parameter to obtain the modulated appearance feature.

[0087] It should be noted that semantic extraction refers to a process of extracting high-level semantic information from an appearance image using a pre-trained visual language model, and the obtained semantic feature can be used to describe abstract visual content such as identity attributes and dressing styles of a person. The encoder refers to the encoding network part in the variational autoencoder (VAE), which is used to map the input appearance image from the pixel space to the low-dimensional latent space to generate a compact latent vector containing rich visual information. The latent vector refers to the representation of the appearance image in the latent space after processing by the VAE encoder. High-pass filtering is a signal processing operation that preserves high-frequency components in the latent vector while suppressing low-frequency components to highlight the detailed information of the image. The modulation parameter refers to the scaling parameter γ and the bias parameter β generated by the high-frequency latent variable, which is used for feature enhancement on the original appearance feature channel by channel.

[0088] It should also be noted that the embodiment maps the appearance image to the latent space through the VAE encoder, thereby reducing the data dimension, so that the model can more efficiently learn and process information in the latent space without directly operating in the high-dimensional pixel space. The VAE encoder also has a corresponding VAE decoder, which restores the representation in the latent space to an image or video frame through the VAE decoder, that is, after the generative model completes tasks such as denoising, replacement or reconstruction in the latent space, the decoder is finally relied on to obtain the visualized results.

[0089] The extraction of the appearance feature in the embodiment is performed by an appearance feature extraction module (ReferenceNet). The appearance feature extraction module inherits the model structure and weights of Stable-Diffusion-1.5 (Stable-Diffusion Model 1.5 version, hereinafter referred to as SD-1.5) when initialized. That is, the appearance feature extraction module of the embodiment includes:

[0090] 4 down-sampling modules, namely a zeroth down-sampling module, a first down-sampling module, a second down-sampling module and a third down-sampling module, wherein the zeroth down-sampling module, the first down-sampling module and the second down-sampling module are cross-attention down-sampling modules for two-dimensional data, the cross-attention down-sampling module includes two Transformer (a kind of self-attention mechanism deep learning model architecture) layers, and the third down-sampling module is a down-sampling module for two-dimensional data.

[0091] 1 intermediate module, wherein the intermediate module comprises 1 Transformer layer.

[0092] 4 up-sampling modules, namely a zeroth up-sampling module, a first up-sampling module, a second up-sampling module and a third up-sampling module, wherein the zeroth up-sampling module is a two-dimensional data up-sampling module, the first up-sampling module, the second up-sampling module and the third up-sampling module are cross-attention up-sampling modules of two-dimensional data, and the cross-attention up-sampling module comprises 2 Transformer layers.

[0093] The distribution of each module and the Transformer layer is as follows:

[0094] The resolution of the layer feature of the zeroth down-sampling module and the third up-sampling module is (h, w), the channel number is 320, and each comprises 2 Transformer layers; the resolution of the layer feature of the first down-sampling module and the second up-sampling module is (h / 2, w / 2), the channel number is 640, and each comprises 2 Transformer layers; the resolution of the layer feature of the second down-sampling module and the first up-sampling module is (h / 4, w / 4), the channel number is 1280, and each comprises 2 Transformer layers; the resolution of the layer feature of the third down-sampling module is (h / 8, w / 8), the channel number is 1280, and no Transformer layer is contained.

[0095] The resolution of the layer feature of the intermediate module is (h / 8, w / 8), and 1 Transformer layer is contained; the resolution of the layer feature of the zeroth up-sampling module is (h / 8, w / 8), and no Transformer layer is contained. The architecture diagram of the appearance feature extraction module of the embodiment can be referred to Figure 3 .

[0096] Further, the structures of the denoising generation module and the appearance feature extraction module in the embodiment are consistent, that is, each sampling module and the resolution contained in the denoising generation module are consistent with the appearance feature extraction module. In the denoising generation module, the Transformer layer in each sampling module and the intermediate module will additionally receive the reference feature from the appearance feature extraction module. The specific method is as follows: the input of the current layer of the denoising generation module is spliced with the appearance feature extracted by the corresponding appearance feature extraction module along the height dimension (H dimension), and then is sent to the self-attention (Self-Attention) module for fusion. In this way, the generation process not only depends on the input noise and the background, but also can be constrained by the reference feature, so as to ensure the consistency of the appearance of the person.

[0097] The specific steps of the appearance feature extraction module of the embodiment extracting the feature and injecting it into the denoising generation module are as follows:

[0098] 1. In each Transformer layer of the appearance feature extraction module, the input tensor of the current layer is A copy is generated as a reference feature input;

[0099] 2. The copied reference input is concatenated with the input of the Transformers layer corresponding to the denoising generation module Along the height dimension The formula for concatenation is:

[0100]

[0101] Wherein, is the concatenated tensor, is the concatenation function is the input of the Transformers layer corresponding to the denoising generation module, is the input tensor of the current layer of each Transformer layer of the appearance feature extraction module.

[0102] 3. The concatenated tensor is sent to the self-attention module of the denoising generation module to realize modeling of the reference feature, and the specific formula is:

[0103]

[0104] Wherein, is the tensor after self-attention.

[0105] 4. The first half of In the height dimension is taken as the output, keeping the shape consistent with the original input.

[0106] Further, since the denoising generation model only supports image restoration generation and does not have video generation capability. Therefore, the embodiment adds a Transformer layer that only performs self-attention in the time dimension after each sampling module of the denoising generation module. This layer can capture the temporal dependence between adjacent frames, thereby enabling the model to generate continuous video sequences while maintaining the original image restoration capability.

[0107] It can be understood that by adding semantic features, the model not only focuses on pixel-level details, but also understands the overall visual meaning of the characters, enhancing the consistency of the generated results at the semantic level and avoiding problems such as identity confusion or style conflict.

[0108] Further, the embodiment accurately extracts high-frequency information in the appearance image by performing frequency domain analysis and high-pass filtering in the latent space, and converts the high-frequency information into modulation parameters that can act on the feature map. Since the modulation parameters are learned by mapping the high-frequency components, the video face replacement model can more effectively retain and enhance the fine visual details such as hair, eyelashes, and skin texture, which are small but crucial, so that the final generated target video not only maintains global consistency but also has rich local detail expression ability, significantly improving the realism of the appearance after the character replacement.

[0109] In a feasible implementation, the specific implementation of high-pass filtering and mapping the latent vector in the frequency domain to obtain the modulation parameters corresponding to the appearance feature can also be:

[0110] performing Fourier transform on the latent vector to obtain a latent vector spectrum of the latent vector, moving a part with a frequency of zero in the latent vector spectrum from an original position to a spectrum center of the latent vector spectrum to obtain a target spectrum, performing high-pass filtering on the target spectrum to obtain a high-frequency spectrum, moving the part with the frequency of zero in the high-frequency spectrum to the original position, and performing inverse Fourier transform on the high-frequency spectrum after the frequency is moved to obtain a high-frequency latent vector, and determining the modulation parameters based on the high-frequency latent vector and a preset high-frequency refinement mapping method.

[0111] It should be noted that Fourier transform is a mathematical method for converting a signal from a spatial domain to a frequency domain, which is used to analyze the distribution of different frequency components in an image, and is used in this embodiment to separate high-frequency and low-frequency information from the latent vector. The latent vector spectrum refers to the representation of the latent vector in the frequency domain after Fourier transform, which contains amplitude and phase information and reflects the intensity of each frequency component in the latent variable. The part with a frequency of zero represents the most average and smooth part in the image, which usually corresponds to the overall brightness or color shift. In frequency domain processing, the zero-frequency component is usually moved from the corner of the spectrum to the center position by spectrum centering operation, so as to facilitate the design of a symmetric filter for frequency selection.

[0112] The target spectrum refers to the latent vector spectrum after spectrum centering processing, and the zero-frequency component is located at the center and the high-frequency component is diffused outward, which is suitable for structured frequency selection. High-pass filtering is a filtering operation that retains high-frequency components and suppresses low-frequency components, which is used in this embodiment to extract detail information such as edges and textures. The high-frequency spectrum refers to the result obtained by performing high-pass filtering on the target spectrum, which only retains the frequency components in the high-frequency region. Inverse Fourier transform is an operation for restoring the processed frequency domain signal to a spatial domain signal, which is used to convert the high-frequency spectrum back to the high-frequency latent vector that can be used for subsequent processing.

[0113] It can be understood that, in order to introduce the high-frequency feature information of the character, the embodiment models the high-frequency information of the character appearance image as a modulation parameter, which is an enhanced unit of the appearance feature extraction module, and is used to modulate the output of the appearance feature extraction module. By introducing the high-frequency feature information, the detail restoration ability of the character in the replaced video can be significantly improved, especially in the high-frequency areas such as facial features and clothing textures, and the realism and consistency of the generated results are enhanced.

[0114] Specifically, the embodiment first maps the character reference image to the latent space through the VAE encoder, denoted as L; then performs high-pass filtering on L in the frequency domain, and restores it to the latent space through inverse Fourier transform to obtain the high-frequency latent variable. On this basis, the method designs a high-frequency feature mapping layer (HFML) for further mapping the high-frequency latent variable into a modulation parameter for subsequent feature modulation and optimization of the output of the appearance feature extraction module.

[0115] The embodiment sets the appearance image of the target character , wherein channel 3 represents RGB (Red Green Blue), and the values of each channel are normalized to . The formula for mapping to the latent space using the variational autoencoder is:

[0116]

[0117] where C is the channel, C = 4, h = H0 / 8, and w = W0 / 8.

[0118] After obtaining the latent variable, a two-dimensional fast Fourier transform is performed on each channel of the latent variable, and the zero-frequency component is shifted to the center of the frequency spectrum through fftshift (Fast Fourier Transform Shift), and the transformation formula is:

[0119]

[0120] Then calculate its spectral amplitude (remove phase information):

[0121] where M is the calculated amplitude, and represent the real and imaginary parts, respectively.

[0122] In a feasible implementation, the specific implementation of the high-pass filtering of the target spectrum to obtain the high-frequency spectrum can also be:

[0123] Using the center of the target spectrum as the origin, the distance between each frequency in the target spectrum and the origin is determined to obtain the radius set of the target spectrum. The amplitudes of the frequencies corresponding to each radius in the radius set are accumulated and summed, and the summed amplitudes are sorted to obtain the cumulative amplitude distribution of the target spectrum. The cumulative amplitude distribution includes the cumulative amplitude of each distribution and the frequency corresponding to the cumulative amplitude. Based on the cumulative amplitude distribution and a preset amplitude threshold, the corresponding target cumulative amplitude is determined, and the frequency corresponding to the target cumulative amplitude is used as the cutoff frequency. The high-frequency frequencies in the target spectrum that are higher than the cutoff frequency are determined to obtain the high-frequency spectrum.

[0124] It should be noted that the radius set refers to the set of Euclidean distances from each frequency point to the origin in the target spectrum after spectral centering, with the spectrum center as the origin. This set is used to characterize the spatial frequency level of each frequency component. Amplitude refers to the magnitude of the amplitude at each frequency point in the target spectrum, reflecting the energy intensity of that frequency component in the latent vector. Cumulative summation refers to the summation of the amplitudes of frequency points within the same distance according to the radius, obtaining the overall energy contribution corresponding to each radius. Cumulative amplitude distribution refers to the distribution relationship formed by arranging the cumulative amplitude values ​​corresponding to different radii in ascending order, including each cumulative amplitude and its corresponding frequency, used to analyze the energy ratio of low and high frequencies. The preset amplitude threshold refers to a pre-set proportion value, representing the proportion of total energy to be retained, used to determine the cutoff frequency from the cumulative amplitude distribution. In this embodiment, the preset amplitude threshold can be 30%. The cutoff frequency refers to the frequency value in the cumulative amplitude distribution that causes the cumulative energy to reach the preset amplitude threshold, serving as the boundary between low and high frequencies.

[0125] It should also be noted that in this embodiment, the radius function r(u, v) is constructed with the spectral center (u0, v0) = (|W / 2|, |H / 2|) as the origin, and the formula is as follows:

[0126]

[0127] in,( , () represents the coordinates of the center point of the spectrum, u represents the horizontal frequency coordinate, and v represents the vertical frequency coordinate. The coordinates of the horizontal center are Here are the coordinates of the vertical center, W is the image width, and H is the image height.

[0128] Then calculate the amplitude spectrum, using the formula:

[0129]

[0130] in, For the calculated amplitude spectrum, The absolute value of the log magnitude spectrum, i.e., taking the logarithm of the magnitude spectrum to enhance the visual effect.

[0131] And on the radius set According to the spectral amplitude energy distribution, sorting and accumulation are performed. Specifically, the amplitude corresponding to each radius is accumulated and summed to obtain the cumulative energy distribution curve. According to the cumulative energy distribution, the cutoff radius corresponding to the top p=30% of low-frequency energy is selected as the cutoff frequency:

[0132]

[0133] wherein, is the calculated cutoff frequency, r is the radius, and p is a preset amplitude threshold, represents the sum of the amplitudes of all frequency points within a distance of rrfrom the center, i.e., the total amount of low-frequency energy contained in the current radius, is the sum of the amplitudes of all frequency points in the entire spectrum.

[0134] Finally, according to the cutoff frequency, a low / high frequency mask is constructed:

[0135]

[0136]

[0137] And the high-frequency spectrum is obtained accordingly:

[0138]

[0139] wherein, is the low-frequency mask, is the high-frequency mask; is an indicator function; represents element-wise multiplication, is the obtained high-frequency spectrum.

[0140] After obtaining the high-frequency spectrum , it needs to be remapped from the frequency domain back to the latent space representation. The specific method is to first move the zero-frequency component back to its original position by ifftshift, and then perform a two-dimensional inverse Fourier transform (IFFT2) on each channel and take the amplitude as the high-frequency latent variable H:

[0141]

[0142] wherein, ifftshift(·) represents moving the zero-frequency component of the spectrum back to the origin, IFFT2(·) represents a two-dimensional inverse Fourier transform, and represents taking the amplitude of a complex number.

[0143] It can be understood that the embodiment automatically identifies the demarcation point between the energy concentrated area and the detail dispersed area according to the characteristics of the frequency energy distribution of each appearance image itself, and ensures that the filtering operation can effectively remove the dominant low-frequency component and accurately retain the high-frequency part which is crucial to visual details.

[0144] Further, since the cutoff frequency is dynamically generated according to the image content, the high-frequency latent variable determined by the method of the embodiment has good robustness and universality, and can adapt to target person images of different resolutions and different texture complexities. For example, for a person image with rich facial texture, the proportion of high-frequency energy is relatively high, and the algorithm will automatically select a higher cutoff frequency to retain more details, and for an image with more smooth areas, it avoids excessive extraction of invalid noise. Therefore, the embodiment improves the quality of the high-frequency latent vector by the above-mentioned manner, so that it more accurately reflects the real detail features of the target person, and further improves the accuracy of the subsequent modulation parameter generation.

[0145] In a possible implementation, the modulation parameter includes a scaling parameter and a bias parameter, and the specific implementation of determining the modulation parameter based on the high-frequency latent vector and a preset high-frequency refinement mapping method can be:

[0146] The high-frequency latent vector is convolved through a preset first mapping convolution layer to obtain a first target latent vector with a first target shape, wherein the first target shape includes a first target channel number. The first target latent vector is convolved through a preset second mapping convolution layer to obtain a second target latent vector with a second target shape, wherein the second target shape includes a second target channel number, and the second target channel number is twice the first target channel number. The second target latent vector is split based on the first target channel number to obtain the scaling parameter with the first target channel number and the bias parameter with the second target channel number.

[0147] It should be noted that the first mapping convolution layer refers to a group of lightweight convolutional neural network layers, which are used to convert the high-frequency latent vector from the original shape to the first target shape with a specified channel number and spatial size, and realize preliminary feature transformation. The first target latent vector refers to the intermediate feature representation obtained after the first mapping convolution layer processing, and its vector shape (channel number, height, and width) matches the input required by the subsequent module. The first target channel number refers to the number of feature channels contained in the first target latent vector, which is pre-set by the model structure and is consistent with the channel number output by the corresponding layer in the appearance feature extraction module in the embodiment.

[0148] The second mapping convolutional layer acts on the first target latent vector to further map it into a second target latent vector with a higher number of channels for generating joint modulation parameters. The second target latent vector refers to a feature tensor output by the second mapping convolutional layer, which has twice the number of channels of the first target number of channels, facilitating subsequent splitting into paired scaling and bias parameters.

[0149] It should also be noted that in the present embodiment, each HFML (High-Frequency Mapping Layer) first completes channel expansion and resolution alignment through Conv-A (2D convolutional layer) in one time to map the input high-frequency latent variable to a target shape Cb, Hb, Wb, where Cb e {320, 640, 1280}, (Hb, Wb) e {h, w, h / 2, w / 2, h / 4, w / 4, h / 8, w / 8}. Subsequently, Conv-B (2D convolutional layer) further generates a 2Cb channel output and splits it into a pair of scaling and bias parameters in the channel dimension . The Conv-A output is followed by SiLU activation, while the Conv-B output is not nonlinear and is initialized to zero to ensure that the modulation effect is close to zero (γ ≈ 0, β ≈ 0) during the initial training phase.

[0150] In the present embodiment, each sampling module in the appearance feature extraction module has a corresponding independent HFML module, and the inputs of all HFMLs are the same high-frequency latent variable. The HFML extracts detailed information related to the appearance of the person from the high-frequency latent variable and outputs a set of modulation parameters . These modulation parameters will act on the output of the sampling module in the corresponding appearance feature extraction module through a preset modulation formula, thereby further optimizing and enhancing the feature injection process of the appearance feature extraction module to make it carry high-frequency information of the appearance features of the person. The modulation formula is:

[0151]

[0152] wherein is the input tensor of each current layer of the appearance feature extraction module, is the scaling parameter, is the bias parameter. The overall process of feature modulation in the present embodiment can be referred to in Figure 4 , and the details of modulation for different sampling modules of the appearance feature extraction module can be referred to in Figure 5 . The specific setting parameters of each HFML module can be seen in the following table:

[0153]

[0154] Down0 to Down2 in the table are a zeroth downsampling module to a second downsampling module, Mid is an intermediate module, Up0 to Up2 are a zeroth upsampling module to a second upsampling module, in_dim is an input dimension, out_dim is an output dimension, KernelSize is a convolution kernel size, and stride is a step length.

[0155] It can be understood that the embodiment eliminates the dimension difference between the high-frequency latent vector and the target feature space through the first mapping convolution layer to preliminarily extract features and adapt the shape of the high-frequency latent vector, and further expands the high-frequency latent vector into a joint representation with double channel numbers through the second mapping convolution layer, thereby providing a data basis for simultaneously generating two types of modulation parameters.

[0156] Further, the embodiment splits the second target latent vector into two parts with equal channel numbers, respectively as scaling parameters and bias parameters, so that the two modulation functions are optimized in the same learning process, which can independently control the gain and offset of the features, and share the context expression of the high-frequency information, thereby enhancing the consistency and coordination of the modulation process. Since the two parameters are generated by the high-frequency latent vector, the high-frequency information such as edges and textures carried by the high-frequency latent vector is effectively converted into fine-tuning ability of the feature map, thereby significantly enhancing the detail restoration degree of key areas such as human faces, hair, and clothes without destroying the overall structure.

[0157] In summary, the embodiment obtains an original video and an appearance image of a target person that need to be replaced, inputs the original video and the appearance image into a preset video face replacement model, replaces a person in the original video with the target person, wherein the video face replacement model includes a high-frequency refinement module, the high-frequency refinement module extracts a high-frequency latent variable of the appearance image, modulates appearance features of the appearance image based on the high-frequency latent variable to obtain modulated appearance features after high-frequency modulation, and appearance of the target person in the target video is determined based on the modulated appearance features.

[0158] In view of the problem that the existing method is difficult to capture high-frequency detail features such as facial features and clothing textures, so that the generated result loses a lot of high-frequency details, thereby resulting in low realism of the generated face replacement video, the embodiment modulates appearance features of the appearance image based on a high-frequency latent variable. Since the appearance of the target person in the target video is determined based on the modulated appearance features, and the appearance features are obtained based on the high-frequency latent variable modulation, the embodiment can make the model better learn the high-frequency details in the high-frequency latent variable by modulating the appearance features, thereby better depicting the high-frequency details of the person in the output target video, and improving the realism of the generated face replacement video.

[0159] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar content as the above embodiment one can be introduced externally, and the subsequent will not be described. On this basis, please refer to Figure 6 , the step of training the preset to-be-trained model based on the preset Gaussian noise, the person mask, the person skeleton graph, the video background, the modulated appearance feature, the semantic feature and the appearance image further comprises steps S100-S300:

[0160] Step S100, based on the preset action convolution module and the person skeleton graph, the action feature of the target person is extracted;

[0161] It should be noted that the action convolution module refers to a lightweight neural network structure, which is usually composed of multiple convolution layers and normalization layers, and is used to encode the input person skeleton graph into a high-dimensional action feature vector to capture the spatial structure and dynamic changes of human posture. The person skeleton graph refers to the human key point connection graph extracted from the original video through pose estimation technology, which represents the limb position and action posture of the person. The action feature refers to the vector representation extracted from the person skeleton graph by the action convolution module, which contains the geometric structure and motion information of the person's action, and is used to constrain the action of the target person to be consistent with the original video during the generation process.

[0162] It can be understood that the embodiment further extracts the action feature of the target person based on the preset action convolution module and the previously extracted person skeleton graph. Specifically, the embodiment inputs the person skeleton graph into the action convolution module, which abstracts the spatial structure in the skeleton graph layer by layer and extracts features through a series of convolution operations. The convolution kernel slides on the skeleton graph, captures the relative position relationship between the key points and the limb connection mode, and enhances the expression ability of the model through a nonlinear activation function. After multiple convolution processing, a high-dimensional feature tensor with a dimension matching the generation model is output, which is the action feature of the target person. The action feature can effectively represent the posture information of the person in the current frame and serve as an action constraint condition in the subsequent training process to guide the to-be-trained model to generate a replacement person that is synchronized with the action of the original video.

[0163] Step S200, based on the Gaussian noise, the person mask, the video background, the modulated appearance feature, the semantic feature and the action feature, an initial person-replaced video is generated through the preset to-be-trained model;

[0164] It should be noted that the action control module of the embodiment extracts motion control features through convolution structure. The output motion control features are consistent with the first down-sampling module (Down0 Block) of the denoising generation module in terms of resolution and channel number. Then, the motion control features and the input of the first down-sampling module of the denoising generation module are fused together through residual addition, so that the motion information can directly participate in the feature modeling of the diffusion process. In order to ensure the stability of the output, zero initialization is used for output convolution, which ensures that the original feature distribution of the denoising generation module will not be destroyed in the early stage of training, and at the same time, effective pose control signals are gradually learned during the training process.

[0165] It should also be noted that the semantic features in the embodiment will be input as additional features before the tensor is input to the appearance feature extraction module and the Transformer layer of the denoising generation module, thereby providing semantic constraints.

[0166] It can be understood that since the input explicitly includes the action features extracted by the action convolution module, the model can perceive and follow the body movement trajectory of the character in the original video, thereby actively maintaining the pose consistency during the generation process. At the same time, the introduction of the modulated appearance features and semantic features makes the generation result not only close to the reference image in appearance, but also consistent in high-level semantics, enhancing the model's understanding ability of the overall image of the character. The joint input of the character mask and the video background ensures that the generation process only acts on the target area, effectively preventing background modification and ensuring scene stability. In addition, the diffusion mechanism starting from Gaussian noise gives the model strong generation ability, enabling it to gradually restore the video content with reasonable structure and rich details under complex conditions.

[0167] Step S300, based on the character mask, the appearance image and the initial face swapping video, adjusting the parameters of the to-be-trained model to obtain the video face swapping model.

[0168] It can be understood that since the character mask clearly defines the replacement area to be concerned, the loss calculation can be focused on the target character itself. In the embodiment, the appearance image is taken as a reference benchmark based on the character mask, and compared with the corresponding area in the initial face swapping video, so that the model can perceive the edge difference between the current generation result and the target identity, and gradually reduce the gap through gradient back propagation, thereby improving the naturalness of the transition between the character and the background.

[0169] In a feasible implementation manner, the specific implementation manner of adjusting the parameters of the to-be-trained model based on the character mask, the appearance image and the initial face swapping video to obtain the video face swapping model can also be:

[0170] Based on a preset dilation operation, increase the pixel points of the person mask to obtain an expansion region, based on a preset erosion operation, reduce the pixel points of the person mask to obtain a contraction region, calculate the difference between the expansion region and the contraction region to obtain a ring belt region, calculate the similarity loss of the appearance image and the initial person video in the ring belt region, adjust the parameters of the to-be-trained model based on the similarity loss to obtain the video person model.

[0171] It should be noted that the dilation operation is a morphological image processing operation, which expands the boundary pixels of the person mask to increase its coverage, obtains a larger region than the original mask, called expansion region, which is used to capture the adjacent background outside the person edge. The erosion operation is another morphological operation, which shrinks the boundary pixels of the person mask to reduce its coverage, obtains a smaller region than the original mask, called contraction region, which is used to retain the internal region of the person body. The expansion region is the expanded version of the person mask after the dilation operation, which contains the original person region and the peripheral boundary belt of a certain width. The contraction region is the reduced version of the person mask after the erosion operation, which only retains the core region of the person and excludes the fuzzy border area. The ring belt region is a ring-shaped region obtained by set difference operation of the expansion region and the contraction region, which is located near the boundary of the person mask, forming a narrow belt-shaped region, which is used to focus on the fusion quality at the junction of the person and the background. The similarity loss refers to the calculation of the structural similarity difference between the initial person video and the appearance image in the ring belt region, which is commonly measured by SSIM (Structural Similarity Index) and other indicators, which is used to evaluate the natural degree of boundary transition.

[0172] It should also be noted that the specific implementation of obtaining the ring belt region and calculating the structural similarity in the above embodiment includes:

[0173] Let the person mask be M, dilate the mask by morphological operation and erode to obtain the expansion and contraction regions, and the set difference forms the ring belt region:

[0174]

[0175] In the ring belt region B, the structural similarity loss (SSIM Loss) is introduced to force the generated result to keep the structure consistent with the reference data in the boundary transition zone. The constraint loss function is defined as:

[0176]

[0177] wherein, represents pixel-by-pixel multiplication, is the generated result after one-step denoising, is the target reference image.

[0178] It can be understood that in the video character replacement task based on the diffusion model, the boundary region of the character foreground and the background tends to have artifacts, resulting in lack of naturalness and temporal stability in the visual perception of the video. Therefore, the embodiment introduces the mask strip loss function in the training process on the basis of the prediction noise loss of the video face changing model , and the model is forced to keep the generated result consistent with the structure of the target reference image in the boundary region.

[0179] Specifically, when the character is in a motion state or the background is continuously changing, the traditional method tends to produce unstable contour jitter at the edge, and by introducing a structural similarity constraint in the ring band region, the model generates texture and brightness distribution in this region that is closer to the real image, thereby significantly improving the boundary fusion effect. Thus, when performing video face changing, not only can the naturalness of the transition between the character and the background in the spatial dimension be improved, but also the bright edges, flickering and other artifact phenomena caused by the difference between consecutive frames in the time dimension can be reduced.

[0180] And compared to the way of applying uniform constraints to the entire image, the loss function of the embodiment only acts on the mask ring band region, avoiding excessive restrictions on the details inside the main body and the overall texture of the background, while ensuring the freedom of the generated model, it accurately provides structural guidance to the most error-prone boundary part. That is, the embodiment does not constrain the entire image, but only constrains the mask ring band region, thereby avoiding affecting the generation of the main character or background details, while effectively improving the naturalness of the transition between the character and the background, and reducing the bright edges, flickering and other artifact phenomena caused by the difference between consecutive frames in the time dimension, and improving the naturalness and temporal stability of the generated face-changing video.

[0181] In summary, the embodiment extracts the motion features of the target character based on the preset motion convolution module and the character skeleton map, generates an initial face-changing video based on the Gaussian noise, the character mask, the video background, the modulated appearance features, the semantic features and the motion features through a preset to-be-trained model, adjusts the parameters of the to-be-trained model based on the character mask, the appearance image and the initial face-changing video, and obtains the video face changing model.

[0182] In view of the problem that in the current video character replacement task, the boundary region of the character foreground and the background is prone to artifacts, resulting in lack of naturalness and temporal stability in the visual perception of the video, the embodiment forms a ring belt region through morphological dilation and erosion operations, and additionally calculates a structural similarity loss in the ring belt region of the generated video and appearance image, so that the model is forced to keep the generated result consistent with the structure of the target reference image in the boundary region, improving the naturalness of the transition between the character and the background, and reducing the bright edges, flickering and other artifact phenomena caused by the difference between consecutive frames in the time dimension. Moreover, the embodiment does not constrain the entire picture, but only constrains the mask ring belt region, thereby avoiding affecting the generation of the main character or the background details, and improving the naturalness and temporal stability of the generated character replacement video.

[0183] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the video character replacement method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0184] The present application provides a video character replacement method device, which comprises at least one processor and a memory in communication connection with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the video character replacement method in Embodiment One.

[0185] Reference will now be made to Figure 7 which shows a structural schematic diagram of a video character replacement method device suitable for implementing the embodiments of the present application. The video character replacement method device in the embodiments of the present application can include, but is not limited to, mobile terminals such as mobile phones, tablet computers, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant: Personal Digital Assistant), PMPs (Portable Media Player: Portable Multimedia Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, as well as fixed terminals such as digital TVs, desktop computers, and the like. Figure 7 The video character replacement method device shown is only an example and should not limit the functions and use range of the embodiments of the present application in any way.

[0186] As Figure 7As shown, the video face swapping method device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the operation of the video face swapping method device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the video face swapping method device to communicate with other devices wirelessly or by wire to exchange data. Although the video face swapping method device with various systems is shown in the figure, it should be understood that all the systems shown are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0187] In particular, according to the embodiments disclosed in the present application, the processes described in the above appearance flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product including a computer program carried on a computer readable medium, the computer program containing program codes for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are performed.

[0188] The video face swapping method device provided in the present application adopts the video face swapping method in the above-mentioned embodiments, and can solve the technical problem that the generated face swapping video has low realism. Compared with the prior art, the video face swapping method device provided in the present application has the same beneficial effects as the video face swapping method provided in the above-mentioned embodiments, and other technical features in the video face swapping method device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0189] It should be understood that portions of the application disclosed can be implemented in hardware, software, firmware, or combinations thereof. In the description of the embodiments above, specific features, structures, materials or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0190] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any changes and modifications that can be made to the application in light of the teachings described herein are contemplated in the broad scope of the application. Accordingly, the scope of the application should be determined not with reference to the above description but with reference to the claims appended hereto.

[0191] The application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e., a computer program) for performing the video face swapping method in the above embodiments.

[0192] The computer readable storage medium provided by the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), etc., or any suitable combination of the above.

[0193] The computer readable storage medium described above can be included in the video face swapping method device; or can exist separately and not be assembled into the video face swapping method device.

[0194] The computer readable storage medium described above carries one or more programs, which, when executed by the video face swapping method device, cause the video face swapping method device to perform the video face swapping method described above.

[0195] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0196] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0197] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0198] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the video replacement method described above, and can solve the technical problem of low realism of the generated replacement video. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the video replacement method provided by the above-mentioned embodiments, which will not be described here.

[0199] The user-related data involved in the present application is obtained after obtaining the permission or consent of the user, with reference to Figure 8 That is, when the present application is applied to a specific product or technology, the user's permission is required to realize the acquisition and processing of relevant data, and the processing of relevant data needs to comply with relevant laws, regulations and supervision standards of relevant countries and regions.

[0200] The above is only some embodiments of the present application, and does not limit the protection scope of the present application. Any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A video-based method for replacing actors, characterized in that, The method includes: Obtain the original video that needs to be replaced and the appearance image of the target person; The original video and the appearance image are input into a preset video person-replacing model, and the original video person in the original video is replaced with the target person to obtain the target video. The video person-replacing model extracts the high-frequency latent variables of the appearance image, and modulates the appearance features of the appearance image based on the high-frequency latent variables to obtain high-frequency modulated appearance features. The appearance of the target person in the target video is determined based on the modulated appearance features. The step of inputting the original video and the appearance image into a preset video person-replacing model, replacing the original video character with the target character, and obtaining the target video, further includes: The characters and background of the original video are extracted to obtain the character mask, the character skeleton diagram, and the video background of the original video. Feature extraction is performed on the appearance image to obtain the modulated appearance features and the semantic features of the target person; Based on the preset Gaussian noise, the person mask, the person skeleton diagram, the video background, the modulation appearance features, the semantic features, and the appearance image, the preset training model is trained to obtain the video person replacement model.

2. The method as described in claim 1, characterized in that, The step of extracting features from the appearance image to obtain the modulated appearance features and the semantic features of the target person includes: Semantic extraction is performed on the appearance image to obtain the semantic features; Based on a preset encoder, the appearance image is mapped to the latent space to obtain the latent vector of the appearance image; Based on the latent vector, the appearance features of the appearance image are extracted; The latent vector is high-pass filtered and mapped in the frequency domain to obtain the modulation parameters corresponding to the appearance feature; Based on the modulation parameters, the appearance feature is modulated to obtain the modulated appearance feature.

3. The method as described in claim 2, characterized in that, The step of performing high-pass filtering and mapping on the latent vector in the frequency domain to obtain the modulation parameters corresponding to the appearance feature includes: Perform a Fourier transform on the latent vector to obtain the latent vector spectrum; The portion of the latent vector spectrum with zero frequency is moved from its original position to the center of the latent vector spectrum to obtain the target spectrum; The target spectrum is high-pass filtered to obtain the high-frequency spectrum; The portion of the high-frequency spectrum with a frequency of zero is moved to the original position, and an inverse Fourier transform is performed on the high-frequency spectrum after the frequency shift to obtain the high-frequency latent variable. The modulation parameters are determined based on the high-frequency latent variables and the preset high-frequency refinement mapping method.

4. The method as described in claim 3, characterized in that, The modulation parameters include scaling parameters and bias parameters. The step of determining the modulation parameters based on the high-frequency latent variable and a preset high-frequency refinement mapping method includes: The high-frequency latent variables are convolved by a preset first mapping convolution layer to obtain a first target latent vector with a vector shape of a first target shape, wherein the first target shape includes a first target number of channels; The first target latent vector is convolved by a preset second mapping convolution layer to obtain a second target latent vector with a vector shape that is the shape of the second target. The second target shape includes a second target channel number, which is twice the number of the first target channels. Based on the first target number of channels, the second target latent vector is split to obtain the scaling parameter with the number of channels equal to the first target number of channels, and the bias parameter with the number of channels equal to the second target number of channels.

5. The method as described in claim 3, characterized in that, The step of performing high-pass filtering on the target spectrum to obtain the high-frequency spectrum includes: Using the center of the target spectrum as the origin, determine the distance between each frequency in the target spectrum and the origin to obtain the set of radii of the target spectrum; The amplitude of the frequency corresponding to each radius in the radius set is accumulated and summed, and the summed amplitudes are sorted to obtain the cumulative amplitude distribution of the target spectrum, wherein the cumulative amplitude distribution includes the cumulative amplitude of each distribution and the frequency corresponding to the cumulative amplitude; Based on the cumulative amplitude distribution and the preset amplitude threshold, a corresponding target cumulative amplitude is determined, and the frequency corresponding to the target cumulative amplitude is used as the cutoff frequency. The high-frequency spectrum is obtained by identifying the high-frequency frequencies above the cutoff frequency in the target spectrum.

6. The method as described in claim 1, characterized in that, The step of training a preset training model based on preset Gaussian noise, the person mask, the person skeleton diagram, the video background, the modulated appearance features, the semantic features, and the appearance image to obtain the video person replacement model includes: Based on the preset action convolution module and the character skeleton diagram, the action features of the target character are extracted; Based on the Gaussian noise, the person mask, the video background, the modulation appearance features, the semantic features, and the action features, an initial person-replacement video is generated using a preset training model. Based on the character mask, the appearance image, and the initial replacement video, the parameters of the model to be trained are adjusted to obtain the video replacement model.

7. The method as described in claim 6, characterized in that, The step of adjusting the parameters of the model to be trained based on the person mask, the appearance image, and the initial person-replacement video to obtain the video person-replacement model includes: Based on a preset dilation operation, the number of pixels in the character mask is increased to obtain the expanded area; Based on a preset erosion operation, the number of pixels in the character mask is reduced to obtain a shrunken area; Calculate the difference between the expanded region and the contracted region to obtain the annular region; Calculate the similarity loss between the appearance image and the initial person-swapping video in the annular region; Based on the similarity loss, the parameters of the model to be trained are adjusted to obtain the video human replacement model.

8. A video-based human replacement device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the video replacement method as described in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the video replacement method as described in any one of claims 1 to 7.

Images