Face identity exchange method, system and device

Abstract

The present application relates to the field of artificial intelligence and computer vision technology, and particularly relates to a face identity exchange method, system, device and storage medium, and specifically as follows: inputting a source face image and a target face image into a preset face identity exchange model to perform face identity exchange, the model comprising a latent space encoder, an identity feature extractor, a Transformer mapping network, a latent space identity injection module, a distillation diffusion refining module and a reconstruction decoder, after feature extraction, knowledge distillation learning and decoding, an exchanged face image is obtained, face identity exchange is realized, and the model is optimized and trained through calculation of a loss function. The present application can present better effects in identity preservation, expression restoration and visual fidelity during face identity exchange by introducing a distillation diffusion and attention fusion mechanism.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to a method, system and device for exchanging facial identities. Background Technology

[0002] Existing face identity swapping methods are mainly based on two paradigms: Generative Adversarial Networks (GANs) and Diffusion Models. Traditional GAN-based methods achieve identity transfer and expression preservation by fusing the source and target faces at the pixel or feature level. However, these methods are prone to pattern collapse and artifact generation during training, and their performance is unstable under complex poses, lighting changes, or occlusion conditions. Diffusion models, as a new generation generative framework in recent years, have been widely adopted in face editing and face swapping tasks due to their significant advantages in image quality and controllability caused by their progressive denoising generation mechanism.

[0003] However, existing diffusion-based face swapping methods still have three shortcomings: First, the identity feature injection mechanism is crude, resulting in insufficient consistency of identity in key areas such as eyes, nose, and lips in the generated results; second, attributes and identity features are strongly coupled in the latent space, making it difficult to effectively separate expressions from identity, thus affecting the naturalness of the target person and the preservation of expressions; third, the multi-step reasoning process of the diffusion model leads to low reasoning efficiency, which cannot meet the needs of real-time application scenarios.

[0004] Therefore, this invention proposes a method, system, and device for exchanging facial identities to solve the above problems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention develops a method, system, and device for face identity exchange. By introducing a distillation diffusion and attention fusion mechanism, this invention can achieve better results in terms of identity preservation, expression restoration, and visual realism during the face identity exchange process.

[0006] The technical solution of this invention to solve the technical problem is a method for face identity exchange, comprising the following steps:

[0007] The source face image and the target face image are input into a preset face identity exchange model for face identity exchange. The model includes a latent space encoder, an identity feature extractor, a Transformer mapping network, a latent space identity injection module, a distillation diffusion refinement module, and a reconstruction decoder.

[0008] The latent space encoder processes the target face image to obtain the target latent space features, and the identity feature extractor processes the source face image to obtain the source identity vector.

[0009] The source identity vector is input into the Transformer mapping network to obtain a local identity token sequence. Then, the local identity token sequence is fused with the target latent space features and input into the latent space identity injection module to obtain fused latent space features.

[0010] The distillation-diffusion refinement module adopts a teacher-student dual-network structure. After fusing latent space features and adding random noise, it is input into the teacher diffusion network to perform denoising operations, and outputs noise-free latent space features. The student network learns the knowledge of the teacher diffusion network through knowledge distillation, and then inputs the fused latent space features to obtain refined latent space features. The refined latent space features are input into the reconstruction decoder to output swapped face images.

[0011] Calculate the loss function of the model, and iterate the training and parameter optimization of the model.

[0012] The latent space encoder is built upon a pre-trained VAE encoder and includes an input convolutional layer, double residual blocks, multi-level downsampling layers, and a latent variable mapping layer. The target is a human face image. The target latent space features are output after being processed sequentially by the latent space encoder. The specific operations in the latent space encoder are as follows:

[0013] (1) The input convolutional layer includes two-dimensional convolution operations, batch normalization layers, and the SiLU activation function, and the target face image. After being fed into the input convolutional layer, the data undergoes various processing operations in sequence, outputting a feature map containing low-level texture information of the target face. ;

[0014] (2) The dual residual block consists of two cascaded residual units. Each residual unit contains two 3×3 convolutional layers, a batch normalization layer, a SiLU activation function, and a SiLU activation function. Skip connections are set within each residual unit. In the first residual unit, the input features are... First, after a first 3×3 convolution, batch normalization, and SiLU activation function, features are extracted using a second 3×3 convolution. Then, the output of the second 3×3 convolution is compared with the input features. Element-wise addition; in the second residual unit, the output of the first residual unit is passed sequentially through the first 3×3 convolutional layer, the batch normalization layer, the SiLU activation function, and the second 3×3 convolutional layer. Then, the output of the first residual unit is added element-wise to the output of the second 3×3 convolutional layer in the second residual unit to output the enhanced feature. ;

[0015] (3) The multi-level downsampling layer consists of four downsampling stages stacked sequentially. Each stage consists of a convolutional downsampling layer and a max pooling layer to enhance features. Through four levels of downsampling, the number of channels increases sequentially while the spatial size is halved each time, resulting in high semantic features. ;

[0016] (4) The latent variable mapping layer includes a 1×1 convolutional layer, a transposed convolutional layer, and a normalization layer, with high semantic features. After being input into the latent variable mapping layer, the target latent space features are output through operations at each layer in sequence. .

[0017] The identity feature extractor is built on a pre-trained face recognition network, ArcFace, and includes an input convolutional layer, four bottleneck residual blocks, a global average pooling layer, and a fully connected layer, with the source face image as the input. The identity feature extractor processes the data sequentially through each operation, outputting an identity embedding vector. The specific steps are as follows:

[0018] (1) The input convolutional layer includes a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function. The source face image After being fed into the input convolutional layer, the data undergoes various operations in sequence, resulting in the output of the initial feature map. ;

[0019] (2) Each of the four bottleneck residual blocks consists of a 1×1 convolutional layer, a 3×3 convolutional layer, a 1×1 convolutional layer, and a batch normalization layer. Skip connections are set within the block to add the input features to the output of the second 1×1 convolution element by element; initial feature map After passing through four bottleneck residual blocks, the final output is the identity semantic feature. ;

[0020] (3) Incorporate identity semantic features The input is fed into a global average pooling layer for global feature aggregation, resulting in a compact global feature description vector. Then the global feature description vector The input is fed into a fully connected layer for feature mapping, and the output is an identity embedding vector. .

[0021] The Transformer mapping network consists of a linear mapping layer, a learnable query vector set, and a four-layer stacked Transformer decoder that embeds identity into the vector. The input is fed into a Transformer mapping network and transformed into a structured sequence of local identity tokens. The specific steps are as follows:

[0022] (1) The linear mapping layer includes a fully connected layer, a normalized layer, and a SiLU activation function, with identity embedding vectors. After being input into the linear mapping layer, the data undergoes various operations in sequence, and the basic identity embedding vector u is output.

[0023] (2) The learnable query vector set consists of several learnable query vectors, each of which corresponds to a semantic region of the face. A total of K query vectors are introduced to form the initial query matrix. , , Indicates the first One query vector;

[0024] (3) Integrate the basic identity embedding vector u with the initial query matrix The input is fed into a four-layer stacked Transformer decoder. Each Transformer decoder layer includes a multi-head self-attention layer, a multi-head cross-attention layer, a feedforward network, and a normalization and residual connection structure.

[0025] 1) Basic identity embedding vector u and initial query matrix First, they are all input to the first-layer Transformer decoder;

[0026] Multi-head self-attention layer with initial query matrix As input, K attention heads are used to calculate the correlation between different query vectors, and the updated query matrix is ​​output. ;

[0027] The multi-head cross-attention layer uses the basic identity embedding vector u as the key and value to update the query matrix. As a query, the attention weight matrix is ​​calculated and the fused features are generated. ;

[0028] The fused features The input is fed into a feedforward network, which consists of two linear mapping layers, a SiLU activation function, residual connections, and a normalization layer. The fused features... First, the feature is passed through a first linear mapping layer, then a nonlinear response is introduced through the SiLU activation function to obtain the activated feature. The activated feature is then passed through a second linear mapping layer to obtain the compressed feature. Then, the fused features are combined using residual connections. and compression features The results are added pixel by pixel, and then normalized by a normalization layer to output the first layer result. ;

[0029] 2) The second-layer decoder uses Using the basic identity embedding vector u as input, repeat step 1) above to output... ;

[0030] 3) The third-layer decoder uses Using the basic identity embedding vector u as input, repeat step 1) above to output... ;

[0031] 4) The fourth layer decoder uses Using the basic identity embedding vector u as input, repeat step 1) above to output... This forms eight identity sub-vectors, ultimately outputting an identity token sequence. .

[0032] The latent space identity injection module includes transform convolutional layers, cross-modal multi-head attention layers, feedforward networks, and upsampled residual blocks to inject target latent space features. and identity token sequence Input latent space identity injection module to obtain fused latent space features The specific steps are as follows:

[0033] (1) The transformation convolutional layer includes a 1×1 convolutional layer, a batch normalization layer, and a SiLU activation function. First, the target latent space features are transformed. The input is fed into a transform convolutional layer to obtain the mapped features. ;

[0034] (2) Mapping features With identity token sequence The input is fed into a cross-modal multi-head attention layer, which is used to... For keys and values, to map features For querying, a multi-head attention mechanism is used to calculate the correlation weights between identity features and latent space features, and the fused features are output. ;

[0035] (3) Integrating features The input is fed into a feedforward network, which consists of two linear mapping layers, a SiLU activation function, and a normalization layer. The first linear mapping layer performs feature upscaling, and the SiLU activation function introduces non-linearity. The second linear layer then reduces the dimensionality to restore the original number of channels. The output is then combined with the fused features. By adding the residual connections and then performing a normalization layer operation, we obtain semantically stable and uniformly distributed reconstructed features. ;

[0036] (4) Reconstruct features The input is fed into an upsampled residual block, which consists of a transposed convolutional layer, two 3×3 convolutional layers, a normalization layer, and a SiLU activation function to reconstruct features. First, spatial upsampling is performed using a transposed convolutional layer, followed by a 3×3 convolutional layer, a normalization layer, a SiLU activation function, and another 3×3 convolutional layer. Finally, residual connections are used to reconstruct the features. The latent space features are obtained by adding them element-wise to the output of the 3×3 convolutional layer. .

[0037] The distillation-diffusion refinement module adopts a teacher-student dual-network structure, including a teacher diffusion network and a student network. The specific operation is as follows:

[0038] (1) The teacher diffusion network adopts a multi-step diffusion structure based on U-Net, including an encoder part, a bottleneck layer and a decoder part, and inputs the latent space features after fusion. According to preset noise scheduling parameters Input latent space features Gaussian noise injection is performed to generate noisy latent space samples, which are then input into a teacher diffusion network for denoising, generating denoised latent space features. T-step iterative denoising operations are then performed, using the denoised latent space features generated in the previous step as input for the next step. Finally, after T-step iterative operations, noise-free latent space features are generated. That is, the output of the last iteration. , = ;

[0039] Each noise reduction step is detailed below:

[0040] The first Denoising latent space samples The input is fed into the encoder, which passes through four encoder layers in sequence, and outputs high semantic features. High semantic features The input is then fed into a bottleneck layer, which consists of two residual blocks and a multi-head self-attention layer. The fused features are then output from the bottleneck layer. ; Fusion features The input is fed into the decoder, which consists of four sequentially stacked decoding layers. Each decoding layer includes a transposed convolutional layer, a normalization layer, and a SiLU activation function. These layers are then fused with the corresponding layer features from the encoder via skip connections, ultimately outputting predicted noise. The teacher diffusion network performs a denoising update process based on the predicted noise. The denoising calculation formula is as follows:

[0041] ,

[0042] in, Indicates the first The denoising latent space characteristics of the step. Indicates the first Step-by-step preset noise scheduling parameters, Indicates the first Predictive noise of the step, Indicates the first The denoising latent space characteristics of the step;

[0043] (2) The student network adopts the same U-Net structure as the teacher network, and the un-noised latent space features are displayed. The input is fed into the student network, which directly learns the denoising mapping relationship formed by the teacher network in the multi-step diffusion process through single-step forward inference. After knowledge distillation training, the student network learns a high-fidelity denoising and reconstruction capability equivalent to that of the teacher network. The student network finally outputs the refined latent space features. .

[0044] The reconstruction decoder employs a symmetrical deconvolution structure, including an input projection layer, multi-level upsampling modules, and an output convolutional layer. It inputs refined latent space features into the reconstruction decoder and outputs swapped face images. The specific operation is as follows:

[0045] The refined latent space features The input is fed into the input projection layer, which consists of a 1×1 convolutional layer and a layer normalization layer (LayerNorm). The output is the projection feature. ;

[0046] Aligned features The input is fed into a multi-level upsampling module, which consists of three upsampling layers. Each layer includes a transposed convolutional layer, a batch normalization layer, and a SiLU activation function. Each upsampling layer doubles the size of the feature map space and gradually restores the global contour and local texture details of the face during the upsampling process, outputting a high-resolution feature map. ;

[0047] High-resolution feature mapping The input is fed into the output convolutional layer, which consists of a 3×3 convolutional layer and a Tanh activation function, ultimately generating the swapped face image. .

[0048] The loss function is calculated as follows:

[0049] The total loss function includes identity loss, latent space consistency constraint loss, and distillation loss;

[0050] The identity embedding vectors of the source face image and the swapped face image are extracted using a face recognition model, and the cosine similarity between the identity embedding vectors is calculated as the identity loss. ;

[0051] Teacher diffusion network output noiseless latent space characteristics Student's network output of refined latent space characteristics The consistency of their distributions in the characteristic space is calculated using the mean square error, thus yielding the distillation loss. ;

[0052] Extract the latent space features of the target before identity injection. Features of the merged latent space By constraining structural differences through pixel-by-pixel mean square error, the latent space consistency constraint loss is calculated. ;

[0053] Total loss function for:

[0054] ,

[0055] in, , , These represent the weighting coefficients for each type of loss.

[0056] The present invention also provides a face identity exchange system, which performs one of the face identity exchange methods described above, including:

[0057] The feature extraction module includes a latent space encoder and an identity feature extractor. The latent space encoder processes the target face image to obtain the target latent space features, and the identity feature extractor processes the source face image to output the identity embedding vector.

[0058] The feature fusion and transformation module includes a Transformer mapping network and a latent space identity injection module. The Transformer mapping network converts the identity embedding vector into a local identity token sequence. The latent space identity injection module fuses the target latent space features and the identity token sequence to obtain fused latent space features.

[0059] The refinement and reconstruction module includes a distillation-diffusion refinement module and a reconstruction decoder. The distillation-diffusion refinement module includes a teacher diffusion network and a student network. The student network learns the denoising mapping relationship of the teacher network through knowledge distillation and outputs refined latent space features. The reconstruction decoder reconstructs the refined latent space features into swapped face images.

[0060] The present invention also provides an electronic device,

[0061] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a face identity exchange method.

[0062] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. The above technical solutions have the following advantages or beneficial effects:

[0063] This invention proposes a high-fidelity face identity swapping method based on cascaded latent space fusion. The method first extracts the latent space feature representation of the target face and the identity vector of the source face through a latent space encoder and an identity feature extractor, respectively. A lightweight Transformer identity mapping network is then used to generate a structured sequence of identity tokens, achieving regionalized modeling of identity information. Subsequently, a latent space identity injection module, based on a cross-modal multi-head attention mechanism, fuses the source identity tokens with the target latent space features, achieving accurate injection of identity semantics and attribute preservation. A distillation-diffusion refinement module further refines the fused latent space features for high-fidelity reconstruction, effectively improving the detail realism and illumination consistency of the generated image. Finally, a reconstruction decoder decodes the refined latent space features to generate swapped face images, ensuring identity consistency and visual naturalness. This invention introduces a distillation-diffusion and attention fusion mechanism, which significantly outperforms existing methods in terms of identity preservation, expression restoration, and visual realism, while providing higher-quality data support and a technical foundation for face forgery detection and identity transfer research. Attached Figure Description

[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0065] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0066] Figure 2 This is a schematic diagram of the operation process in the model.

[0067] Figure 3 This is a comparison diagram of the face identity swapping effects of the method of the present invention and existing methods. Detailed Implementation

[0068] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0069] Example 1

[0070] like Figure 1 As shown, a method for exchanging facial identities includes the following steps:

[0071] The source face image and the target face image are input into a preset face identity exchange model for face identity exchange. The model includes a latent space encoder, an identity feature extractor, a Transformer mapping network, a latent space identity injection module, a distillation diffusion refinement module, and a reconstruction decoder.

[0072] like Figure 2 As shown, the specific operations in the face identity swapping model are as follows:

[0073] The latent space encoder processes the target face image to obtain the target latent space features, and the identity feature extractor processes the source face image to obtain the source identity vector.

[0074] The source identity vector is input into the Transformer mapping network to obtain a local identity token sequence. Then, the local identity token sequence is fused with the target latent space features and input into the latent space identity injection module to obtain fused latent space features.

[0075] The distillation-diffusion refinement module adopts a teacher-student dual-network structure. After fusing latent space features and adding random noise, it is input into the teacher diffusion network to perform denoising operations, and outputs noise-free latent space features. The student network learns the knowledge of the teacher diffusion network through knowledge distillation, and then inputs the fused latent space features to obtain refined latent space features. The refined latent space features are input into the reconstruction decoder to output swapped face images.

[0076] Calculate the loss function of the model, and iterate the training and parameter optimization of the model.

[0077] In a specific implementation, the latent space encoder is constructed based on a pre-trained VAE encoder, including an input convolutional layer, dual residual blocks, multi-level downsampling layers, and a latent variable mapping layer, targeting a human face image. The target latent space features are output after being processed sequentially by the latent space encoder. The specific operations in the latent space encoder are as follows:

[0078] (1) The input convolutional layer includes a two-dimensional convolution operation, a batch normalization layer, and a SiLU activation function. The kernel size of the two-dimensional convolution operation is 7×7, the stride is 2, and the padding is 3. The target face image After being fed into the input convolutional layer, the data undergoes batch normalization to stabilize the numerical distribution, and then the SiLU activation function is used to enhance the non-linear expressive power, outputting a feature map containing low-level texture information of the target face. ;

[0079] (2) The double residual block consists of two cascaded residual units. The feature representation capability and semantic level are gradually enhanced through the two residual units. Each residual unit contains two 3×3 convolutional layers, a batch normalization layer, a SiLU activation function, and a SiLU activation function. Skip connections are set in each residual unit. In the first residual unit, the input features are... First, after a first 3×3 convolution, batch normalization, and SiLU activation function, features are extracted using a second 3×3 convolution. Then, the output of the second 3×3 convolution is compared with the input features. Element-wise addition; in the second residual unit, the output of the first residual unit is passed sequentially through the first 3×3 convolutional layer, the batch normalization layer, the SiLU activation function, and the second 3×3 convolutional layer. Then, the output of the first residual unit is added element-wise to the output of the second 3×3 convolutional layer in the second residual unit to ensure feature consistency and output enhanced features. ;

[0080] (3) The multi-level downsampling layer consists of four downsampling stages stacked sequentially. Each stage consists of a convolutional downsampling layer and a max pooling layer to enhance features. Semantic compression is performed sequentially through four downsampling stages, with the number of channels increasing and the spatial size gradually halved in each stage, resulting in high semantic features. ;

[0081] (4) The latent variable mapping layer includes a 1×1 convolutional layer, a transposed convolutional layer, and a normalization layer, with high semantic features. After being input into the latent variable mapping layer, the latent space representation is transformed through operations in each layer in sequence, and the target latent space features are output. .

[0082] In a specific implementation, the identity feature extractor is built based on the pre-trained face recognition network ArcFace, including an input convolutional layer, four bottleneck residual blocks, a global average pooling layer, and a fully connected layer, with the source face image as the input. The identity feature extractor processes the data sequentially through each operation, outputting an identity embedding vector. The specific steps are as follows:

[0083] (1) The input convolutional layer includes a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function, used to extract low-level texture and local structural features, and the source face image. After being fed into the input convolutional layer, the data undergoes various operations in sequence, resulting in the output of the initial feature map. ;

[0084] (2) Each of the four bottleneck residual blocks consists of a 1×1 convolutional layer, a 3×3 convolutional layer, a 1×1 convolutional layer, and a batch normalization layer. Skip connections are set within the block to add the input features to the output of the second 1×1 convolution element by element to maintain feature consistency and prevent gradient vanishing; initial feature map After passing through four bottleneck residual blocks, the final output is the identity semantic feature. ;

[0085] (3) Incorporate identity semantic features The input is fed into a global average pooling layer for global feature aggregation, resulting in a compact global feature description vector. Then the global feature description vector The input is fed into a fully connected layer for feature mapping, and the output is an identity embedding vector. This serves as a global representation of the identity features of the source face.

[0086] In a specific implementation, the Transformer mapping network includes a linear mapping layer, a learnable query vector set, and a four-layer stacked Transformer decoder, embedding identity into the vector. The input is fed into a Transformer mapping network and transformed into a structured sequence of local identity tokens. The specific steps are as follows:

[0087] (1) The linear mapping layer includes a fully connected layer, a normalized layer, and a SiLU activation function, with identity embedding vectors. After being input into the linear mapping layer, the data undergoes various operations in sequence, and the basic identity embedding vector u is output.

[0088] (2) The learnable query vector set consists of several learnable query vectors, each of which corresponds to a semantic region of the face (e.g., forehead, temporal region, orbital region, nasal region, lip region, chin region, cheek region, and zygomatic region). A total of 8 query vectors are introduced to form the initial query matrix. , , Indicates the first There are 8 query vectors, where K=8 and K represents the number of query vectors.

[0089] (3) Integrate the basic identity embedding vector u with the initial query matrix The input is fed into a four-layer stacked Transformer decoder. Each Transformer decoder layer includes a multi-head self-attention layer, a multi-head cross-attention layer, a feedforward network, and a normalization and residual connection structure.

[0090] 1) Basic identity embedding vector u and initial query matrix First, they are all input to the first-layer Transformer decoder;

[0091] Multi-head self-attention layer with initial query matrix As input, eight attention heads are used to calculate the correlation between different query vectors, and the updated query matrix is ​​output. ;

[0092] The multi-head cross-attention layer uses the basic identity embedding vector u as the key and value to update the query matrix. As a query, the attention weight matrix is ​​calculated and the fused features are generated. ;

[0093] The fused features The input is fed into a feedforward network, which consists of two linear mapping layers, a SiLU activation function, residual connections, and a normalization layer. The fused features... First, the feature is passed through a first linear mapping layer, then a nonlinear response is introduced through the SiLU activation function to obtain the activated feature. The activated feature is then passed through a second linear mapping layer to obtain the compressed feature. Then, the fused features are combined using residual connections. and compression features The results are added pixel by pixel, and then normalized by a normalization layer to output the first layer result. ;

[0094] 2) The second-layer decoder uses Using the basic identity embedding vector u as input, repeat step 1) above to output... ;

[0095] 3) The third-layer decoder uses Using the basic identity embedding vector u as input, repeat step 1) above to output... ;

[0096] 4) The fourth layer decoder uses Using the basic identity embedding vector u as input, repeat step 1) above to output... This forms eight identity sub-vectors, ultimately outputting an identity token sequence. .

[0097] In a specific implementation, the latent space identity injection module includes a transform convolutional layer, a cross-modal multi-head attention layer, a feedforward network, and an upsampled residual block, which injects the target latent space features. and identity token sequence Input latent space identity injection module to obtain fused latent space features The specific steps are as follows:

[0098] (1) The transformation convolutional layer includes a 1×1 convolutional layer, a batch normalization layer, and a SiLU activation function. First, the target latent space features are transformed. The input is fed into a transform convolutional layer to obtain the mapped features. ;

[0099] (2) Mapping features With identity token sequence The input is fed into a cross-modal multi-head attention layer, which is used to... For keys and values, to map features For querying, a multi-head attention mechanism is used to calculate the correlation weights between identity features and latent space features, and the fused features are output. ;

[0100] (3) Integrating features The input is fed into a feedforward network, which consists of two linear mapping layers, a SiLU activation function, and a normalization layer. The first linear mapping layer performs feature upscaling, and the SiLU activation function introduces non-linearity. The second linear layer then reduces the dimensionality to restore the original number of channels. The output is then combined with the fused features. By adding the residual connections and then performing a normalization layer operation, we obtain semantically stable and uniformly distributed reconstructed features. ;

[0101] (4) Reconstruct features The input is fed into an upsampled residual block, which consists of a transposed convolutional layer, two 3×3 convolutional layers, a normalization layer, and a SiLU activation function to reconstruct features. First, spatial upsampling is performed using a transposed convolutional layer, followed by a 3×3 convolutional layer, a normalization layer, a SiLU activation function, and another 3×3 convolutional layer. Finally, residual connections are used to reconstruct the features. The latent space features are obtained by adding them element-wise to the output of the 3×3 convolutional layer. .

[0102] In a specific implementation, the distillation-diffusion refinement module adopts a teacher-student dual-network structure and collaboratively trains through a knowledge distillation mechanism to achieve high-fidelity reconstruction of latent space features. The distillation-diffusion refinement module includes a teacher diffusion network and a student network. The teacher network is responsible for performing multi-step diffusion denoising to generate high-quality reference features, while the student network learns the denoising mapping of the teacher network to compress the multi-step diffusion process into a single-step fast inference. The specific operation is as follows:

[0103] (1) The teacher diffusion network adopts a multi-step diffusion structure based on U-Net, including an encoder part, a bottleneck layer and a decoder part, and inputs the latent space features after fusion. According to preset noise scheduling parameters Input latent space features Gaussian noise injection is performed to generate noisy latent space samples, which are then input into a teacher diffusion network for denoising, generating denoised latent space features. T-step iterative denoising operations are then performed, using the denoised latent space features generated in the previous step as input for the next step. Finally, after T-step iterative operations, noise-free latent space features are generated. That is, the output of the last iteration. , = ;

[0104] Each noise reduction step is detailed below:

[0105] The first Denoising latent space samples The input is fed into the encoder, which passes through four encoder layers in sequence, and outputs high semantic features. High semantic features The input is then fed into a bottleneck layer, which consists of two residual blocks and a multi-head self-attention layer. The fused features are then output from the bottleneck layer. ; Fusion features The input is fed into the decoder, which consists of four sequentially stacked decoding layers. Each decoding layer includes a transposed convolutional layer (with a stride of 2), a normalization layer, and a SiLU activation function. These layers are then fused with the corresponding layer features from the encoder via skip connections, ultimately outputting predicted noise. The teacher diffusion network performs a denoising update process based on the predicted noise. The denoising calculation formula is as follows:

[0106] ,

[0107] in, Indicates the first The denoising latent space characteristics of the step. Indicates the first Step-by-step preset noise scheduling parameters, Indicates the first Predictive noise of the step, Indicates the first The denoising latent space characteristics of the step;

[0108] (2) The student network adopts the same U-Net structure as the teacher network, and the un-noised latent space features are displayed. The input is fed into the student network, which directly learns the denoising mapping relationship formed by the teacher network in the multi-step diffusion process through single-step forward inference. After knowledge distillation training, the student network learns a high-fidelity denoising and reconstruction capability equivalent to that of the teacher network. The student network finally outputs the refined latent space features. .

[0109] In a specific implementation, the reconstruction decoder adopts a symmetrical deconvolution structure, including an input projection layer, a multi-level upsampling module, and an output convolutional layer. The refined latent space features are input to the reconstruction decoder, and the swapped face image is output. The specific operation is as follows:

[0110] The refined latent space features The input is fed into the input projection layer, which consists of a 1×1 convolutional layer and a layer normalization layer (LayerNorm). The output is the projection feature. ;

[0111] Aligned features The input is fed into a multi-level upsampling module, which consists of three upsampling layers. Each layer includes a transposed convolutional layer, a batch normalization layer, and a SiLU activation function. Each upsampling layer doubles the size of the feature map space and gradually restores the global contour and local texture details of the face during the upsampling process, outputting a high-resolution feature map. ;

[0112] High-resolution feature mapping The input is fed into the output convolutional layer, which consists of a 3×3 convolutional layer and a Tanh activation function, ultimately generating the swapped face image. .

[0113] In a specific implementation, the loss function is calculated as follows:

[0114] The total loss function includes identity loss, latent space consistency constraint loss, and distillation loss;

[0115] The identity embedding vectors of the source face image and the swapped face image are extracted using a face recognition model, and the cosine similarity between the identity embedding vectors is calculated as the identity loss. ;

[0116] Teacher diffusion network output noiseless latent space characteristics Student's network output of refined latent space characteristics The consistency of their distributions in the characteristic space is calculated using the mean square error, thus yielding the distillation loss. ;

[0117] Extract the latent space features of the target before identity injection. Features of the merged latent space By constraining structural differences through pixel-by-pixel mean square error, the latent space consistency constraint loss is calculated. ;

[0118] Total loss function for:

[0119] ,

[0120] in, , , These represent the weighting coefficients for each loss, which are 1, 5, and 0.2, respectively.

[0121] Example 2

[0122] A facial identity exchange system, comprising executing a facial identity exchange method, including:

[0123] The feature extraction module includes a latent space encoder and an identity feature extractor. The latent space encoder processes the target face image to obtain the target latent space features, and the identity feature extractor processes the source face image to output the identity embedding vector.

[0124] The feature fusion and transformation module includes a Transformer mapping network and a latent space identity injection module. The Transformer mapping network converts the identity embedding vector into a local identity token sequence. The latent space identity injection module fuses the target latent space features and the identity token sequence to obtain fused latent space features.

[0125] The refinement and reconstruction module includes a distillation-diffusion refinement module and a reconstruction decoder. The distillation-diffusion refinement module includes a teacher diffusion network and a student network. The student network learns the denoising mapping relationship of the teacher network through knowledge distillation and outputs refined latent space features. The reconstruction decoder reconstructs the refined latent space features into swapped face images.

[0126] Example 3

[0127] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a face identity exchange method.

[0128] Example 4

[0129] A series of qualitative and quantitative experiments were conducted on the CelebA-HQ dataset, a high-quality face dataset that is an extension and improvement upon the CelebA dataset. The CelebA dataset was originally released in 2014 by the Computer Graphics Laboratory at Carnegie Mellon University and contains over 200,000 facial images of celebrities (mainly famous figures), primarily used for research in computer vision and image processing.

[0130] The performance was evaluated using five key metrics: identity retrieval accuracy (ID), face shape error (Shape), expression error (Exp), head pose error (Pose), and Fréchet Inception Distance (FID). Among them, Fréchet Inception Distance is an indicator for evaluating the quality of the generated image. It measures the generation effect by comparing the distribution difference between the generated image and the real image in the feature space. The lower the FID value, the higher the quality of the generated image and the closer it is to the real image.

[0131] Existing methods include HifiFace face-swapping model, FaceShifter face-swapping model, SimSwap face-swapping model, UniFace face-swapping model, DiffSwap face-swapping model, and LatentSwap face-swapping model;

[0132] The comparison results between the method of the present invention and the existing methods are shown in Table 1, where ↑ indicates that the larger the data, the better, and ↓ indicates that the smaller the data, the better.

[0133] Table 1. Performance comparison results of the method of the present invention and existing methods

[0134]

[0135] As can be seen from the results in Table 1, the present invention significantly outperforms existing methods in all indicators. Its ID accuracy is 98.23%, indicating the most accurate identity preservation. At the same time, the errors in Shape, Exp, and Pose are all the lowest, which fully demonstrates that the model has stronger fidelity in maintaining facial structure, expression, and pose. In addition, the FID value is only 7.85, which is much lower than other models, indicating that the generated results have reached the optimal level in terms of visual quality and realism, fully verifying the comprehensive advantages of the model in identity consistency and visual naturalness.

[0136] like Figure 2 As shown, different face-swapping methods exhibit significant differences in their performance on face identity swapping tasks. For instance, while achieving identity transfer, the SimSwap and HifiFace face-swapping models are prone to insufficient fusion of identity features, blurred details, and artifacts. Although the InfoSwap and FaceDancer face-swapping models have improved image quality and detail fidelity, they still suffer from insufficient consistency in local features such as eyes, nose, and lips.

[0137] In comparison, the method of the present invention can more accurately inject the identity features of the source face while maintaining the naturalness of the target face's pose and expression. The generated results are superior to existing methods in terms of identity consistency, texture clarity, and overall realism.

[0138] Although the specific embodiments of the invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the invention. Based on the technical solutions of the invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the invention.

Claims

1. A method for exchanging facial identities, characterized in that, Includes the following steps: The source face image and the target face image are input into a preset face identity exchange model for face identity exchange. The model includes a latent space encoder, an identity feature extractor, a Transformer mapping network, a latent space identity injection module, a distillation diffusion refinement module, and a reconstruction decoder. The latent space encoder processes the target face image to obtain the target latent space features, and the identity feature extractor processes the source face image to obtain the source identity vector. The source identity vector is input into the Transformer mapping network to obtain a local identity token sequence. Then, the local identity token sequence is fused with the target latent space features and input into the latent space identity injection module to obtain fused latent space features. The distillation-diffusion refinement module adopts a teacher-student dual network structure. After fusing latent space features and adding random noise, it is input into the teacher diffusion network to perform denoising operations. After denoising, it outputs noise-free latent space features. After learning the knowledge of the teacher diffusion network through knowledge distillation, the student network inputs the fused latent space features to obtain refined latent space features; the refined latent space features are then input into the reconstruction decoder to output swapped face images. Calculate the loss function of the model, and iterate the training and parameter optimization of the model.

2. The face identity exchange method according to claim 1, characterized in that, The latent space encoder is built upon a pre-trained VAE encoder and includes an input convolutional layer, double residual blocks, multi-level downsampling layers, and a latent variable mapping layer. The target is a human face image. The target latent space features are output after being processed sequentially by the latent space encoder. The specific operations in the latent space encoder are as follows: (1) The input convolutional layer includes two-dimensional convolution operations, batch normalization layers, and the SiLU activation function, and the target face image. After being fed into the input convolutional layer, the data undergoes various processing operations in sequence, outputting a feature map containing low-level texture information of the target face. ; (2) The dual residual block consists of two cascaded residual units. Each residual unit contains two 3×3 convolutional layers, a batch normalization layer, a SiLU activation function, and a SiLU activation function. Skip connections are set within each residual unit. In the first residual unit, the input features are... First, after a first 3×3 convolution, batch normalization, and SiLU activation function, features are extracted using a second 3×3 convolution. Then, the output of the second 3×3 convolution is compared with the input features. Element-wise addition; in the second residual unit, the output of the first residual unit is passed sequentially through the first 3×3 convolutional layer, the batch normalization layer, the SiLU activation function, and the second 3×3 convolutional layer. Then, the output of the first residual unit is added element-wise to the output of the second 3×3 convolutional layer in the second residual unit to output the enhanced feature. ; (3) The multi-level downsampling layer consists of four downsampling stages stacked sequentially. Each stage consists of a convolutional downsampling layer and a max pooling layer to enhance features. Through four levels of downsampling, the number of channels increases sequentially while the spatial size is halved each time, resulting in high semantic features. ; (4) The latent variable mapping layer includes a 1×1 convolutional layer, a transposed convolutional layer, and a normalization layer, with high semantic features. After being input into the latent variable mapping layer, the target latent space features are output through operations at each layer in sequence. .

3. The facial identity exchange method according to claim 2, characterized in that, The identity feature extractor is built on a pre-trained face recognition network, ArcFace, and includes an input convolutional layer, four bottleneck residual blocks, a global average pooling layer, and a fully connected layer, with the source face image as the input. The identity feature extractor processes the data sequentially through each operation, outputting an identity embedding vector. The specific steps are as follows: (1) The input convolutional layer includes a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function. The source face image After being fed into the input convolutional layer, the data undergoes various processing operations in sequence, resulting in the output of the initial feature map. ; (2) Each of the four bottleneck residual blocks consists of a 1×1 convolutional layer, a 3×3 convolutional layer, a 1×1 convolutional layer, and a batch normalization layer. Skip connections are set within the block to add the input features to the output of the second 1×1 convolution element by element; initial feature map After passing through four bottleneck residual blocks, the final output is the identity semantic feature. ; (3) Incorporate identity semantic features The input is fed into a global average pooling layer for global feature aggregation, resulting in a compact global feature description vector. Then the global feature description vector The input is fed into a fully connected layer for feature mapping, and the output is an identity embedding vector. .

4. The face identity exchange method according to claim 3, characterized in that, The Transformer mapping network consists of a linear mapping layer, a learnable query vector set, and a four-layer stacked Transformer decoder that embeds identity into the vector. The input is fed into a Transformer mapping network and transformed into a structured sequence of local identity tokens. The specific steps are as follows: (1) The linear mapping layer includes a fully connected layer, a normalized layer, and a SiLU activation function, with the identity embedding vector... After being input into the linear mapping layer, the data undergoes various operations in sequence, and the basic identity embedding vector u is output. (2) The learnable query vector set consists of several learnable query vectors, each of which corresponds to a semantic region of the face. A total of K query vectors are introduced to form the initial query matrix. , , Indicates the first One query vector; (3) Integrate the basic identity embedding vector u with the initial query matrix The input is fed into a four-layer stacked Transformer decoder. Each Transformer decoder layer includes a multi-head self-attention layer, a multi-head cross-attention layer, a feedforward network, and a normalization and residual connection structure. 1) Basic identity embedding vector u and initial query matrix First, they are all input to the first-layer Transformer decoder; Multi-head self-attention layer with initial query matrix As input, K attention heads are used to calculate the correlation between different query vectors, and the updated query matrix is ​​output. ; The multi-head cross-attention layer uses the basic identity embedding vector u as the key and value to update the query matrix. As a query, the attention weight matrix is ​​calculated and the fused features are generated. ; The fused features The input is fed into a feedforward network, which consists of two linear mapping layers, a SiLU activation function, residual connections, and a normalization layer. The fused features... First, the feature is passed through a first linear mapping layer, then a nonlinear response is introduced through the SiLU activation function to obtain the activated feature. The activated feature is then passed through a second linear mapping layer to obtain the compressed feature. Then, the fused features are combined using residual connections. and compression features The results are added pixel by pixel, and then normalized by a normalization layer to output the first layer result. ; 2) The second-layer decoder uses Using the basic identity embedding vector u as input, repeat step 1) above to output... ; 3) The third-layer decoder uses Using the basic identity embedding vector u as input, repeat step 1) above to output... ; 4) The fourth layer decoder uses Using the basic identity embedding vector u as input, repeat step 1) above to output... This forms eight identity sub-vectors, ultimately outputting an identity token sequence. .

5. The face identity exchange method according to claim 4, characterized in that, The latent space identity injection module includes transform convolutional layers, cross-modal multi-head attention layers, feedforward networks, and upsampled residual blocks to inject target latent space features. and identity token sequence Input latent space identity injection module to obtain fused latent space features The specific steps are as follows: (1) The transformation convolutional layer includes a 1×1 convolutional layer, a batch normalization layer, and a SiLU activation function. First, the latent space features of the target are transformed. The input is fed into a transform convolutional layer to obtain the mapped features. ; (2) Mapping features With identity token sequence The input is fed into a cross-modal multi-head attention layer, which is used to... For keys and values, to map features For querying, a multi-head attention mechanism is used to calculate the correlation weights between identity features and latent space features, and the fused features are output. ; (3) Incorporate fusion features The input is fed into a feedforward network, which consists of two linear mapping layers, a SiLU activation function, and a normalization layer. The first linear mapping layer performs feature upscaling, and the SiLU activation function introduces non-linearity. The second linear layer then reduces the dimensionality to restore the original number of channels. The output is then combined with the fused features. By adding the residual connections and then performing a normalization layer operation, we obtain semantically stable and uniformly distributed reconstructed features. ; (4) Reconstruct features The input is fed into an upsampled residual block, which consists of a transposed convolutional layer, two 3×3 convolutional layers, a normalization layer, and a SiLU activation function to reconstruct features. First, spatial upsampling is performed using a transposed convolutional layer, followed by a 3×3 convolutional layer, a normalization layer, a SiLU activation function, and another 3×3 convolutional layer. Finally, residual connections are used to reconstruct the features. The latent space features are obtained by adding them element-wise to the output of the 3×3 convolutional layer. .

6. A face identity exchange method according to claim 5, characterized in that, The distillation-diffusion refinement module adopts a teacher-student dual-network structure, including a teacher diffusion network and a student network. The specific operation is as follows: (1) The teacher diffusion network adopts a multi-step diffusion structure based on U-Net, including an encoder part, a bottleneck layer and a decoder part, and inputs the latent space features after fusion. According to preset noise scheduling parameters Input latent space features Gaussian noise injection is performed to generate noisy latent space samples, which are then input into a teacher diffusion network for denoising, generating denoised latent space features. T-step iterative denoising operations are then performed, using the denoised latent space features generated in the previous step as input for the next step. Finally, after T-step iterative operations, noise-free latent space features are generated. That is, the output of the last iteration. , = ; Each noise reduction step is detailed below: The first Step-by-step denoising latent space samples The input is fed into the encoder, which passes through four encoder layers in sequence, and outputs high semantic features. High semantic features The input is then fed into a bottleneck layer, which consists of two residual blocks and a multi-head self-attention layer. The fused features are then output from the bottleneck layer. ; Fusion features The input is fed into the decoder, which consists of four sequentially stacked decoding layers. Each decoding layer includes a transposed convolutional layer, a normalization layer, and a SiLU activation function. These layers are then fused with the corresponding layer features from the encoder via skip connections, ultimately outputting predicted noise. The teacher diffusion network performs a denoising update process based on the predicted noise. The denoising calculation formula is as follows: ， in, Indicates the first The denoising latent space characteristics of the step. Indicates the first Step-by-step preset noise scheduling parameters, Indicates the first Predictive noise of the step, Indicates the first The denoising latent space characteristics of the step; (2) The student network adopts the same U-Net structure as the teacher network, and the un-noised latent space features are displayed. The input is fed into the student network, which directly learns the denoising mapping relationship formed by the teacher network in the multi-step diffusion process through single-step forward inference. After knowledge distillation training, the student network learns a high-fidelity denoising and reconstruction capability equivalent to that of the teacher network. The student network finally outputs the refined latent space features. .

7. The facial identity exchange method according to claim 1, characterized in that, The reconstruction decoder employs a symmetrical deconvolution structure, including an input projection layer, multi-level upsampling modules, and an output convolutional layer. It inputs refined latent space features into the reconstruction decoder and outputs swapped face images. The specific operation is as follows: The refined latent space features The input is fed into the input projection layer, which consists of a 1×1 convolutional layer and a layer normalization layer (LayerNorm). The output is the projection feature. ; Aligned features The input is fed into a multi-level upsampling module, which consists of three upsampling layers. Each layer includes a transposed convolutional layer, a batch normalization layer, and a SiLU activation function. Each upsampling layer doubles the size of the feature map space and gradually restores the global contour and local texture details of the face during the upsampling process, outputting a high-resolution feature map. ; High-resolution feature mapping The input is fed into the output convolutional layer, which consists of a 3×3 convolutional layer and a Tanh activation function, ultimately generating the swapped face image. .

8. The face identity exchange method according to claim 7, characterized in that, The loss function is calculated as follows: The total loss function includes identity loss, latent space consistency constraint loss, and distillation loss; The identity embedding vectors of the source face image and the swapped face image are extracted using a face recognition model, and the cosine similarity between the identity embedding vectors is calculated as the identity loss. ; Teacher diffusion network output noiseless latent space characteristics Student's network output of refined latent space characteristics The consistency of their distributions in the characteristic space is calculated using the mean square error, thus yielding the distillation loss. ; Extract the latent space features of the target before identity injection. Features of the merged latent space By constraining structural differences through pixel-by-pixel mean square error, the latent space consistency constraint loss is calculated. ; Total loss function for: ， in, , , These represent the weighting coefficients for each type of loss.

9. A facial identity exchange system, executing a facial identity exchange method as described in any one of claims 1 to 8, characterized in that, include: The feature extraction module includes a latent space encoder and an identity feature extractor. The latent space encoder processes the target face image to obtain the target latent space features, and the identity feature extractor processes the source face image to output the identity embedding vector. The feature fusion and transformation module includes a Transformer mapping network and a latent space identity injection module. The Transformer mapping network converts the identity embedding vector into a local identity token sequence. The latent space identity injection module fuses the target latent space features and the identity token sequence to obtain fused latent space features. The refinement and reconstruction module includes a distillation-diffusion refinement module and a reconstruction decoder. The distillation-diffusion refinement module includes a teacher diffusion network and a student network. The student network learns the denoising mapping relationship of the teacher network through knowledge distillation and outputs refined latent space features. The reconstruction decoder reconstructs the refined latent space features into swapped face images.

10. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a face identity exchange method as described in any one of claims 1 to 8.

Images