Face identity exchange method based on multi-scale feature transfer
Through the face identity exchange method of multi-scale feature transfer, the problem of poor retention of identity details such as skin color and face shape in the existing technology is solved, the high-fidelity face exchange effect is achieved, and the development of forgery detection technology is promoted.
Patent Information
- Application Number
- CN202511052933.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-30
- Publication Date
- 2025-08-29
- Estimated Expiration
- 2045-07-30
AI Technical Summary
The existing face exchange technology is not effective in handling the retention of identity details in complex postures and expressions, especially in retaining key identity details such as skin color and face shape.
The face identity exchange method based on multi-scale feature transfer is adopted, and the multi-scale features of the source face and the target face are extracted through a unified multi-scale encoder, and the identity information is extracted using the identity mapping network, and the identity feature transfer is carried out through cross-attention fusion and multi-scale feature transfer modules, and finally the exchange face is obtained through image generation.
It realizes the precise transfer of identity information such as face shape and skin color. The generated exchanged faces are highly fidelity, improves the authenticity and detail retention ability of generated images, and provides more diverse training and testing data to promote the development of forgery detection technology.
Smart Images

Figure CN120563308A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image-level deep fake technology, and specifically relates to a face identity exchange method based on multi-scale feature transfer. Background Art
[0002] Face swapping is a key technology in computer vision and image processing. It aims to naturally transfer key identity features from a source face to a target face while preserving the target face's attributes (such as head pose, expression, and background). With the recent development of deep learning, this technology has made significant progress and is widely used in film and television production, virtual reality, privacy protection, and other fields. The development of face swapping technology has undergone a significant leap from traditional image processing to deep learning. Early methods primarily relied on 3D morphable models (3DMMs) and feature point matching for facial alignment and texture transfer. However, due to the limited expressiveness of handcrafted features, they struggled to preserve identity details in complex poses and expressions. With the introduction of generative adversarial networks (GANs), some deep learning-based face swapping methods have significantly improved the realism of generated images. However, these methods still face numerous challenges in accurately transferring identity information, particularly in preserving key identity details such as skin color and face shape. Summary of the Invention
[0003] To achieve the above-mentioned purpose, the present invention is implemented through the following technical solutions: The present invention provides a face identity exchange method based on multi-scale feature transfer, comprising the following steps: S1. Get the source face image and target face image ; S2. Source face image and target face image Input into a unified multi-scale encoder to obtain multi-scale source face features and multi-scale target face features; S3 establishes an identity mapping network ID Mapping, the multi-scale source face features are extracted through the identity mapping network identity information to obtain a multi-scale source face identity vector; S4. Establish a cross attention fusion module CAFM, the first source face identity vector in the multi-scale source face identity vector and the first target face feature in the multi-scale target face feature After fusion through the cross attention fusion module, the preliminary fusion features are obtained ; S5. Establish a multi-scale feature transfer decoder MFTD, the multi-scale source face identity vector and the multi-scale target face feature are input into the multi-scale feature transfer decoder MFTD, and the multi-scale fusion feature is obtained through the feature transfer module FTB; the multi-scale fusion feature and the preliminary fusion feature are combined. After upsampling residual block URB processing, the output features are obtained ; The output characteristics After the convolution operation, the exchanged face image is obtained ; S6. Iteratively optimize the above process through the loss function.
[0004] Furthermore, the unified multi-scale encoder is constructed based on the pre-trained ResNet-50 model, including a convolutional layer, a first residual block, a second residual block, a third residual block and a fourth residual block; after the source face image is preliminarily processed by the convolutional layer, it passes through the first residual block, the second residual block, the third residual block and the fourth residual block in sequence to obtain the first source face feature respectively. , second source facial features , third-source facial features , the fourth source of facial features Similarly, the target face image is input into a unified multi-scale encoder for processing to obtain the first target face feature , second target facial features , the third target facial features , the fourth target facial feature .
[0005] Furthermore, the identity mapping network ID Mapping includes a channel attention module, an adaptive average pooling layer, a batch normalization layer, and a LeakyReLU activation function. The channel attention module includes a multi-scale pooling module and a fully connected network; the multi-scale pooling module includes a global average pooling layer and a global maximum pooling layer; the fully connected network includes a fully connected layer, a ReLU activation function, and a Sigmoid function: The first source facial features The overall distribution characteristics are obtained through the multi-scale pooling module and local salient features ; The overall distribution characteristics and local salient features Splicing to obtain fusion features The fusion feature Generate channel weight W through the fully connected network and combine the channel weight W with the first source face feature Multiply channel by channel to get the feature map of enhanced identity ; The feature map of the enhanced identity The first source face identity vector is obtained after the adaptive average pooling layer, batch normalization layer and LeakyReLU activation function ; Similarly, the second source facial features , third-source facial features , the fourth source of facial features Input into the identity mapping network ID Mapping, through the first source face feature The same process is used to obtain the second source face identity vector , the third source face identity vector and the fourth source face identity vector .
[0006] Furthermore, the first source face identity vector and the first target facial features Input into the cross attention fusion module, the first source face identity vector As a query vector , the first target facial feature As a key vector Sum vector ; By calculating the query vector With key vector The similarity between them is used to obtain the query vector For the key vector The attention weight ; The obtained attention weight Applied to a vector of values The initial fusion features are obtained by weighted fusion .
[0007] Furthermore, the multi-scale feature transfer decoder MFTD includes four feature transfer upsampling residual blocks and a convolutional layer.
[0008] Furthermore, the four feature transfer upsampling residual blocks adopt a unified structure; each feature transfer upsampling residual block adopts a serial cascade architecture, which is composed of a feature transfer module FTB and an upsampling residual block URB in sequence; the feature transfer module FTB performs identity fusion operation, and the upsampling residual block URB performs feature space resolution recovery and residual connection enhancement.
[0009] Furthermore, in the feature transfer module FTB, the multi-scale source face identity vector and the multi-scale target face features are processed to obtain the multi-scale feature transfer preliminary fusion feature , the formula is as follows: , in, Represents the i-th target face feature; represents the identity vector of the i-th source face; ; Represents element-wise multiplication operation; Represents element-by-element addition operation; the multi-scale feature transfer initially fuses features After processing by the global fusion network Global Mapping network and the local fusion network Local Mapping network, the global fusion features are obtained respectively. and local fusion features ; The global fusion feature and local fusion features Adaptive fusion is performed through the gating mechanism to obtain multi-scale fusion features ; The multi-scale fusion feature Including the first fusion feature , the second fusion feature , the third fusion feature , the fourth fusion feature ; The global fusion network Global Mapping network includes an adaptive average pooling layer, three 1×1 convolutional layers, a batch normalization layer, a LeakyReLU activation function, and a residual connection; the local fusion network Local Mapping network includes three 3×3 convolutional layers, an instance normalization layer, a LeakyReLU activation function, and a residual connection; The global fusion feature and local fusion features Adaptive fusion is performed through a gating mechanism. The formula for this process is as follows: , in, Represents a dynamic attention mask; the dynamic attention mask is a global fusion feature and local fusion features After addition, it is processed by Sigmoid function.
[0010] Furthermore, the upsampling residual block (URB) adopts a unified structure. Each upsampling residual block (URB) includes an upsampling layer, a 3×3 convolution layer, an instance normalization layer, a LeakyReLU activation function, a 1×1 convolution layer, and a residual connection. The preliminary fusion features and the first fusion feature Input into the first upsampled residual block URB, and perform feature addition to obtain the first feature map , the first feature map After the upsampling layer, 3×3 convolution layer, instance normalization layer, LeakyReLU activation function dimension expansion and spatial information extraction, the processing result is obtained; the residual connection is used to connect the processing result with the first fusion feature map Fusion is performed to retain the original information while enhancing the model's expressiveness; information extraction and channel adjustment are performed on the fusion result through a 1×1 convolutional layer to obtain the first upsampled residual block URB output feature ; The first up-sampled residual block URB and the output first fusion feature The feature transfer module FTB is connected in series; The first block upsamples the residual block URB output features and the second fusion feature Input into the second upsampled residual block URB to obtain the output features of the second upsampled residual block URB ; The second block upsamples the residual block URB and outputs the second fusion feature Similarly, repeat the above operation to obtain the fourth up-sampled residual block URB output feature , as the final output feature ; The fourth up-sampled residual block URB and the output fourth fusion feature The feature transfer modules FTB are connected in series.
[0011] Furthermore, step S6 specifically includes: The loss function includes identity loss , reconstruction losses , fight against losses , Perceptual Loss And expression coefficient loss : The face recognition model is used to extract the identity embedding vectors of the source face image and the exchanged face image, and the cosine similarity between the identity embedding vectors is calculated as the identity loss. ; According to the target face image and the exchanged face image, the square error of each element is summed and the average is taken to obtain the reconstruction loss ; Use a multi-scale discriminator to distinguish high-resolution images and obtain adversarial loss , the formula is as follows: , Where D represents the multi-scale discriminator; Represents the source face image The probability distribution to be obeyed is the expectation; Represents the source face image and swap face images The joint probability distribution takes the expectation; In order to improve the visual effect of the generated image at the pixel level, the pre-trained VGG network is used as a feature extractor , perceptual loss The formula is as follows: , in, represents the feature extractor; express norm; Use pre-trained 3DMM model Extract target face images separately and swap face images 3D face coefficient, extract the expression coefficient in the 3D face coefficient, and calculate the expression coefficient loss , the formula is as follows: , in, Represents the expression coefficient extraction operation; Represents the pre-trained 3DMM model.
[0012] Furthermore, the final loss function for: ;in, 、 、 、 、 Represents identity loss , reconstruction losses , fight against losses , Perceptual Loss And expression coefficient loss The weight coefficient of .
[0013] The advantages of the present invention are: The present invention uses a unified multi-scale encoder to extract the multi-scale features of the source face and the target face respectively, and uses the identity mapping network to accurately extract the identity information of the multi-scale features of the source face, and then uses the cross-attention fusion and multi-scale feature transfer modules to transfer the identity features. Finally, the fused information is used to obtain the final exchanged face through image generation. This method introduces the attention mechanism and multi-scale feature transfer method to promote the accurate transfer of identity information details such as face shape and skin color. At the same time, an optimized loss function is added during the training process to ensure the generation of high-fidelity exchanged faces with source identity and target attributes. The present invention also provides more diverse training and test data for deep fake detection research, which helps to promote the development of fake detection technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.
[0015] Figure 1 is a flow chart of the steps of the method of the present invention; Figure 2 This is a qualitative comparison of the results of the present invention on the CelebA-HQ dataset; Figure 3 This is a qualitative comparison of the effects of the present invention on the FF++ dataset. DETAILED DESCRIPTION
[0016] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments derived by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.
[0017] Example 1 In this embodiment, Figure 1 As shown, the present invention provides a face identity exchange method based on multi-scale feature transfer, which specifically includes the following steps: S1. Get the source face image and target face image .
[0018] S2. Source face image and target face image The multi-scale source face features and the multi-scale target face features are obtained by inputting them into a unified multi-scale encoder; the unified multi-scale encoder is constructed based on the pre-trained ResNet-50 model, including a convolution layer, a first residual block, a second residual block, a third residual block, and a fourth residual block; after the source face image is preliminarily processed by the convolution layer, it passes through the first residual block, the second residual block, the third residual block, and the fourth residual block in sequence to obtain the first source face features. , second source facial features , third-source facial features , the fourth source of facial features Similarly, the target face image is input into a unified multi-scale encoder for processing to obtain the first target face feature , second target facial features , the third target facial features , the fourth target facial feature .
[0019] S3. Establish an identity mapping network ID Mapping, and extract identity information from the multi-scale source face features through the identity mapping network to obtain a multi-scale source face identity vector.
[0020] The identity mapping network ID Mapping includes a channel attention module, an adaptive average pooling layer, a batch normalization layer and a LeakyReLU activation function; the channel attention module includes a multi-scale pooling module and a fully connected network; the multi-scale pooling module includes a global average pooling layer and a global maximum pooling layer; the fully connected network includes a fully connected layer, a ReLU activation function and a Sigmoid function.
[0021] In the channel attention module, the first source face feature The overall distribution features are obtained through the global average pooling layer , after the global maximum pooling layer local salient features ; The overall distribution characteristics and local salient features Splicing to obtain fusion features The fusion feature Generate channel weight W through the fully connected network and combine the channel weight W with the first source face feature Multiply channel by channel to get the feature map of enhanced identity .
[0022] The feature map of the enhanced identity The first source face identity vector is obtained after the adaptive average pooling layer, batch normalization layer and LeakyReLU activation function .
[0023] Similarly, the second source facial features , third-source facial features , the fourth source of facial features Input into the identity mapping network ID Mapping, through the first source face feature The same process is used to obtain the second source face identity vector , the third source face identity vector and the fourth source face identity vector .
[0024] S4. Establish a cross attention fusion module CAFM, the first source face identity vector and the first target facial features After fusion through the cross attention fusion module, the preliminary fusion features are obtained .
[0025] In the cross attention fusion module, the first source face identity vector As a query vector , the first target facial feature As a key vector Sum vector ; By calculating the query vector With key vector The similarity between them is used to obtain the query vector For the key vector The attention weight , the formula is as follows: , Among them, softmax means normalization processing; Represents the key vector The transpose of represents the dimension of the feature vector; Represents the factor used to scale the attention score; the resulting attention weight Applied to a vector of values The initial fusion features are obtained by weighted fusion , the formula is as follows: , in, Represents a multiplication operation.
[0026] S5. Establish a multi-scale feature transfer decoder MFTD, the multi-scale source face identity vector and the multi-scale target face feature are input into the multi-scale feature transfer decoder MFTD, and the multi-scale fusion feature is obtained through the feature transfer module FTB; the multi-scale fusion feature and the preliminary fusion feature are combined. After upsampling residual block URB processing, the output features are obtained ; The output characteristics After the convolution operation, the exchanged face image is obtained The multi-scale feature transfer decoder (MFTD) includes four feature transfer upsampling residual blocks and a convolutional layer. The four feature transfer upsampling residual blocks adopt a unified structure. Each feature transfer upsampling residual block adopts a serial cascade architecture, consisting of a feature transfer module (FTB) and an upsampling residual block (URB). The feature transfer module (FTB) performs identity fusion, and the upsampling residual block (URB) performs feature space resolution restoration and residual connection enhancement.
[0027] In the feature transfer module FTB, the multi-scale source face identity vector and the multi-scale target face features are processed to obtain the multi-scale feature transfer preliminary fusion feature , the formula is as follows: , in, Represents the i-th target face feature; represents the identity vector of the i-th source face; ; Represents element-wise multiplication operation; Represents element-by-element addition operation; the multi-scale feature transfer initially fuses features After processing by the global fusion network Global Mapping network and the local fusion network Local Mapping network, the global fusion features are obtained respectively. and local fusion features ; The global fusion feature and local fusion features Adaptive fusion is performed through the gating mechanism to obtain multi-scale fusion features ; The multi-scale fusion feature Including the first fusion feature , the second fusion feature , the third fusion feature , the fourth fusion feature .
[0028] The global fusion network Global Mapping network includes an adaptive average pooling layer, three 1×1 convolutional layers, a batch normalization layer, a LeakyReLU activation function, and a residual connection; the adaptive average pooling layer is the same as the adaptive average pooling layer (Avg Pool) in the previous step (such as ID Mapping). Multi-scale feature transfer preliminary fusion features First, it is processed by the adaptive average pooling layer to obtain the global feature representation. Then, it is processed by the 1×1 convolution layer, batch normalization layer and LeakyReLU activation function to adjust the channel and perform nonlinear transformation. The transformed feature map is initially fused with the multi-scale feature transfer. Fusion is performed through residual connections (1×1 convolutional layers are used in the residual branches to process multi-scale feature transfer and initially fuse features , and then added to the transformed feature map), retaining the original information while enhancing the model's expressiveness; the feature map after residual connection fusion is further extracted and channel adjusted through the 1×1 convolution layer to obtain the final global mapping feature .
[0029] The local fusion network Local Mapping network includes three 3×3 convolutional layers, instance normalization layer, LeakyReLU activation function and residual connection; multi-scale feature transfer preliminary fusion feature First, local spatial information is extracted and nonlinearly transformed through a 3×3 convolutional layer, instance normalization, and LeakyReLU activation function; the transformed feature map is initially fused with multi-scale feature transfer. Fusion is performed through residual connections (a 3×3 convolutional layer is used in the residual branch to process the multi-scale feature transfer and the initial fusion feature , and then added to the transformed feature map), retaining the original information while enhancing the model's expressive power; the feature map after residual connection fusion is further extracted through a 3×3 convolutional layer and normalized through instance normalization to obtain local mapping features that focus more on features representing identity such as eyes, mouth, and nose .
[0030] The global fusion feature and local fusion features Adaptive fusion is performed through a gating mechanism. The formula for this process is as follows: , in, Represents a dynamic attention mask; the dynamic attention mask is a global fusion feature and local fusion features After addition, it is processed by Sigmoid activation function.
[0031] The upsampling residual block (URB) adopts a unified structure. Each upsampling residual block (URB) includes an upsampling layer, a 3×3 convolution layer, an instance normalization layer, a LeakyReLU activation function, a 1×1 convolution layer, and a residual connection.
[0032] The preliminary fusion features and the first fusion feature Input into the first upsampled residual block URB, and perform feature addition to obtain the first feature map , the first feature map After the upsampling layer, 3×3 convolution layer, instance normalization layer, LeakyReLU activation function dimension expansion and spatial information extraction, the processing result is obtained; the residual connection is used to connect the processing result with the first fusion feature map Fusion is performed to retain the original information while enhancing the model's expressiveness; information extraction and channel adjustment are performed on the fusion result through a 1×1 convolutional layer to obtain the first upsampled residual block URB output feature ; The first up-sampled residual block URB and the output first fusion feature The feature transfer module FTB is connected in series; The first block upsamples the residual block URB output features and the second fusion feature Input into the second upsampled residual block URB to obtain the output features of the second upsampled residual block URB ; The second block upsamples the residual block URB and outputs the second fusion feature Similarly, repeat the above operation to obtain the fourth up-sampled residual block URB output feature , as the final output feature ; The fourth up-sampled residual block URB and the output fourth fusion feature The feature transfer modules FTB are connected in series.
[0033] S6. Iteratively optimize the above process through the loss function.
[0034] The loss function includes identity loss , reconstruction losses , fight against losses , Perceptual Loss And expression coefficient loss : The identity embedding vectors of the source face image and the swapped face image are extracted through the pre-trained ArcFace face recognition model, and the cosine similarity between the identity embedding vectors is calculated as the identity loss. ; The formula is as follows: , in, Represents a face recognition model; Indicates cosine similarity calculation; According to the target face image and the exchanged face image, the square error of each element is summed and the average is taken to obtain the reconstruction loss ; The formula is as follows: , in, express norm; Use a multi-scale discriminator to distinguish high-resolution images and obtain adversarial loss , the formula is as follows: , Where D represents the multi-scale discriminator; Represents the source face image The probability distribution to be obeyed is the expectation; Represents the source face image and swap face images The joint probability distribution takes the expectation; In order to improve the visual effect of the generated image at the pixel level, the pre-trained VGG network is used as a feature extractor , perceptual loss The formula is as follows: , in, represents the feature extractor; Use pre-trained 3DMM model Extract target face images separately and swap face images 3D face coefficient, extract the expression coefficient in the 3D face coefficient, and calculate the expression coefficient loss , the formula is as follows: , in, Represents the expression coefficient extraction operation; Represents the pre-trained 3DMM model.
[0035] The final loss function for: ;in, 、 、 、 、 Represents identity loss , reconstruction losses , fight against losses , Perceptual Loss And expression coefficient loss The weight coefficients are set to 0.3, 0.05, 0.5, 0.1, and 0.05 respectively.
[0036] Example 2 In this example, a series of qualitative and quantitative experiments were conducted on the CelebA-HQ dataset and the FaceForensics++ (FF++) dataset, respectively. Four key metrics, including identity retrieval accuracy (ID), face shape error (Shape), expression error (Exp), and head pose error (Pose), were used to evaluate the performance. The results are shown in Tables 1 and 2. Bold text indicates optimal performance, and underlined text indicates suboptimal performance.
[0037] Table 1 Quantitative analysis results of the exchange model on the CelebA-HQ dataset As can be seen in Table 1, our method achieves excellent results across multiple metrics. In terms of identity retrieval accuracy (ID), it outperforms the next-best method, the RAFSwap face-swapping model, by approximately 0.25%, demonstrating its significant advantage in identity transfer. This improvement is primarily attributed to the design of the identity mapping network, which more accurately extracts source identity information. Furthermore, our method also performs well in the face shape error (Shape) metric, achieving the best result of 0.428. This indicates that the face shape of the generated image is more similar to that of the source face, while other methods tend to generate faces closer to the target face. This further validates the effectiveness of the feature transfer module in enhancing the transfer of source face geometry.
[0038] Table 2 Quantitative analysis results of the exchange model on the FF++ dataset As can be seen in Table 2, our method still achieves optimal results in terms of identity retrieval accuracy (ID) and face shape error (Shape), further demonstrating its superior ability to preserve the source identity features and geometric structure. Although our method does not achieve optimal results in terms of expression error (Exp) and head pose error (Pose), this is primarily because the generated images tend to retain the source face's facial structure, which has a certain impact on the preservation of expression and pose. Therefore, our method achieves slightly higher errors compared to other face-swapping models.
[0039] according to Figure 2 As shown in the figure, our method not only successfully transfers the identity features of the source face but also achieves better skin color consistency. In contrast, most other methods tend to retain the skin color of the target face. Furthermore, our method excels in preserving facial details. For example, in the fifth row, our method accurately transfers local structures such as eyebrows and nose from the source face to the target face while preserving their original shapes.
[0040] according to Figure 3 As shown, the proposed method not only retains most attribute features but also effectively transfers identity features (including skin color, mouth, and nose), fully demonstrating the effectiveness of the proposed method in extracting and merging identity information. Notably, in addition to successfully transferring identity and skin color features, the proposed method also effectively transfers structural information such as facial shape from the source face to the target face. In contrast, while other methods perform well in preserving attributes, they have significant limitations in preserving the fine-grained identity details of the source face.
[0041] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.
Claims
1. A face identity exchange method based on multi-scale feature transfer, characterized in that: The following steps are involved: S1. Get the source face image and target face image ; S2. Source face image and target face image Input into a unified multi-scale encoder to obtain multi-scale source face features and multi-scale target face features; S3 establishes an identity mapping network ID Mapping, the multi-scale source face features are extracted through the identity mapping network identity information to obtain a multi-scale source face identity vector; S4. Establish a cross attention fusion module CAFM, the first source face identity vector in the multi-scale source face identity vector and the first target face feature in the multi-scale target face feature After fusion through the cross attention fusion module, the preliminary fusion features are obtained ; S5. Establish a multi-scale feature transfer decoder MFTD, the multi-scale source face identity vector and the multi-scale target face feature are input into the multi-scale feature transfer decoder MFTD, and the multi-scale fusion feature is obtained through the feature transfer module FTB; the multi-scale fusion feature and the preliminary fusion feature are combined. After upsampling residual block URB processing, the output features are obtained ; The output characteristics After the convolution operation, the exchanged face image is obtained ; S6. Iteratively optimize the above process through the loss function.
2. The face identity exchange method based on multi-scale feature transfer according to claim 1, characterized in that: In step S2, the unified multi-scale encoder is constructed based on the pre-trained ResNet-50 model, including a convolutional layer, a first residual block, a second residual block, a third residual block, and a fourth residual block; after the source face image is preliminarily processed by the convolutional layer, it passes through the first residual block, the second residual block, the third residual block, and the fourth residual block in sequence to obtain the first source face feature. , second source facial features , third-source facial features , the fourth source of facial features ; Similarly, the target face image is input into a unified multi-scale encoder for processing to obtain the first target face feature , second target facial features , the third target facial features , the fourth target facial feature .
3. The face identity exchange method based on multi-scale feature transfer according to claim 2, characterized in that: In step S3, the identity mapping network ID Mapping includes a channel attention module, an adaptive average pooling layer, a batch normalization layer, and a LeakyReLU activation function. The channel attention module includes a multi-scale pooling module and a fully connected network; the multi-scale pooling module includes a global average pooling layer and a global maximum pooling layer; the fully connected network includes a fully connected layer, a ReLU activation function, and a Sigmoid function: The first source facial features The overall distribution characteristics are obtained through the multi-scale pooling module and local salient features ; The overall distribution characteristics and local salient features Splicing to obtain fusion features The fusion feature Generate channel weight W through the fully connected network and combine the channel weight W with the first source face feature Multiply channel by channel to get the feature map of enhanced identity ; The feature map of the enhanced identity The first source face identity vector is obtained after the adaptive average pooling layer, batch normalization layer and LeakyReLU activation function ; Similarly, the second source facial features , third-source facial features , the fourth source of facial features Input into the identity mapping network ID Mapping, through the first source face feature The same process is used to obtain the second source face identity vector , the third source face identity vector and the fourth source face identity vector .
4. The face identity exchange method based on multi-scale feature transfer according to claim 3, characterized in that: In step S4, the first source face identity vector and the first target facial features Input into the cross attention fusion module, the first source face identity vector As a query vector , the first target facial feature As a key vector Sum value vector ; By calculating the query vector With key vector The similarity between them is used to obtain the query vector For the key vector The attention weight ; The obtained attention weight Applied to a vector of values The initial fusion features are obtained by weighted fusion .
5. The face identity exchange method based on multi-scale feature transfer according to claim 4, characterized in that: In step S5, the multi-scale feature transfer decoder MFTD includes four feature transfer upsampling residual blocks and a convolutional layer.
6. The face identity exchange method based on multi-scale feature transfer according to claim 5, characterized in that: The four feature transfer upsampling residual blocks adopt a unified structure; each feature transfer upsampling residual block adopts a serial cascade architecture, which is composed of a feature transfer module FTB and an upsampling residual block URB in sequence.
7. The face identity exchange method based on multi-scale feature transfer according to claim 6, characterized in that: In the feature transfer module FTB, the multi-scale source face identity vector and the multi-scale target face features are processed to obtain the multi-scale feature transfer preliminary fusion feature , the formula is as follows: , in, Represents the i-th target face feature; represents the identity vector of the i-th source face; ; Represents element-wise multiplication operation; Represents element-by-element addition operation; the multi-scale feature transfer initially fused features After processing by the global fusion network Global Mapping network and the local fusion network Local Mapping network, the global fusion features are obtained respectively. and local fusion features The global fusion network Global Mapping network includes an adaptive average pooling layer, a 1×1 convolution layer, a batch normalization layer, a LeakyReLU activation function, and a residual connection; the local fusion network Local Mapping network includes a 3×3 convolution layer, an instance normalization layer, a LeakyReLU activation function, and a residual connection; the global fusion feature and local fusion features Adaptive fusion is performed through the gating mechanism to obtain multi-scale fusion features , the formula of this process is as follows: , in, Represents a dynamic attention mask; the dynamic attention mask is a global fusion feature and local fusion features After addition, it is processed by Sigmoid function to obtain the multi-scale fusion feature Including the first fusion feature , the second fusion feature , the third fusion feature , the fourth fusion feature .
8. The face identity exchange method based on multi-scale feature transfer according to claim 7, characterized in that: The upsampling residual block (URB) adopts a unified structure. Each upsampling residual block (URB) includes an upsampling layer, a 3×3 convolution layer, an instance normalization layer, a LeakyReLU activation function, a 1×1 convolution layer, and a residual connection: The preliminary fusion features and the first fusion feature Input into the first upsampled residual block URB, and perform feature addition to obtain the first feature map , the first feature map After the upsampling layer, 3×3 convolution layer, instance normalization layer, LeakyReLU activation function dimension expansion and spatial information extraction, the processing result is obtained; the residual connection is used to connect the processing result with the first fusion feature map Fusion is performed; information extraction and channel adjustment are performed on the fusion result through a 1×1 convolution layer to obtain the first upsampled residual block URB output feature ; The first up-sampled residual block URB and the output first fusion feature The feature transfer module FTB is connected in series; The first block upsamples the residual block URB output features and the second fusion feature Input into the second upsampled residual block URB to obtain the output features of the second upsampled residual block URB ; The second block upsamples the residual block URB and outputs the second fusion feature Similarly, repeat the above operation to obtain the fourth up-sampled residual block URB output feature , as the final output feature ; The fourth up-sampled residual block URB and the output fourth fusion feature The feature transfer modules FTB are connected in series.
9. The face identity exchange method based on multi-scale feature transfer according to claim 8, characterized in that: Step S6 specifically includes: The loss function includes identity loss , reconstruction losses , fight against losses , Perceptual Loss And expression coefficient loss : The face recognition model is used to extract the identity embedding vectors of the source face image and the exchanged face image, and the cosine similarity between the identity embedding vectors is calculated as the identity loss. ; According to the target face image and the exchanged face image, the square error of each element is summed and the average is taken to obtain the reconstruction loss ; Use a multi-scale discriminator to distinguish high-resolution images and obtain adversarial loss , the formula is as follows: , Where D represents the multi-scale discriminator; Represents the source face image The probability distribution to be obeyed is the expectation; Represents the source face image and swap face images The joint probability distribution takes the expectation; Perceptual loss The formula is as follows: , in, represents the feature extractor; express norm; Use pre-trained 3DMM model Extract target face images separately and swap face images 3D face coefficient, extract the expression coefficient in the 3D face coefficient, and calculate the expression coefficient loss , the formula is as follows: , in, Represents the expression coefficient extraction operation; Represents the pre-trained 3DMM model.
10. The face identity exchange method based on multi-scale feature transfer according to claim 9, characterized in that: The final loss function for: ;in, 、 、 、 、 Represents identity loss , reconstruction losses , fight against losses , Perceptual Loss And expression coefficient loss The weight coefficient of .
Citation Information
Patent Citations
Face replacement method based on multistage attribute encoder and attention mechanism
CN112766160A
Face image identity synthesis method based on GAN and 3D coefficient reconstruction
CN116612211A
Cited By
Face identity exchange method, system and equipment
CN121190619A
