A face beautification method and system based on sketch and style transfer

Abstract

The present application belongs to the technical field of image processing, and particularly relates to a face beautifying method and system based on sketch and style transfer, the method comprising: obtaining a data set containing an original face image and a beautified face image; obtaining an expected face appearance sketch corresponding to the face image through image processing technology; pre-processing and data enhancing the image to obtain a standard input image; obtaining deformation features and style features according to the image; generating a beautified face image by using the deformation features for deformation processing and simultaneously adding the style features; calculating the loss of the generated beautified face image to guide the training of a neural network and iteratively training a face beautifying and editing model, the present application introduces a face sketch, combines style transformation, realizes a new face beautifying mode, and the model is simple and can be widely used in the field related to face image processing.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a face beautification method and system based on sketch and style transfer. Background Technology

[0002] Facial image beautification refers to the automatic editing and enhancement of original facial images through neural networks, transforming them into high-quality images that conform to human aesthetics. With further technological advancements, applications using AI-driven photo and image editing to streamline facial beautification workflows are becoming increasingly common. Simultaneously, facial beauty prediction technology has been extensively researched. Possessing knowledge related to facial beauty, it provides feedback and guidance for facial beauty transformations and demonstrates that computers can accurately predict facial beauty to a certain extent.

[0003] Traditional image editing methods for facial beautification mostly focus on facial skin and geometric features. They employ traditional image processing techniques to enhance the skin, alter the pixel matrix of the facial image, and influence the beautification effect by improving image quality and adding head ornaments, thus preserving the facial structure. Other methods attempt to modify facial structure through face averaging or operators to change facial contours. All of these operations have corresponding automated algorithms; combining these algorithms can automatically complete the facial beautification process without any user intervention.

[0004] In existing methods, once the algorithm is set, its style cannot be changed; generally, only the degree can be adjusted. There is a lack of automated control over beautification. In the field of image editing, it is also difficult to achieve fine-grained control over facial regions; each operation requires a dedicated algorithm. Beautification methods utilizing style transfer are often trained on unpaired images. Due to the lack of beautification datasets with paired images, research on algorithms for paired beautification images is limited, especially in the field of deep learning beautification algorithms. Therefore, it is necessary to explore beautification methods that take into account both facial shape editing and skin enhancement, and achieve beautification from paired images. Summary of the Invention

[0005] The purpose of this invention is to provide a face beautification method and system based on sketches and style transfer. By mapping the shape of the face through sketches and using style coding to achieve skin beautification, the controllable beautification of the face can be effectively realized.

[0006] In a first aspect, this invention provides a face beautification method based on sketches and style transfer, and proposes a network structure and training method for mining information from pairs of beautified images. The method involves acquiring a face image to be beautified, inputting the face image to be beautified into a trained face beautification and editing model, and obtaining the beautified image. The training of the face beautification and editing model includes the following steps:

[0007] S1. Obtain a dataset containing multiple pairs of face images, each pair of face images including an original face image and a beautified face image;

[0008] S2. Obtain the original face sketch based on the original face image in the face image pair, and modify the original face sketch to obtain the desired face appearance sketch.

[0009] S3. Perform the same preprocessing on the original face image and the desired face sketch respectively;

[0010] S4. Perform the same data augmentation on the preprocessed original face image and the expected face appearance sketch respectively to obtain the standard original face image and the standard expected face appearance sketch.

[0011] S5. Sample the standard expected face sketch and encode the deformation features;

[0012] A sketch parsing encoding module is proposed, which uses max pooling and average pooling to downsample the standard expected face sketch, and then takes the average of the two results to obtain the deformation features;

[0013] S6. Classify and label the styles of the original face images and the beautified face images, and perform non-linear mapping on the labels to obtain style features;

[0014] S7. Stitch together the standard original face image and the standard expected face sketch, and input the stitched image, deformation features and style features into the face beautification and editing model to generate a beautified face image;

[0015] Furthermore, the specific process of generating beautified facial images includes:

[0016] Extract low-level semantic features from the stitched image and add style features to the low-level semantic features;

[0017] High-level semantic features are obtained by deforming low-level semantic features that incorporate style features.

[0018] Style features are then added to the high-level semantic features to ultimately generate a beautified face image.

[0019] The process of adding stylistic features is represented as follows:

[0020]

[0021] Where x represents low-level semantic features or high-level semantic features, z represents style features, μ(z) and σ(z) are input feature statistics, and y represents an affine transformation parameter;

[0022] S8. Calculate the loss function on the generated beautified face image to guide the training of the neural network. Iteratively train the face beautification and editing model until convergence, and save the model parameters. The training steps include the following, and the proposed loss function is expressed as:

[0023]

[0024] L adv This is the decision loss, used to train the generative adversarial network structure. D(·) represents the output of the discriminator network, where i represents the style of the current image, and s j represents the deformation encoding of the label image sketch, and j represents the one-hot style encoding of the label image, where This represents the image x with style i. i By adding x j The deformation encoding and style j are used to generate the output image;

[0025]

[0026] L rec This is the reconstruction loss, used to train the network to generate the original image. This represents the image x with style i. i By adding x i The deformation encoding and style i are used to generate the output image, where ||·||1 represents the L1 norm;

[0027] L cyc =‖G j,si,i (x i,sj,j )-x i ||1

[0028] L cyc It is a cyclic loss used to train the network to generate faces that retain identity features, where G j,si,i (x i,sj,j ) indicates that the generated image x i,sj,j The image is then input into network G again, along with the original image style i and the original image deformation code s. i and compared with the original image x i To incur losses;

[0029] L trans =‖V(x) i,sj,j )-V(x j )‖1+‖F(xi,sj,j )-j‖2

[0030] L trans It is the transformation loss, used to train the network to generate the desired beautified face, where x j It is image x i The corresponding label image, V(·) represents the output feature of a certain layer in the VGG network, F(·) represents the prediction of the image style, and ‖·‖2 represents the L2 norm;

[0031] The final loss is the weighted sum of the above losses, L. total , represented as:

[0032] L total =λ1L adv +λ2L rec +λ3L cyc +λ4L trans

[0033] Wherein, λ1, λ2, λ3, and λ4 are the first, second, third, and fourth loss weights, respectively, used to control the training process.

[0034] In a second aspect, the present invention provides a face beautification system based on sketches and style transfer, comprising an image acquisition module, an image processing module, and an image display module. The image acquisition module acquires a face image to be beautified and inputs it into the image processing module. The image processing module processes the face image to be beautified using a trained face beautification and editing model to obtain a beautified image. The image display module obtains and displays the beautified image. The training of the face beautification and editing model by the image processing module includes the following modules:

[0035] The data acquisition module is used to acquire multiple sets of face image pairs, each set of face image pairs including one original face image and one beautified face image;

[0036] The sketch acquisition module is used to perform threshold segmentation on the original face image to obtain a segmented image, extract the edge information of the segmented image to obtain an edge information image, expand the edge information image to obtain a 256×256×1 original face sketch, modify the features of the original face sketch to obtain the desired face appearance sketch.

[0037] The face image preprocessing module is used to scale the original face image and the desired face shape sketch, and to center-crop and normalize the scaled image to obtain a 256×256×3 preprocessed image.

[0038] The data augmentation module generates random numbers to randomly rotate the output image of the face image preprocessing module, and combines random permutations and combinations of scaling, vertical flipping and horizontal flipping data augmentation methods to perform data augmentation.

[0039] The deformation encoding module samples the standard expected human face sketch and encodes the deformation features.

[0040] The style encoding module divides the original face image and the beautified face image in the face image pair into original style and beautified style, finds the boundary between the two styles, converts the two styles into one-hot encoding, and converts the one-hot encoding into a one-dimensional vector style feature through multiple fully connected layers.

[0041] The face image generation module stitches together the standard original face image and the standard expected face shape sketch, accepts different style features and deformation features, and generates the desired beautified face image as output.

[0042] The neural network in the face image generation module is divided into downsampling layer, intermediate layer and upsampling layer;

[0043] When adding deformation features to a neural network, the deformation features are spliced ​​in the middle layer of the neural network to ensure that conditional information is injected into each block of the network.

[0044] When adding style features to a neural network, they are added to both the downsampling layer and the upsampling layer. The downsampling layer is the layer before the deformation feature injection network, and the upsampling layer is the layer after the deformation feature injection network.

[0045] The training module extracts facial features from the beautified face image, performs loss calculations on the beautified face image and the beautified face image in the image pair, and guides the training of the face beautification and editing model to make the correct deformation to the input image through gradient descent;

[0046] The deformation encoding module and the style encoding module are isolated from each other, and the roles of the two modules are optimized separately when training the face beautification and editing model.

[0047] The advantages and beneficial effects of this invention are as follows:

[0048] (1) This invention proposes a style coding module and a deformation coding module to mine these two types of information respectively, and achieves simultaneous face shape editing and skin beautification on the same network.

[0049] (2) This invention designs a novel face editing method that supports us to customize the shape of the face and change the skin color style.

[0050] By using the original face sketch, fine-tuning is performed to obtain the desired face appearance sketch, so that the generated face can change accordingly with the sketch, achieving fine control over the face region.

[0051] Then, by inputting different style features, the skin style of the face can be changed, ultimately realizing face beautification editing, achieving automated control of beautification, and simplifying the steps of face beautification. Attached Figure Description

[0052] Figure 1 This is a flowchart of a face beautification method and system based on sketch and style transfer provided in an embodiment of the present invention;

[0053] Figure 2 This is a system block diagram of the face beautification and editing model of the present invention;

[0054] Figure 3 This is a schematic diagram of a network structure based on sketching and style transfer according to an embodiment of the present invention;

[0055] Figure 4 This is an application effect diagram of an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them.

[0057] This invention provides a face beautification method based on sketches and style transfer, such as... Figure 1 As shown, it includes the following steps:

[0058] S1. Obtain a dataset containing multiple pairs of face images. Each pair of face images includes an original face image and a beautified face image. The paired images should be registered as much as possible, while ensuring that the faces before and after beautification have some common features.

[0059] S2. Obtain the original face sketch based on the original face image in the face image pair. Modify the facial features of the original face sketch using programmatic means or drawing tools. Modify or replace a certain face area of ​​the sketch to obtain the desired face appearance sketch.

[0060] The face region is cropped and scaled to the size allowed by the generator network as input. The input image size must be kept consistent when training the network and using the model. The cropped face region in the image is scaled and then normalized.

[0061] S4. Perform the same data augmentation on the preprocessed original face image and the expected face sketch to expand the dataset;

[0062] By generating random numbers, the preprocessed image matrix is ​​randomly rotated according to the first and second dimensions, and combined with scaling, vertical flipping and horizontal flipping operations, and data augmentation methods are randomly arranged and combined. Different data augmentation methods are used on the same image data in each training session to obtain the standard original face image and the standard expected face sketch.

[0063] S5. Sample the standard expected face sketch and encode the deformation features;

[0064] The open-source Python library function open-cv is used to perform threshold segmentation on the face image, processing the image into a binary image that is either black or white; then the Cannay operator is used to extract the edge of the image, adjust the high and low thresholds, obtain the face edge information, retain the edges of the facial features, ignore other detailed edges, and then expand the face sketch into a matrix form of (256, 256, 1) dimensions.

[0065] S6. Divide the original face image and the beautified face image of the face image pair into original style and beautified style, and train a binary classifier to find the boundary between the two styles. Convert the two styles into one-hot encoding, and convert the one-hot encoding into style features through non-linear mapping.

[0066] S7. Stitch together the standard original face image and the standard expected face sketch;

[0067] After quantization in a computer, the standard original face image presents as a matrix A = (256, 256, 3) with dimensions (width, height, number of channels) and 3 channels. The standard desired face sketch, after quantization, presents as a matrix B1 = (256, 256) with dimensions (width, height). By expanding the dimensions of the standard desired face sketch, it becomes a matrix B2 = (256, 256, 1) with dimensions (width, height, number of channels) and 1 channel. Since matrices A and B have the same width and height, they can be concatenated. Concatenating them along the channel dimension results in a matrix C with dimensions (256, 256, 4). Matrix C then serves as the input to the generator network.

[0068] The stitched image, deformation features, and style features are input into the face beautification and editing model to generate a beautified face image. The specific process of generating a beautified face image includes:

[0069] Extract low-level semantic features from the stitched image and add style features to the low-level semantic features;

[0070] High-level semantic features are obtained by deforming low-level semantic features that incorporate style features.

[0071] Style features are then added to the high-level semantic features to ultimately generate a beautified face image.

[0072] The process of adding stylistic features is represented as follows:

[0073]

[0074] Where x represents low-level semantic features or high-level semantic features, z represents style features, μ(z) and σ(z) are input feature statistics, and y represents an affine transformation parameter;

[0075] S8. Perform loss calculation on the generated beautified face image. Perform L1 loss calculation on the generated beautified face image and the beautified face to obtain pixel-level loss. Perform feature loss calculation on the generated beautified face image and the generated beautified face image to obtain feature loss.

[0076] The results of loss calculation are used to guide the training of the neural network through gradient descent. The face beautification and editing model is iteratively trained until convergence, and the model parameters are saved.

[0077] The training steps include the following, and the proposed loss function is expressed as:

[0078]

[0079] L adv This is the decision loss, used to train the generative adversarial network structure. D(·) represents the output of the discriminator network, where i represents the style of the current image, and s j represents the deformation encoding of the label image sketch, and j represents the one-hot style encoding of the label image, where This represents the image x with style i. i By adding x j The deformation encoding and style j are used to generate the output image;

[0080]

[0081] L rec This is the reconstruction loss, used to train the network to generate the original image. This represents the image x with style i. i By adding x i The deformation encoding and style i are used to generate the output image, where ||·||1 represents the L1 norm;

[0082] L cyc =‖G j,sj,i (x i,sj,j )-x i ||1

[0083] L cycIt is a cyclic loss used to train the network to generate faces that retain identity features, where G j,si,i (x i,sj,j ) indicates that the generated image x i,sj,j The image is then input into network G again, along with the original image style i and the original image deformation code s. i and compared with the original image x i To incur losses;

[0084] L trans =‖V(x) i,sj,j )-V(x j )‖1+‖F(x i,sj,j )-j‖2

[0085] L trans It is the transformation loss, used to train the network to generate the desired beautified face, where x j It is image x i The corresponding label image, V(·) represents the output feature of a certain layer in the VGG network, F(·) represents the prediction of the image style, and ‖·‖2 represents the L2 norm;

[0086] The final loss is the weighted sum of the above losses, L. total , represented as:

[0087] L total =λ1L adv +λ2L rec +λ3L cyc +λ4L trans

[0088] Wherein, λ1, λ2, λ3, and λ4 are the first, second, third, and fourth loss weights, respectively, used to control the training process.

[0089] This invention provides a face beautification system based on sketches and style transfer, such as... Figure 2 As shown, it includes an image acquisition module, an image processing module, and an image display module. The image acquisition module acquires the face image to be beautified and inputs it into the image processing module. The image processing module uses a trained face beautification and editing model to process the face image to obtain the beautified image. The image display module obtains and displays the beautified image. The training of the face beautification and editing model in the image processing module includes the following modules:

[0090] The data acquisition module is used to acquire multiple sets of face image pairs, each set of face image pairs including one original face image and one beautified face image;

[0091] The sketch acquisition module is used to perform threshold segmentation on the original face image to obtain a segmented image, extract the edge information of the segmented image to obtain an edge information image, expand the edge information image to obtain a 256×256×1 original face sketch, modify the features of the original face sketch to obtain the desired face appearance sketch.

[0092] The face image preprocessing module is used to scale the original face image and the desired face shape sketch. If the image is shorter in width, height and height, the image is scaled by a ratio of 256 for height and height. If the image is shorter in width and longer in height, the image is scaled by a ratio of 256 for width and height. The scaled image is then cropped at the center and normalized to obtain a preprocessed image of 256×256×3.

[0093] The data augmentation module generates random numbers to randomly rotate the output image of the face image preprocessing module, and combines random permutations and combinations of scaling, vertical flipping and horizontal flipping data augmentation methods to perform data augmentation.

[0094] The deformation encoding module samples the standard expected human face sketch and encodes the deformation features.

[0095] The style encoding module divides the original face image and the beautified face image in the face image pair into original style and beautified style, finds the boundary between the two styles, converts the two styles into one-hot encoding, and converts the one-hot encoding into a one-dimensional vector style feature through multiple fully connected layers.

[0096] The face image generation module stitches together the standard original face image and the standard expected face shape sketch, accepts different style features and deformation features, and generates the desired beautified face image as output.

[0097] The neural network in the face image generation module is divided into downsampling layer, intermediate layer and upsampling layer;

[0098] When adding deformation features to a neural network, the deformation features are spliced ​​in the middle layer of the neural network to ensure that conditional information is injected into each block of the network.

[0099] When adding style features to a neural network, they are added to both the downsampling layer and the upsampling layer. The downsampling layer is the layer before the deformation feature injection network, and the upsampling layer is the layer after the deformation feature injection network.

[0100] The training module extracts facial features from the beautified face image, performs loss calculations on the beautified face image and the beautified face image in the image pair, and guides the training of the face beautification and editing model to make the correct deformation to the input image through gradient descent;

[0101] The deformation encoding module and the style encoding module are isolated from each other, and the roles of the two modules are optimized separately when training the face beautification and editing model.

[0102] The trained convolutional neural network is as follows Figure 2 As shown:

[0103] The standard expected facial sketch is processed by a deformation encoding module to obtain deformation features. The standard original facial image and the standard expected facial sketch are then stitched together at the input layer of the face image generation module. After the face image generation module extracts features from the stitched image, deformation features are stitched together in the intermediate layers of the face image generation module, ensuring that conditional information is injected into each block of the network. When adding style features to the face image generation module, they are added at both the shallow and deep layers of the network; the shallow layer is the layer before deformation feature injection, and the deep layer is the layer after deformation feature injection.

[0104] The generated face effect is as follows Figure 3 As shown:

[0105] The left side shows the standard original face and standard expected face edge map of the input face beautification editing model, and the beautified face image on the right is generated after beautification.

[0106] The above embodiments should be understood as illustrative only and not as limiting the scope of protection of the present invention. After reading the description of the present invention, those skilled in the art can make various alterations or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A sketch-based and style transfer-based face beautification method, obtaining a face image to be beautified, inputting the face image to be beautified into a trained face beautification editing model, and obtaining a beautified image, characterized in that, The training of the face beautification editing model comprises: S1, obtaining a data set containing multiple groups of face image pairs, each group of face image pairs comprising an original face image and a beautified face image; S2, obtaining an original face sketch according to the original face image in the face image pair, and modifying the original face sketch to obtain an expected face appearance sketch; Obtaining an original face sketch according to the original face image in the face image pair comprises: Using an open-source python library function open-cv to perform threshold segmentation on the face image, processing the image into a binary image of black and white; then using a Cannay operator to perform edge extraction on the image, adjusting the high and low thresholds to obtain face edge information, retaining the edge of the face features and ignoring other detail edges; S3, the original face image and the expected face appearance sketch are respectively preprocessed in the same way; S4, the preprocessed original face image and the expected face appearance sketch are respectively subjected to the same data enhancement to obtain a standard original face image and a standard expected face appearance sketch; S5, sampling the standard expected face appearance sketch to obtain a deformation feature; S6, classifying and labeling the styles of the original face image and the beautified face image, and performing nonlinear mapping on the labels to obtain a style feature; S7, the standard original face image and the standard expected face appearance sketch are spliced, and the spliced image, the deformation feature and the style feature are input into the face beautification editing model to generate a beautified face image; S8, loss calculation is performed on the generated beautified face image to guide the training of the neural network, and the face beautification editing model is iteratively trained until convergence, and the model parameters are saved.

2. The sketch-based and style transfer-based face beautification method of claim 1, wherein, The deformation feature obtained in step S5 is obtained by using maximum pooling and average pooling to downsample the standard expected face appearance sketch, and then taking the average of the two results.

3. The sketch-based and style transfer-based face beautification method of claim 1, wherein, The specific process of generating a beautified face image in step S7 comprises: S11. Extracting low-level semantic features of the spliced image, and adding style features to the low-level semantic features; S12. Deforming the low-level semantic features with added style features through the deformation feature to obtain high-level semantic features; S13. Adding style features to the high-level semantic features again to finally generate a beautified face image.

4. The sketch-based and style transfer-based face beautification method of claim 3, wherein, The process of adding style features is represented as: Wherein, x represents low-level semantic features or high-level semantic features, z represents style features, μ(z) and σ(z) are input feature statistics, and y represents an affine transformation parameter.

5. The sketch-based and style transfer-based face beautification method of claim 1, wherein, The loss calculation on the generated beautified face image in step S8 is represented by the loss function as follows: L adv is the decision loss for training the adversarial generative network structure, D(·) represents the output of the discriminator network, where i indicates the style of the current image, s j represents the morphological encoding of the sketch of the label image, j represents the one-hot style encoding of the label image, where represents the output image generated by adding the morphological encoding of x j and the style j; i , the image x L rec is the reconstruction loss, used to train the network to generate the original image; wherein represents the image x i of style i is generated by adding the morphological encoding of x i and the style i, and ‖·‖1 represents the L1 norm. L cyc is a cycle loss, used to train the network to generate a face preserving identity features, wherein G j,si,i (x i,sj,j ) represents inputting the generated image x i,sj,j into the network G again, adding the original image style i and the original image morphological encoding s i , and making a loss with the original image x i ; L trans =‖V(x i,sj,j )-V(x j )‖1+‖F(c i,sj,j )-j‖2 L trans is the transformation loss for training the network to generate the expected beautified face, where x j is the image x i is the corresponding label image, V(·) represents the output feature of a certain layer in the VGG network, F(·) represents the prediction of the image style, and ‖·‖2 represents the L2 norm. The final loss is a weighted sum of the above losses L total is represented as: L total = λ1L adv + λ2L rec + λ3L cyc + λ4L trans Wherein, λ1, λ2, λ3, λ4 are respectively the first, second, third and fourth loss weights, which are used to control the training process.

6. A face beautification system based on sketch and style transfer, comprising an image acquisition module, an image processing module and an image display module, the image acquisition module acquires a face image to be beautified and inputs into the image processing module, the image processing module processes the face image to be beautified by using a trained face beautification editing model to obtain a beautified image, and the image display module obtains and displays the beautified image, characterized in that, The training of the face beautification editing model by the image processing module comprises: A data acquisition module for acquiring multiple groups of face image pairs, each group of face image pairs comprising an original face image and a beautified face image; The sketch obtaining module is configured to perform threshold segmentation on the original face image to obtain a segmented image, extract edge information of the segmented image to obtain an edge information image, expand the edge information image to obtain a 256x256x1 original face sketch, perform feature modification on the original face sketch, and obtain an expected face appearance sketch. The face image preprocessing module is configured to perform scaling on the original face image and the expected face appearance sketch, perform center cropping and normalization on the scaled images to obtain a 256x256x3 preprocessed image. The data enhancement module is configured to perform random rotation on the output image of the face image preprocessing module by generating random numbers, and perform data enhancement by combining random permutation and combination of scaling, vertical flipping and horizontal flipping. The morphing encoding module is configured to sample the standard expected face appearance sketch to obtain morphing features. The style encoding module is configured to divide the original face image and the beautified face image in the face image pair into original image styles and beautified styles, find boundaries of the two styles, convert the two styles into one-hot encoding, convert the one-hot encoding into a one-dimensional vector style feature through multiple fully connected layers. The face image generation module is configured to splice the standard original face image and the standard expected face appearance sketch, accept different style features and morphing features, and generate an expected beautified face image as an output. The training module is configured to extract face features of the beautified face image, perform loss calculation on the beautified face image and the beautified face image in the image pair, and guide the training of the face beautification editing model to make correct morphing on the input image through gradient descent.

7. The sketch and style transfer based face beautification system of claim 6, wherein, The morphing encoding module and the style encoding module are isolated from each other, and the functions of the two modules are optimized respectively when training the face beautification editing model.

8. The sketch and style transfer based face beautification system of claim 6, wherein, The neural network in the face image generation module includes a down-sampling layer, an intermediate layer and an up-sampling layer.

9. The sketch and style transfer based face beautification system of claim 8, wherein, When adding morphing features to the neural network, the morphing features are spliced in the intermediate layer of the neural network, so that the block of each layer of the network is injected with conditional information once.

10. The sketch and style transfer based face beautification system of claim 8, wherein, When adding style features to the neural network, the style features are added to the down-sampling layer and the up-sampling layer of the neural network, respectively, the down-sampling layer is in front of the layer where the morphing features are injected into the network, and the up-sampling layer is in back of the layer where the morphing features are injected into the network.

Images