Lightweight face image generation method based on improved StarGAN and knowledge distillation

Abstract

The invention belongs to the field of image data processing or generation, and relates to an improved StarGAN and knowledge distillation-based lightweight face image generation method, which comprises the following steps of: firstly constructing a student model and a teacher model, pre-training the teacher model by adopting a StarGAN v2 model, then performing knowledge distillation, setting a loss function during knowledge distillation, and finally performing knowledge distillation. Training data are input into the teacher model and the student model, in the knowledge distillation process, activation values of all layers of the generator in forward propagation are obtained, characteristics of the pre-trained teacher model are output to serve as soft labels of the student model, and a loss function is minimized; the loss function comprises pixel-level MSE loss, perception loss, triple loss, GKA loss and feature-based wavelet loss; by means of the mode, the student model can keep high generation quality, meanwhile, the model parameter quantity and the calculation complexity can be reduced, and therefore the light weight target is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of image data processing or generation, and relates to computer vision face image generation, and specifically, to a method for face image generation based on an improved generative adversarial network (StarGAN) and knowledge distillation. Background Art

[0002] With the continuous advancement of artificial intelligence technology, the field of computer vision has made significant progress, especially in image processing and generation. In the existing technology, the generative adversarial network (GAN) model, especially the high-performance image generation model such as StarGAN, can generate high-quality images; however, in the optimization of the GAN model, there is usually a balance between the generation quality and the model size. Larger models usually provide higher generation quality, but overly large models are slow in inference and difficult to adapt to the needs of real-time generation.

[0003] StarGAN v2 is an image-to-image translation model developed by the Clova AI team, which aims to address the limitations of existing models in terms of diversity and multi-domain scalability. It achieves diversified image generation and multi-domain scalability through a single framework, significantly surpassing existing baseline models. The generator only receives the input image x and the encoding s of a specific style, injects s ​​into the generation process through adaptive instance normalization (AdaIN), and generates an image with the style of s; the mapping network converts the latent code into style encodings of multiple domains; the style encoder extracts the style encoding from a given image; the discriminator adopts multi-task output branches, each branch distinguishes between real images and fake images from a specific area. The model has strong multi-domain conversion capabilities and generates high-quality images, but due to its complex network structure and numerous parameters, it consumes a lot of computing power during training and inference, and has high computing and storage overheads, making it difficult to deploy on resource-constrained devices, thus limiting its scope of application. Therefore, how to fully consider the need for lightweight without reducing the quality of image generation (ensuring high generation quality), further reduce the model size, reduce the number of model parameters and computational complexity, and improve the reasoning speed has become an urgent problem to be solved.

[0004] Knowledge distillation technology has achieved remarkable success in the field of deep learning, especially in lightweight model training. However, knowledge distillation is still rarely used in complex image generation models such as StarGAN. At present, knowledge distillation is usually applied in the field of classification. There is not enough research and exploration on the optimization of generation tasks, especially in generative adversarial models. Therefore, how to use knowledge distillation technology to improve the lightweight and performance of the StarGAN model is a shortcoming of current technology. Summary of the invention

[0005] In view of the above technical problems and defects, the present invention provides a lightweight face image generation method based on improved StarGAN and knowledge distillation. The method introduces knowledge distillation technology into the StarGAN v2 model, calls the pre-trained model of StarGANv2 as the teacher model, and designs a series of loss functions, and uses the trained teacher model to guide the student model; in this way, the student model can reduce the number of model parameters and computational complexity while maintaining a high generation quality, thereby achieving the lightweight goal.

[0006] To achieve the above object, the present invention adopts the following technical solution: A lightweight face image generation method based on improved StarGAN and knowledge distillation, the method comprising the following steps: Step 1. Pre-train the teacher model, where the teacher model adopts the StarGAN v2 model; Step 2. Construct a student model, and then perform knowledge distillation; wherein the student model is obtained after improvement on the basis of the teacher model, and the student model uses depthwise separable convolution to replace the standard convolution operation in downsampling and upsampling in the teacher model, and uses movable residual blocks to replace the residual blocks of downsampling and upsampling in the teacher model, and the movable residual blocks of downsampling and upsampling in the student model are set to 2, which reduces the number of convolutional layers and the number of channels of each layer, and the max_conv_dim in the student model is 128; in addition, when designing the student model, a movable mapping network is added after the mapping network, and multiple fully connected layers are designed between the two, and the nonlinear representation capability is enhanced by the ReLU activation function, the hidden layer dimension of the movable mapping network is max_hid_dim=128, and the hidden layer dimension of the mapping network is max_hid_dim=512; When performing knowledge distillation, first set the loss function, then input the training data into the teacher model and the student model for knowledge distillation. During the knowledge distillation process, obtain the activation values ​​of each layer of the generator in the forward propagation, use the pre-trained teacher model feature output as the soft label of the student model, minimize the loss function, so that the student model not only learns to generate similar output images from the input image, but also learns the intermediate layer representations of each layer of the teacher model, helping the student model to be closer to the feature representation of the teacher model in the feature space, and continuously optimizing and improving the performance of the student model itself; Among them, the loss function includes pixel-level MSE loss , Perceptual Loss , triplet loss , GKA loss , feature-based wavelet loss ; ; in, and denote the output images of the student and teacher generators respectively, P is the number of pixels of the image;

[0007] in, and Represent the feature extraction networks of the teacher model and the student model respectively, N is the number of image samples, that is, the number of images in the batch image; Represents the feature extraction network extracted using the teacher model The high-level features in Represents the feature extraction network extracted using the student model The high-level features in The feature difference between the image generated by the student model and the image generated by the teacher model under the student model feature extraction network; ; in, represents the feature extraction function, is a real image, Represents the margin of triplet loss. In the setting of triplet loss, each training sample consists of a triplet: (Anchor, Positive, Negative), where Anchor is an anchor point, which is a sample in the input data. Positive is a sample that belongs to the same category or has similar features as the anchor point sample. Negative is a sample that belongs to a different category or has different features from the anchor point sample. ; Among them, X and Y are the outputs of the student and teacher models at a certain layer, respectively. N is the number of image samples, T represents the transposed matrix; ; in, represents the feature extraction function, are generated by the teacher model and the student model. i The features extracted from the image are calculated by the feature extraction network of the teacher model and the student model; is the wavelet loss, is the GKA loss;

[0008] in, and Represents the teacher and student images after wavelet transformation. i High-frequency details; Step 3. Deploy the trained student model to the actual application scenario to generate face images.

[0009] As a preferred embodiment of the present invention, in step 1, when pre-training the teacher model, the original image needs to be pre-processed, and the specific steps are: scaling the original image to unify the pixels; then normalizing the data to stabilize the distribution; then cropping different areas of the image and scaling the cropped areas; finally, randomly flipping the image horizontally, exchanging the left and right pixels during flipping, and then mapping each pixel value in the image from the integer range of [0, 255] to the floating range of [0.0, 1.0], and normalizing it to adapt it to the training model of the teacher network.

[0010] As a preferred embodiment of the present invention, the feature extraction network of the teacher model and the student model adopts a trained VGG network.

[0011] As a further preferred embodiment of the present invention, when cropping an image, it is allowed to crop the image randomly during training, and the cropping probability prob=0.5 is set. If the generated random number is less than the cropping probability, cropping is performed, otherwise cropping is not performed and the original image is returned.

[0012] Advantages and beneficial effects of the present invention:

[0013] (1) The present invention provides a new method for generating facial images based on improved StarGAN and knowledge distillation. The model is distilled on the basis of StarGAN v2. In addition, in order to ensure that the quality of the student model obtained by distillation is higher and the model is more lightweight, the present invention designs a series of loss functions to train the generator. Finally, based on the teacher model, not only the excellent performance of the model is learned, but also the purpose of being more lightweight is achieved.

[0014] (2) When constructing the student model, the present invention replaces the convolution layer of the teacher model and uses depthwise separable convolution instead of traditional convolution to achieve the characteristics of lightweight and fast. In addition, the present invention replaces the traditional modules with more flexible MoblieResBlk, MobileMapping Network, and MoblieStyleEcoder, and designs a series of loss functions, so that the trained student model can reduce the number of model parameters, improve computational efficiency, and enhance robustness while retaining the image generation quality.

[0015] (3) The present invention adds pixel-level MSE loss, perceptual loss, triplet loss, GKA loss, and wavelet loss to the generator and discriminator. The design of the loss function enables the generative model to optimize image quality at multiple levels, including details, texture, and style consistency, thereby improving the authenticity and diversity of the generated facial images.

[0016] (4) By using knowledge distillation, the student model of the present invention not only learns the advantages of the teacher model, achieving the excellent generation technology of the teacher model and the purpose of lightweight, but also enhances the generalization ability of the model by designing a sophisticated loss function and network architecture, making it have stronger generalization ability.

[0017] (5) The present invention has strong adaptability and can be flexibly applied to different types of generation tasks. Due to its lightweight design, the model can perform transfer learning on a variety of different tasks, thereby better adapting to various data distributions. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of raw image pre-data processing; Figure 2 This is a schematic diagram of the student model network structure based on the improvement of the teacher model network structure; Figure 3 It is a knowledge distillation flow chart; Figure 4 It is a graph of the knowledge distillation model; Figure 5 It is a partial face image generated by the teacher model (not a real face image); Figure 6 It is a partial face image generated by the trained student model (not a real face image). DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solution and advantages of the present invention, the present application is described in detail below in conjunction with the accompanying drawings, but it is not intended to limit the protection scope of the present invention.

[0020] like Figures 1 to 4 As shown, this embodiment provides a lightweight face image generation method based on improved StarGAN and knowledge distillation. The method mainly includes the following three stages: The first stage: image data preprocessing, model training, model evaluation of the teacher model (the existing StarGAN v2), and finally adversarial learning to output the model characteristics of the generator and discriminator.

[0021] The second stage: build a student model (improved StarGAN v2), use lightweight strategies to reduce and replace some layers of the teacher model to achieve the purpose of lightweight; then perform knowledge distillation and set new loss functions (pixel-level MSE loss, perceptual loss, triple loss, GKA loss, wavelet loss); input the training data into the teacher model and the student model, and use the characteristics of the teacher model in the first stage as the soft label of the student model. Learning is carried out on the basis of the teacher model. During the learning process, the student model's own performance is continuously optimized and improved to ensure the quality of the generated images.

[0022] The third stage: evaluate the performance of the student model and conduct a comparative test with the teacher model; then deploy the trained student model to the actual application scenario for face image generation.

[0023] In this embodiment, in the first stage, the existing StarGAN v2 model is directly selected as the teacher model, and the dataset CelebA-HQ is selected to train the teacher model. The dataset contains 30,000 high-resolution (1024*1024) face images, each of which is equipped with 40 binary attribute labels; in the raw image data preprocessing stage, it mainly includes image scaling, data normalization, data enhancement, etc.

[0024] Specifically, Figure 1 As shown, in this embodiment, the steps of preprocessing the original image data are as follows: The original image is scaled to unify the pixels; then the data is normalized to stabilize the distribution; finally, in order to enhance the robustness of the teacher model and increase the diversity of the dataset, different regions of the image are cropped and the cropped regions are scaled to simulate different shooting scenes or changes in viewing angles of the target.

[0025] The area of ​​the cropped region is ; in, is the scaling factor, H and W are the length and width of the input image, H⋅W is the area of ​​the input image, and the width of the cropped area is and high And the ratio for: ; ; ; in, .

[0026] In this embodiment, a custom lambda function is applied to perform the cropping operation during cropping, that is, whether to crop the image is determined based on a certain probability, so as to help the model better adapt to different image inputs and thus improve the training effect; specifically, if the generated random number is less than the cropping probability prob (prob=0.5), cropping is performed, otherwise no cropping is performed and the original image is returned.

[0027] Furthermore, in this embodiment, as a conditional cropping operation, it is packaged into transforms.Lambda so that it can be added to transform as part of the preprocessing step, which allows images to be randomly cropped during training, enhancing the diversity of data and thus improving the generalization ability of the model.

[0028] Then randomly flip the image horizontally, swapping the left and right pixels. Then map each pixel value in the image from the integer range of [0, 255] to the floating range of [0.0, 1.0] and normalize it to fit the training model of the teacher network. The formula is as follows: ; in, , , is the pixel value of the original image, is the converted Tensor value; finally, each channel of the image is standardized to make the image pixel value conform to a certain mean and standard deviation. Assuming that the mean of the kth channel of the input image is , the standard deviation is , the standardized pixel formula is:

[0029] in, is the pixel value after conversion to Tensor, is the normalized pixel value.

[0030] After the original image data is preprocessed, this embodiment inputs the processed image into the teacher model for training, and the size of the input image is (batch_size, 3, H, W). The target domain label (Target Domain Label) is usually a multi-hot encoded vector, representing the target domain (different facial styles). In the embedding process of the target domain label, the target domain label maps the multi-hot encoded vector to a low-dimensional space through an Embedding Layer (fully connected layer) to reduce complexity. The Embedding Layer maps each class to a dense low-dimensional vector, which more efficiently represents the information between categories and captures the "semantic" similarity between the corresponding vectors of different data images; then the embedded domain label is feature fused with the input image (occurs in the initial layer of the generator) to form a combination of information between the input image and the target domain label, and this information is input into the generator of the teacher model. The generator will use the fused latent vector (including domain label information) to generate a conditioned image. In the network structure of the generator, the latent vector (the result after embedding and concatenation with the domain label) is passed as input to the initial convolution layer, which maps the input image and the latent vector to a higher dimension, followed by downsampling (encoding) and upsampling (decoding). Finally, a convolution layer is used to map the number of channels back to the RGB image to generate the image.

[0031] Specifically, after the data flow enters the generator of the teacher model, it will pass through 4 residual blocks. Each residual block includes convolution, activation, and normalization operations. These operations together complete the downsampling operation. The main purpose is to gradually reduce the spatial resolution and increase the number of feature map channels, which helps to extract higher-level features while maintaining the content information of the image.

[0032] In this embodiment, the formula for the downsampling convolution operation is: ;in, is the feature map, is the convolution kernel, is the bias term, is the output of the convolution operation, the convolution kernel w On the input image Slide up to perform a dot product operation and calculate the output at each position. s Control the sliding pace of the convolution kernel. During downsampling, the stride s Set to 1 to reduce the spatial resolution of the output image, ensuring that the convolution operation does not cause too much loss of input size; padding Set to 1 to control the edges of the input image to maintain the spatial dimensions of the output image.

[0033] The activation function mainly uses the LeakyReLU function, which aims to improve the nonlinear expression ability of the network. The activation function formula is as follows:

[0034] Among them, a = 0.2 is a negative slope. When the input value is less than zero, the output value is 0.2 times the input. This operation allows a portion of negative gradients to flow, avoiding the "dead neuron" problem of ReLU and maintaining good gradient propagation during training.

[0035] The normalization operation uses Adaptive Instance Normalization (AdaIN), which is used in the generator to adjust the style of the feature map. The input is normalized and adjusted through two operations, namely normalization (Instance Normalization) and style transformation. In Adaptive Instance Normalization, s is the style vector, which is usually generated by another network and is input to the original feature image. x After instance normalization, the scaling factor (gamma) and offset (bet) are calculated through the style vector s, and finally the normalized features are used for scaling and translation.

[0036] In this embodiment, after the teacher model completes the downsampling operation, the generator will start the upsampling stage. During the upsampling process, the network restores the size of the image by gradually increasing the spatial resolution. Each upsampling stage will further extract features through the residual block and use the same structure as the downsampling stage (convolution, activation function, normalization, etc.). The residual block uses a skip connection (also called a residual connection) to add the convolutional and normalized feature map to the input original feature map. The residual connection alleviates the gradient vanishing problem during training.

[0037] Specifically, transposed convolution is used for upsampling. This operation performs upsampling by learning convolution kernels. In fact, it is the inverse process of convolution and is used to map low-resolution feature maps to higher-resolution feature maps. Transposed convolution generates a local area around each pixel to restore spatial details. The size of the feature map is expanded by twice the original size each time until the resolution of the target image is reached to achieve spatial expansion of the feature map. In this way, the generator is able to maintain high-level semantic information of the image while restoring spatial resolution.

[0038] In addition, in order to enhance the learning ability of the generator, the last layer of the generator usually uses a convolutional layer to map the output feature map back to the pixel space of the target image. The output of this layer is usually a generated image with a size of (batch_size, 3, H, W), that is, the output image has the same size as the input image, but has the style characteristics of the target domain.

[0039] In this embodiment, the teacher model is trained to generate high-quality face images, and then the student model is designed based on the pre-trained teacher model. The student model construction strategy is: introduce depthwise separable convolution to replace the traditional convolution in downsampling and upsampling in the teacher model, significantly reducing the number of model parameters and computational complexity; design a lightweight network, and the student model reduces the maximum convolution dimension and hidden layer dimension to design a more lightweight generator student model (Generator Student).

[0040] Specifically, the depthwise separable convolution is introduced in the student model to replace the standard convolution operation used in the teacher model for downsampling and upsampling, and the number of parameters and computational complexity in the convolution operation are optimized. In the depthwise separable convolution, the convolution operation is divided into two steps: depthwise convolution and pointwise convolution. Assume that the input image is , whose size is (height * width * number of input channels), where the depth convolution is to perform a convolution operation on each channel of the image separately. Assuming that the convolution kernel size is K*K, for each input channel , the output of the depthwise convolution for:

[0041] in, is the input image x The pixel value at position (h,w) and channel c in is the weight of the deep convolution kernel. For each channel c, there is a K*K convolution kernel. is the output after depth convolution, and its size is The key to this innovation is that it effectively reduces the amount of computation and parameters by decomposing the traditional convolution operation into two steps: depthwise convolution and point-by-point convolution. It is particularly suitable for resource-constrained devices or scenarios with low computing power.

[0042] In addition, in the traditional StarGAN v2 (Star Generative Adversarial Network), the network structure of the generator is complex and a larger convolution layer is used (a larger residual block (ResBlk) is used in the teacher model to extract style features), and in the student model (Generator Student) after knowledge distillation, a movable residual block (MoblieResBlk) is used instead of the traditional residual block (ResBlk). The MobileStyle Encoder in the student model of this embodiment further reduces the number of convolution layers and uses a smaller number of channels, which makes the extraction process of style information more efficient.

[0043] Specifically, for the student model, this embodiment adjusts the structure of the style encoder (downsampling) and decoder (upsampling), reduces the number of convolutional layers in the network and the number of channels in each layer, and the max_conv_dim used in the teacher model is 512. This parameter is set to 128 in the student model, which means that each layer of convolution operation in the student model has fewer output channels, which can effectively reduce the computational complexity and the number of parameters, achieving a lightweight level. Moreover, the number of convolutional layers in the student model is reduced by 2 layers compared to the teacher model, that is, the number of downsampling and upsampling residual blocks in the student model is reduced to 2, in order to reduce the model complexity and maintain a certain generation capability, improve the computational efficiency of the student model, and reduce computational and memory consumption.

[0044] Finally, in StarGAN v2, the core task of the generator is to map random noise to different styles. In order to generate stylized images, it is necessary to map the vectors in the latent space to the style space through the mapping network. However, the efficiency and diversity of the style mapping network directly affect the performance of the model. In this embodiment, when designing the student model, a mobile mapping network is added after the mapping network, and multiple (specifically 4) fully connected layers (nn.Linear) are designed between the two, and the nonlinear representation ability is enhanced by the ReLU activation function (nn.ReLU()). The Mobile Mapping Network uses a smaller hidden layer dimension (max_hid_dim=128), while the Mapping Network uses a larger hidden layer dimension (max_hid_dim=512). This optimization reduces the amount of computation and memory consumption by limiting the hidden layer dimension.

[0045] After the student model is constructed in this embodiment, knowledge distillation is performed. During the knowledge distillation process, it is necessary to obtain the activation values ​​of each layer of the generator in the forward propagation. The activation values ​​of each layer are the outputs of each layer. The get_activations method can be used to obtain the activation values ​​of each layer. Specifically, the input image is converted into the form of model input; then, the input data (such as image and style information) is forward propagated layer by layer through the encode (encoding) and decode (decoding) modules; each layer is passed through, and the activation value of the layer is saved in the lst list for subsequent loss calculation and distillation process. During the distillation process, the activation values ​​of each layer of the teacher model (t_act) are compared with the activation values ​​of each layer of the student model (s_act) through the loss function (such as GKA or perceptual loss), so that the student model not only learns to generate similar output images from the input image, but also learns the intermediate layer representations of each layer of the teacher model, helping the student model to be closer to the feature representation of the teacher model in the feature space to enhance the generation ability. For example: By minimizing the perceptual loss, the student model adjusts its internal activation (s_act) to try to match the activation of the teacher model (t_act). This process usually causes the student model to adjust its weights, prompting it to learn a representation that is more consistent with the characteristics of the teacher model.

[0046] In order to narrow the gap between the teacher model and the student model, and enable the student model to be lightweight and flexibly switch based on the excellent image generation of the teacher model, this embodiment sets a series of loss functions to narrow the gap between the two.

[0047] The pixel-level MSE loss is added to the classic adversarial loss of the student model based on the mean squared error (MSE) between the output of the student generator and the teacher generator. This loss function measures the pixel difference between the output of the student generator and the output of the teacher generator. By minimizing the pixel-level MSE, the student model can learn pixel-level details directly from the teacher model. The mathematical expression is as follows:

[0048] in, and denote the output images of the student and teacher generators respectively, P is the number of pixels in the image, which helps the output of the student model to be as close to the teacher model as possible, so that the generated quality is consistent with the teacher model.

[0049] When the student model is small or has limited training data, the student model can more effectively learn important information such as image style and details captured by the teacher model through the perceptual loss function, making up for the shortcomings of the student model. The mathematical expression is as follows:

[0050] in, and They represent the feature extraction networks of the teacher model and the student model respectively. The feature extraction network is an evaluation tool used to extract image features to measure whether the image quality between the teacher and student models is similar. The output metric is the L2 norm, and the trained VGG network can be used. N is the number of image samples, that is, the number of images in the batch image; Represents the feature extraction network extracted using the teacher model The high-level features in Represents the feature extraction network extracted using the student model The high-level features in The feature difference between the image generated by the student model and the image generated by the teacher model under the student model feature extraction network; Since the quality of generated images and the perception of the human eye are key, the perceptual loss function encourages the images generated by the student model to be more consistent with the output of the teacher model at the perceptual level, improving the quality and naturalness of the image. It emphasizes the high-level visual features of the image, rather than just the pixel-level similarity. This is especially important for image generation tasks.

[0051] To ensure the real image Generate images with teachers The distance between them is smaller than the real image Generate images with students The distance between them helps the student model learn better generation capabilities to improve the quality of generated images. This embodiment introduces triplet loss. The mathematical expression is as follows:

[0052] in, represents the feature extraction function, Represents the margin of triplet loss. In the triplet loss setting, each training sample consists of a triplet: (Anchor, Positive, Negative), where the anchor (Anchor, e) is a sample in the input data, usually a benchmark sample in the positive sample. The positive sample (Positive, p) belongs to the same category as the anchor sample or has similar features. The purpose is to let the model learn that the features of similar samples should be closer. The negative sample (Negative, n) belongs to a different category or has different features from the anchor sample. The purpose is to let the model learn that the features of dissimilar samples should be far away from the anchor.

[0053] In order to enable the student model to better learn the teacher network, this embodiment adopts a loss function for knowledge distillation. The main goal is to make the student model better imitate the intermediate features of the teacher model through feature comparison and alignment. It combines feature distillation and global kernel alignment (GKA) methods to optimize the feature transfer process. The mathematical expression of the Transfer Features Distillation GKA Loss function is:

[0054] Among them, X and Y are the outputs of the student and teacher models at a certain layer, N is the number of image samples, and T represents the transposed matrix. This method emphasizes transferring features by minimizing the L2 (also called Euclidean distance) difference between layer outputs. It solves channel differences through learnable layers and measures output similarity through global kernel alignment (GKA), which can be directly compared.

[0055] In order to enhance the details in the student-generated image, minimize the difference in high-frequency details, and solve the limitations of traditional GAN, this embodiment introduces discrete wavelet transform loss (DWT). Wavelet transform is a signal processing technique that allows the characteristics of the signal to be analyzed at different scales (image resolution) by decomposing the signal (the image in the present invention) into components of different frequencies. The main operation is first to perform wavelet decomposition, decomposing the input image (generated image and real image) into multiple subbands (subbands) through discrete wavelet transform, usually including low-frequency (LL) and high-frequency (LH, HL, HH) components. The low-frequency subband contains the overall structural information of the image, and the high-frequency subband contains detail information such as edges, textures, etc. Secondly, the loss is calculated. For the generated image and the real image, the difference between the high-frequency parts after wavelet transform is calculated respectively. These high-frequency components have an important influence on the details and edges of the image. By minimizing the difference in these high-frequency parts, the generator is able to retain the details of the image without only generating smooth rough structures. Finally, optimization is performed. In this way, the generator can focus on improving the details during training, reduce the difference between the generated image and the real image in high-frequency features such as texture and edge, and thus generate a more realistic and detailed image. The mathematical expression of discrete wavelet transform (DWT) is as follows:

[0056] in, and Represents the teacher and student images after wavelet transformation. i High frequency details.

[0057] This embodiment proposes a feature-based wavelet loss (Wavelet-based Feature Loss), which combines the intermediate feature extraction between frequency components with the GKA metric of high-frequency component similarity. By applying discrete wavelet transform (DWT) to the generated image and intermediate feature map, high-frequency components can be effectively compared and the difference in the number of channels between models can be accommodated. Feature-based wavelet loss The mathematical expression is as follows: ; in, represents the feature extraction function, are generated by the teacher model and the student model. i The features extracted from the image are calculated by the feature extraction network (VGG network) of the teacher model and the student model; is the wavelet loss, is the GKA loss; During the knowledge distillation process, the above loss function is minimized to continuously optimize the student model. After that, the trained student model is deployed to the actual application scenario, and the face image is generated using the student model.

[0058] In this embodiment, in order to evaluate the association between the teacher model and the student model, the traditional Fréchet distance (FID) and perceptual similarity (LPIPS) are used to evaluate the gap between the two models. Fréchet (FID) is used to measure the gap between the generated image and the real image. It is based on the feature space of the Inception network and compares the generated image and the real image by calculating the distance between the statistical properties (mean and covariance). More specifically, FID calculates the Frechet distance between the feature distributions of the two sets of images. The mathematical expression of FID is as follows:

[0059] in, and are the means of the features of the real image and the generated image, respectively. and are the covariance between the real image and the generated image, Represents the trace of the matrix.

[0060] Perceptual similarity (LPIPS) is more than just pixel-level similarity. It learns the perceptual differences between images by training a deep network that measures the similarity of images by comparing the differences in image features. LPIPS simulates the perceptual process of the human visual system, so it has more advantages than traditional pixel-level indicators (such as MSE or PSNR) when evaluating the subjective quality of images. The lower the LPIPS value, the more similar the two images are in perception. The specific evaluation results of the student model and the teacher model in this embodiment are shown in Table 1.

[0061] Table 1 shows the evaluation results of the student model and the teacher model. Evaluation Metrics Reference Teacher Student Trend FID 23.84±0.03 13.73 11.51 2.22 LPIPS 0.39 0.45 0.418 0.032 Elapsed Time \ 49min 40.5min 8.5min In this embodiment, Figure 5 It shows the sample graph generated by the teacher model. Figure 6 The sample graph generated by the student model is shown. From the observation point of view, the student model perfectly possesses the excellent generation performance of the teacher model, and the colors are more vivid, the texture is clear, the saturation is higher, and the changes in the face image are more obvious. From the experimental data, the FID score (Frechet distance) shows the intersection of the generated fake image and the real image, that is, the larger the FID, the closer the fake image is to the real image, and the smaller the face diversity; and the LPIPS parameter shows the perceived similarity of the generated image. The lower the LPIPS, the higher the image perceived similarity and the better the quality; the student model dropped by 2.22 in the FID score comparison and dropped by 0.032 in the LPIPS comparison, and the time was shortened by 8.5min, as shown in Table 1. In summary, it can be seen that the student model trained by the present invention has achieved both the excellent generation technology of the teacher model and the purpose of lightweight, and it has certain advantages over the teacher model in terms of image quality and speed.

[0062] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art of the present invention, according to the idea of ​​the present invention, some simple deductions, deformations or substitutions can be made. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A lightweight face image generation method based on improved StarGAN and knowledge distillation, characterized in that: The method comprises the following steps: Step 1. Pre-train the teacher model, where the teacher model adopts the StarGAN v2 model; Step 2. Construct a student model, and then perform knowledge distillation; wherein the student model is obtained after improvement on the basis of the teacher model, and the student model uses depthwise separable convolution to replace the standard convolution operation in downsampling and upsampling in the teacher model, and uses movable residual blocks to replace the residual blocks of downsampling and upsampling in the teacher model, and the movable residual blocks of downsampling and upsampling in the student model are set to 2, which reduces the number of convolutional layers and the number of channels of each layer, and the max_conv_dim in the student model is 128; in addition, when designing the student model, a movable mapping network is added after the mapping network, and multiple fully connected layers are designed between the two, and the nonlinear representation capability is enhanced by the ReLU activation function, the hidden layer dimension of the movable mapping network is max_hid_dim=128, and the hidden layer dimension of the mapping network is max_hid_dim=512; When performing knowledge distillation, first set the loss function, then input the training data into the teacher model and the student model for knowledge distillation. During the knowledge distillation process, obtain the activation values ​​of each layer of the generator in the forward propagation, use the pre-trained teacher model feature output as the soft label of the student model, minimize the loss function, so that the student model not only learns to generate similar output images from the input image, but also learns the intermediate layer representations of each layer of the teacher model, helping the student model to be closer to the feature representation of the teacher model in the feature space, and continuously optimizing and improving the performance of the student model itself; Among them, the loss function includes pixel-level MSE loss , Perceptual Loss , triplet loss , GKA loss , feature-based wavelet loss ; ； in, and denote the output images of the student and teacher generators respectively, P is the number of pixels of the image; ； in, and Represent the feature extraction networks of the teacher model and the student model respectively, N is the number of image samples, that is, the number of images in the batch image; Represents the feature extraction network extracted using the teacher model The high-level features in Represents the feature extraction network extracted using the student model The high-level features in The feature difference between the image generated by the student model and the image generated by the teacher model under the student model feature extraction network; ； in, represents the feature extraction function, is a real image, Represents the margin of triplet loss. In the setting of triplet loss, each training sample consists of a triplet: (Anchor, Positive, Negative), where Anchor is an anchor point, which is a sample in the input data. Positive is a sample that belongs to the same category or has similar features as the anchor point sample. Negative is a sample that belongs to a different category or has different features from the anchor point sample. ； Among them, X and Y are the outputs of the student and teacher models at a certain layer, respectively. N is the number of image samples, T represents the transposed matrix; ； in, represents the feature extraction function, are generated by the teacher model and the student model. i The features extracted from the image are calculated by the feature extraction network of the teacher model and the student model; is the wavelet loss, is the GKA loss; ； in, and Represents the teacher and student images after wavelet transformation. i High-frequency details; Step 3. Deploy the trained student model to the actual application scenario to generate face images.

2. According to claim 1, a lightweight face image generation method based on improved StarGAN and knowledge distillation is characterized in that: Step 1: When pre-training the teacher model, the original image needs to be pre-processed. The specific steps are: scale the original image to unify the pixels; then normalize the data to stabilize the distribution; then crop different areas of the image and scale the cropped areas; finally, randomly flip the image horizontally, swap the left and right pixels during flipping, and then map each pixel value in the image from the integer range of [0, 255] to the floating range of [0.0, 1.0], and normalize it to make it suitable for the training model of the teacher network.

3. The lightweight face image generation method based on improved StarGAN and knowledge distillation according to claim 1, characterized in that: The feature extraction network of the teacher model and the student model uses the trained VGG network.

4. The lightweight face image generation method based on improved StarGAN and knowledge distillation according to claim 2, characterized in that: When cropping images, it is allowed to crop images randomly during training, and the cropping probability prob=0.5 is set. If the generated random number is less than the cropping probability, cropping is performed, otherwise no cropping is performed and the original image is returned.

Images