A compression method of a generative adversarial network model based on reserved factor pruning

Abstract

The application discloses a compression method of a generative adversarial network model, and belongs to the technical field of artificial intelligence, which comprises the following steps: given a pre-training teacher generative adversarial network model and a to-be-compressed student generative adversarial network model, the pre-training teacher generative adversarial network model comprises a teacher generator and a teacher discriminator, and the to-be-compressed student generative adversarial network model comprises a student generator and a student discriminator; a learnable reservation factor is set for each convolution kernel of the student generator, and the reservation factor is converted into a corresponding forward operation participation ratio of the convolution kernel according to a boundary parameter; the convolution kernel with a zero forward operation participation ratio is determined by adding a constraint condition and a shrinkage boundary parameter to the reservation factor; the convolution kernel with a zero forward operation participation ratio is removed, and a lightweight student generator is obtained; the student generator is fine-tuned according to the knowledge in the teacher generator and the teacher discriminator; the student discriminator is removed, the student generator is reserved, and the student generative adversarial network model is deployed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a compression method of a generative adversarial network model based on reserved factor pruning. BACKGROUND

[0002] In recent years, with the rapid development of artificial intelligence, neural networks have rapidly been applied in computer vision, natural language processing, pattern recognition and other fields. Alex Krizhevsky et al. designed AlexNet neural network to achieve a breakthrough on the ImageNet picture set of millions of orders of magnitude, which has a wide influence on the development of neural networks in recent years. AlexNet contains more than 60 million floating point operations, and more than one billion floating point operations are required to process a color image with a length and width of 224 pixels. The subsequently developed VGG-16 neural network further improves the prediction accuracy on the ImageNet dataset, but it is a huge network model with a size of more than 500M and more than 28 billion floating point operations required for single prediction. The pursuit of prediction accuracy has given birth to the ResNet neural network with 1001 layers. The rapid development of neural networks cannot be separated from the powerful parallel computing capability of GPU graphics cards. The power consumption of the current mainstream GPU graphics card is 275W, which is equivalent to consuming the battery of an iPhone X in 270 seconds. High power consumption also brings the power consumption of the additional cooling system. The bottlenecks of computing power and high power consumption limit the application of neural networks to more resource-constrained devices, such as mobile phones, embedded hardware and other devices. These devices cannot bear these powerful neural networks due to the harsh restrictions of battery capacity or computing power. In addition, generative adversarial networks (GANs) have made great achievements in the field of image processing and have many interesting applications in image generation, such as image processing, image editing, style transfer, super-resolution, etc. These applications have gradually penetrated into people's entertainment life. However, compared with traditional convolutional neural networks (CNNs), GANs have taken another step in computing and parameter quantity. For example, the classic CycleGAN needs more than 56.8G MACs (Multiply-Accumulate Operations) to process an input picture of 256*256 size, which is 13 times the computing amount of ResNet-50; in addition, the classic Pix2Pix network needs 54.4M parameter quantity, which is close to 2 times the parameter quantity of ResNet-50. Therefore, how to obtain a lightweight generative adversarial network model is also an urgent problem to be solved.

[0003] In summary, the prior art generative adversarial network model has too many parameters and a large amount of calculation, resulting in low operation efficiency. SUMMARY

[0004] The embodiment of the application aims to provide a compression method of a generative adversarial network model based on reserved factor pruning, which can solve the technical problem of the prior art generative adversarial network model having too many parameters and a large amount of calculation, resulting in low operation efficiency.

[0005] To solve the above technical problems, the application is implemented as follows:

[0006] First aspect:

[0007] The embodiment of the application provides a compression method of a generative adversarial network model based on reserved factor pruning, comprising:

[0008] S101: Given a pre-trained teacher generative adversarial network model and a student generative adversarial network model to be compressed, the pre-trained teacher generative adversarial network model comprises a teacher generator and a teacher discriminator, and the student generative adversarial network model to be compressed comprises a student generator and a student discriminator;

[0009] S102: Set a learnable reserved factor for each convolution kernel of the student generator, and convert the reserved factor into a corresponding forward operation participation ratio of the convolution kernel according to a boundary parameter;

[0010] S103: Determine the convolution kernel with a forward operation participation ratio of zero by adding a constraint condition and a shrinkage boundary parameter to the reserved factor;

[0011] S104: Remove the convolution kernel with a forward operation participation ratio of zero to obtain a lightweight student generator;

[0012] S105: Fine-tune the student generator according to the knowledge in the teacher generator and the teacher discriminator;

[0013] S106: Remove the student discriminator, retain the student generator, and deploy the student generative adversarial network model.

[0014] Second aspect:

[0015] The embodiment of the application provides a compression device of a generative adversarial network model based on reserved factor pruning, comprising:

[0016] A given module is configured to give a pre-trained teacher generative adversarial network model and a student generative adversarial network model to be compressed, the pre-trained teacher generative adversarial network model comprising a teacher generator and a teacher discriminator, and the student generative adversarial network model to be compressed comprising a student generator and a student discriminator;

[0017] The setting module is configured to set a learnable reservation factor for each convolution kernel of the student generator, and convert the reservation factor into a participation proportion of the corresponding convolution kernel in forward operation according to a boundary parameter.

[0018] The determining module is configured to determine the convolution kernel with the participation proportion of zero in the forward operation by adding a constraint condition to the reservation factor and shrinking the boundary parameter.

[0019] The removing module is configured to remove the convolution kernel with the participation proportion of zero in the forward operation, and obtain the lightweight student generator.

[0020] The training module is configured to fine-tune the student generator according to the knowledge in the teacher generator and the teacher discriminator.

[0021] The deploying module is configured to remove the student discriminator, retain the student generator, and deploy the student generative adversarial network model.

[0022] In the embodiment of the present application, a learnable reservation factor is set for each convolution kernel of the student generator, and the reservation factor is converted into a participation proportion of the corresponding convolution kernel in forward operation according to a boundary parameter. The convolution kernel with the participation proportion of zero in the forward operation is determined by adding a constraint condition to the reservation factor and shrinking the boundary parameter, and is pruned to remove unnecessary convolution kernels. The parameter amount is reduced, the calculation amount is reduced, and the operation efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a flowchart of a compression method of a generative adversarial network model based on reservation factor pruning provided by the embodiment of the present application.

[0024] The implementation of the object, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0025] In order to make the object, technical scheme and advantages of the present application more clear, the technical scheme in the embodiment of the present application will be described clearly and completely below with reference to the drawings in the embodiment of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0026] The compression method and compression device of the generative adversarial network model provided by the embodiment of the present application will be described in detail below with reference to the drawings and specific embodiments and application scenarios.

[0027] Embodiment one

[0028] ReferenceFigure 1 Fig. 1 shows a flowchart of a compression method of a generative adversarial network model based on reserved factor pruning provided by an embodiment of the present application.

[0029] The compression method of the generative adversarial network model based on reserved factor pruning provided by the embodiment of the present application comprises:

[0030] S101: Given a pre-trained teacher generative adversarial network model and a student generative adversarial network model to be compressed, the pre-trained teacher generative adversarial network model comprises a teacher generator and a teacher discriminator, and the student generative adversarial network model to be compressed comprises a student generator and a student discriminator.

[0031] S102: Set a learnable reserved factor for each convolution kernel of the student generator, and convert the reserved factor into a participation ratio in forward operation of the corresponding convolution kernel according to a boundary parameter.

[0032] In a possible implementation, S102 specifically comprises:

[0033] S1021: Convert the reserved factor into the participation ratio in forward operation of the corresponding convolution kernel by formula 1:

[0034]

[0035] Wherein, p ij represents the reserved factor, b represents the boundary, ij represents the jth convolution kernel of the ith convolution layer, m ij represents the participation ratio in forward operation.

[0036] S103: Determine the convolution kernel with a participation ratio in forward operation of zero by adding a constraint condition to the reserved factor and shrinking the boundary parameter.

[0037] Wherein, the constraint condition is added to the reserved factor during the training process, so that the participation ratio in forward operation converted from the reserved factor constantly approaches zero.

[0038] Specifically, adding the constraint condition means adding the absolute value of the reserved factor to the optimization objective function.

[0039] Wherein, the boundary b is constantly shrunk during the training process, so that the variable region can become smaller until zero, at which time m ij becomes a step function with a non-zero value of one.

[0040] In a possible implementation, S103 specifically comprises:

[0041] S1031: Convert the output of the convolution kernel from O ij to O′ ij :

[0042] O ij =m ij *O ij Formula 2 wherein O ij represents the original output of the convolution kernel, O' ij represents the real output of the convolution kernel.

[0043] In a possible implementation, S103 specifically includes:

[0044] S1032: adding a constraint to the reservation factor by Formula 3, so that the participation ratio in the forward operation continuously approaches zero, and the reservation factor is close to the boundary:

[0045]

[0046] S1033: shrinking the boundary parameter by Formula 4, and continuously reducing the boundary parameter until it is zero:

[0047]

[0048] wherein e represents the current iteration number, and E represents the total iteration number.

[0049] S104: removing the convolution kernel with the participation ratio in the forward operation being zero, to obtain a lightweight student generator.

[0050] Optionally, the number of training times can be set in advance, and when the number of training times reaches the number set in advance, the pruning on the convolution kernel with the participation ratio in the forward operation being zero is stopped.

[0051] S105: fine-tuning the student generator according to the knowledge in the teacher generator and the teacher discriminator.

[0052] In a possible implementation, S105 specifically includes:

[0053] S1051: for the i-th layer feature map O i , the attention map of the i-th layer feature map O i is obtained by adding channel by channel:

[0054]

[0055] S1052: extracting the attention maps in the teacher generator G T and the teacher discriminator and fusing them to obtain:

[0056]

[0057] wherein G T represents the teacher generator, and D T represents the teacher discriminator. denotes the fusion attention map;

[0058] S1053: The student generator imitates the extracted fusion attention map The corresponding optimization target is:

[0059]

[0060] S1054: The image generated by the student generator is input into the last layer of the teacher discriminator, and the output is approximated to the image generated by the teacher generator. The corresponding optimization target is:

[0061]

[0062] wherein, denotes the output of the last layer of the teacher discriminator;

[0063] S1055: The optimization target of the student generator is:

[0064]

[0065] wherein, is the basic loss function of the generative adversarial network, and alpha, beta and gamma are hyperparameters for balancing different loss terms.

[0066] S106: Remove the student discriminator, retain the student generator, and deploy the student adversarial network model.

[0067] In the embodiment of the application, a learnable retention factor is set for each convolution kernel of the student generator. The retention factor is converted into a participation ratio of the corresponding convolution kernel in forward operation according to the boundary parameter. By adding a constraint condition to the retention factor and shrinking the boundary parameter, the convolution kernel with a participation ratio of zero in forward operation is determined and pruned, and unnecessary convolution kernels are removed. The parameter amount is reduced, the calculation amount is reduced, and the operation efficiency is improved.

[0068] Further, the embodiment of the application also provides a simulation experiment to verify the superiority of the compression method of the generative adversarial network provided by the application.

[0069] Simulation conditions: The application is developed on a GTX 1080Ti GPU environment, and the developed deep learning framework is based on Pytorch. The language mainly used in the application is Python.

[0070] Simulation content: we use CycleGAN to conduct experiments on unpaired datasets Horse2zebra and Summer2winter, and the size of all input images is adjusted to 256*256 pixels for training. We use FID (Frechet Inception Distance) to evaluate the quality of generated images, and the lower the FID value, the closer the quality of generated images to the original images. In addition, we also conduct experiments on image-to-image translation tasks, and use the classic Pix2Pix model to translate paired images, respectively on Edges2shoes and Cityscapes datasets, and all images are adjusted to 256*256 pixels for training. For the Edge2shoes dataset, we still use the FID index to evaluate the quality of generated images. For the Cityscpes dataset, we run DRN-D-105 to calculate the mean intersection over union (mIoU) to evaluate the quality of generated images, and the higher the score, the better the image generation performance. A large number of experimental verification proves the effectiveness and advancement of the proposed method.

[0071] Referring to Table 1, Table 1 shows a compression effect comparison table of each compression method based on the Pix2Pix model provided by the embodiment of the application.

[0072] Table 1

[0073]

[0074] Referring to Table 2, Table 2 shows a compression effect comparison table of each compression method based on the CycleGAN model provided by the embodiment of the application.

[0075] Table 2

[0076]

[0077] Embodiment two

[0078] The compression device for the generative adversarial network model based on the reserved factor pruning provided by the embodiment of the application comprises:

[0079] The compression device 20 for the generative adversarial network model based on the reserved factor pruning provided by the embodiment of the application comprises:

[0080] The given module 201 is configured to give a pre-trained teacher generative adversarial network model and a to-be-compressed student generative adversarial network model, the pre-trained teacher generative adversarial network model comprising a teacher generator and a teacher discriminator, and the to-be-compressed student generative adversarial network model comprising a student generator and a student discriminator.

[0081] The setting module 202 is configured to set a learnable reservation factor for each convolution kernel of the student generator, and convert the reservation factor into a participation proportion of the corresponding convolution kernel in the forward operation according to a boundary parameter;

[0082] The determining module 203 is configured to determine the convolution kernel with a participation proportion of zero in the forward operation by adding a constraint condition to the reservation factor and shrinking the boundary parameter;

[0083] The removing module 204 is configured to remove the convolution kernel with the participation proportion of zero in the forward operation, and obtain the lightweight student generator;

[0084] The training module 205 is configured to fine-tune the student generator according to the knowledge in the teacher generator and the teacher discriminator;

[0085] The deploying module 206 is configured to remove the student discriminator, retain the student generator, and deploy the student generative adversarial network model.

[0086] Optionally, the setting module 202 specifically includes:

[0087] The conversion submodule is configured to convert the reservation factor into the participation proportion of the corresponding convolution kernel in the forward operation by using Formula 1:

[0088]

[0089] wherein p ij represents the reservation factor, b represents the boundary, ij represents the jth convolution kernel of the ith convolution layer, and m ij represents the participation proportion in the forward operation.

[0090] Optionally, the determining module 203 specifically includes:

[0091] The conversion submodule is configured to convert the output of the convolution kernel from O ij to O′ ij :

[0092] O′ ij = m ij *O ij Formula 2 wherein O ij represents the original output of the convolution kernel, and O′ ij represents the real output of the convolution kernel.

[0093] Optionally, the determining module 203 specifically includes:

[0094] The constraint submodule is configured to add a constraint condition to the reservation factor by using Formula 3, so that the participation proportion in the forward operation tends to zero, and the reservation factor is close to the boundary:

[0095]

[0096] The shrinking submodule is used to shrink the boundary parameters using Formula 4, continuously reducing the boundary parameters until they become zero:

[0097]

[0098] Where e represents the current iteration number and E represents the total number of iterations.

[0099] Optionally, training module 205 specifically includes:

[0100] The addition submodule is used to perform addition on the i-th layer feature map O. i The feature map O of the i-th layer is obtained by adding each channel sequentially. i Attention map:

[0101]

[0102] Extraction submodule, used to extract the teacher generator G T The attention map from the teacher discriminator is fused to obtain:

[0103]

[0104] Among them, G T D represents the teacher generator. T This indicates a teacher discriminator. This represents the fusion attention map;

[0105] The imitation submodule is used for the fusion attention map extracted by the student generator. The corresponding optimization objective is:

[0106]

[0107] The output submodule takes the image generated by the student generator as input and outputs it in the last convolutional layer of the teacher discriminator to approximate the image generated by the teacher generator. The corresponding optimization objective is:

[0108]

[0109] in, This indicates that the output of the last convolutional layer of the teacher discriminator is being returned.

[0110] The defined submodule is used to determine the optimization objective of the student generator.

[0111]

[0112] in, is the basic loss function of the generative adversarial network, and alpha, beta and gamma are hyperparameters used to balance different loss terms.

[0113] The data monitoring device 20 provided by the embodiments of the present application can realize the various processes realized in the method embodiments, and thus details are not repeated here.

[0114] In the embodiments of the present application, a learnable reservation factor is set for each convolution kernel of the student generator, the reservation factor is converted into a participation forward operation ratio of the corresponding convolution kernel according to a boundary parameter, the participation forward operation ratio of the convolution kernel is determined to be zero by adding a constraint condition to the reservation factor and contracting the boundary parameter, and the convolution kernel is pruned to remove unnecessary convolution kernels. The parameter amount is reduced, the calculation amount is reduced, and the operation efficiency is improved.

[0115] The virtual device in the embodiments of the present application can be a device, or a component, an integrated circuit, or a chip in a terminal.

[0116] The above only describes the embodiments of the present application and is not used to limit the present application. The present application can have various changes and modifications for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A compression method of a generative adversarial network model based on reserved factor pruning, characterized in that, Comprise: S101: Given a pre-training teacher generative adversarial network model and a student generative adversarial network model to be compressed, the pre-training teacher generative adversarial network model comprises a teacher generator and a teacher discriminator, and the student generative adversarial network model to be compressed comprises a student generator and a student discriminator; S102: Set a learnable retention factor for each convolution kernel of the student generator, and convert the retention factor into the participation proportion of the corresponding convolution kernel in forward operation according to a boundary parameter; S1021: Convert the retention factor into the participation proportion of the corresponding convolution kernel in forward operation by formula 1: ； wherein, p ij denotes a retention factor, b denotes a boundary, Ij denotes a first i layer convolutional layer, j convolutional kernel of the first m ij denotes a participation in the forward operation proportion; S103: Determine the convolution kernel whose participation proportion in forward operation is zero by adding a constraint condition to the retention factor and shrinking the boundary parameter; S1032: Add a constraint condition to the retention factor by formula 3, so that the participation proportion in forward operation tends to zero, and the retention factor is aligned to the boundary: ； S1033: Shrink the boundary parameter by formula 4, and continuously reduce the boundary parameter until it is zero: ； wherein e denotes the current iteration number, E denotes the total iteration number; S104: Remove the convolution kernel whose participation proportion in forward operation is zero, and obtain a lightweight student generator; S105: Fine-tune the student generator according to the knowledge in the teacher generator and the teacher discriminator; S106: Remove the student discriminator, retain the student generator, and deploy the student generative adversarial network model; Wherein, the student generative adversarial network model is used for image-to-image translation task, and FID (Frechet Inception Distance) is used to evaluate the quality of generated images; The image translation task includes training based on Cityscapes image dataset; The model performs image generation operation on the deployment platform, and uses FID (Frechet Inception Distance) and calculates the average intersection over union (Mean Intersection over Union, mIoU) to evaluate the quality of generated images; The pruning method is used to reduce the number of generator convolution kernels and reduce the model parameter quantity and calculation quantity.

2. The compression method of claim 1, wherein, S103 specifically includes: S1031: converting the output of the convolution kernel from to : ； wherein, denotes the original output of the convolution kernel, denotes the real output of the convolution kernel.

3. The compression method of claim 1, wherein, The S105 specifically includes: S1051: For the i-th layer feature map , the attention map of the i-th layer feature map is obtained by adding channels one by one. ； S1052: extracting the teacher generator G T and the attention map in the teacher discriminator and fusing them to obtain: ； wherein, G T denotes the teacher generator, D T denotes the teacher discriminator, denotes the fused attention map; S1053: The student generator imitates the extracted fusion attention map , the corresponding optimization target is obtained as: ； S1054: The image generated by the student generator is input into the last layer of convolution of the teacher discriminator for output, so as to approximate the picture generated by the teacher generator, and the corresponding optimization target is: ； wherein, represents the last layer convolution output returned to the teacher discriminator; S1055: Determine the optimization target of the student generator as: ； where, is the basic loss function of the generative adversarial network, α、β and Gamma is a hyperparameter for balancing different loss terms.

4. A compression device of a generative adversarial network model based on reserved factor pruning, characterized in that, Comprise: A given module is used to give a pre-training teacher generative adversarial network model and a student generative adversarial network model to be compressed, the pre-training teacher generative adversarial network model comprises a teacher generator and a teacher discriminator, and the student generative adversarial network model to be compressed comprises a student generator and a student discriminator; A setting module is used to set a learnable retention factor for each convolution kernel of the student generator, and convert the retention factor into the participation proportion of the corresponding convolution kernel in forward operation according to a boundary parameter, the setting module specifically comprises: The conversion submodule is configured to convert the reserved factor into a corresponding participation forward operation proportion of the convolution kernel by Formula 1: ； wherein, p ij denotes a retention factor, b denotes a boundary, ij denotes a first i layer convolutional layer, j denotes a first m ij denotes a participation in the forward operation proportion; The determination module is configured to determine the convolution kernel with the participation forward operation proportion of zero by adding a constraint condition to the reserved factor and shrinking the boundary parameter, and specifically includes: The constraint submodule is configured to add a constraint condition to the reserved factor by Formula 3, so that the participation forward operation proportion is constantly close to zero, and the reserved factor is constantly close to the boundary: ； The shrinking submodule is configured to shrink the boundary parameter by Formula 4, and constantly reduce the boundary parameter until it is zero: ； wherein e denotes the current iteration number, E denotes the total number of iterations; The removal module is configured to remove the convolution kernel with the participation forward operation proportion of zero, and obtain the lightweight student generator; The training module is configured to fine-tune the student generator according to the knowledge in the teacher generator and the teacher discriminator; The deployment module is configured to remove the student discriminator, retain the student generator, and deploy the student generative adversarial network model. The device is suitable for deploying the student generator in an image-to-image translation task, the image translation task includes performing image generation inference on the Horse2Zebra and Cityscapes data sets, and the quality of the generated image can be evaluated in combination with FID (Frechet Inception Distance) and the calculation of the average intersection over union (Mean Intersection over Union, mIoU).

5. The compression device of claim 4, wherein, The determination module specifically includes: a transforming sub-module, configured to transform the output of the convolution kernel from to : ； wherein, denotes the original output of the convolution kernel, denotes the real output of the convolution kernel.

6. The compression device of claim 4, wherein, The training module specifically includes: an adding sub-module, configured to add the feature maps of the i-th layer in each channel to obtain an attention map of the i-th layer feature map of the i-th layer feature map ​ ； extracting sub-module, configured to extract the teacher generator G T and the attention map in the teacher discriminator are fused to obtain ； wherein, G T denotes the teacher generator, D T denotes the teacher discriminator, denotes the fusion attention map; a mimic sub-module configured to cause the student generator to mimic the extracted fusion attention map and the corresponding optimization objective is obtained as: ； The output submodule is configured to input the image generated by the student generator into the last layer of convolution of the teacher discriminator for output, so as to approximate the picture generated by the teacher generator, and the corresponding optimization target is: ； wherein, denotes the last layer convolution output returned to the teacher discriminator; The determination submodule is configured to determine that the optimization target of the student generator is: ； wherein, is the basic loss function of the generative adversarial network, α、β and γ is a hyperparameter for balancing different loss terms.

Images