Lightweight method of SAR optical image mapping model based on conditional generative adversarial network
By using neural architecture search and knowledge distillation technology to perform channel pruning in the SAR optical image mapping model, the problem of excessive feature differences between SAR images and optical images and instability in the generative adversarial network training is solved, and the model is lightweight and performance retention is achieved.
Patent Information
- Application Number
- CN202111429672.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-29
- Publication Date
- 2025-04-04
- Estimated Expiration
- 2041-11-29
AI Technical Summary
The prior art is difficult to effectively solve the problem of excessive feature differences between SAR images and optical images and unstable training of generative adversarial networks, resulting in poor results of conventional compression methods.
Neural architecture search is used for channel pruning, and supplemented with knowledge distillation operation, a SAR optical image mapping model based on condition-generating adversarial network is built to solve the problems of excessive differences and instability in training, while compressing network parameters without loss of performance.
It realizes significant compression of network parameters without losing performance, improves the lightweightness of the model, and improves the performance quality of the model in SAR image to optical image mapping tasks.
Smart Images
Figure CN114202017B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of model compression, and in particular relates to a lightweight method for SAR optical image mapping model based on a conditional generative adversarial network. Background Art
[0002] In recent years, remote sensing technology has been continuously developing. Synthetic Aperture Radar (SAR) has been continuously used as an important means of earth observation. Thanks to its use of microwave bands, it can detect all day and all weather. However, due to its special imaging mechanism, the obtained SAR images are difficult to recognize. Although optical remote sensing is subject to weather and light, its imaging results are easy to identify. Therefore, it will be of great significance to be able to map the easily acquired SAR images to easily recognizable optical images.
[0003] Generative adversarial networks (GANs) have achieved results in many fields and have become the preferred method for generative tasks. As the name of the generative adversarial network indicates, this method is a generative model that is mainly used to generate data that conforms to the target distribution; the word "adversarial" refers to the game between the generator and the discriminator contained in this method. The generator wants to generate better data, and the discriminator wants to better judge the authenticity of the generated data. The two compete with each other and promote each other to obtain better generated target data. However, conventional generative adversarial networks ignore the huge feature differences between SAR images and ordinary optical images and the large amount of high-frequency multiplicative noise contained in SAR images, and often cannot achieve satisfactory results.
[0004] The conditional generative adversarial network has achieved good performance in the task of mapping SAR images to optical images. The parameters of its network structure are relatively large. When considering only its generator, the amount of calculation is one order of magnitude higher than some actual deployments of target recognition networks. Since the generative adversarial network contains generators and discriminators, the training process is unstable compared to conventional convolutional neural networks, and due to the particularity of SAR images themselves, directly using the common compression method of convolutional neural networks will produce poor results. This patent uses neural architecture search for channel pruning, supplemented by knowledge distillation operations to solve the problems of excessive differences and unstable training. Summary of the invention
[0005] Purpose of the invention: The present invention proposes a lightweight method for SAR optical image mapping model based on conditional generative adversarial network, which adopts neural architecture search for channel pruning and is supplemented by knowledge distillation operation to solve the problems of large difference between SAR images and optical images, unstable training of generative adversarial network, and compresses parameters without losing performance.
[0006] Technical solution: The lightweight method of SAR optical image mapping model based on conditional generative adversarial network described in the present invention comprises the following steps:
[0007] (1) Construct a generative adversarial network consisting of a discriminant network and a generative network. Use a paired SAR optical image dataset to train the generative adversarial network to obtain a generative network that can map SAR images to optical images. Use the generative network as a teacher network to provide prior knowledge for the student network.
[0008] (2) Replace the standard convolution of the residual mapping module in the generative network with a depth-wise separable convolution, and use it as a supernet containing the results of all subnets. The supernet will serve as a student network to receive the knowledge of the teacher network and is fully trained using the knowledge distillation method to obtain a pre-distilled supernet.
[0009] (3) Use weight sharing to train all subnets in the pre-distillation supernet and update the parameters of each subnet so that each subnet can have good performance;
[0010] (4) Using genetic algorithm to search for the best subnet structure in the supernet;
[0011] (5) Fine-tune the optimal subnetwork structure on the dataset, load the original training parameters, and further train the structure to improve the performance of the generative adversarial network.
[0012] Furthermore, the generation network in step (1) includes an encoding module, a decoding module and a residual mapping module; the encoding module is composed of five sub-units of the same structure, each unit is composed of a convolution, an instance normalization, and a correction rectifier unit, and the input image is downsampled four times to obtain an image feature tensor; the residual mapping module combines a residual learning mechanism to deepen the network depth and obtain a better feature representation; the decoding module decodes the optical features completed by the mapping module into a generated image, which includes four transposed convolution modules and one convolution module; the transposed convolution upsamples the input tensor, and after four transposed convolution operations, it reaches the same length and width as the input image; the identification network adopts a multi-scale discriminator and PatchGAN technology, the multi-scale discriminator includes a high-resolution discriminator and a low-resolution discriminator with the same structure, the original input image and the downsampled image are respectively sent to the two, and then the identification results of the high-resolution image and the low-resolution image are obtained; the PatchGAN technology removes the fully connected layer of the conventional discriminator, and replaces the overall judgment result with the independent authenticity of each N×N size area in the image.
[0013] Furthermore, the loss function of the generative adversarial network in step (1) is described by the following formula:
[0014] min G maxD L(D,G)=min G max D (L GAN +λ VGG L VGG +λ DCT L DCT )
[0015] Among them, L GAN , L VGG , L DCT They are adversarial loss function, neural network feature matching function, discrete cosine loss function, λ VGG and λ DCT They are the weights of the neural network feature matching function and the discrete cosine loss function respectively;
[0016] The adversarial loss function is described as:
[0017] L GAN (X,Y)=E (X,Y) [logD(X,Y)]+E X [log(1-D(X,G(X)))]
[0018] Where X is the input SAR image, Y is the real optical image, G(·) is the generator, D(·) is the discriminator, and E represents the mathematical expectation;
[0019] The neural network feature matching function can be described by the following formula
[0020]
[0021] Among them, φ i (·) is the output feature map of the i-th maximum pooling layer of the VGG19 network, λ i , H i , W i , C i are the weight value, length, width and number of channels of the i-th maximum pooling output, respectively, ||·|| 1 represents the L1 norm;
[0022] The discrete cosine loss function is described as:
[0023] L DCT (X,Y)=||DCT(G(X))-DCT(Y)|| 1
[0024] Wherein, DCT(·) is discrete cosine transform.
[0025] Furthermore, the knowledge distillation process described in step (2) is:
[0026]
[0027] Among them, x is the network input, that is, the input SAR image, f k Transform the convolution module for the kth channel, is the kth intermediate feature output of the student network, is the kth intermediate feature output of the teacher network, and the L2 norm is used to shorten the distance between the intermediate layer output features of the student network and the teacher network;
[0028] Therefore, the overall loss function of the generative adversarial network training is:
[0029] min G max D L(D,G)=min G max D (L GAN +λ VGG L VGG +λ DCT L DCT +λ distil L distil )
[0030] Among them, L distil is the knowledge distillation loss function, λ distil is the weight coefficient of the knowledge distillation loss function.
[0031] Furthermore, the implementation process of step (3) is as follows:
[0032] The supernet contains all subnet parameters. The difference between subnets only exists in the number of channels in each convolutional layer. It is the set of all subnets. If the number of channels in the first convolutional layer of the supernet is C 1 , the number of convolutional layer channels corresponding to any subnet is So And the front of the convolutional layer corresponding to the supernet The channel weight is the weight of the subnet; during each training, a subnet is randomly sampled for forward reasoning and back propagation, and only the parameters of this subnet are updated; due to the coupling between subnet parameters, some parameters of other subnets will also be updated accordingly.
[0033] Furthermore, the implementation process of step (4) is as follows:
[0034] (41) Randomly select P subnets from all subnet structures contained in the supernet as the first population;
[0035] (42) The FID image quality evaluation index is used as the fitness function to evaluate the quality of the images generated by all subnets in the population, and the S subnet structures with the best performance are selected, where S is less than P, as the parents of the next population, and the structures with unsatisfactory performance are eliminated;
[0036] (43) Randomly select a number of pairs of parents from the S selected subnetwork structures to produce offspring and add them to the population;
[0037] (44) Randomly select several subnetworks in the new population for mutation operation, and the generated new structure is added to the population;
[0038] (45) Repeat steps (42) to (44) until the preset termination condition is met, and then select the subnet with the best performance from the final population as the final result.
[0039] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects: The present invention targets the physical properties of SAR images, generates adversarial network structural characteristics, adopts neural architecture search for channel pruning, and is supplemented by knowledge distillation operations, which can solve the problems of large differences between SAR images and optical images, unstable training of generative adversarial networks, and ultimately compresses network parameters without losing performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a lightweight model framework of SAR-optical image mapping model based on conditional generative adversarial network;
[0041] Figure 2 It is a schematic diagram of the internal structure of the depth-wise convolution in the present invention;
[0042] Figure 3 Schematic diagram of the knowledge distillation structure in the present invention;
[0043] Figure 4 It is a schematic diagram of the shared weight training structure in the present invention;
[0044] Figure 5 It is the flow chart of genetic algorithm in the present invention. DETAILED DESCRIPTION
[0045] The present invention is further described in detail below with reference to the accompanying drawings.
[0046] The lightweight method of SAR optical image mapping model based on conditional generative adversarial network mainly includes pre-training, pre-distillation, weight sharing training, substructure search and subnetwork fine-tuning processes, such as Figure 1 As shown, the specific process is as follows:
[0047] Step 1: Construct a generative adversarial network including a discriminative network and a generative network. Use a paired SAR optical image dataset to train the generative adversarial network to obtain a generative network that can map SAR images to optical images. Use the generative network as a teacher network to provide prior knowledge for the student network.
[0048] The loss function designed for the SAR image feature mechanism is used, which can be described by the following formula:
[0049] min G max D L(D,G)=min G max D (L GAN +λ VGG L VGG +λ DCT L DCT )
[0050] Among them, L GAN , L VGG , L DCT They are adversarial loss function, neural network feature matching function, discrete cosine loss function, λ VGG and λ DCT They are the weights of the neural network feature matching function and the discrete cosine loss function respectively.
[0051] The adversarial loss function can be described by the following formula:
[0052] L GAN (X,Y)=E (X,Y) [logD(X,Y)]+E X [log(1-D(X,G(X)))]
[0053] Among them, X is the input SAR image, Y is the real optical image, G(·) is the generator, D(·) is the discriminator, and E represents the mathematical expectation.
[0054] The neural network feature matching function can be described by the following formula:
[0055]
[0056] Among them, φ i (·) is the output feature map of the i-th maximum pooling layer of the VGG19 network, λ i , H i , W i , C i are the weight value, length, width and number of channels of the i-th maximum pooling output, respectively, ||·|| 1 Represents the L1 norm.
[0057] The discrete cosine loss function can be described as:
[0058] L DCT (X,Y)=||DCT(G(X))-DCT(Y)|| 1
[0059] Wherein, DCT(·) is discrete cosine transform.
[0060] The generative network consists of three modules: encoding, decoding, and mapping. The encoding module consists of five sub-units with the same structure, each of which consists of convolution, instance normalization, and rectified rectifier unit (ReLU). The input image is downsampled four times to obtain the image feature tensor. The residual mapping module has 9 identical sub-units, combined with the residual learning mechanism, to deepen the network depth and obtain better feature representation. The decoding module decodes the mapped optical features into a generated image. It contains four transposed convolution modules and one convolution module. The transposed convolution module has the same structure as the sub-unit of the encoding module, except that the convolution operation is replaced by transposed convolution. The transposed convolution upsamples the input tensor, and after four transposed convolution operations, it finally reaches the same length and width as the input image.
[0061] The identification network uses a multi-scale discriminator and PatchGAN technology. The multi-scale discriminator contains a high-resolution discriminator and a low-resolution discriminator with the same structure. The original input image and the downsampled image are sent to the two respectively, and then the identification results of the high-resolution image and the low-resolution image are obtained; PatchGAN technology removes the fully connected layer of the conventional discriminator and replaces the overall judgment result with the independent authenticity of each N×N area in the image.
[0062] The final model will be used as a teacher network to provide high-dimensional feature knowledge to the student network during the distillation process.
[0063] Step 2: Replace the standard convolution of the residual mapping module in the generative network with a depth-wise separable convolution, and use it as a supernet containing the results of all subnets. The supernet will serve as a student network to receive the knowledge of the teacher network and is fully trained using the knowledge distillation method to obtain a pre-distilled supernet.
[0064] Depthwise convolution includes two operations: depthwise convolution and pointwise convolution. In standard convolution, the input feature map needs to be convolved with the convolution kernel of the number of input channels, and then the results are added to form one of the channels in the output feature map. This is repeated the number of output channels to form the output feature map. Figure 2As shown in the figure, depth-wise convolution only convolves a single channel of the feature map with a single convolution kernel, without any weighted addition operation in the middle, and then concatenates all the channels received into the output feature map. While reducing the amount of computation, depth-wise convolution also brings two problems: the number of output channels in standard convolution is determined by the number of convolution kernels, but in depth-wise convolution the number of output channels is always equal to the number of input channels, which means that depth-wise convolution cannot change the number of output channels; at the same time, due to the special convolution method of depth-wise convolution, there is a lack of interaction between the channels of the feature map, and information only flows in the current channel. In order to solve these problems, point-wise convolution is usually used after depth-wise convolution. The essence of point-by-point convolution is 1×1 standard convolution. Point-by-point convolution can freely change the number of channels while also promoting the fusion of features between channels. Finally, a supernet model is obtained that can greatly reduce the number of parameters but maintain the convolution capacity.
[0065] In the pre-distillation process, in order to assist the training of small-scale networks, knowledge distillation is an effective means. Figure 3 As shown in the figure, a knowledge distillation method based on the feature output of the intermediate layer of the teacher network is adopted. The intermediate layer of the generative adversarial network contains higher dimensions and can provide more high-dimensional information to achieve the purpose of improving the quality of the generated images of the student model. The intermediate feature output of the generator is the intermediate output of the residual mapping module. The residual mapping module contains a total of 9 residual sub-modules, so the intermediate features of every three modules are used as output features. Due to the problem of inconsistent number of channels during the training process of the teacher network and the student network, it is necessary to introduce a channel conversion convolution module. The essence of the channel conversion convolution module is a 1×1 standard convolution, which can upgrade the intermediate output features of the student network to the same dimension as the intermediate features of the teacher network. During the training process, the channel conversion module also updates its parameters to achieve higher conversion efficiency without losing the high-dimensional information of the teacher network. The entire knowledge distillation process can be written as the following formula:
[0066]
[0067] Among them, x is the network input, that is, the input SAR image, f k Transform the convolution module for the kth channel, is the kth intermediate feature output of the student network, is the kth intermediate feature output of the teacher network, and the L2 norm is used to shorten the distance between the intermediate layer output features of the student network and the teacher network. According to the characteristics of SAR images, the overall loss function uses the discrete cosine loss function and the neural network feature matching loss function, plus the adversarial loss function of the conditional generative adversarial network. The final optimization goal is:
[0068] min G max D L(D,G)=minG max D (L GAN +λ VGG L VGG +λ DCT L DCT +λ distil L distil )
[0069] Among them, L distil is the knowledge distillation loss function, λ distil is the weight coefficient of the knowledge distillation loss function.
[0070] Step 3: Use weight sharing to train all subnets in the pre-distillation supernet and update the parameters of each subnet so that each subnet can have good performance.
[0071] In the process of training subnets by weight sharing, the supernet contains all subnet parameters and is the set of all subnets. Figure 4 As shown, assuming that the number of channels of the first convolutional layer of the supernet is C 1 , the number of convolutional layer channels corresponding to any subnet is So And the front of the convolutional layer corresponding to the supernet The weight of each channel is the weight of the subnet. During each training, the number of subnet channels is randomly sampled, and the weight of the corresponding subnet is updated through back propagation. It can be seen that there is a lot of overlap in the weights of the subnets, and the weights with the highest number of channels will be shared by a large number of subnets, which shows that these weights play a more important role in the network. During each training, a subnet is randomly sampled for forward reasoning and back propagation, and only the parameters of the subnet are updated. Due to the coupling of parameters between subnets, some parameters of other subnets will also be updated accordingly. With the continuous random sampling during the training process, although many subnets are not sampled, their parameters are constantly updated, and each subnet can perform well on the data set.
[0072] Step 4: Use genetic algorithm to search for the best subnet structure in the supernet. Figure 5 As shown in the figure, the process of genetic algorithm mainly includes initialization conditions, fitness evaluation, screening, crossover, mutation and other processes. Specifically, the process of search algorithm is:
[0073] (1) Randomly select P subnets from all subnet structures contained in the supernet as the first population;
[0074] (2) The FID image quality evaluation index is used as the fitness function to evaluate the quality of the images generated by all subnets in the population, and the S subnet structures with the best performance are selected, where S is less than P, as the parents of the next population, and the structures with poor performance are eliminated;
[0075] (3) Randomly select a number of pairs of parents from the S selected subnetwork structures to produce offspring and add them to the population;
[0076] (4) Randomly select several subnets in the new population for mutation operations, and the resulting new structures are added to the population;
[0077] (5) Repeat steps (2) to (4) until the preset termination condition is met, and then select the subnet with the best performance from the final population as the final result.
[0078] Step 5: Fine-tune the optimal subnet structure on the dataset, load the original training parameters, and further train the structure to improve its performance, and finally obtain a lightweight network structure with almost no performance loss. The final results are shown in Table 1.
[0079] On the optimized SEN1-2 dataset, it can be seen that the number of model parameters is reduced to about one-third of the original after compression, but the generated images have no loss in the FID evaluation index, where the smaller the FID value, the better the quality.
[0080] Table 1 Comparison of the original model and the compressed model results
[0081] Model Parameter quantity FID Original Model 182.44M 140.85 After compression 65.79M 138.66
[0082] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.
Claims
1. A lightweight method for SAR optical image mapping model based on conditional generative adversarial network, characterized in that: The following steps are involved: (1) Construct a generative adversarial network consisting of a discriminant network and a generative network. Use a paired SAR optical image dataset to train the generative adversarial network to obtain a generative network that can map SAR images to optical images. Use the generative network as a teacher network to provide prior knowledge for the student network. (2) Replace the standard convolution of the residual mapping module in the generative network with a depth-wise separable convolution, and use it as a supernet containing the results of all subnets. The supernet will serve as a student network to receive the knowledge of the teacher network and is fully trained using the knowledge distillation method to obtain a pre-distilled supernet. (3) Use weight sharing to train all subnets in the pre-distillation supernet and update the parameters of each subnet so that each subnet can have good performance; (4) Using genetic algorithm to search for the best subnet structure in the supernet; (5) Fine-tune the optimal subnetwork structure on the dataset, load the original training parameters, and further train the structure to improve the performance of the generative adversarial network; Step (1) The generation network includes an encoding module, a decoding module and a residual mapping module; the encoding module is composed of five sub-units of the same structure, each unit is composed of a convolution, an instance normalization, and a correction rectifier unit, and the input image is downsampled four times to obtain an image feature tensor; the residual mapping module combines a residual learning mechanism to deepen the network depth and obtain a better feature representation; the decoding module decodes the optical features completed by the mapping module into a generated image, which includes four transposed convolution modules and one convolution module; the transposed convolution upsamples the input tensor, and after four transposed convolution operations, it reaches the same length and width as the input image; the identification network adopts a multi-scale discriminator and PatchGAN technology, the multi-scale discriminator includes a high-resolution discriminator and a low-resolution discriminator with the same structure, the original input image and the downsampled image are respectively sent to the two, and then the identification results of the high-resolution image and the low-resolution image are obtained; the PatchGAN technology removes the fully connected layer of the conventional discriminator, and replaces the overall judgment result with the independent authenticity of each N×N size area in the image; The loss function of the generative adversarial network in step (1) is described by the following formula: min G max D L(D,G)=min G max D (L GAN +λ VGG L VGG +λ DCT L DCT ) Among them, L GAN , L VGG , L DCT They are adversarial loss function, neural network feature matching function, discrete cosine loss function, λ VGG and λ DCT They are the weights of the neural network feature matching function and the discrete cosine loss function respectively; The adversarial loss function is described as: L GAN (X,Y)=E (X,Y) [logD(X,Y)]+E X [log(1-D(X,G(X)))] Where X is the input SAR image, Y is the real optical image, G(·) is the generator, D(·) is the discriminator, and E represents the mathematical expectation; The neural network feature matching function can be described by the following formula Among them, φ i (·) is the output feature map of the i-th maximum pooling layer of the VGG19 network, λ i , H i , W i , C i are the weight value, length, width and number of channels of the i-th maximum pooling output, respectively, and ||·||1 represents the L1 norm; The discrete cosine loss function is described as: L DCT (X,Y)=||DCT(G(X))-DCT(Y)||1 Where DCT(·) is discrete cosine transform; The knowledge distillation process described in step (2) is: Among them, x is the network input, that is, the input SAR image, f k Transform the convolution module for the kth channel, is the kth intermediate feature output of the student network, is the kth intermediate feature output of the teacher network, and the L2 norm is used to shorten the distance between the intermediate layer output features of the student network and the teacher network; Therefore, the overall loss function of the generative adversarial network training is: min G max D L(D,G)=min G max D (L GAN + VGG L VGG +λ DCT L DCT +λ distil L distil ) Among them, L distil is the knowledge distillation loss function, λ distil is the weight coefficient of the knowledge distillation loss function.
2. The lightweight method for SAR optical image mapping model based on conditional generative adversarial network according to claim 1 is characterized in that: The implementation process of step (3) is as follows: The supernet contains all subnet parameters. The difference between subnets only exists in the number of channels in each convolutional layer. It is a set of all subnets. If the number of channels of the first convolutional layer of the supernet is C1, the number of channels of the convolutional layer corresponding to any subnet is So And the front of the convolutional layer corresponding to the supernet The channel weight is the weight of the subnet; during each training, a subnet is randomly sampled for forward reasoning and back propagation, and only the parameters of this subnet are updated; due to the coupling between subnet parameters, some parameters of other subnets will also be updated accordingly.
3. The lightweight method for SAR optical image mapping model based on conditional generative adversarial network according to claim 1 is characterized in that: The implementation process of step (4) is as follows: (41) Randomly select P subnets from all subnet structures contained in the supernet as the first population; (42) The FID image quality evaluation index is used as the fitness function to evaluate the quality of the images generated by all subnets in the population, and the S subnet structures with the best performance are selected, where S is less than P, as the parents of the next population, and the structures with unsatisfactory performance are eliminated; (43) Randomly select a number of pairs of parents from the S selected subnetwork structures to produce offspring and add them to the population; (44) Randomly select several subnetworks in the new population for mutation operation, and the generated new structure is added to the population; (45) Repeat steps (42) to (44) until the preset termination condition is met, and then select the subnet with the best performance from the final population as the final result.