A lightweight semantic image translation method based on feature pyramid

Abstract

The application discloses a kind of lightweight semantic image translation methods based on feature pyramid, it is related to computer image processing technical field.The application applies feature pyramid to the generator of generative adversarial network, produces several hierarchical different feature maps, the number is determined by the scale of input image, so that different scale pictures use different features, greatly reduce the parameter quantity of generator.In the discriminator based on block generative adversarial network, weight branch is introduced, and the loss function of training is improved, so that training is carried out normally, ensure that generated image and real image match and improve the quality of output image.The semantic image is translated using the application, can generate more real, higher quality natural scene picture, and network scale is smaller, more lightweight, can be run under lower hardware conditions, can adapt to more rich actual production environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer image processing, and particularly relates to a lightweight semantic image translation method based on a feature pyramid. BACKGROUND

[0002] Image translation technology is widely used in image creation, image stylization, image inpainting, color filling and other scenes, aiming to establish a mapping relationship from an image source domain to a target domain, so that the representation of the original image is converted to another image representation. This mapping can be diverse, such as conversion from a contour map to a color map, image day and night, seasonal conversion, and semantic image to natural scene image conversion. This conversion is usually end-to-end, that is, the input and output are both images.

[0003] With the rapid development of deep learning, using a generative adversarial network to process image translation tasks has become a popular research direction. In the training of a generative adversarial network, the features of the source domain image can be learned, and a target domain image with similar features can be generated based on the learned features, thereby realizing image translation. Generally, in order to generate higher quality images, the generative adversarial network used is usually large-scale, which brings a huge amount of calculation and requires higher hardware conditions, thereby limiting the application in limited hardware conditions and large-scale data conditions. In fact, the features learned by such a large-scale network are redundant for small-scale images, and using such a network does not significantly improve the quality of the generated images, but wastes hardware resources. SUMMARY

[0004] In view of the above problems, the present application proposes a lightweight semantic image translation method based on a feature pyramid. The present application applies the idea of a feature pyramid to a generative adversarial network, realizing a lightweight semantic image translation method.

[0005] The technical scheme adopted by the present application is as follows:

[0006] A lightweight semantic image translation method based on a feature pyramid, the method comprising the following steps:

[0007] Step 1, constructing a semantic image translation network model;

[0008] The semantic image translation network model comprises a generative network and a discriminative network;

[0009] The generative network comprises an encoder, a decoder and a feature map fusion module.

[0010] The encoder outputs a plurality of feature maps with different levels based on a feature pyramid manner, the feature maps of different levels are in a pyramid structure according to the scale of the feature maps, and the feature maps of different levels contain different semantic information, and the number of levels of the feature maps output by the encoder is defined as M;

[0011] The decoder includes M decoding branches, the input of each decoding branch is a feature map output by the encoder, and each decoding branch is used for feature encoding processing on the input feature map and outputs a feature map with the same size; the network structure of each decoding branch is the same, and sequentially includes a plurality of residual networks and a plurality of deconvolution operation modules;

[0012] The input of the feature map fusion module is the output of each decoding branch, which is used for fusing the M input feature maps and converting them into a generated image with a specified size;

[0013] The discriminator network includes a weighting branch, a patch GAN branch and a second fusion module, wherein the input of the weighting branch and the patch GAN branch is the generated image output by the feature map fusion module; the patch GAN branch is used for dividing the input image into patches according to a specified patch size, and generating corresponding patch features based on the content of each patch; the weighting branch divides the input image into patches based on the same patching manner as the patch GAN branch, and generates weights for each patch based on the content of each patch; then the outputs of the weighting branch and the patch GAN branch are input into the second fusion module through Hadamard product operation to obtain a second generated image, and the second fusion module is used for outputting the true or false discrimination result of the second generated image and the real image corresponding to the input image of the generation network (i.e. the input image of the encoder);

[0014] Step 2, based on the training image set, the semantic image translation network model is trained by deep learning, when the preset training end condition is met (for example, the maximum training number or the loss function value converges is reached, etc.), stop, and based on the trained generation network, a semantic image translator is obtained for image translation processing of the to-be-translated image;

[0015] During the deep learning training process of the semantic image translation network model, the generation network and the discriminator network are trained alternately, and the loss used by the generation network during training is the weighted sum of the mean absolute error loss, the feature matching loss and the total variation loss.

[0016] Further, the encoder further comprises, based on a preset mapping relationship between image size and feature map layer number, determining the number of feature maps of the current input image, and starting from the bottom layer of the pyramid structure of the feature map corresponding to the maximum number of feature maps that the encoder can extract, extracting a corresponding number of feature maps, and triggering the corresponding decoding branch to decode the current extracted feature map of the layer number. That is, in the present application, the maximum number of extracted feature maps of the encoder is preset, and the decoding branch corresponding to each scale (each layer of feature map) is also mapped in advance.

[0017] Further, the encoder is a network structure based on a deep convolutional network, preferably a feature extraction backbone network of a VGG network (such as VGG16, VGG19, etc.).

[0018] Further, each decoding branch comprises in turn a plurality of residual operation modules and a plurality of deconvolution operation modules.

[0019] Further, the deconvolution operation module comprises in turn a deconvolution layer, a batch normalization layer and an activation function. Preferably, the number of deconvolution operation modules is inversely related to the size of the output feature map of the decoding branch, that is, it decreases by N as the size increases.

[0020] Further, the weighting branch comprises in turn a convolution layer with a 3x3 convolution kernel, and three average pooling layers.

[0021] The technical scheme provided by the present application at least brings the following beneficial effects:

[0022] (1) The present application can realize high-quality semantic image translation under the condition of small parameter amount, and can run under low hardware conditions.

[0023] (2) The present application uses different features for pictures of different scales, which greatly reduces the parameter amount of the generator.

[0024] (3) The present application designs a weighted discriminator and an improved loss function to improve the quality of the generated image without increasing the parameter amount of the generator. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0026] Figure 1 The overall structure diagram of the network used in the embodiments of the present application.

[0027] Figure 2 The decoding branch structure diagram of the embodiment of the present application.

[0028] Figure 3 The feature map fusion module structure diagram of the embodiment of the present application.

[0029] Figure 4 The schematic diagram of picture partitioning of the embodiment of the present application.

[0030] Figure 5 The structure diagram of the weighted branch of the embodiment of the present application.

[0031] Figure 6 The effect comparison diagram of the existing image translation network and the present application on semantic image translation on the Cityscapes dataset.

[0032] Among them, from left to right are input image, pix2pixHD (high-definition pixel to pixel), SPADE (spatial adaptive non-normalization) and generated image of the embodiment of the present application.

[0033] Figure 7 The effect comparison diagram of the existing image translation network and the present application on semantic image translation on the Facades dataset. Among them, from left to right are input image, pix2pix (pixel to pixel network), pix2pixHD (high-definition pixel to pixel network), SPADE (spatial adaptive non-normalization) and generated image of the embodiment of the present application.

[0034] Figure 8 The effect comparison diagram of the existing image translation network and the present application on semantic image translation on the LaPa dataset. Among them, from left to right are input image, pix2pix (pixel to pixel network), pix2pixHD (high-definition pixel to pixel network), SPADE (spatial adaptive non-normalization) and generated image of the embodiment of the present application.

[0035] Figure 9 The effect comparison diagram of the existing image translation network and the present application on semantic image translation on the ADE20K dataset. Among them, the lower left corner of the leftmost image is the input image, from left to right are pix2pix (pixel to pixel network), pix2pixHD (high-definition pixel to pixel network), SPADE (spatial adaptive non-normalization) and generated image of the embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the drawings.

[0037] Although the prior art has very good performance in the semantic image translation task, they usually rely on a large and complex network, because such a large-scale network often requires higher hardware conditions, thereby limiting its application scenarios, and for some smaller resolution images, part of the structure of these complex networks may be redundant. In order to adapt to more application scenarios, an embodiment of the present application proposes a lightweight semantic image translation method based on a feature pyramid, which can translate semantic images into natural scene images with high enough quality under the condition of greatly reducing the number of parameters. In an embodiment of the present application, the generative adversarial network model constructed includes a generator and a discriminator. The significant feature of the embodiment of the present application is to apply the idea of a feature pyramid to the generator. For input images of different sizes, the network can dynamically adjust the number of generator feature maps to achieve lightweight. At the same time, the embodiment of the present application also sets a weighted discriminator and an improved loss function, so that the discriminator focuses more on the area with rich details, so that the generator can output better images.

[0038] As a possible implementation manner, the overall framework of the network used in the lightweight semantic image translation method based on a feature pyramid proposed by an embodiment of the present application is as shown in Figure 1 The encoder and the decoder constitute the generator of the embodiment of the present application. In the embodiment of the present application, cGAN (Conditional GAN, conditional generative adversarial network) is selected as the basis, and the generative adversarial network generator and the discriminator are improved, and the loss function is improved to improve the performance of the network.

[0039] As shown in Figure 1 In the embodiment of the present application, the generator adopts an encoder-decoder structure. Generally, the role of the encoder is to extract the features of the input image (semantic image, i.e. an image with image semantics, such as an image obtained by image semantic segmentation) to generate a feature map, and the role of the decoder is to output an image according to the feature map. In order to make the feature map contain more information, the feature map will contain more channels, resulting in an increase in the number of parameters. In the embodiment of the present application, the idea of a feature pyramid is introduced in the encoder part. The encoder will output at most three levels of different feature maps. Different levels of feature maps will contain different semantic information. The decoder constructs a corresponding number of decoding branches. Each decoding branch contains a number of residual operation modules (for example, including 6 residual operation modules) and deconvolution operation modules (N represents the number of deconvolution operation modules. Each deconvolution module includes a deconvolution layer, a batch normalization layer and an activation function (such as a Relu activation function) in turn), and outputs a feature map of the same size, as shown in Figure 2As shown in the figure, the residual module is defined consistent with that in ResNet (Residual Network). To ensure that the output feature maps are of the same scale, different numbers of deconvolution operations are required for branches at different levels. The number N in the figure is determined by the size of the branch feature map; N is 3 for the smallest size and decreases as the size increases. After obtaining feature maps of the same scale, they are input into... Figure 3 The feature map fusion module shown mainly includes a channel connection module and a convolution operation. It can fuse feature maps (concatenate them according to the channel dimension) and then transform them (through a convolutional layer with a 1×1 kernel) into the final generated image. To reduce the number of parameters and make the network more lightweight, this embodiment of the invention reduces the number of channels of the feature maps to 256 and designs a variable network structure. The number of feature maps generated by the encoder changes according to the size of the input image, which also changes the decoding branches of the decoder accordingly. For a 128p image, the number of feature maps generated is 1, for 256p it is 2, and for 512p it is 3.

[0040] like Figure 1 As shown, the discriminant network in this embodiment of the invention includes two branches: a weighting branch and a PatchGAN (Patch Generative Adversarial Network) branch. The PatchGAN branch divides the image into blocks, generating different outputs for different regions. Figure 4 As shown, dividing the image into 10×10 blocks reveals significant differences in the content contained within different regions. For instance, the region containing the car contains more edge and texture information. Therefore, different discrimination weights should be assigned to different regions. However, the PatchGAN branch assigns the same weight to each region. To address this issue, this embodiment introduces a weighted branch, assigning higher weights to regions with richer details. Its structure is as follows: Figure 5 As shown, it includes, in sequence: a convolutional layer with a 3×3 kernel, and three average pooling layers. The weighted branch and the PatchGAN branch are processed using the Hadamard product and then input into the fusion module. The fusion module contains a convolutional layer with a 4×4 kernel and a sigmoid activation function. Therefore, the discriminator output expression is:

[0041]

[0042] Where ⊙ represents the Hadamard product. The input image is represented by P, the PatchGAN branch is represented by W, the weighting branch is represented by W, and the fusion module is represented by Fusion.

[0043] It should be noted that the discriminative network is only used in the training process, in the training process, the discriminative network and the generative network are trained alternately, and after the training converges, the image translation result of the target image is directly obtained based on the output of the feature map fusion module.

[0044] In order to make the training proceed normally, ensure that the generated image matches the real image and improve the quality of the output image, the training target is improved in the embodiment of the application, and the improved training target expression is:

[0045]

[0046] Among them, is the original training target of cGAN (Conditional GAN, conditional generative adversarial network), and the expression is:

[0047]

[0048] Among them, x represents a semantic image, y represents a real image, z is a random noise vector, G is a generator, D is a discriminator, D(·) and G(·) represent the outputs of the discriminator D and the generator G respectively, represents an expected operation.

[0049] The training target newly added in the embodiment of the application introduces the average absolute error loss feature matching loss and the total variation loss to train. The expression of is:

[0050]

[0051] Among them, λ1, λ2, λ3 are the proportion coefficients of the three losses, and the values are empirical preset values, preferably, in the embodiment of the application, all of them are set to 1.

[0052] The expression of the average absolute error loss is as follows:

[0053]

[0054] Among them, is the output image of the generator (i.e. the generated image output by the feature map fusion module), is the corresponding real image, k represents the pixel position, and n represents the number of generated image pixels.

[0055] The expression of the feature matching loss is as follows:

[0056]

[0057] wherein, is the output image of the generator, is the real image, is the output feature map of the i-th layer network of the pre-trained feature extraction network (i.e. the i-th layer feature map output by the encoder), the encoder used in the embodiments of the present application is the feature extraction backbone network in the pre-trained VGG19, N i is the number of elements of the i-th layer feature map, for example, the i-th layer feature map has a dimension of m x n x c, where m x n represents the size of the feature map, and c represents the number of channels, then N i = m x n x c.

[0058] Total variation loss is expressed as follows:

[0059]

[0060] wherein, is the output image of the generator, and p and q represent pixel positions.

[0061] As shown in Figure 6 , Figure 7 , Figure 8 , Figure 9 , the embodiments of the present application compare the performance of several existing image translation networks and the method of the embodiments of the present application on four different data sets, the leftmost side of the figure is the input semantic image, and the output results of different models are shown from left to right, and the rightmost side is the result of the present application. It can be seen that, compared with existing networks, the output of the embodiments of the present application is closer to reality and has more rich texture details.

[0062] The size and performance of the existing image translation network and the present application are shown in Table 1 and Table 2, the size index uses the number of parameters, and the performance index uses IS (Inception Score) and FID (Frechet Inception Distance). The higher the IS, the better the performance, and the lower the FID, the better the performance.

[0063] Table 1

[0064]

[0065]

[0066] Table 2

[0067]

[0068] The results prove that, compared with the existing image translation network, the application has obvious advantages: in terms of parameter quantity, the application is much lower than the existing image translation network, can run under lower hardware conditions, and can adapt to a wider production environment; in terms of performance, compared with the existing network, better performance can be achieved under the Facades data set, and some indicators can also surpass the existing network on other data sets. In summary, the application reduces the parameter quantity of the network without obvious performance loss, and realizes a lightweight semantic image translation network.

[0069] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0070] The above only describes some embodiments of the present application. For those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of protection of the present application.

Claims

1. A lightweight semantic image translation method based on a feature pyramid, characterized in that, The method comprises the following steps: Step 1, constructing a semantic image translation network model; The semantic image translation network model comprises a generation network and a discrimination network; The generation network comprises an encoder, a decoder and a feature map fusion module; The encoder outputs a plurality of feature maps of different levels based on a feature pyramid mode, the feature maps of different levels form a pyramid structure according to the feature map size, and the feature maps of different levels contain different semantic information, and the number of levels of the feature maps output by the encoder is defined as M; The decoder comprises M decoding branches, the input of each decoding branch is a feature map output by the encoder, and each decoding branch is used for performing feature coding processing on the input feature map and outputting a feature map of the same size; each decoding branch has the same network structure and comprises a plurality of residual networks and a plurality of deconvolution operation modules in sequence; The input of the feature map fusion module is the output of each decoding branch, and the feature map fusion module is used for converting the M feature maps into a generated image of a specified size after fusion; The discrimination network comprises a weighting branch, a patch GAN branch and a second fusion module, the input of the weighting branch and the patch GAN branch is the generated image output by the feature map fusion module; the patch GAN branch is used for dividing the input image into patches according to a specified patch size, and generating corresponding patch features based on the content of each patch; the weighting branch divides the input image into patches based on the same patch mode as the patch GAN branch, and generates weights of each patch based on the content of each patch; then the outputs of the weighting branch and the patch GAN branch are subjected to Hadamard product operation to obtain a second generated image, and the second generated image is input into the second fusion module, and the second fusion module is used for outputting a true-false discrimination result between the second generated image and a real image corresponding to the input image of the generation network; Step 2, performing deep learning training on the semantic image translation network model based on a training image set, stopping when a preset training end condition is met, and obtaining a semantic image translator based on the trained generation network; During the deep learning training of the semantic image translation network model, the generation network and the discrimination network are trained alternately, and the loss adopted by the generation network during training is a weighted sum of the mean absolute error loss, the feature matching loss and the total variation loss.

2. The method of claim 1, wherein, The encoder further comprises determining the number of feature maps of the current input image based on a preset mapping relationship between the image size and the number of feature map levels, and starting from the bottom layer of the pyramid structure of the feature maps corresponding to the maximum number of feature maps that the encoder can extract, extracting a corresponding number of feature maps, and triggering the corresponding decoding branch to decode the current extracted feature maps of different levels.

3. The method of claim 1 or 2, wherein, The mean absolute error loss, the feature matching loss and the total variation loss contained in the improved loss function in step 2 are respectively: mean absolute error loss The expression for the mean absolute error loss is: Feature matching loss is expressed as: total variation loss The expression for the total variation loss is: wherein, denotes a generated image representing the network output, denotes a corresponding real image, k denotes a pixel position, n denotes a number of generated image pixels; denotes an i-th layer feature map output by the encoder, N i denotes a number of elements of the i-th layer feature map; generated image both superscripts p and q denote a pixel position.

4. The method of claim 1 or 2, wherein, The encoder is a network structure based on a deep convolutional network.

5. The method of claim 1 or 2, wherein, Each decoding branch comprises a plurality of residual operation modules and a plurality of deconvolution operation modules in sequence.

6. The method of claim 5, wherein, The deconvolution operation module comprises a deconvolution layer, a batch normalization layer and an activation function in sequence.

7. The method of claim 1 or 2, wherein, The feature map fusion module of the generation network comprises, in sequence, a channel dimension splicing layer and a convolution layer with a 1x1 convolution kernel.

8. The method of claim 1 or 2, wherein, The weighted branch comprises, in sequence, a convolution layer with a 3x3 convolution kernel and three average pooling layers.

Images