A deformable instance-level image translation method

Abstract

The present application relates to the field of image processing, and particularly relates to a deformable instance-level image translation method, which solves the problem that instances are difficult to deform and instances cannot be consistent with mask information due to too large gap between domains in the prior art. The method fuses feature encoding of edge information of foreground, background mask information and target domain label information to obtain a hybrid mask; then, the background feature and the hybrid mask are input into a generator, a decoding network of the generator decodes the input background feature, extracts additional information from the hybrid mask, and applies the extracted additional information to the normalized decoding output, and the additional information is used to perform affine transformation on the normalized decoding output, so that fusion information including foreground information corresponding to the target domain mask and position information indicating the position of the foreground is obtained; finally, the generated foreground information and the source domain background image are fused by using the position information, and a target domain picture retaining the source domain background is output.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a deformable instance-level image translation method. BACKGROUND

[0002] In recent years, with the rapid development of deep neural networks, some image processing techniques based on neural networks have gradually replaced the traditional image processing methods which are time-consuming and laborious, thus making some high-level image semantic editing tasks possible, such as image translation tasks. The image translation task aims to convert the source domain image to the target domain image through a designed model, that is, to learn the mapping between the source domain and the target domain, which is a set of pictures with certain characteristics. Since the generative adversarial network was proposed, a considerable part of visual tasks can be converted into image translation tasks, such as style transfer, super-resolution, label-guided image generation, and image inpainting.

[0003] Among them, the instance-level image translation task mainly acts on specific foreground instances in the image, which is generally divided into two types:

[0004] I. Translate the foreground instance while not constraining the background information;

[0005] II. Preserve the original background while translating the specific foreground instance.

[0006] For the first task, the common design paradigm is that the model learns to distinguish the foreground instance and the background in the input image, and then performs different image translation processing on the foreground and the background. However, this method sometimes incorrectly distinguishes the foreground and the background, resulting in unexpected results.

[0007] Therefore, the second task has more application prospects in real life, such as virtual fitting, post-processing of film and television images, etc., which often require replacing specific instances while preserving the original background. However, early models only achieve the translation of low-level features such as texture and style when performing image translation, and cannot give reasonable results when facing high-level feature changes such as foreground shape changes.

[0008] Therefore, in recent years, in order to meet this demand, some people have proposed the definition of unsupervised deformable instance image translation task. This task aims to preserve the background information while translating the foreground to the target domain, accompanied by significant changes in the shape of the foreground.

[0009] From this point, in recent years, attempts have been made to introduce external masks as guidance information, and attempts have been made to guide the model to learn the cross-domain mapping relationship in shape by setting the target domain mask as the learning goal. But the experiments show that the current model has several problems. First, unreasonable target domain masks often appear in the generated results, which shows that the model's ability to generate target domain masks is insufficient. Although this method, in the training stage, guides the generation of cross-domain masks by constructing mask data pairs, the way of introducing mask information is only to splice it into the image information, and the mask information is not fully utilized. Second, the translated foreground images often do not match the generated masks, which reflects that the target domain mask does not provide reasonable guidance and constraints for the generated images. In addition, the current method uses a serial translation method when facing multiple foreground instances of a single image, which greatly increases the time overhead of training. SUMMARY

[0010] The technical problem to be solved by the present application is to provide a deformable instance-level image translation method, which solves the problem of instance deformation difficulty and inconsistency between instance and mask information caused by too large gap between domains.

[0011] The technical solution adopted by the present application to solve the above technical problems is:

[0012] A deformable instance-level image translation method, comprising the following steps:

[0013] C1, input the image mask pair of the source domain, the label information and the instance mask of the target domain into the image translation model, the image mask pair includes a group of instance masks and corresponding images; the image translation model includes a pre-trained image completion model and an instance generation network;

[0014] C2, based on the source domain image mask pair input in step C1, first aggregate all instance masks of the source domain to obtain a source domain mask, then remove the foreground of the corresponding source domain image according to the source domain mask to obtain a remaining image without the mask part, and use the image completion model to complete the remaining image to obtain a background image of the source domain;

[0015] C3, based on the target domain instance mask input in step C1, first aggregate all instance masks of the target domain to obtain a target domain mask, then input the target domain mask and the source domain background image B S obtained in step C2 into the instance generation network; the instance generation network includes an encoder and a generator, and processes as follows:

[0016] Based on the input source domain background image, the background features of the source domain background image are extracted through the encoder;

[0017] Based on the input target domain mask, edge information of the foreground is obtained through an edge extraction algorithm; background mask information is obtained by inverting the target domain mask; feature encoding is performed on the label information of the target domain; then, the feature encodings of the foreground edge information, background mask information, and target domain label information are fused to obtain a hybrid mask.

[0018] The background features and the blending mask are input into the generator, which includes a decoding network that decodes the input background features. At the same time, the generator extracts additional information from the blending mask and applies the extracted additional information to the normalized decoded output. The normalized decoded output is then subjected to an affine transformation using the additional information to obtain fused information that includes foreground information corresponding to the target domain mask and position information indicating the foreground position.

[0019] Finally, the generated foreground information and the source domain background image are fused using location information to output a target domain image that retains the source domain background.

[0020] Specifically, the image completion model is the HiFill model; the encoder of the instance generation network is a multi-layer residual neural network; the edge extraction algorithm is the CANNY edge detection algorithm; the label information of the target domain is feature-encoded using one-hot encoding; and the feature encodings of the foreground edge information, background mask information, and target domain label information are fused using matrix multiplication.

[0021] Specifically, the generator of the instance generation network is an APADE-ResNet network. The APADE-ResNet network is a neural network formed by adding APADE blocks between each convolutional layer and ReLU layer of the ResNet network. That is, the decoding network of the generator is a ResNet network, and additional information is extracted from the mixing mask through APADE blocks.

[0022] The APADE block includes two inputs: one is the output of the convolutional layer, and the other is a mixing mask.

[0023] After scaling the blending mask to the APADE block input feature dimension, it is input into the APADE block. After input, it is first passed through a convolutional layer Conv. s Then, Conv s The outputs are input into two independent convolutional layers, Conv1 and Conv2, respectively, to generate two embedding vectors γ and β.

[0024] The output of the convolutional layer, after being input into the APADE block, first passes through a batch normalization layer. Then, the output of the batch normalization layer, along with the embedding vectors γ and β, is calculated as the output of the APADE block using the following formula:

[0025] F out = γ · F in + β

[0026] wherein, F in represents the output of the batch normalization layer, F out represents the output of the APADE block.

[0027] Further, the APADE-ResNet network is a neural network composed of an APADE block based on a two-layer ResNet network, that is:

[0028] Each layer of the APADE-ResNet network includes a forward neural network branch and a shortcut branch, wherein the forward neural network branch includes a first convolutional layer, a first APADE block, a first ReLU layer, a second convolutional layer, a second APADE block and a second ReLU layer connected in series, and the shortcut branch is constructed between the second APADE block and the input to add the input to the output of the second APADE block to form the input of the second ReLU layer.

[0029] Further, the APADE-ResNet network is a multi-layer APADE-ResNet network, the input of the first layer is the background feature and the mixed mask, and the input of each layer after the first layer is the output of the previous layer and the mixed mask; the interlayer is enlarged by upsampling, so that the final output has the same size as the source domain image.

[0030] Further, the instance generation network is trained according to the following steps:

[0031] D1, training sample data preparation:

[0032] Collect data from the data set to define the domain of the foreground category, construct image mask pairs of each domain, and the image mask pair includes a group of instance masks and corresponding images;

[0033] D2, input sample data, train the instance generation network, including:

[0034] D21, input the image mask pair of at least one domain including the target domain specified by the translation task;

[0035] D22, for each domain image mask pair input, process according to the following steps:

[0036] The image mask pair of the input domain is used as the source domain image mask pair of the input image translation model; the label information and instance mask of the input domain are used as the label information and instance mask of the target domain of the input image translation model; the image I' is generated by the image translation model according to the input iwherein i represents the i-th sample data corresponding to the generated image input;

[0037] The image in the input domain image mask pair is taken as the real image I i , and the generated image I' i and the generated image I' i constitute a positive and negative sample pair;

[0038] D23, input the positive and negative sample pair obtained in step D22 into the discriminator to perform adversarial training on the instance generation network;

[0039] D24, if the set number of iterations is reached or the instance generation network converges, the training is completed, otherwise return to step D22.

[0040] Preferably, in step D23, the loss function of the adversarial training is:

[0041]

[0042]

[0043] wherein, is the loss function of the generator, is the loss function of the discriminator, is the adversarial loss function of the generator, is the adversarial loss function of the discriminator, L fmap is the fusion graph loss function, and λ is a hyperparameter for adjusting the balance of each loss function;

[0044] The calculation formula of the adversarial loss function of the generator is as follows:

[0045]

[0046] The calculation formula of the adversarial loss function of the discriminator is as follows:

[0047]

[0048] wherein, D img represents the discriminator for adversarial training of the instance generation network, I' i represents the generated image based on the i-th domain image mask pair input; I i represents the image of the i-th domain image mask pair input, and P is the number of sample data input in step D21;

[0049] The calculation formula of the fusion graph loss function L fmap is as follows:

[0050]

[0051] Among them, M i The input domain mask is formed by aggregating the instance masks of the image mask pairs of the i-th domain of the input; α′ i This represents the positional information indicating the foreground location in the fused information generated by the generator based on the image mask of the i-th domain.

[0052] To address the issue of lacking target domain masks in certain scenarios, this invention also provides a deformable instance-level image translation method, which includes two stages: mask deformation and image generation, comprising the following steps:

[0053] A. Mask distortion

[0054] A1. Input the instance mask of the source domain and the label of the target domain into a pre-trained mask deformation network; the mask deformation network includes an encoder and a generator;

[0055] A2. The mask deformation network deforms the mask according to the following steps:

[0056] A21. Aggregate all instance masks of the source domain to obtain the source domain mask; extract features from the source domain mask using an encoder to obtain the overall features F of the source domain mask. img Mask each instance of the source domain. Respectively with overall feature F img The fusion is performed to obtain the instance mask features F corresponding to each instance mask. mask(i) Then, feature encoding is performed on the label information of the target domain, and the feature encoding of the target domain label information is then embedded into the mask features F of each instance. mask(i) ;

[0057] A22. Input the instance mask features that incorporate the label information feature encoding into the generator, and generate a mask from the target domain output by the generator. As an instance mask for the corresponding target domain.

[0058] B. Image Generation

[0059] Using the source domain image mask pair, the target domain label information, and the target domain instance mask obtained in step A as input, a deformable instance-level image translation method according to any one of claims 1 to 7 is used to generate a target domain image that retains the source domain background.

[0060] Specifically, the encoder of the mask deformation network is a multi-layer convolutional neural network, and it masks the instances of the source domain through matrix multiplication. With overall characteristics F img To merge;

[0061] The label information of the target domain is encoded by one-hot encoding, and the instance mask feature F mask(i) is fused with the feature encoding of the target domain label information; or, the label information of the target domain is encoded by a convolutional neural network, and then the instance mask feature F mask(i) is spliced with the feature encoding of the target domain label information.

[0062] Specifically, the generator of the mask deformation network includes a multi-layer residual neural network and a multi-layer convolutional neural network; first, the input instance mask feature F mask(i) that fuses the label information is scaled to match the input dimension of the multi-layer convolutional neural network, and then decoded by the multi-layer convolutional neural network to generate a target domain generated mask; the layers of the multi-layer convolutional neural network are enlarged by upsampling, so that the final output has the same size as the source domain image.

[0063] Further, the mask deformation network is trained according to the following steps:

[0064] B1, training sample data preparation:

[0065] Collect masks from the data set to define the domain of the foreground category, construct sample pairs based on the constructed domains, and each sample pair contains two domains, one of which is the source domain and the other is the target domain. In all sample pairs, each constructed domain is used as a target domain at least once.

[0066] B2, training the mask deformation network:

[0067] B21, input at least one sample pair including the source domain and the target domain specified by the translation task to be performed;

[0068] B22, for each input sample pair, the mask deformation network is processed according to the following steps:

[0069] Randomly sample a set number of instance masks from the instance masks of the source domain and the instance masks of the target domain of the sample pair; pair the sampled source domain instance masks and target domain instance masks two by two, i.e. one source domain instance mask corresponds to one target domain instance mask wherein subscript i represents the i-th sample pair, ranging from 1 to P, P is the number of input sample pairs, j represents the j-th mask of the corresponding domain, ranging from 1 to Q, Q is the set sampling number, superscript T represents the target domain, and S represents the source domain;

[0070] input the target domain label and the instance mask of the source domain obtained by sampling into the mask deformation network to generate a target domain generated mask corresponding to each instance mask of the source domain

[0071] based on the pairing information of the source domain instance mask and the target domain instance mask and the correspondence between the target domain generated mask and the source domain instance mask , a triple consisting of the corresponding source domain instance mask , the target domain instance mask and the target domain generated mask is constructed;

[0072] B23, based on the triple obtained in step B22, for each triple, the target domain instance mask is scaled so that it matches the size of the corresponding target domain generated mask , serving as a target domain real mask; and a positive-negative sample pair is constructed from the corresponding target domain generated mask and the target domain real mask;

[0073] B24, input the positive-negative sample pair obtained in step B23 into the discriminator to conduct adversarial training on the mask deformation network;

[0074] B25, if the set number of iterations or the convergence of the mask deformation network is reached, the training is completed, otherwise return to step B22.

[0075] Further, in the step B22, after the instance masks of the source domain and the instance masks of the target domain obtained by sampling are paired two by two, the center positions of the two-by-two paired instance masks are aligned.

[0076] Further, the generator of the mask deformation network is a multi-layer network, and in the step B22, for each input instance mask feature fused with label information feature encoding, the multi-layer network of the generator generates a target domain generated mask corresponding to different sizes when decoding, and the last K layers of the generator output the target domain generated mask corresponding to different sizes layer by layer That is for the target domain generated mask sequence composed of the target domain generated mask corresponding to different sizes output by the last K layers of the generator, and the K is the number of network layers selected for layer-by-layer output in the multi-layer network;

[0077] In step B23, for each triple, the target domain instance mask is scaled to obtain a target domain real mask matching the size of the target domain generated mask sequence and the corresponding target domain generated mask and target domain real mask constitute an adversarial sample, and n is the serial number of the mask sequence.

[0078] Further, the loss function of the adversarial training in step B24 is:

[0079]

[0080]

[0081] wherein, the loss function of the generator, the loss function of the discriminator, the adversarial loss function of the generator, the adversarial loss function of the discriminator, L pc the mask pseudo closed loop loss function, L const the mask consistency loss function, L reg the mask regularization function, and λ is a hyperparameter for adjusting the balance of each loss function;

[0082] the adversarial loss function of the generator is calculated as follows:

[0083]

[0084] the adversarial loss function of the discriminator is calculated as follows:

[0085]

[0086] wherein, D mask represents a discriminator for adversarial training of the mask deformation network, represents the target domain generated mask output by the nth output layer corresponding to the jth instance mask of the source domain in the ith sample pair; represents the nth target domain real mask obtained by scaling the target domain instance mask corresponding to the jth instance mask of the source domain in the ith sample pair;

[0087] the calculation formula of the mask pseudo closed loop loss function L pc is as follows:

[0088]

[0089] wherein, represents the source domain instance mask sequence in the ith sample pair represents the label information of the source domain in the ith sample pair; represents the generator G of the mask deformation network based on the source domain instance mask and the label information in the ith sample pairmask reconstructed source domain instance mask;

[0090] The calculation formula of the mask consistency loss function L const The calculation formula is as follows:

[0091]

[0092] Wherein, d(·) represents a downsampling function;

[0093] The calculation formula of the mask regularization function L reg The calculation formula is as follows:

[0094]

[0095] Wherein, The Kth target domain generated mask corresponding to the jth instance mask of the source domain of the ith sample, the Kth is the final output of the generator, and sum(·) represents the numerical summation of all pixel points in the input mask.

[0096] The beneficial effects of the present application are:

[0097] The deformable instance-level translation method of the present application introduces target domain mask information in the translation process, extracts additional information from the mixed mask through the generator in the decoding stage, and applies the extracted additional information to the normalized decoding output. The fusion information including the foreground information corresponding to the target domain mask and the position information indicating the foreground position is obtained by performing affine transformation on the normalized decoding output through the additional information. Therefore, the corresponding instance can be generated by using the target domain mask information as guidance information, which greatly relieves the inconsistency between the translated mask and the corresponding instance and the difficulty of successfully deforming the instance.

[0098] In the foreground replacement task, when there is a matching target domain instance image and mask as a reference, the foreground can be easily converted between domains in shape and image through the first method of the present application.

[0099] However, in reality, more often than not, only the target domain label can be obtained, that is, only the category of the target domain image to be converted is known, and the mask is lacking. Although there are other ways to obtain the target domain mask in the prior art, which can be adapted to the first method of the present application. In order to improve the scope of application of the present application, the present application provides another translation method, which adds a mask deformation stage to translate the source domain mask into the target domain mask on the basis of the first method, so that the translation task can be effectively completed under the premise of only knowing the target domain label information.

[0100] Meanwhile, the application designs a training method with effective supervision information for two unsupervised training stages, the supervision information is obtained by mining the knowledge of itself, without additional data, which ensures the stability of the training, and provides an effective and efficient training paradigm for unsupervised deformable instance translation. BRIEF DESCRIPTION OF DRAWINGS

[0101] Figure 1 The inference data flow diagram of the two-stage deformable instance-level image translation method of the application;

[0102] Figure 2 The network structure diagram of the multi-layer APADE-ResNet network of the deformable instance-level image translation method of the application;

[0103] Figure 3 The comparison diagram of the simulation experiment of the deformable instance-level image translation method of the application. DETAILED DESCRIPTION

[0104] The application aims to provide a deformable instance-level image translation method, which introduces target domain mask information in the translation process, takes the target domain mask information as guidance information, and generates corresponding instances, so as to solve the problem that the mask and the corresponding instance of the translation are inconsistent and the instance is difficult to deform successfully.

[0105] The deformable instance-level image translation method provided by the application includes only an image generation stage, and another includes a mask deformation and image generation two stages, the image generation stages of the two adopt the same model, therefore, in order to simplify the description, the application will be further described below in combination with the drawings and the specific embodiments of the deformable instance-level image translation method including two stages.

[0106] Embodiment:

[0107] As shown in the figure, a deformable instance-level image translation method includes the following steps: Figure 1

[0108] S1, mask deformation

[0109] S11, input the instance mask of the source domain and the target domain label into the pre-trained mask deformation network; the mask deformation network includes an encoder and a generator. The above-mentioned instance mask is also the mask corresponding to a foreground instance in the image, which is essentially a segmentation mask formed by segmenting the foreground mask of the image by instance, and is aggregated by all instance masks corresponding to the image to form the mask of the corresponding image, for example, there are two people in the image, one person corresponds to an instance mask, and the aggregation of the instance masks corresponding to the two people forms the mask of the image.

[0110] S12, the mask deformation network deforms the mask according to the following steps:​

[0111] S121. Aggregate all instance masks of the source domain to obtain the source domain mask; extract features from the source domain mask using an encoder to obtain the overall features F of the source domain mask. img Mask each instance of the source domain. Respectively with overall feature F img The fusion is performed to obtain the instance mask features F corresponding to each instance mask. mask(i) Then, feature encoding is performed on the label information of the target domain, and the feature encoding of the target domain label information is then embedded into the mask features F of each instance. mask(i) .

[0112] The encoder described above can employ any existing encoding network used for image feature extraction. In this embodiment, specifically, a multi-layer convolutional neural network is used as the encoder for the mask deformation network, and the instance mask of the source domain is achieved through matrix dot multiplication. With overall characteristics F img To integrate.

[0113] Similarly, any existing encoding method can be used to encode the label information of the target domain. For example, one-hot encoding can be used, such as encoding sheep as [0,1] and giraffes as [1,0], and then fusing them through matrix multiplication. In this embodiment, specifically, the label information of the target domain is encoded using a convolutional neural network, and then the instance mask feature F is... mask(i) It is concatenated with the feature encoding of the target domain label information.

[0114] S122. Input the instance mask features that incorporate the label information feature encoding into the generator, and generate a mask from the target domain output by the generator. As an instance mask for the corresponding target domain.

[0115] The generator described above can also employ any existing decoding network used for image generation. In this embodiment, specifically, the generator of the mask deformation network includes a multi-layer residual neural network and a multi-layer convolutional neural network; firstly, the multi-layer residual neural network processes the input instance mask feature F, which incorporates label information. mask(i) The image is scaled to match the input dimension of the multi-layer convolutional neural network. Then, the multi-layer convolutional neural network decodes the image to generate a target domain mask. The layers of the multi-layer convolutional neural network are upsampled to amplify the image so that the final output has the same size as the source domain image.

[0116] S2, Image Generation

[0117] S21, input the image mask pair of the source domain, the label information of the target domain and the target domain instance mask obtained in step S1 into the image translation model, the image mask pair includes a set of instance masks and corresponding images; the image translation model includes a pre-trained image inpainting model and an instance generation network.

[0118] S22, based on the source domain image mask pair input in step S21, first aggregate all instance masks of the source domain to obtain a source domain mask, then remove the foreground of the corresponding source domain image according to the source domain mask to obtain a remaining image with the mask part removed, use the image inpainting model to complete the remaining image to obtain the background image of the source domain.

[0119] In this embodiment, the image inpainting model uses the HiFill model, which is published in the paper "Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting" in CVPR 2020. Of course, other existing image inpainting models can also be used and pre-trained in the existing manner through a large picture data set.

[0120] S23, based on the target domain instance mask input in step S21, first aggregate all instance masks of the target domain to obtain a target domain mask, then input the target domain mask and the source domain background image B S obtained in step S22 into the instance generation network; the instance generation network includes an encoder and a generator, and processes as follows:

[0121] Based on the input source domain background image, the background features of the source domain background image are extracted through the encoder;

[0122] Based on the input target domain mask, the edge information of the foreground is obtained through an edge extraction algorithm; the background mask information is obtained by taking the inverse of the target domain mask; the label information of the target domain is feature encoded; then, the foreground edge information, the background mask information and the feature encoding of the target domain label information are fused to obtain a hybrid mask;

[0123] The background features and the hybrid mask are input into the generator, the generator includes a decoding network, the decoding network decodes the input background features, and the generator extracts additional information from the hybrid mask and applies the extracted additional information to the normalized decoding output, and the additional information is used to perform affine transformation on the normalized decoding output, thereby obtaining fusion information including the foreground information corresponding to the target domain mask and the position information indicating the position of the foreground.

[0124] The core of the present application: by introducing target domain mask information in the translation process, the target domain mask information is used as guidance information to generate corresponding instances, which is embodied in the design of the mixed mask, and the additional information extracted from the mixed mask by the generator is applied to the normalized decoding output, and the normalized decoding output is affine transformed by the additional information. Therefore, the generator can be any decoder for image generation in the prior art that can extract additional information from the mixed mask and apply the additional information to the normalized decoding output.

[0125] Specifically, in the present embodiment, the encoder of the instance generation network is a multi-layer residual neural network; the edge extraction algorithm is the CANNY edge detection algorithm; the label information of the target domain is feature encoded by one-hot encoding; the foreground edge information, background mask information and feature encoding of the target domain label information are fused by matrix multiplication.

[0126] As shown in Figure 2 , the generator of the instance generation network is an APADE-ResNet network, which is a neural network formed by adding an APADE block (Adapted Spatially-Adaptive Denormalization) between each convolutional layer and ReLU layer of the ResNet network (Residual Neural Network). That is, the decoding network of the generator is a ResNet network, and the additional information is extracted from the mixed mask by the APADE block; the APADE block includes two inputs, one of which is the output of the convolutional layer and the other of which is the mixed mask; after the mixed mask is scaled to the input feature dimension of the APADE block, it is input into the APADE block. After input, it is first passed through a convolutional layer Conv s , then the output of Conv s is input into two independent convolutional layers Conv1 and Conv2 to generate two embedding vectors γ and β; the output of the convolutional layer is first passed through a batch normalization layer after being input into the APADE block, then the output of the batch normalization layer is calculated with the embedding vectors γ and β according to the following formula as the output of the APADE block:

[0127] T out =γ·F in +β

[0128] Where F in represents the output of the batch normalization layer, and F out represents the output of the APADE block.

[0129] Further, in the embodiment, the APADE-ResNet network is a neural network composed of two-layer ResNet network and APADE blocks, that is, each layer of the APADE-ResNet network includes a forward neural network branch and a shortcut branch, wherein the forward neural network branch includes a first convolutional layer, a first APADE block, a first ReLU layer, a second convolutional layer, a second APADE block and a second ReLU layer connected in sequence, and the shortcut branch is constructed between the second APADE block and the input, and the input is added to the output of the second APADE block to form the input of the second ReLU layer. And the APADE-ResNet network is a multi-layer APADE-ResNet network, the input of the first layer is the background feature and the mixed mask, and the input of each layer after the first layer is the output of the previous layer and the mixed mask; the layers are enlarged through upsampling, so that the final output has the same size as the source domain image.

[0130] Finally, the generated foreground information I' fg and the source domain background image B are fused by using the position information a' to output the target domain picture I' retaining the source domain background. The process is represented as follows:

[0131] I' fg , a' = G img (B, M')

[0132] I' = I' fg · a' + B · (1-a')

[0133] Wherein, G img represents the generator.

[0134] In terms of performance, the more layers of the network, the better the effect can be obtained, but the training overhead will be increased. Therefore, in order to balance the performance and overhead, each multi-layer network in the above embodiment adopts a structure of 4-6 layers.

[0135] The mask deformation network and the image translation model of the application are essentially a kind of generation network for translating part of the semantic information in the image, so the training thereof can be based on the existing GAN generation adversarial network.

[0136] But in order to train more effectively and efficiently, the above model is trained according to the following steps:

[0137] First, the training sample data is prepared:

[0138] Data is collected from a data set, such as an MS COCO data set, which can provide a plurality of domain image-foreground mask data pairs. A domain is defined according to a category to which a foreground belongs, and an image mask pair of each domain is constructed, the image mask pair including a set of instance masks and corresponding images; based on the constructed domains, sample pairs are constructed by two-by-two combination, each sample pair including two domains, one of which is a source domain and the other of which is a target domain, and in all sample pairs, each constructed domain serves as a target domain at least once.

[0139] Then, the model is trained, wherein the mask deformation network is trained, including:

[0140] S31, input at least one sample pair including a source domain and a target domain specified by a translation task;

[0141] S32, for each sample pair input, the mask deformation network is processed according to the following steps:

[0142] From the instance masks of the source domain and the instance masks of the target domain of the sample pair, a set number of instance masks are randomly sampled respectively; the sampled instance masks of the source domain and the instance masks of the target domain are paired two by two, that is, one instance mask of the source domain corresponds to one instance mask of the target domain Wherein, subscript i represents the i-th sample pair, ranging from 1 to P, P is the number of sample pairs input, j represents the j-th mask of the corresponding domain, ranging from 1 to Q, Q is the set sampling number, superscript T represents the target domain, and S represents the source domain. At the same time, after pairing the sampled instance masks of the source domain and the instance masks of the target domain two by two, the center positions of the two-by-two paired instance masks are aligned to achieve better spatial matching.

[0143] The target domain label and the sampled instance masks of the source domain are input into the mask deformation network to generate target domain generated masks corresponding to each instance mask of the source domain

[0144] Based on the pairing information of the source domain instance mask and the target domain instance mask , and the corresponding relationship between the target domain generated mask and the source domain instance mask , a triple consisting of the corresponding source domain instance mask target domain instance mask and target domain generated mask is constructed.

[0145] Since in the embodiment, the generator of the mask deformation network is a multi-layer network, in order to achieve better training effect, the loss can be used to realize the constraint of each layer of the multi-layer network, and the smooth progress of the adversarial training is ensured from multiple scales, therefore, for each input instance mask feature fused with label information feature encoding, the multi-layer network of the generator outputs the target domain generated mask corresponding to different sizes layer by layer when decoding That is The target domain generated mask sequence composed of K is the number of network layers selected for layer-by-layer output in the multi-layer network.

[0146] S33, according to the triplets obtained in step S32, for each triplet, the target domain instance mask is scaled so as to match the size of the corresponding target domain generated mask , as the target domain real mask; and the positive and negative sample pairs are composed of the corresponding target domain generated mask and the target domain real mask.

[0147] Since The target domain generated mask sequence composed of , therefore, in this step, the target domain instance mask is scaled to obtain the target domain real mask which respectively matches the size of the target domain generated mask sequence , and the adversarial samples are composed of the corresponding target domain generated mask and the target domain real mask , and n is the sequence number of the mask sequence.

[0148] S34, input the positive and negative sample pairs obtained in step S33 into the discriminator, and perform adversarial training on the mask deformation network.

[0149] Wherein, the calculation formula of the adversarial loss function of the generator is as follows:

[0150]

[0151] The calculation formula of the adversarial loss function of the discriminator is as follows:

[0152]

[0153] Wherein, D mask represents the discriminator used for adversarial training of the mask deformation network, represents the target domain generated mask output by the nth output layer corresponding to the jth instance mask of the source domain in the ith sample pair; denotes the n-th target domain real mask obtained by scaling the target domain instance mask corresponding to the j-th instance mask of the source domain in the i-th sample pair.

[0154] Meanwhile, by reconstructing the source domain mask, a mask pseudo closed-loop loss function is constructed to supervise the mask generation process. The mask pseudo closed-loop loss function L pc , and the calculation formula is as follows:

[0155]

[0156] wherein, denotes the source domain instance mask sequence in the i-th sample pair denotes the source domain label information in the i-th sample pair; denotes the source domain instance mask reconstructed by the generator G mask of the mask deformation network based on the source domain instance mask and the label information in the i-th sample pair.

[0157] In order to stabilize the training, the semantic consistency of the masks generated in different scales needs to be evaluated, therefore, the masks other than the mask output by the first layer are down-sampled to the size of the mask of the first layer, and a mask consistency loss function L const is constructed, and the calculation formula is as follows:

[0158]

[0159] wherein, d(·) denotes a down-sampling function.

[0160] Because the mask is binary, noise will be generated by the activation function in the generation process, in order to suppress this noise, a mask regularization function L reg is constructed, and the calculation formula is as follows:

[0161]

[0162] wherein, denotes the K-th target domain generated mask corresponding to the j-th instance mask of the source domain in the i-th sample pair, and the K-th is the final output of the generator, and sum(·) denotes the numerical summation of all pixel points in the input mask.

[0163] In summary, the loss function of the adversarial training is:

[0164]

[0165]

[0166] wherein, is the loss function of the generator, is the loss function of the discriminator, Ladversarial is an adversarial loss function for the generator, Ladversarial is an adversarial loss function for the discriminator, pc Lmask is a mask pseudo-closed loop loss function, const Lmask is a mask consistency loss function, reg Lmask is a mask regularization function, and λ is a hyperparameter for adjusting the balance of each loss function.

[0167] S35, if the set number of iterations or the mask deformation network converges, the training is completed, otherwise, returning to step S32.

[0168] Wherein, the instance generation network is trained, including: the instance generation network is trained according to the following steps:

[0169] S41, inputting an image mask pair including at least one domain of a specified target domain of a to-be-translated task;

[0170] S42, for each domain image mask pair input, respectively according to the following steps:

[0171] The input domain image mask pair is input as the source domain image mask pair of the image translation model; the label information and instance mask of the input domain are input as the label information and instance mask of the target domain of the image translation model; the image translation model generates an image I i ′ Wherein, i represents the i-th sample data corresponding to the input of the generated image.

[0172] The image in the input domain image mask pair is input as the real image I i , and the real image I i and the generated image I i ′ constitute a positive and negative sample pair.

[0173] S43, inputting the positive and negative sample pair obtained in step S42 into the discriminator to perform adversarial training on the instance generation network.

[0174] The loss function of adversarial training is:

[0175]

[0176]

[0177] Wherein, Lgenerator is a loss function for the generator, Ldiscriminator is a loss function for the discriminator, Ladversarial is an adversarial loss function for the generator, Ladversarial is an adversarial loss function for the discriminator, fmapwherein, λ is a hyper-parameter used for adjusting the balance of each loss function;

[0178] The calculation formula of the adversarial loss function of the discriminator is as follows:

[0179]

[0180] The calculation formula of the adversarial loss function of the discriminator is as follows:

[0181]

[0182] wherein, D img represents the discriminator used for adversarial training of the instance generation network, I i ′ represents that the generated generated image is masked based on the input image mask of the i-th domain; I i represents the image of the input image mask pair of the i-th domain; P is the number of sample data input in step D21;

[0183] The calculation formula of the fusion map loss function L fmap

[0184]

[0185] wherein, M i represents the input domain mask composed of the instance mask aggregation of the input image mask pair of the i-th domain; α i ′ represents the position information indicating the foreground position in the fusion information generated by the generator based on the image mask pair of the i-th domain.

[0186] The fusion map loss function L fmap can better guide the position information to change to the correct mask direction.

[0187] S44, reach the set number of iterations or instance generation network convergence, then complete the training, otherwise return to step S42.

[0188] Although the present application is described herein with reference to the embodiments of the present application, the above embodiments are only preferred embodiments of the present application, and the embodiments of the present application are not limited by the above embodiments, it should be understood that those skilled in the art can design many other modifications and embodiments, which will fall within the scope and spirit of the principles disclosed in the present application.

[0189] The effects of the present application are illustrated by combining simulation experiments with the above-mentioned model:

[0190] Simulation experiment one:​​​

[0191] Test conditions:

[0192] System: Ubuntu 20.04, software: Python 3.6, processor: Intel(R) Xeon(R) CPU E5-2678v3@2.50GHz x 2, memory: 256GB.

[0193] Experimental content:

[0194] By comparing the existing unsupervised image translation scheme with the technical scheme of the application, the source domain data, including images and masks, and the target domain label are used as input, and the image containing the target domain foreground is generated under the premise of preserving the background information.

[0195] Four pairs of data sets were tested, including: sheep & giraffe, bottle & cup, orange & banana, and pants & group, and the results are shown in Figure 3 .

[0196] Analysis of experimental results:

[0197] As can be seen from Figure 3 , compared with the previous scheme, that is, instaGAN, the method of the application introduces the target domain mask information as a guide in the image generation stage, and the generated foreground instance also conforms to the shape constraint of the mask more, has more reasonable visual effect, and well completes the conversion from the source domain to the target domain in shape and texture. The mask deformation network can also achieve more reasonable mask cross-domain translation through the matching self-supervised learning method.

[0198] Simulation experiment two:

[0199] In combination with the video question and answer method of the prior art, the effect of the application is compared and illustrated through simulation experiments:

[0200] Test conditions:

[0201] System: Ubuntu 20.04, software: Python 3.6, processor: Intel(R) Xeon(R) CPU E5-2678v3@2.50GHz x 2, memory: 256GB;

[0202] Test description: The sheep & giraffe dataset is used as the dataset in the experiment, and the dataset used in the experiment is in the form of image mask pair, that is, one picture corresponds to several foreground instance masks. Due to the particularity of the task, the training data is in the form of image mask pairs in two different domains and is input into the network for training. Specifically, the present application and the comparative algorithm are trained on the training set in the dataset in turn. After training, the present application and the comparative algorithm are used to generate test on the test set of the dataset, and the generated picture result is obtained. The comparative algorithm is instaGAN.

[0203] In the experiment, the test set is randomly divided into several batches, and each batch includes 2 image mask pairs from two domains respectively.

[0204] 1) Investigate the classification accuracy of the generated picture / foreground instance:

[0205] Test on the test set, generate the target domain picture-mask set, and count the generated picture set in two ways: (1) in image units, use the pre-trained image classification model to classify the image, and count the number of pictures correctly classified into the target domain; (2) in foreground instance units, use the pre-trained instance classification model to count the number of foreground instances correctly classified into the target domain. Adopt these two ways respectively and calculate to obtain the image classification score CS and the instance classification score CS(Masked).

[0206] 2) Investigate the accuracy of the generated foreground instance being correctly detected and recognized:

[0207] Use the pre-trained Mask-RCNN as a detector to get the predicted label, score and mask from each generated picture. Further, three statistical analysis methods are used to evaluate the quality of the generated foreground image.(1) Count the number of masks detected by the detector from the generated picture set and calculate the ratio of the number of masks generated by the model, to obtain the average matching rate (MMR), from the number of angles to evaluate the probability of correctly recognizing the generated mask.(2) The predicted score represents the confidence of being classified into a specific domain, so we calculate the average score of being classified into the target domain to obtain the average object detection score (MODS).(3) By calculating the intersection over union of the predicted mask and the generated mask, the average effective IoU score (MVIS) is obtained, which evaluates whether the shape is successfully translated to the target domain in the translation process from the perspective of whether the generated mask shape fits the predicted mask shape.

[0208] The above experimental results are shown in Tables 1 and 2. Through the data analysis and comparison of Tables 1 and 2, it can be known that the quality of the generated picture of the present application is better, and these results verify the effectiveness of the translation method and the corresponding supervised data construction method of the present application.

[0209] Table 1

[0210]

[0211] Table 2

[0212]

Claims

1. A deformable instance-level image translation method, characterized in that, The method comprises the following steps: C1, inputting an image mask pair of a source domain and label information and an instance mask of a target domain into an image translation model, the image mask pair comprising a set of instance masks and corresponding images; the image translation model comprising a pre-trained image completion model and an instance generation network; C2, based on the source domain image mask pair input in step C1, first aggregating all instance masks of the source domain to obtain a source domain mask, then removing the foreground of the corresponding source domain image according to the source domain mask to obtain a remaining image with the mask part removed, using the image completion model to complete the remaining image to obtain a background image of the source domain; C3, based on the target domain instance mask input in step C1, first aggregate all the instance masks of the target domain to obtain a target domain mask, and then combine the target domain mask with the source domain background image B obtained in step C2 S , an instance generation network; the instance generation network comprises an encoder and a generator, and processes as follows: based on the input source domain background image, extracting the background features of the source domain background image through an encoder; based on the input target domain mask, obtaining the edge information of the foreground through an edge extraction algorithm; obtaining the background mask information by taking the inverse of the target domain mask; encoding the features of the target domain label information; then, fusing the edge information of the foreground, the background mask information and the feature encoding of the target domain label information to obtain a hybrid mask; inputting the background features and the hybrid mask into a generator, the generator comprising a decoding network, the decoding network decoding the input background features, the generator extracting additional information from the hybrid mask and applying the extracted additional information to the normalized decoding output, and the additional information being used to perform affine transformation on the normalized decoding output to obtain fusion information comprising the foreground information corresponding to the target domain mask and the position information indicating the position of the foreground; finally, fusing the generated foreground information and the source domain background image using the position information to output a target domain picture retaining the source domain background; the image completion model is a HiFill model; the encoder of the instance generation network is a multi-layer residual neural network; the edge extraction algorithm is a CANNY edge detection algorithm; the label information of the target domain is encoded into features by one-hot encoding; the edge information of the foreground, the background mask information and the feature encoding of the target domain label information are fused by matrix multiplication; the generator of the instance generation network is an APADE-ResNet network, which is a neural network comprising an APADE block added between each convolutional layer and ReLU layer of a ResNet network, i.e., the decoding network of the generator is a ResNet network, and the APADE block extracts additional information from the hybrid mask; the APADE block comprises two inputs, one of which is the output of the convolutional layer and the other of which is the hybrid mask; After the mixed mask is scaled to the input feature dimension of the APADE block, it is input into the APADE block. After the input, firstly, a convolutional layer Conv s is used to process the input, and then the output of Conv s is input into two independent convolutional layers Conv1 and Conv2 respectively to generate two embedding vectors γ and β. after the output of the convolutional layer is input into the APADE block, it is first subjected to a batch normalization layer, and then the output of the batch normalization layer is calculated with the embedding vectors γ and β according to the following formula to serve as the output of the APADE block: F out = γ · F in + β where F in represents the output of the batch normalization layer, F out represents the output of the APADE block; the APADE-ResNet network is a neural network comprising an APADE block added between two layers of a ResNet network, i.e., Each layer of the APADE-ResNet network comprises a forward neural network branch and a shortcut branch, wherein the forward neural network branch comprises a first convolutional layer, a first APADE block, a first ReLU layer, a second convolutional layer, a second APADE block and a second ReLU layer connected in series, and the shortcut branch is constructed between the second APADE block and the input to add the input to the output of the second APADE block to form the input of the second ReLU layer; The APADE-ResNet network is a multi-layer APADE-ResNet network, the input of the first layer is a background feature and a mixed mask, and the input of each layer after the first layer is the output of the previous layer and the mixed mask; the layers are enlarged through upsampling, so that the final output has the same size as the source domain image.

2. The deformable exemplar-based image translation method of claim 1, wherein, The instance generation network is trained according to the following steps: D1, training sample data preparation: Collect data from a data set to define a domain of a foreground category, and construct an image mask pair for each domain, wherein the image mask pair comprises a group of instance masks and corresponding images; D2, input sample data, and train the instance generation network, including: D21, input an image mask pair comprising at least one domain of a target domain specified by a translation task; D22, for each image mask pair of each domain input, process according to the following steps: the image mask pair of the input domain as the source domain image mask pair of the input image translation model; the label information and instance mask of the input domain as the label information and instance mask of the target domain of the input image translation model; generating an image I' by the image translation model according to the input i wherein i represents the i-th sample data corresponding to the input of the generated image. The image in the input domain image mask pair is taken as the real image I i , and the generated image I' i is taken as the negative sample. i The positive and negative sample pair is composed of the real image I D23, input the positive and negative sample pairs obtained in step D22 into the discriminator to perform adversarial training on the instance generation network; D24, if a set number of iterations is reached or the instance generation network converges, the training is completed, otherwise return to step D22.

3. The deformable exemplar-based image translation method of claim 2, wherein, In step D23, the loss function of the adversarial training is: wherein, Lg is a loss function of the generator, Ld is a loss function of the discriminator, Lgadversarial is an adversarial loss function of the generator, Ldadversarial is an adversarial loss function of the discriminator, L fmap Lfusion is a fusion graph loss function, and λ is a hyperparameter for adjusting the balance of each loss function. The generator's adversarial loss function The formula is as follows: The adversarial loss function of the discriminator The calculation formula is as follows: wherein D img denotes a discriminator for instance generation network adversarial training, I' i denotes the generated generated image based on the input i-th domain image mask pair; I i denotes the input i-th domain image mask pair image; P is the number of sample data input in step D21; The fusion graph loss function L fmap The calculation formula is as follows: wherein M i represents an input domain mask constituted by aggregating the instance masks of the input image mask pair of the i-th domain; a' i represents position information indicating the position of the foreground in the fusion information generated by the generator based on the image mask pair of the i-th domain.

4. A deformable instance-level image translation method, characterized in that, Including the following steps: A, mask deformation A1, input the instance mask of the source domain and the target domain label into the pre-trained mask deformation network; the mask deformation network comprises an encoder and a generator; A2, the mask deformation network deforms the mask according to the following steps: A21, aggregate all instance masks of the source domain to obtain a source domain mask; extract features of the source domain mask through an encoder to obtain overall features F of the source domain mask img ; fuse each instance mask of the source domain with the overall features F img respectively to obtain instance mask features F mask(i) corresponding to each instance mask; then, encode features of label information of the target domain, and embed the feature encoding of the label information of the target domain into the mask features F mask(i) of each instance respectively; A22, input each instance mask feature fused with the label information feature code into the generator respectively, and generate a target domain generated mask as the final output of the generator as the instance mask of the corresponding target domain; B, image generation Input the image mask pair of the source domain, the label information of the target domain and the target domain instance mask obtained in step A into the deformable instance-level image translation method according to any one of claims 1-3 to generate a target domain image that retains the source domain background.

5. The deformable instance-level image translation method of claim 4, wherein: The encoder of the mask deformation network is a multi-layer convolutional neural network, and the instance mask of the source domain is multiplied by a matrix to obtain a mask feature with the overall feature F img is fused; The label information of the target domain is encoded by one-hot encoding, and the instance mask feature F mask(i) is fused with the feature encoding of the target domain label information; or, the label information of the target domain is encoded by a convolutional neural network, and then the instance mask feature F mask(i) is spliced with the feature encoding of the target domain label information.

6. The deformable instance-level image translation method of claim 4, wherein: The generator of the mask deformation network comprises a multi-layer residual neural network and a multi-layer convolutional neural network; first, the multi-layer residual neural network is used to process the input instance mask feature F mask(i) The multi-layer residual neural network is scaled to match the input dimension of the multi-layer convolutional neural network, and then the multi-layer convolutional neural network is used for decoding to generate the target domain generated mask; the layers of the multi-layer convolutional neural network are enlarged through upsampling, so that the final output has the same size as the source domain image.

7. A deformable instance-level image translation method as defined in any one of claims 4, 5 or 6, wherein, The mask deformation network is trained according to the following steps: B1, training sample data preparation: Collect masks from a data set to define a domain of a foreground category, and construct sample pairs by combining each two domains based on the constructed domains, wherein each sample pair contains two domains, one of which is a source domain and the other of which is a target domain, and in all sample pairs, each constructed domain is used as a target domain at least once; B2, train the mask deformation network: B21, input at least one sample pair comprising a source domain and a target domain specified by a translation task; B22, for each sample pair input, the mask deformation network processes according to the following steps: From the instance masks of the source domain and the instance masks of the target domain of the sample pair, a set number of instance masks are randomly sampled respectively; the instance masks of the source domain and the instance masks of the target domain obtained by sampling are paired two by two, that is, one instance mask of the source domain is paired with one instance mask of the target domain corresponding to one instance mask of the target domain wherein, subscript i represents the i-th sample pair, ranging from 1 to P, P is the number of input sample pairs, j represents the j-th mask corresponding to the domain, ranging from 1 to Q, Q is the set sampling number, superscript T represents the target domain, and S represents the source domain; The target domain label and the instance mask of the source domain obtained by sampling are input into the mask deformation network to generate a target domain generated mask corresponding to each instance mask of the source domain pairing information based on a source domain instance mask and a target domain instance mask , and a target domain generated mask and a correspondence relationship between the source domain instance mask , a target domain instance mask corresponding to the source domain instance mask , and a target domain generated mask ​ B23. Triplet obtained from step B22, for each triplet, mask its target domain instance Scale so that it matches the corresponding target domain generated mask Match the size as the target domain real mask; and form positive and negative sample pairs by the corresponding target domain generated mask and the target domain real mask; B24, input the positive and negative sample pairs obtained in step B23 into the discriminator to perform adversarial training on the mask deformation network; B25, if a set number of iterations or the mask deformation network converges, the training is completed, otherwise, return to step B22. 8.The deformable instance-level image translation method of claim 7, wherein: In the step B22, after the instance masks of the source domain and the instance masks of the target domain obtained by sampling are paired two by two, the center positions of the two-by-two paired instance masks are aligned. 9.The deformable instance-level image translation method of claim 7, wherein: The generator of the mask deformation network is a multi-layer network, and in the step B22, for each input instance mask feature fused with label information feature code, the multi-layer network of the generator generates a target domain generated mask corresponding to different sizes when decoding, and the last K layers of the generator output different sizes layer by layer That is For The generated target domain generated mask sequence, and K is the number of network layers selected for layer-by-layer output in the multi-layer network; In step B23, for each triple, a target domain instance mask is created. Scaling is performed to obtain the mask sequences generated from the target domain. True mask of target domain with matching size And generate a mask from the corresponding target domain. and the target domain real mask This constitutes an adversarial example, where n is the sequence number of the mask sequence. 10.The deformable instance-level image translation method of claim 9, wherein: In the step B24, the loss function of the adversarial training is: wherein, Lg is a loss function for the generator, Ld is a loss function for the discriminator, Lg-adversarial is an adversarial loss function for the generator, Ld-adversarial is an adversarial loss function for the discriminator, L pc Lmask-pseudo-closed-loop is a mask pseudo-closed-loop loss function, const Lmask-consistency is a mask consistency loss function, reg Lmask-regularization is a mask regularization function, and λ is a hyperparameter for adjusting the balance of each loss function. The generator's adversarial loss function The formula for calculating the generator's adversarial loss function is as follows: The adversarial loss function of the discriminator The calculation formula is as follows: wherein D mask denotes a discriminator for the adversarial training of the mask morphing network, denotes the target domain generated mask output by the n-th output layer corresponding to the j-th instance mask of the source domain in the i-th sample pair; denotes the n-th target domain real mask obtained by scaling the target domain instance mask corresponding to the j-th instance mask of the source domain in the i-th sample pair; The mask pseudo-closed loop loss function L pc The calculation formula is as follows: wherein, denotes a sequence of source domain instance masks in the ith sample pair denotes label information of the ith sample pair source domain; denotes a sequence of source domain instance masks in the ith sample pair mask the reconstructed source domain instance mask; The mask consistency loss function L const The calculation formula is as follows: wherein d(·) represents a down-sampling function. The mask regularization function L reg The calculation formula is as follows: wherein, denotes the K-th target domain generated mask corresponding to the j-th instance mask of the i-th sample pair source domain, the K-th being the final output of the generator, and sum(·) denotes the numerical summation of all pixel points in the input mask.

Images