Training of an image translation model, image translation method, device and storage medium

Abstract

The application provides a kind of training of image translation model, image translation method, equipment and storage medium, it is related to image processing technical field.The training method is by using initial image translation model, input source image in sample image pair, obtain training image;Sample image pair includes: source image, target image, wherein, target image is the reference standard of source image translation;Using initial image discrimination model, input sample image pair and training image pair, obtain image pair discrimination result;Wherein, training image pair includes: source image, training image;According to image pair discrimination result, adjust the parameters of initial image translation model, initial image discrimination model, obtain target image translation model.Thereby, by using image pair for model training, and using the combination of image translation model and image discrimination model, form unsupervised model training network, improve model training precision and training speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to training of an image translation model, an image translation method, equipment and a storage medium. BACKGROUND

[0002] As a very common information medium in nature, images contain rich semantic information. In the Internet era, pictures are the most widely and frequently disseminated information form in addition to natural language. The human ability to understand the unique semantic information of images is called vision, and the simplest one is image classification. Classification is generally based on the common points and differences of certain features to divide the field to which each sample belongs. For images, these classification fields include low-level semantics such as resolution and color, and high-level semantics such as content, style and object relationship. Just as in the translation between various languages in natural language, there is also a translation task between images, which is called image-to-image translation, simply referred to as image translation. Generally, an image translation model needs to generate and transform the input image to some extent to have new features while retaining other irrelevant features.

[0003] At present, there are generally two types of image classification: the first type of scheme trains a model using offline real data sets collected in reality, and the second type of scheme uses a pre-trained image generator to perform latent variable reverse reconstruction and latent variable editing.

[0004] Generally, the second type of scheme is used more, but it has a slow training speed, which is difficult to meet the requirements of reconstruction quality and running speed in real scenarios. SUMMARY

[0005] The present application provides a training method of an image translation model, an image translation method, equipment and a storage medium to solve the problem of slow image model training speed in the prior art.

[0006] To achieve the above object, the technical scheme adopted by the embodiments of the present application is as follows:

[0007] In a first aspect, the embodiments of the present application provide a training method of an image translation model, comprising:

[0008] An initial image translation model is used to input a source image in a sample image pair to obtain a training image; the sample image pair comprises the source image and a target image, wherein the target image is a reference standard for translation of the source image;

[0009] An initial image discrimination model is used to input the sample image pair and a training image pair to obtain an image pair discrimination result; wherein the training image pair comprises the source image and the training image;

[0010] According to the image pair discrimination result, the parameters of the initial image translation model and the initial image discrimination model are adjusted to obtain a target image translation model.

[0011] Optionally, before the initial image translation model is used to input the source image in the sample image pair to obtain the training image, the method further comprises:

[0012] A plurality of sets of noises are input to a preset image generator to obtain a plurality of sets of the source images and the target images, which constitute a plurality of sets of the sample image pairs.

[0013] Optionally, the initial image translation model comprises an encoder and a reparameterization generator, and the reparameterization generator comprises a plurality of modulation convolutional layers and a mapper.

[0014] The source image, preset noise and constant feature map are input;

[0015] The preset noise is converted into a noise offset vector in a feature space by using the mapper;

[0016] The source image is converted into a source image vector in the feature space by using the encoder;

[0017] The training image is generated according to the source image vector, the noise offset vector and the constant feature map by using the reparameterization generator.

[0018] Optionally, before the reparameterization generator is used to generate the training image according to the source image vector, the noise offset vector and the constant feature map, the method further comprises:

[0019] The pre-training generator is reparameterized according to the first affine parameter and the second affine parameter to obtain the reparameterization generator.

[0020] Optionally, the initial image discrimination model is used to input the sample image pair and the training image pair to obtain the image pair discrimination result, comprising:

[0021] The pre-training image discrimination model is reparameterized according to the third affine parameter and a preset convolutional block to obtain a reparameterization image discrimination model;

[0022] The reparameterization image discrimination model is used to input the sample image pair and the training image pair to obtain the image pair discrimination result.

[0023] Optionally, the parameters of the initial image translation model and the initial image discrimination model are adjusted according to the image pair discrimination result to obtain a target image translation model, comprising:

[0024] determine an image decision loss function according to the image pair discrimination result;

[0025] adjust the parameters of the initial image translation model and the initial image discrimination model in a loop until a deviation of the image decision loss function is less than a preset threshold, and obtain a target image translation model.

[0026] Optionally, the adjusting the parameters of the initial image translation model and the initial image discrimination model in a loop until the deviation of the image decision loss function is less than the preset threshold, and obtaining the target image translation model, comprises:

[0027] keeping the parameters of the initial image translation model unchanged, adjusting the parameters of the initial image discrimination model, and obtaining an adjusted image discrimination model;

[0028] keeping the parameters of the adjusted image discrimination model unchanged, adjusting the parameters of the initial image translation model until the deviation of the image decision loss function is less than the preset threshold, and obtaining the target image translation model.

[0029] In a second aspect, an embodiment of the present application provides an image translation method, and the method comprises:

[0030] obtaining an image to be translated;

[0031] processing the image to be translated by using the target image translation model according to any one of the first aspect, and obtaining a target image.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a storage medium, the processor and the storage medium are communicatively connected through a bus, the storage medium stores program instructions executable by the processor, and the processor invokes the program stored in the storage medium to execute the steps of the training method of the image translation model according to any one of the first aspect or the image translation method according to the second aspect.

[0033] In a fourth aspect, an embodiment of the present application provides a storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to perform the steps of the training method of the image translation model according to any one of the first aspect or the image translation method according to the second aspect.

[0034] Compared with the prior art, the present application has the following beneficial effects:

[0035] The application provides a training method of an image translation model, an image translation method, equipment and a storage medium. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0037] Figure 1 A flowchart of a training method of an image translation model provided by the present application is shown in the figure.

[0038] Figure 2 A flowchart of a method for obtaining a training image provided by the present application is shown in the figure.

[0039] Figure 3 A flowchart of an image pair discrimination method provided by the present application is shown in the figure.

[0040] Figure 4 A flowchart of a model determination method provided by the present application is shown in the figure.

[0041] Figure 5 A flowchart of a model adjustment method provided by the present application is shown in the figure.

[0042] Figure 6 A flowchart of an image translation method provided by the present application is shown in the figure.

[0043] Figure 7 A schematic diagram of a training device of an image translation model provided by the embodiments of the present application is shown in the figure.

[0044] Figure 8 A schematic diagram of an image translation device provided by the embodiments of the present application is shown in the figure.

[0045] Figure 9A schematic diagram of an electronic device provided in an embodiment of the present application.

[0046] Icon: 701 - translation module, 702 - discrimination module, 703 - adjustment module, 704 - generation module, 801 - acquisition module, 802 - processing module, 901 - processor, 902 - storage medium. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0048] Therefore, the detailed description of the embodiments of the present application provided in the drawings below is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making creative efforts fall within the scope of protection of the present application.

[0049] It should be noted that: similar reference numbers and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0050] In addition, if the terms "first", "second", etc. are used only to distinguish description, and cannot be understood as indicating or implying relative importance.

[0051] It should be noted that: the features in the embodiments of the present application can be combined with each other without conflict.

[0052] To improve the training speed and training accuracy of the image translation model, the present application provides a training method of an image translation model, an image translation method, an equipment and a storage medium.

[0053] The training method of an image translation model provided in the present application is explained and described below through specific examples. Figure 1 A flowchart of a training method of an image translation model provided in the present application, the execution subject of the method can be an electronic device, which can be a device with computing processing function, such as desktop computer, notebook computer, etc. As shown in the figure, the method comprises: Figure 1 As shown in the figure, the method comprises:

[0054] S101, using an initial image translation model, inputting a source image in a sample image pair, and obtaining a training image.

[0055] The sample image pair includes a source image and a target image, wherein the target image is a reference standard for translation of the source image, and the source image is an image to be translated, that is, the target image is a standard image translated from the source image.

[0056] The initial image translation model is used to input the source image, and the source image is translated to obtain a training image. The training image is a translation result of the source image in this round of image translation.

[0057] In the process of training the image translation model, in order to achieve the expected effect of model training, the user can pre-process the source image according to the image translation requirements (for example, generate a hairline for a portrait, and perform animation processing on the portrait) to obtain a target image. The training image obtained by the image translation model will take the target image as a reference standard, that is, the more similar the training image is to the target image, the better the effect of the trained image translation model.

[0058] S102, an initial image discrimination model is used to input the sample image pair and the training image pair to obtain an image pair discrimination result.

[0059] The training image pair includes a source image and a training image.

[0060] After obtaining the training image translated from the source image by the image translation model, the source image and the training image can be combined to form a training image pair, and then combined with the sample image pair to form two pairs of images. It can be understood that the sample image pair represents the gap between the source image and the target image. The training image pair represents the gap between the source image and the training image. Compared with using the target image and the training image for discrimination, using the sample image pair and the training image pair for discrimination is equivalent to introducing the source image in the discrimination process, so that the discrimination result is more accurate and the discrimination speed is improved.

[0061] S103, according to the image pair discrimination result, adjusting the parameters of the initial image translation model and the initial image discrimination model to obtain a target image translation model.

[0062] In the process of training the model, the image translation model and the image discrimination model are used. The purpose of the image translation model is to translate a training image closer to the target image to pass the discrimination of the image discrimination model (successfully deceive the image discrimination model). The purpose of the image discrimination model is to strictly discriminate (so that it is not deceived by the image translation model). Through the combination of the image translation model and the image discrimination model, the two models supervise each other and perform adversarial training to form an unsupervised model adversarial training network.

[0063] According to the image pair discrimination result, the parameters of the initial image translation model and the initial image discrimination model are adjusted. The purpose of the adjustment is to make the image translation model more accurate in translating images (more successful in deceiving the image discrimination model) and the image discrimination model more strict in discrimination (more successful in not being deceived by the image translation model).

[0064] A plurality of sets of sample image pairs are input, and the above steps S101 and S102 are cycled. Each time, the parameters of the initial image translation model and the initial image discrimination model are adjusted according to the image pair discrimination result, until the model obtained by training meets the user's demand, and the model training is completed. The trained image translation model is used as the target image translation model. By using image pairs for model training and combining the image translation model and the image discrimination model, an unsupervised model training network is formed, and the model training accuracy and speed are improved.

[0065] In summary, in the embodiment, the initial image translation model is used to input the source image in the sample image pair to obtain a training image. The sample image pair includes a source image and a target image, where the target image is a reference standard for the translation of the source image. The initial image discrimination model is used to input the sample image pair and a training image pair to obtain an image pair discrimination result. The training image pair includes a source image and a training image. According to the image pair discrimination result, the parameters of the initial image translation model and the initial image discrimination model are adjusted to obtain a target image translation model. Thus, by using image pairs for model training and combining the image translation model and the image discrimination model, an unsupervised model training network is formed, and the model training accuracy and speed are improved.

[0066] In the above Figure 1 Based on the corresponding embodiment, before the initial image translation model is used to input the source image in the sample image pair to obtain a training image in S101, the method further includes:

[0067] A plurality of sets of noise are input to a preset image generator to obtain a plurality of sets of source images and target images, and form a plurality of sets of sample image pairs.

[0068] For example, the preset image generator is a pair of pre-trained generators, including a source image generator and a target image generator. The input of the preset image generator is noise, and the output is an image. A plurality of sets of noise can be randomly sampled from a standard Gaussian distribution. In each round, a set of noise is input into the source image generator (containing initial source image information) and the target image generator (containing initial target image information) respectively to obtain a set of source images and a set of target images. A set of source images and a set of target images form a set of sample image pairs. The specific image generation method can be shown in the following formula (1):

[0069] (x,y)=(G x(z), y (z))(1)

[0070] wherein z is noise, G x is an expression of the target image generator, G y is an expression of the source image generator, x is the target image, and y is the source image.

[0071] By inputting multiple sets of noise into the preset image generator, only one set of real initial source image and initial target image is needed to generate multiple sets of source images and target images. Multiple sets of source images do not need to be obtained from a real scene, and the target images are obtained by processing the multiple sets of source images in the real scene. The number of training samples is greatly expanded, the cost of obtaining samples is reduced, and the efficiency of obtaining samples is improved.

[0072] In summary, in the embodiment, multiple sets of noise are input into the preset image generator to obtain multiple sets of source images and target images, which constitute multiple sets of sample image pairs. Thus, the number of training samples is greatly expanded, the cost of obtaining samples is reduced, and the efficiency of obtaining samples is improved.

[0073] In another embodiment, the application also provides a method for translating to obtain training images. Figure 2 A flowchart of a method for translating to obtain training images provided by the application is shown in FIG. 1. As shown in FIG. 1, the initial image translation model includes an encoder and a reparameterization generator, and the reparameterization generator includes multiple modulation convolution layers and a mapper. Figure 2

[0074] wherein the reparameterization generator refers to a reparameterization generator, reparameterization refers to sampling from a distribution, and the distribution is with parameters. If direct sampling (the sampling action is discrete, and discrete data is not differentiable) is performed, there is no gradient information, and the parameter gradient will not be updated during neural network backpropagation. Reparameterization can ensure sampling from the distribution while preserving gradient information.

[0075] In S101, the initial image translation model is used to input the source image in the sample image pair to obtain the training image, including:

[0076] S201, input the source image, the preset noise, and the constant feature map.

[0077] ​For example, when inputting the source image into the image translation model, the preset noise and the constant feature map are also inputted at the same time. The preset noise and the constant feature map are the noise and the constant feature map corresponding to the reparameterization generator. Taking the reparameterization generator as a pre-trained styleGAN generator as an example, the constant feature map is the constant input in the styleGAN generator, and the constant feature map is a preset fixed constant vector as the input of the styleGAN generator. The preset noise can generate a vector in the feature space, and the vector is input into the styleGAN generator as the style information of the image.

[0078] S202, converting the preset noise into a noise offset vector in the feature space by using a mapper.

[0079] For example, the feature space can be a W space, and the specific conversion manner is shown in the following formula (2):

[0080] w=M(z) (2)

[0081] Wherein, M represents the expression of the mapper, z represents the preset noise, and w represents the noise offset vector.

[0082] S203, converting the source image into a source image vector in the feature space by using an encoder.

[0083] Wherein, the encoder can be represented as Wherein, f i represents the i-th layer of the encoder. Further, the output of the i-th layer of the encoder can be shown in the following formula (3):

[0084] e i = i (e i+1 )=Conv i (e i+1 )(3)

[0085] Wherein, e n+1 is the source image, i=(0, 1, …, n), and e i represents the output of the i-th layer of the encoder.

[0086] When i=0, the final output of the encoder is e0={w1,…,w n , which is the W+ space offset of the encoding output, that is, the noise offset vector in step S202.

[0087] Therefore, the output of the encoder includes two parts, which are the output of each layer of the encoder and the noise offset vector.

[0088] S204, generating a training image by using the reparameterization generator according to the source image vector, the noise offset vector and the constant feature map.

[0089] Repeat steps S203 and S204 to iterate through all layer outputs of the encoder and reparameter generator to generate a complete training image.

[0090] The image translation model employs a reparameter generator, transforming it into a reparameter network structure. The reparameter generator produces training images based on the source image vector, noise offset vector, and constant feature maps. This allows for smooth transfer between tasks with minimal cost, preserving the properties of a reparameter network and improving training speed.

[0091] In summary, in this embodiment, the initial image translation model includes an encoder and a reparameter generator. The reparameter generator includes multiple modulation convolutional layers and a mapper. It takes a source image, preset noise, and a constant feature map as input; uses the mapper to convert the preset noise into a noise offset vector in the feature space; uses the encoder to convert the source image into a source image vector in the feature space; and uses the reparameter generator to generate training images based on the source image vector, the noise offset vector, and the constant feature map. This ensures the properties of the reparameter network and improves the training speed.

[0092] In the above Figure 2 Based on the corresponding embodiments, in this application embodiment, before generating the training image using a reparameter generator in S204 according to the source image vector, noise offset vector, and constant feature map, the method further includes:

[0093] The pre-trained generator is re-parameterized based on the first and second affine parameters to obtain a re-parameterized generator.

[0094] The output features of the i-th layer of the pre-trained generator can be represented by the following formula (4):

[0095] c i =ModConv i (c i-1 ,w)(4)

[0096] Among them, c i Let w be the output feature of the i-th layer of the pre-trained generator, and w be the noise offset vector.

[0097] Based on the first and second affine parameters, the pre-trained generator is re-parameterized to change its structure and obtain a re-parameterized generator. The output features of the i-th layer of the re-parameterized generator can be represented by the following formula (5):

[0098] c i =ModConv i (c i-1 ,w+αw i )+ei (5)

[0099] where a is the first affine parameter and b is the second affine parameter, e i The output of the i-th layer of the encoder, w i is the W+ spatial offset.

[0100] The first affine parameter and the second affine parameter are introduced, and the output e i of each layer of the encoder is re-parameterized, and the final output W+ spatial offset is re-parameterized. The first affine parameter and the second affine parameter are affine parameters that can be learned and optimized, and they are initialized to 0, so that the prior of the re-parameterized generator network can be utilized, and the task can be smoothly transferred between tasks at a minimum cost, to ensure the properties of the re-parameterized network, and a fast convergence speed is obtained, and the training speed is improved.

[0101] In summary, in the embodiment, the pre-trained generator is re-parameterized according to the first affine parameter and the second affine parameter, and a re-parameterized generator is obtained. Thus, the properties of the re-parameterized network are ensured, and the training speed is improved.

[0102] In another embodiment, the application also provides an image pair discrimination method. Figure 3 A flowchart of an image pair discrimination method provided by the application is shown in FIG. 1. As shown in FIG. 1, in S102, an initial image discrimination model is used to input a sample image pair and a training image pair to obtain an image pair discrimination result, including: Figure 3

[0103] S301, re-parameterize the pre-trained image discrimination model according to the third affine parameter and the preset convolution block to obtain a re-parameterized image discrimination model.

[0104] The image discrimination model is used to discriminate images. For the pre-trained image discrimination model (i.e., the original image discrimination model), the input is a single image, while in the above embodiments of the application, paired data (image pairs) are generated, so the input of the image discrimination model is also paired images (image pairs), so the pre-trained image discrimination model also needs to be re-parameterized.

[0105] The image discrimination model can be regarded as a stack of n down-sampling residual blocks d, and the first layer d n The encoded feature map is regarded as a whole, as shown in the following formula (6):

[0106]

[0107] The re-parameterized image discrimination model obtained according to the third affine parameter and the preset convolution block is in the form of the following formula (7): ​

[0108]

[0109] wherein, y is a third affine parameter, the third affine parameter is an affine parameter initialized as 0 and can be learned by optimization, d ′ n The preset convolution block can be a discriminator convolution block and is randomly initialized.

[0110] S302, input the sample image pair and the training image pair into the re-parameterized image discrimination model to obtain an image pair discrimination result.

[0111] The sample image pair includes a source image and a target image, and the training image pair includes a source image and a training image. The sample image pair is input into the re-parameterized image discrimination model to obtain a discrimination result of the sample image pair. The training image pair is input into the re-parameterized image discrimination model to obtain a discrimination result of the training image pair.

[0112] In summary, in the embodiment, the pre-trained image discrimination model is re-parameterized according to the third affine parameter and the preset convolution block to obtain a re-parameterized image discrimination model, and the sample image pair and the training image pair are input into the re-parameterized image discrimination model to obtain an image pair discrimination result. Thus, the image pair is discriminated by re-parameterizing the image discrimination model, and the discrimination efficiency is improved.

[0113] In another embodiment, the application further provides a model determination method. Figure 4 A flowchart of a model determination method provided by the application is shown in FIG. 3. Figure 4 As shown in FIG. 3, in S103, the parameters of the initial image translation model and the initial image discrimination model are adjusted according to the image pair discrimination result to obtain a target image translation model, including:

[0114] S401, determine an image judgment loss function according to the image pair discrimination result.

[0115] The image pair discrimination result includes a discrimination result of a sample image pair and a discrimination result of a training image pair. The image judgment loss function determined according to the discrimination result of the sample image pair and the discrimination result of the training image pair is shown in the following formula (8):

[0116]

[0117] wherein, D(x,y) is the discrimination result of the sample image pair, D(T(y),) is the discrimination result of the training image pair, and T(y) is the training image.

[0118] The loss function represents which image pair is the sample image pair and which image pair is the training image pair (represented as ) that needs to be accurately distinguished by the image discrimination model, and the image translation model needs to make the training image pair more likely to pass the judgment of the image discrimination model, so that the image discrimination model judges the training image pair as the sample image pair (so that the gap between the training image pair and the sample image pair is smaller, and the image translation model successfully deceives the image discrimination model, which is represented as ) in the loss function.

[0119] S402, the parameters of the initial image translation model and the initial image discrimination model are adjusted in a loop until the deviation of the image judgment loss function is less than a preset threshold, and a target image translation model is obtained.

[0120] For each input image pair discrimination result, the deviation of the corresponding image judgment loss function is obtained. If the deviation of the image judgment loss function is greater than or equal to the preset threshold, it means that the image judgment loss function is still not stable enough and the function curve is not smooth enough, and the parameters of the initial image translation model and the initial image discrimination model (for example, the first affine parameter, the second affine parameter, and the third affine parameter in the above embodiment) need to be continuously adjusted. If the deviation of the image judgment loss function is less than the preset threshold, it means that the image judgment loss function is stable enough and the function curve is smooth enough, so there is no need to continue adjusting the parameters of the initial image translation model and the initial image discrimination model, and the adjusted image translation model is taken as the target image translation model.

[0121] For example, the preset threshold can be set according to the actual needs of model training, and the deviation of the image judgment loss function can be the mean square error of the image judgment loss function.

[0122] In summary, in this embodiment, the image judgment loss function is determined according to the image pair discrimination result, and the parameters of the initial image translation model and the initial image discrimination model are adjusted in a loop until the deviation of the image judgment loss function is less than a preset threshold, and a target image translation model is obtained. Therefore, through the loss function, the model training is more accurate.

[0123] In another embodiment, the present application also provides a model adjustment method. Figure 5 A flowchart of a model adjustment method provided by the present application is shown in FIG. Figure 5 As shown in S402, the parameters of the initial image translation model and the initial image discrimination model are adjusted in a loop until the deviation of the image judgment loss function is less than a preset threshold, and a target image translation model is obtained, which includes:

[0124] S501, keep the parameters of the initial image translation model unchanged, adjust the parameters of the initial image discrimination model, and obtain an adjusted image discrimination model.

[0125] In each round of parameter adjustment, the parameters of the initial image translation model can be kept unchanged (e.g., the first affine parameter and the second affine parameter are kept unchanged), and the parameters of the initial image discrimination model are adjusted (e.g., the third affine parameter is adjusted). In the loss function, the translation structure of T(y) is kept unchanged, and the discrimination structure of D(x, y) and D(T(y),) is changed. And the adjusted image discrimination model is obtained.

[0126] The expression of the corresponding loss function can be as shown in the following formula (9):

[0127]

[0128] According to formula (9), the model loss is calculated, and then the parameters of the initial image discrimination model are optimized and adjusted by the gradient descent back propagation algorithm, and the parameters of the initial image discrimination model are updated.

[0129] S502, the parameters of the adjusted image discrimination model are kept unchanged, and the parameters of the initial image translation model are adjusted until the deviation of the image judgment loss function is less than a preset threshold, and a target image translation model is obtained.

[0130] Further, the parameters of the initial image discrimination model are kept unchanged (e.g., the third affine parameter is kept unchanged), and the parameters of the initial image translation model are adjusted (e.g., the first affine parameter and the second affine parameter are adjusted). In the loss function, the discrimination structure of D(x, y) and D(T(y),) is kept unchanged, and the translation structure of T(y) is changed. And the adjusted image discrimination model is obtained.

[0131] The expression of the corresponding loss function can be as shown in the following formula (10):

[0132]

[0133] According to formula (10), the model loss is calculated, and then the parameters of the initial image translation model are optimized and adjusted by the gradient descent back propagation algorithm, and the parameters of the initial image translation model are updated.

[0134] In this way, through the adversarial training manner, the training image generated by the image translation model will be closer and closer to the target image.

[0135] In summary, in the embodiment, the parameters of the initial image translation model are kept unchanged, the parameters of the initial image discrimination model are adjusted, and the adjusted image discrimination model is obtained; the parameters of the adjusted image discrimination model are kept unchanged, and the parameters of the initial image translation model are adjusted until the deviation of the image judgment loss function is less than a preset threshold, and a target image translation model is obtained. Thus, through the adversarial training manner, the model training precision is improved.

[0136] The image translation method provided in the application is explained and described below through specific examples. Figure 6 The flowchart of the image translation method provided in the application, and the execution subject of the method can be an electronic device, which can be a device with computing processing function, such as a desktop computer, a notebook computer, etc. As shown in the figure, Figure 6 The method comprises the following steps.

[0137] S601, obtaining an image to be translated.

[0138] S602, using the target image translation model in any of the above embodiments to process the image to be translated to obtain a target image.

[0139] Through the above training method, the target image translation model with high translation accuracy has been obtained. The target image obtained by processing the image to be translated using the target image translation model will be more accurate.

[0140] In summary, in the embodiment, an image to be translated is obtained, and the target image translation model in any of the above embodiments is used to process the image to be translated to obtain a target image. Thus, the image translation accuracy is improved.

[0141] Further, the simulation experiment environment of the application is further described.

[0142] The application is developed on the Ubuntu platform, and the developed deep learning framework is based on Pytorch. The language mainly used in the application is Python. And DDP (Distributed Data Parallel) is used as the basic parallel architecture for multi-card training.

[0143] The sample image generator and the target image generator use two methods based on StyleCLIP and StyleGAN-nada. For StyleCLIP, the application uses the text pair "Face" and "Face with fringe hair" to realize the generation of local tampering in the form of adding bangs. And for StyleGAN-nada, the application uses the text pair "Photo" and "Pixar" to realize the global tampering target of face animation. The CLIP model in this paper uses Vision Transformer as the visual coding model. After generating the image pair, the two images are subjected to synchronous data enhancement, including horizontal flip, color change, etc.

[0144] The generator of the image translation model adopts a StyleGAN2 structure. The generator and the image discriminator model are both initialized as the generator and the discriminator in the default face generator StyleGAN2 model. The encoder part in the image translation model adopts a structure similar to the discriminator and is randomly initialized.

[0145] The following describes the training of an image translation model provided by the present application, an image translation device, an apparatus, and a storage medium, and the specific implementation process and technical effects are described above. The following will not be described again.

[0146] Figure 7 A schematic diagram of an image translation model training device provided by an embodiment of the present application is shown in FIG. 7, which includes: Figure 7

[0147] The translation module 701 is configured to input a source image in a sample image pair into an initial image translation model to obtain a training image. The sample image pair includes the source image and a target image, where the target image is a reference standard for translation of the source image.

[0148] The discrimination module 702 is configured to input the sample image pair and a training image pair into an initial image discrimination model to obtain an image pair discrimination result. The training image pair includes the source image and the training image.

[0149] The adjustment module 703 is configured to adjust parameters of the initial image translation model and the initial image discrimination model according to the image pair discrimination result to obtain a target image translation model.

[0150] The generation module 704 is configured to input a plurality of groups of noise into a preset image generator to obtain a plurality of groups of source images and target images, and to form a plurality of groups of sample image pairs.

[0151] Further, the translation module 701 is specifically configured to include an encoder and a reparameterization generator, the reparameterization generator includes a plurality of modulation convolution layers and a mapper, and the source image, a preset noise, and a constant feature map are input. The preset noise is converted into a noise offset vector in a feature space by using the mapper. The source image is converted into a source image vector in the feature space by using the encoder. The reparameterization generator is used to generate a training image according to the source image vector, the noise offset vector, and the constant feature map.

[0152] Further, the translation module 701 is specifically configured to reparameterize the pre-training generator according to the first affine parameter and the second affine parameter to obtain a reparameterization generator.

[0153] ​Further, the determining module 702 is specifically configured to re-parameterize the pre-trained image pair discrimination model according to the third affine parameter and the preset convolution block, to obtain a re-parameterized image pair discrimination model; and input the sample image pair and the training image pair into the re-parameterized image pair discrimination model, to obtain the image pair discrimination result.

[0154] Further, the adjusting module 703 is specifically configured to determine an image judgment loss function according to the image pair discrimination result; and cyclically adjust the parameters of the initial image translation model and the initial image pair discrimination model until the deviation of the image judgment loss function is less than a preset threshold, to obtain the target image translation model.

[0155] Further, the adjusting module 703 is specifically configured to keep the parameters of the initial image translation model unchanged, adjust the parameters of the initial image pair discrimination model, to obtain an adjusted image pair discrimination model; keep the parameters of the adjusted image pair discrimination model unchanged, adjust the parameters of the initial image translation model, until the deviation of the image judgment loss function is less than the preset threshold, to obtain the target image translation model.

[0156] Figure 8 A schematic diagram of an image translation device provided by an embodiment of the present application is shown in FIG. 8. Figure 8 As shown in FIG. 8, the device includes:

[0157] The obtaining module 801 is configured to obtain an image to be translated.

[0158] The processing module 802 is configured to use the target image translation model of any one of the above embodiments to process the image to be translated, to obtain a target image.

[0159] Figure 9 A schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 9.

[0160] The electronic device includes a processor 901 and a storage medium 902. The processor 901 and the storage medium 902 are connected through a bus.

[0161] The storage medium 902 is configured to store a program, and the processor 901 invokes the program stored in the storage medium 902 to execute the above method embodiments. The specific implementation manners and technical effects are similar, and will not be described here again.

[0162] Optionally, the present application also provides a storage medium comprising a program, which, when executed by a processor, is used to perform the above-mentioned method embodiments. In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the described apparatus embodiments are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electric, mechanical or in other forms.

[0163] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0164] In addition, each functional unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can be a physically independent unit, or two or more units can be integrated into a unit. The integrated unit can be implemented in the form of hardware, or in the form of hardware plus software functional units.

[0165] The integrated unit implemented in the form of software functional units can be stored in a storage medium. The software functional units stored in the storage medium include a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to execute part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A training method for an image translation model, characterized in that, include: The initial image translation model is used, and the source image in the input sample image pair is used to obtain the training image; The sample image pair includes: the source image and the target image, wherein the target image serves as a reference standard for translating the source image; An initial image discrimination model is used, and the sample image pair and the training image pair are input to obtain the image pair discrimination result; wherein, the training image pair includes: the source image and the training image; Based on the image pair discrimination results, the parameters of the initial image translation model and the initial image discrimination model are adjusted to obtain the target image translation model; The initial image translation model includes an encoder and a reparameter generator. The reparameter generator includes multiple modulation convolutional layers and a mapper. The initial image translation model is used to obtain training images from a pair of input sample images, including: Input the source image, preset noise, and constant feature map; The mapper is used to convert the preset noise into a noise offset vector in the feature space; The encoder is used to transform the source image into a source image vector in the feature space; The training image is generated using the reparameter generator based on the source image vector, the noise offset vector, and the constant feature map. The initial image discrimination model is adopted, and the input sample image pairs and training image pairs are used to obtain the image pair discrimination results, including: Based on the third affine parameter and the preset convolutional block, the pre-trained image discrimination model is reparameterized to obtain the reparameterized image discrimination model. Using the aforementioned parametric image discrimination model, the sample image pairs and the training image pairs are input to obtain the image pair discrimination results.

2. The method according to claim 1, characterized in that, Before obtaining the training image by using the source image from the input sample image pair and employing the initial image translation model, the method further includes: Multiple sets of noise are input into a preset image generator to obtain multiple sets of source images and target images, forming multiple sets of sample image pairs.

3. The method according to claim 1, characterized in that, Before generating the training image using the reparameter generator based on the source image vector, the noise offset vector, and the constant feature map, the method further includes: The pre-trained generator is re-parameterized based on the first affine parameter and the second affine parameter to obtain the re-parameterized generator.

4. The method according to claim 1, characterized in that, The step of adjusting the parameters of the initial image translation model and the initial image discrimination model based on the image pair discrimination results to obtain the target image translation model includes: Determine the image discrimination loss function based on the image pair discrimination results; The parameters of the initial image translation model and the initial image discrimination model are cyclically adjusted until the deviation of the image discrimination loss function is less than a preset threshold, thereby obtaining the target image translation model.

5. The method according to claim 4, characterized in that, The process of cyclically adjusting the parameters of the initial image translation model and the initial image discrimination model until the deviation of the image discrimination loss function is less than a preset threshold, and obtaining the target image translation model, includes: Keeping the parameters of the initial image translation model unchanged, the parameters of the initial image discrimination model are adjusted to obtain the adjusted image discrimination model; Keeping the parameters of the adjusted image discrimination model unchanged, adjust the parameters of the initial image translation model until the deviation of the image discrimination loss function is less than the preset threshold, and obtain the target image translation model.

6. An image translation method, characterized in that, The method includes: Obtain the image to be translated; The target image is obtained by processing the image to be translated using the target image translation model according to any one of claims 1-4.

7. An electronic device, characterized in that, include: The processor and the storage medium are connected via a bus for communication. The storage medium stores program instructions executable by the processor. The processor calls the program stored in the storage medium to execute the steps of the training method of the image translation model as described in any one of claims 1 to 5 or the image translation method as described in claim 6.

8. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, performs the steps of the training method for the image translation model as described in any one of claims 1 to 5 or the image translation method as described in claim 6.

Images