Image processing method, device, terminal, and storage medium

Abstract

Provided are an image processing method and device, a terminal, and a storage medium, wherein the method comprises: obtaining a style image and a content image to be fused; extracting the content image to obtain an internal intermediate style code of the content image; extracting the style image to obtain an external intermediate style code of the style image; and generating a fused image based on the internal intermediate style code of the content image and the external intermediate style code of the style image.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and relates to but is not limited to an image processing method and device, a terminal, and a storage medium. Background Art

[0002] Example-based image stylization aims to render an image in a style consistent with the style of a specified art image example. It has a wide range of applications in daily production and life, such as avatar design and animation poster advertising production. In related technologies, when an image generation model is used to render a style image, the image generation model can only learn the overall style of the data set, and it is difficult to accurately simulate the style of the art portrait of a specified example in the data set. Summary of the invention

[0003] Embodiments of the present disclosure provide an image processing method and device, a terminal, and a storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a training method for an image generation model, wherein the image generation model includes a trained first generator and an external style control module to be trained; the first generator is pre-trained on a real sample set and can generate a real image based on an internal style encoding;

[0005] The method includes: acquiring a style sample set and the real sample set; training the external style control module based on the real sample set so that the first fused image generated by the image generation model can reconstruct the second fused image generated by the first generator; after the training of the external style control module is completed, training the image generation model based on the style sample set and the randomly sampled internal style encoding until a first convergence condition is met.

[0006] In a second aspect, an embodiment of the present disclosure provides an image processing method, the method comprising: obtaining a style image and a content image to be fused; extracting the content image to obtain an internal intermediate style code of the content image; extracting the style image to obtain an external intermediate style code of the style image; and generating a fused image based on the internal intermediate style code of the content image and the external intermediate style code of the style image.

[0007] In a third aspect, an embodiment of the present disclosure provides a training device for an image generation model, wherein the image generation model includes a trained first generator and an external style control module to be trained; the first generator is pre-trained on a real sample set and can generate a real image based on an internal style code; the device includes: a first acquisition module, used to acquire a style sample set and the real sample set; a first training module, used to train the external style control module based on the real sample set, so that the first fused image generated by the image generation model can reconstruct the second fused image generated by the first generator; a second training module, used to train the image generation model based on the style sample set and the randomly sampled internal style code after the training of the external style control module is completed, until a first convergence condition is met.

[0008] In a fourth aspect, an embodiment of the present disclosure provides an image processing device, comprising: a first acquisition module, used to acquire a style image and a content image to be fused; a first extraction module, used to extract the content image to obtain an internal intermediate style code of the content image; a second extraction module, used to extract the style image to obtain an external intermediate style code of the style image; and a generation module, used to generate a fused image based on the internal intermediate style code of the content image and the external intermediate style code of the style image.

[0009] In a fifth aspect, an embodiment of the present disclosure provides a computer device, comprising: a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, the steps in the above method are implemented.

[0010] In a sixth aspect, an embodiment of the present disclosure provides a computer storage medium, wherein the computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the above method.

[0011] In the training method of the image generation model provided in the embodiment of the present disclosure, on the one hand, a progressive transfer learning method is used to train the image generation model, and the progressive transfer learning method includes two stages: first, in the first stage, the external style control module is trained on the real sample data set, and the first generator is pre-trained on the real sample set, so that the style transfer within the real sample set can be achieved; taking StyleGAN as the first generator as an example, the external style control module is trained using the real face sample set to achieve the style transfer within the real face data. Secondly, in the second stage, the image generation model, especially the external style control module, is trained using the style sample set to achieve the style transfer of the external style control module between the face data and the style data. On the other hand, the first generator in the image generation model is pre-trained, so that the image generation model can transfer learning on a small data set to generate high-definition images, which can effectively reduce the problem of requiring a large amount of training data due to learning the mapping relationship between the real image and the style image.

[0012] In the related art, the first generator, such as StyleGAN, can only render the overall style of a single field, and it is difficult to render the specified sample style. The image generation model obtained by the above training method introduces the control of the external style by constructing an external style control module, so that the image generation model can control both the internal style and the external style at the same time, which is more suitable for processing sample-based stylization tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0014] Figure 1A This is a schematic diagram of the composition structure of the StyleGAN model;

[0015] Figure 1B A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0016] Figure 2A Schematic diagram of the implementation process of step S200 in the embodiment of the present disclosure;

[0017] Figure 2B Schematic diagram of the implementation process of step S210 in the embodiment of the present disclosure;

[0018] Figure 2C Schematic diagram of the implementation process of step S213 in the embodiment of the present disclosure;

[0019] Figure 2D Schematic diagram of the implementation process of step S230 in the embodiment of the present disclosure;

[0020] Figure 3 Schematic diagram of the implementation process of step S310 in the embodiment of the present disclosure;

[0021] Figure 4 A schematic diagram of the composition structure of an image generation model provided by an embodiment of the present disclosure;

[0022] Figure 5 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present disclosure;

[0023] Fig. 6A A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present disclosure;

[0024] Figure 6B A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present disclosure;

[0025] Figure 7 A hardware entity schematic diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. The following embodiments are used to illustrate the present disclosure, but are not used to limit the scope of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0027] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0028] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present disclosure are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present disclosure described here can be implemented in an order other than that illustrated or described here.

[0029] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art in the field to which the disclosed embodiments belong. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0030] Before describing the embodiments of the present disclosure, the nouns and terms involved in the embodiments of the present disclosure are described. The nouns and terms involved in the embodiments of the present disclosure are subject to the following interpretations.

[0031] The Adaptive Instance Normalization (AdaIN) layer is a network module that normalizes image features by channel dimension, so that the mean and variance of image features in each channel are normalized to between 0 and 1; the normalized mean and variance are adjusted to the mean and variance of the target style, thereby achieving style transfer.

[0032] Latent space (Latent Code) is the sample space of latent variables.

[0033] Perceptual Loss is used to evaluate the difference between the feature maps of the original image and the reconstructed image.

[0034] A Style-Based Generator Architecture for Generative Adversarial Networks (StyleGAN) is a style-based generative network that can map random style codes to face images. StyleGAN can transform the generation space of the generative network from the face domain to the artistic portrait domain through hundreds of artistic portrait images and hours of training time.

[0035] The internal style can be understood as the content style of the image, such as the real face style of the pre-trained StyleGAN.

[0036] The external style can be understood as the style and color style of the image, such as the artistic portrait style composed of a small-scale artistic portrait dataset.

[0037] Portrait stylization is the process of transferring the style of a sample artistic portrait onto the target face. It is widely used in industries related to comics, animation, posters, and advertising.

[0038] The content image, i.e., the image that provides content in the stylized image process, is generally an image input by the user, such as a real face image. In the process described below, the capital letter I is used to represent the content image. In different contexts, I can refer to a real face image. For the training process, I can refer to a real sample.

[0039] The style image can be an image of any object that is stylized, such as a human face, an animal, a landscape, etc. Since the human face style image is the most common, for ease of understanding, the following embodiments of the present disclosure will be illustrated by taking the human face stylization as an example, and the style image can be understood as an artistic portrait. In the process described below, the capital letter S is used to represent the style image; in different contexts, S can refer to an artistic portrait, and for the training process, S can refer to a style sample.

[0040] Example-based face stylization aims to render face photos so that their style is consistent with the style of the specified artistic portrait examples. It has a wide range of applications in daily production and life, such as avatar design and animation poster advertising production. The current mainstream technology in the industry uses image conversion methods to train the network to learn the mapping relationship between real face images I and artistic portraits S. This type of method relies on a large amount of training data and is only applicable to small-sized images, which is difficult to meet actual needs.

[0041] There is also a technology that uses StyleGAN to transfer learning on a small-scale artistic portrait dataset to achieve high-definition face stylization. However, this technology can only learn the overall style of the dataset and it is difficult to accurately simulate the style of the artistic portrait samples specified in the dataset. How to implement sample-based high-definition face stylization technology on small-scale data is a difficult problem.

[0042] Before introducing the embodiments of the present disclosure, the StyleGAN model is first introduced. Figure 1A As shown, the model 10 includes a mapping network 11 and a generating network 12, wherein:

[0043] The mapping network 11 can map the input content image style code or the randomly sampled style code z to an internal intermediate style code (which can be understood as an intermediate vector), and the internal intermediate style code will be subsequently passed to the generation network 12 as 18 control vectors of the generation network 12, so that different elements of the control vector can control different visual features. The style code of the content image can be obtained after the content image I is encoded by the encoder 111.

[0044] The mapping network 11 includes 8 fully connected (FC) layers, and the internal intermediate style encoding output by the mapping network 11 The same size as the input of the StyleGAN model. The general input size can be (512×1) or (512×18), that is, a 512-dimensional vector, or 18 512-dimensional vectors. If the input is a 512-dimensional vector, the input layer will be copied into 18 identical vectors.

[0045] The generation network 12 includes 9 generation modules that transform from 4*4 to 1024*1024, that is, each module corresponds to 4*4, 8*8, 16*16, 32*32, 64*64, 128*128, 256*256, 512*512 and 1024*1024 transformations. Except for the first generation module and the last generation module, each module will be affected by two control vectors (the input of the affine transformation A in the figure), one of which affects it once after upsampling (Upsample), and the other control vector affects it once after convolution (Convolution). The influence method uses the style adjustment module (AdaIN adaptive instance normalization layer). The first module has no upsampling operation but only convolution operation. Therefore, the first module only uses one control vector; in addition to the control vectors corresponding to the upsampling operation and the convolution operation, the last module also includes a control vector corresponding to the image converted into a 3-channel. Therefore, the internal intermediate style coder A total of 18 control vectors are transformed and passed to the generation network 12.

[0046] To put it another way, the first generation module includes an affine transformation module, a convolution layer, and a style adjustment module, while the first generation module does not have an upsampling module; the other generation modules include an upsampling module, an affine transformation module, a convolution layer, and a style adjustment module.

[0047] The implementation process of the first generation module for sequential style adjustment is as follows: Figure 1A Shown: Encoding the internal intermediate style After passing through a learnable affine transformation module A (which can be a fully connected layer), it is expanded into a scaling factor y_(s,i) and a bias factor y_(b,i) by the style adjustment module. The scaling factor is multiplied by the features output by the normalized convolutional layer and then summed with the bias factor to complete an internal intermediate style encoding. Processes that influence style images.

[0048] The lower the resolution of the StyleGAN layer, the coarser the affected features. These features can be divided into two types: 1) Coarse features - the resolution is no higher than 32*32, and the corresponding resolution layers are 1 to 7 layers, which affect posture, hairstyle, facial shape, etc., as well as facial features, hairstyle, whether the eyes are open or closed, etc.; 2) Fine features - the resolution is 64*64 to 1024*1024, and the corresponding resolution layers are 8 to 18 layers, which affect color (eyes, hair and skin) and micro features.

[0049] StyleGAN can perform well in generating artistic portraits through transfer learning of limited data. However, this method can only learn the overall style of a dataset and it is difficult to accurately simulate the style of the artistic portrait samples specified in the dataset. For ease of understanding, StyleGAN can be understood as a single-channel style generation network.

[0050] In view of the above problems, the embodiment of the present disclosure first provides an image generation model. Compared with StyleGAN, the image generation model can be understood as a dual-path style generation network (DualStyleGAN), that is, an external style control module is added to StyleGAN to construct the DualStyleGAN, and the external style is learned on small-scale data through a progressive transfer learning method to achieve face stylization based on specified artistic portrait samples. The method can flexibly control the dual styles of the original face domain and the extended artistic portrait domain.

[0051] The present disclosure provides a training method for an image generation model, the image generation model comprising a trained first generator and an external style control module to be trained; the first generator is pre-trained on a real sample set and can generate a real image based on internal style coding. Figure 1B The method at least includes the following steps S100, S200 and S300, wherein:

[0052] Step S100, obtaining a style sample set and the real sample set;

[0053] The real sample set may be a real image data set, such as a real face data set, a real animal data set, a real landscape data set, etc. The real sample set is used to pre-train the first generator, for example, the real face data set may be used to pre-train StyleGAN.

[0054] The style sample set may be an art image data set, and the art images in the art image data set may be art portraits, art landscapes, cartoon images, comic images, etc. For example, the images in the art image data set may be images of different forms obtained through modification methods such as rendering and / or deformation.

[0055] Step S200, training the external style control module based on the real sample set, so that the first fused image generated by the image generation model can reconstruct the second fused image generated by the first generator;

[0056] Here, the first generator is pre-trained on a real sample set. In step S200, an external style control module is trained on a real sample data set to achieve style transfer within the real sample set. Taking StyleGAN as the first generator as an example, a real face sample set is used to train the external style control module to achieve style transfer within the face data.

[0057] The external style control module is trained based on the real sample set. During the training process, two associated inputs (two image codes z1 and z2) are used for the first generator and the image generation model. Then, the first fused image generated by the image generation model and the second fused image generated by the first generator can be particularly similar, which is sufficient to use the first fused image to reconstruct the second fused image, that is, it can be considered that the training of the external style control module on the real sample set is completed. The two associated inputs can be one of which is the same and the other is associated. The two image codes z1 and z2 are obtained by random sampling from the real image code set. When the number of image codes in the real image code set is large enough, the two image codes z1 and z2 can also be understood as random noise.

[0058] Step S300, after the training of the external style control module is completed, the image generation model is trained based on the style sample set and the randomly sampled internal style code until a first convergence condition is met.

[0059] In some embodiments, step S300 includes steps S310 to S330, wherein:

[0060] Step S310, iteratively training the image generation model based on the style sample set and the randomly sampled internal style code to obtain each style image generated by the image generation model;

[0061] Step S320, based on each generated style image, and the style sample and internal style code corresponding to each style image, respectively determine and accumulate the adversarial loss, the second reconstruction loss, the external style loss and the internal style loss to obtain a cumulative sum;

[0062] Where: The adversarial loss is the adversarial loss L of the first generator adv The second reconstruction loss L dist is the loss of the generated style image relative to the corresponding style sample, the external style loss L sty is the loss of the generated style image relative to the corresponding style sample and the internal style loss L con is the loss of the generated style image relative to the corresponding internal style encoding;

[0063] Here, in step S310, the loss function (first loss function) of the image generation model is as follows: adv , the second reconstruction loss L dist , the external style loss L sty and internal style loss (content loss) L con :

[0064] L adv +L dist (G(z i ,z e ,1),S)+L sty (G(z,z e ,1),S)+L con (G(z,z e ,1),g(z)) (1);

[0065] Among them, z is the randomly sampled internal style code, and the style loss L sty To measure the consistency between the style transfer result and the style image S, we can use the loss function commonly used in style transfer technology, such as contextual loss or feature matching loss. con To measure the consistency between the style transfer result and the content image g(z), a common content loss function, such as identity loss or perceptual loss, can be used. In addition, the content loss in this method can be implemented by L2 regularization of the parameters of the residual module.

[0066] Step S330, when the accumulated sum satisfies the first convergence condition, completing the training of the image generation model.

[0067] After the training in step S200 is completed, the external style control module realizes the style transfer within the face data; step S300 uses a style sample set to train the image generation model, especially the external style control module, to realize the style transfer of the external style control module between the face data and the style data.

[0068] In the related art, the first generator, such as StyleGAN, can only render the overall style of a single field and it is difficult to render the specified sample style. In the embodiment of the present disclosure, the external style control module must be trained on a style sample set, and the first generator is pre-trained on a real sample set. Since the external style control module added to the image generation model will change the generation space of the first generator, it is difficult to apply the fine-tuning strategy of transfer learning. Therefore, the embodiment of the present disclosure adopts a progressive transfer learning method to train the image generation model. The trained image generation model introduces control of the external style by constructing an external style control module, so that the image generation model can control both the internal style and the external style at the same time, which is more suitable for processing sample-based stylization tasks.

[0069] In the above step S200, the external style control module is trained based on the real sample set so that the first fused image generated by the image generation model can reconstruct the second fused image of the first generator, such as Figure 2A As shown, it includes step S210, step S230, step S250 and step S270, wherein:

[0070] Step S210, using the first generator to generate second fused images based on the real sample set;

[0071] In the process of implementation, Figure 2B As shown, step S210 includes steps S211 to S213, wherein: step S211, respectively determine the image coding of each real sample in the real sample set to obtain a real image coding set; in the implementation process, the image coding of the real sample can be an image feature vector, an image feature matrix, and the image coding can be at least one M-dimensional vector, such as a 512-dimensional or 1024-dimensional vector. Step S212, obtain each pair of real image coding from the real image coding set; in implementation, each real image coding in the real image coding set can be combined to obtain at least one pair of real image coding, wherein each pair of real image coding can be recorded as a first real image coding z1 and a second real image coding z2. Step S213, using the first generator, generate corresponding second fused images based on each pair of the real image coding. In implementation, generate based on the first real image coding z1 and the second real image coding z2 The second fused image is generated using the first generator, denoted as

[0072] Step S230, using the image generation model to generate first fused images based on the real sample set;

[0073] The first fused image and the second fused image are based on the same pair of real image codes in the real sample set, namely, the first real image code z1 and the second real image code z2. In the case where step S210 includes steps S211 to S213, step S230 includes: using the image generation model, generating corresponding first fused images based on each pair of the real image codes, denoted as G(z1, E(g(z2)), 1).

[0074] Step S250, determining a first reconstruction loss between each of the first fused images and the corresponding second fused image; the first reconstruction loss can be expressed as

[0075] In terms of step execution, there is no strict order between step S210 and step S230. In practice, step S210 and step S230 can be executed before step S250.

[0076] Step S270: iteratively train the external style control module based on the first reconstruction loss until a second convergence condition is met.

[0077] Taking StyleGAN as an example, the loss function (second loss function) for iterative training of the external style control module is shown in formula (2). The second loss function includes the adversarial loss L of StyleGAN. adv and the first reconstruction loss L dist :

[0078]

[0079] Among them, the external style code E(g(z2)) is obtained by inputting z2 into StyleGAN to obtain g(z2), and then encoding g(z2) using encoder E.

[0080] In some embodiments, the first generator includes a first mapping network and a generating network; Step S213, using the first generator, based on each pair of the real image encodings, generates corresponding second fused images, such as Figure 2C As shown, it includes steps S2131a to S2133a, wherein:

[0081] Step S2131a, obtaining, based on the first mapping network, intermediate style codes of each pair of real image codes;

[0082] Among them, each of the intermediate style codes includes N element feature sequences; the N element feature sequences can be N vectors with M dimensions, wherein N is an integer greater than or equal to 2, M is 2 to the power of i, and i is a natural number greater than or equal to 1; for example, 18 512-dimensional vectors (when the first generator is StyleGAN), or 18 1024-dimensional vectors.

[0083] Step S2132a, for each pair of the real image codes, extract the first k element features and the last (Nk) element features from the two intermediate style codes, respectively, and fuse them to obtain a fused intermediate style code;

[0084] Wherein, k is an integer greater than or equal to 1 and less than N; assuming that each pair of the real image codes includes a first real image code z1 and a second real image code z2, the first k element features are extracted from z1, and the last (Nk) element features are extracted from z2.

[0085] Step S2133a: using the generation network, generating a corresponding second fused image based on each of the fused intermediate style codes.

[0086] Take StyleGAN as an example. First, we use StyleGAN’s style exchange technology to generate a data set. We randomly select a pair of image codes, such as two image codes z1 and z2. After the first mapping network, we get 18 intermediate style codes. and extract The top k intermediate style encoding and extraction The last (18-k) intermediate style codes form the style code The style-fused image is obtained by generating a network This is the second fused image.

[0087] In some other embodiments, step S213, using the first generator to generate corresponding second fused images based on the encoding of each pair of the real images, includes steps S1231b to S2133b, wherein:

[0088] Step S2131b, for each pair of the real image encoding, repeat k times and (Nk) times respectively. For example, for each pair of z1 and z2 in the real image encoding, repeat k times and (18-k) times respectively.

[0089] Step S2132b, after repeating k times and (Nk) times, each pair of the real image codes are combined to obtain a fused real image code containing N element features. For example, z1 repeated k times and z2 repeated (18-k) times are concatenated to form a fused style code containing 18 style codes.

[0090] Step S2133b: using the first generator, encoding each of the fused real images to generate a corresponding second fused image.

[0091] Taking StyleGAN as an example, we first use StyleGAN's style exchange technology to generate a data set, and train a two-way style generation network within the face data to achieve style transfer. Randomly sample two style codes z1 and z2, repeat k times and (18-k) times respectively, and concatenate them into a fusion style code containing 18 style codes. The style-fused image is obtained through StyleGAN This is the second fused image.

[0092] In the case where step S213 includes steps S2131a to S2133a, or, in the case where step S2131b to S2133b is included, Figure 2D As shown, step S230, using the image generation model to generate corresponding first fused images based on the encoding of each pair of real images, includes steps S231 and S233, wherein:

[0093] Step S231, using the first generator to generate a corresponding real image based on the second real image code in each pair of the real image code pairs;

[0094] Among them, each pair of real image codes includes a first real image code and a second real image code, wherein the first real image code is the real image code corresponding to the first k element features extracted; the second real image code is the real image code corresponding to the last (Nk) element features extracted.

[0095] Step S232, for each pair of real image encodings, using an encoder to encode each generated real image to obtain a corresponding external style code;

[0096] Step S233, using the image generation model to be trained, based on the first real image code and the corresponding external style code in each pair of the real image code pairs, generating a corresponding first fused image.

[0097] In some embodiments, the method further includes: when the first generator is StyleGAN, setting the value of k to 7, 6, and 5 in sequence; for each value of k, obtaining the intermediate style code based on the first mapping network, obtaining the fused intermediate style code, and obtaining the second fused image based on the generation network. As the training of the two-way style generation network proceeds, k is gradually reduced from 7 to 5, so that the external style control module gradually learns the structural style features. In implementation, k in each batch is the same, and the value of k is uniformly modified after training for a period of time.

[0098] In some embodiments, Figure 3 As shown, step S310, based on the style sample set and the randomly sampled internal style coding, the image generation model is iteratively trained to obtain each style image generated by the image generation model, including steps S311 to S313, wherein:

[0099] Step S311, determining the external style code and the internal style code of each style sample;

[0100] In some embodiments, step S311 includes step S3111 and step S3112, wherein: step S3111, using an encoder to encode each of the style samples respectively to obtain an external style code of the corresponding style sample; step S3112, after de-stylizing each of the style samples respectively, obtain an internal style code of each style sample.

[0101] In implementation, step S3112 includes steps a to c: step a, determining the optimized style coding of each style sample, wherein the optimized style coding can reconstruct the corresponding style sample through the trained second generator; step b, using the trained first generator to reconstruct each of the optimized style codings respectively to obtain reconstructed style samples; wherein the first generator and the second generator are trained on a real image dataset and a style image dataset respectively; step c, using the encoder to encode each of the reconstructed style samples to obtain an internal style coding of the corresponding style sample.

[0102] In some embodiments, the method further includes: obtaining a first generator g pre-trained on a real sample set; training the first generator on a style sample set to obtain a second generator g'; during implementation, fine-tuning the first generator to obtain the second generator g'. In the embodiment of the present disclosure, g may be the entire StyleGAN (including the mapping network and the generating network), rather than just the generating network.

[0103] For example, assuming that the style sample is represented by S, the external style encoding of the style sample S can be z e Indicates that the internal style encoding of style sample S can be adopted z i It means that for each style sample S, the encoder E is used to encode the style sample, and the external style code z can be obtained. e =E(S); Then, the style code z is updated by using the optimization-based method as shown in formula (3) e , so that S can be reconstructed after passing through g':

[0104] z′=argmin z L dist (g′(z),S) (3);

[0105] Where L dist (x, y) measures the similarity between two images x and y. Common loss functions can be used, such as L1 distance, perceptual loss, etc. Finally, the reconstructed image g(z') is encoded again using encoder E to obtain the de-stylized style code z i =E(g(z')). Thus, z i and z e It constitutes the internal style coding and external style coding of the artistic image S.

[0106] Step S312, using the internal style code and the external style code of each of the style samples as inputs of the first generator and the external style control module respectively, to obtain a first target style image generated by the image generation model;

[0107] Step S313: Using the randomly sampled internal style code and the external style code of each of the style samples as inputs to the first generator and the external style control module respectively, to obtain a second target style image generated by the image generation model.

[0108] In some embodiments, step S320, based on each generated style image, and the style sample and internal style code corresponding to each style image, respectively determines the second reconstruction loss, the external style loss and the internal style loss, including steps S321 to S323, wherein:

[0109] Step S321, determining the second reconstruction loss based on each of the first target style images and the corresponding style samples; step S322, determining the external style loss based on each of the second target style images and the corresponding style samples; step S323, determining the internal style loss based on each of the second target style images and the corresponding random real images; wherein the random real images are generated by the first generator based on the internal style encoding of the random sampling.

[0110] The disclosed embodiment provides an image generation model. Compared with StyleGAN, the image generation model can be understood as a dual-path style generation network (DualStyleGAN), that is, an external style control module is added to StyleGAN to construct the DualStyleGAN, and the external style is learned on small-scale data through a progressive transfer learning method to achieve face stylization based on a specified artistic portrait sample. The method can flexibly control the dual styles of the original face domain and the extended artistic portrait domain.

[0111] The composition structure of this image generation model can be found in Figure 4 , the image generation model includes an internal style control module 110, a generation network 120 and an external style control module 130, wherein:

[0112] The internal style control module 110 includes a first input encoder 111 and a first mapping network 112, wherein: the first input encoder 111 is used to encode the real face image I to obtain the internal style code; the first mapping network 112 is used to map the internal style code to obtain the internal intermediate style code;

[0113] The generation network 120 is used to generate a high-resolution image based on the internal intermediate style coding;

[0114] The external style control module 130 includes a second input encoder 131 and a second mapping network 132, wherein: the second input encoder 131 is used to encode the style image S to obtain an external style code; the second mapping network 132 is used to map the external style code to obtain an external intermediate style code.

[0115] In one achievable manner, the internal style control module 110 and the generation network 120 form StyleGAN and remain fixed during transfer learning.

[0116] Continue to see Figure 4 The external style control module includes a second mapping network 132 having the same structure as the first mapping network 112, an external structure style control module (T s) 133, residual module (ModRes) 134 and external color style control module (T c )135; In the final training process of the image generation model, the functions of each module are as follows:

[0117] The first mapping network 112 obtains the internal intermediate style code of each style sample based on the internal style code z of each style sample, and obtains the internal intermediate style code of the random sample based on the internal style code z of the random sample;

[0118] The second mapping network 132 obtains the external intermediate style codes of each of the external style codes respectively; each of the external intermediate style codes includes N element feature sequences;

[0119] The external structural style control module 133 determines the structural style code of each style sample based on a part of the external intermediate style code of each style sample;

[0120] The external color style control module 135 extracts the color style code of each style sample based on another part of the external intermediate style code of each style sample;

[0121] The residual module 134 performs structural style adjustment on the target image generated by the generation network based on the structural style encoding of each style sample;

[0122] The generation network 120 generates a first target style image based on the structural style coding and the color style coding of each style sample, and the internal intermediate style coding of each style sample; and generates a second target style image based on the structural style coding and the color style coding of each style sample, and the internal intermediate style coding of the random sampling.

[0123] exist Figure 4 In the external structure style control module (T s ) 133, residual module 134 and external color style control module (T c )135 can be the training part of the image generation model; the convolution layer and the affine transformation module in each generation module, the fully connected layer in the first mapping network 112 and the second mapping network 132, and the first input encoder 111 and the second input encoder 131 can be the fixed parts of the image generation model, and the style adjustment module and the upsampling module in each generation module can be the operation modules of the image generation model.

[0124] In the disclosed embodiments, StyleGAN is used as an example of the first generator for explanation. In view of the problem that existing face stylization methods are difficult to achieve sample-based high-definition face stylization on small-scale data, the disclosed embodiments propose a portrait stylization method based on dual-path style. An external style control module is added to StyleGAN to construct a new dual-path style generation network DualStyleGAN. The external style is learned on small-scale data through a progressive transfer learning method to achieve sample-based high-definition face stylization. The dual-path style generation network is trained because the addition of an external style control module will change the generation space of StyleGAN pre-training, making it difficult to apply the fine-tuning strategy of transfer learning. Therefore, a progressive transfer learning method is required to train the network.

[0125] The image processing method provided by the embodiment of the present disclosure, on the one hand, proposes a DualStyleGAN network model, including an internal style pathway and an external style pathway; the content of the portrait is characterized by the internal style pathway; the artistic style of the portrait is characterized by the external style pathway; wherein the external style pathway of DualStyleGAN enables the network model to hierarchically adjust the content and style of the artistic portrait to accurately imitate the style of the sample. On the other hand, a progressive transfer learning method is proposed, through which the generation space of the network model can be smoothly converted to the target domain.

[0126] The image processing method provided by the embodiment of the present disclosure includes two stages:

[0127] In the first stage, the style dataset is de-stylized to obtain face-style group training data.

[0128] In the second stage, a two-way style generation network is built and trained. A progressive transfer learning method is used to learn the style features of a small-scale style dataset through three steps: initialization, style transfer within the face data, and style transfer between the face and style data, to achieve natural and accurate face stylization.

[0129] The first stage is described below, which is to de-stylize the style dataset; the first stage includes steps 1.1 and 1.2, where:

[0130] Step 1.1, StyleGAN transfer learning and StyleGAN encoder training:

[0131] First, obtain the StyleGAN (which can be understood as the first generator) g pre-trained on the real face dataset; fine-tune StyleGAN g on the artistic portrait dataset to obtain the fine-tuned model (which can be understood as the second generator) g'; that is, fine-tune StyleGAN g pre-trained on the real face dataset on the artistic portrait dataset to obtain the fine-tuned model g'. Then, train a StyleGAN encoder E to map the face image to the style code of StyleGAN, which can reconstruct the input face image after passing through StyleGAN g.

[0132] Step 1.2, Artistic Portrait De-stylization: Face de-stylization aims to restore realistic faces from artistic portraits to form a set of face-style data and use this data for supervision.

[0133] First, for each artistic portrait S, obtain the external style code z e =E(S);

[0134] Then, the style code z is updated by an optimization-based method, see formula (3), so that S can be reconstructed after passing through g':

[0135] z′=argmin z L dist (g′(z),S) (3);

[0136] Among them, the initial value of z is z e , optimize the loss function to change the variable z from z e Update to z', L dist (x, y) measures the similarity between two images x and y. Common loss functions such as L1 distance and perceptual loss can be used.

[0137] Among them, z'=arg min_z(f(z)) is a mathematical description, which means optimizing the variable z (the randomly sampled style code z can also be understood as a variable). The final optimization result is z', and f(z') is the z that minimizes f(z). The optimization-based method is to update the input z of the network through the loss function instead of updating the network itself. The purpose of this step is to obtain the style code z′ that can reconstruct S; the reconstructed S can be defined as S', and S'=g'(z').

[0138] Finally, the reconstructed image g(z') is encoded again using the encoder E to obtain the style code z after S is de-stylized i =E(g(z')). Here, z i and z e It constitutes the internal style coding and external style coding of the artistic image S.

[0139] Part 2: Building and training a two-way style generation network: This part 2 includes steps 2.1 and 2.2, where:

[0140] Step 2.1, build a two-way style generation network: For the 18-layer StyleGAN, Figure 4 As shown, the external style control module 130 on the right is built to form a dual-path style generation network DualStyleGAN G.

[0141] First, the artistic portrait S is passed through the encoder E to obtain the external style code z e , and then 18 external intermediate style codes are obtained through the mapping network. The mapping network is a multi-layer perceptron, which inputs 18 512-dimensional vectors and outputs 18 512-dimensional vectors, that is, 18 style codes with a length of 512.

[0142] Step 2.1.1, external structure style control;

[0143] The first 7 of the 18 outer intermediate style codes are passed through a linear layer T s 133, encoded as external structural style coding. Among them, the external structural style coding adjusts the structure of the face to match the artistic portrait, such as big eyes and big mouth. The features of the first 7 convolutional layers of StyleGAN are adjusted by the residual module 134 to obtain the residual features, and the residual features are multiplied by the weight w i (i<8) is then added to the feature to complete the adjustment of the face structure.

[0144] Among them, the residual module includes two convolutional layers and two adaptive layer instance normalization layers (AdaIN, the style adjustment module in the figure). The external structural style code is passed through the linear layer inside AdaIN to obtain the structural style adjustment parameters for the structural style control of the features. The input of the residual module is two parts: the features output by the convolutional layer and the external structural style code. The first 7 external intermediate style codes are in one-to-one correspondence with the first 7 convolutional layers. Among them, the first 4*4 generation module includes one convolutional layer, the last 1024*1024 generation module includes 3 convolutional layers, and the middle 7 generation modules each include 2 convolutional layers.

[0145] Step 2.1.2, External color style control:

[0146] First, the last 11 of the 18 outer intermediate style codes are passed through a linear layer T c 135, encoded as an external color style code, and then combined with the internal intermediate style code obtained by the internal style control module 110 of StyleGAN to form w i(i>7) After weighted addition, it is input into the style adjustment module of StyleGAN to realize the color style control of the feature.

[0147] Therefore, DualStyleGAN is based on the internal style code z that describes the facial features. i and an external style code describing the artistic style e Generate artistic portraits and adjust the weights w of different layers i Control the style of different scales, formally expressed as G(z i ,z e ,w). Among them, G(z i ,z e ,w) represents the image output by DualStyleGAN, for example, setting the weight w i = 0 (i>7) can retain the internal color style; set w i <1 can reduce the degree of external style and retain the appearance characteristics of the face. Lowercase w represents weight, for example w i , and capital W with angle brackets above Represents the intermediate style encoding (which can be understood as an intermediate vector).

[0148] Step 2.2, training the dual-path style generation network: Adding an external style control module will change the generation space of StyleGAN pre-training, making it difficult to apply the fine-tuning strategy of transfer learning. To this end, a gradual transfer learning method is adopted, which includes three steps:

[0149] Step 2.2.1, initialization of external style control module;

[0150] Initialize the convolution kernel of the convolution layer in the residual module to a random value close to 0, and initialize the linear layer T c The parameter matrix of is the identity matrix. The above initialization makes the residual feature value close to 0, and the external intermediate style coding is T c After that, it remains unchanged, thus keeping the structure almost unchanged, so that StyleGAN with the added external style control module still maintains the original generation space.

[0151] Step 2.2.2, style transfer within face data: Use StyleGAN’s style exchange technology to generate a data set, and train a two-way style generation network within the face data to achieve style transfer.

[0152] First, StyleGAN’s style exchange technology is used to generate a data set. Two style codes z1 and z2 are randomly sampled. After the mapping network, 18 intermediate style codes are obtained. and Pick The first k intermediate style codes and The last (18-k) intermediate style codes form the style code The style-fused image is obtained by generating a network

[0153] Then, the two-way style generation network is trained to reconstruct the internal style code z1 and the external style code E(g(z2)) As the training progresses, k is gradually reduced from 7 to 5, so that the external style control module gradually learns the structural style features.

[0154] The loss function here is as follows, including the adversarial loss L of StyleGAN adv and the reconstruction loss L dist ;

[0155]

[0156] Among them, only the external style control module of the two-way style generation network is trained, the other modules remain unchanged, and the weights w are set to all 1. Among them, the external style code E(g(z2)) is obtained by inputting z2 into StyleGAN to obtain g(z2), and then encoding g(z2) using encoder E.

[0157] In some other embodiments, step 2.2.2 can also be implemented in the following way: first, use StyleGAN's style exchange technology to generate a data set, and train a two-way style generation network within the face data to achieve style transfer. Randomly sample two style codes z1 and z2, repeat k times and (18-k) times respectively, and concatenate them into a fusion style code containing 18 style codes. The style-fused image is obtained through StyleGAN Then, the two-way style generation network is trained to reconstruct the internal style code z1 and the external style code E(g(z2)) As the training progresses, k is gradually reduced from 7 to 5, so that the external style control module gradually learns the structural style features.

[0158] The loss function here, see formula (4), includes the adversarial loss L of StyleGAN adv and the third reconstruction loss L dist :

[0159]

[0160] Among them, only the external style control module of the two-way style generation network is trained, the other modules remain unchanged, and the weights w are set to all 1.

[0161] Step 2.2.3, style transfer between face and style data:

[0162] According to the paired training data (z i ,z e ,S) as supervision, train the two-way style generation network to achieve face stylization, where, see formula (1), the loss function includes the adversarial loss L of StyleGAN adv , reconstruction loss L dist , style loss L sty and content loss L con :

[0163] L adv +L dist (G(z i ,z e ,1),S)+L sty (G(z,z e ,1),S)+L con (G(z,z e ,1),g(z)) (1)

[0164] Among them, z is the randomly sampled internal style code, and the style loss L sty To measure the consistency between the style transfer result and the style image S, we can use the loss function commonly used in style transfer technology, such as contextual loss or feature matching loss. con To measure the consistency between the style transfer result and the content image g(z), a common content loss function, such as identity loss or perceptual loss, can be used. In addition, the content loss in this method can be implemented by L2 regularization of the parameters of the residual module.

[0165] Figure 4 In the image generation model shown, z represents random noise, that is, randomly sampled image code. In the implementation, as mentioned above, the image code can be 18 512-dimensional vectors. Since it is randomly sampled, it can be mathematically regarded as random noise z. In the image generation model, the internal style code on the left can be the content image encoded by the first input encoder, or it can be random noise. Similarly, the external style code on the right can be the style image S encoded by the second input encoder, or it can be the random noise z after the sampling network N c After getting.

[0166] In other embodiments, Figure 4 As shown, the image generation model can also include a sampling network N c , then the training part of the image generation model also includes the sampling network N c In implementation, the training method of the image generation model further includes: training the sampling network N c, learn to model the distribution of external style codes and map random noise z to external style codes, so that the trained sampling network N c , used to map random noise z into external style coding. At this time, after sampling network N c The output external style code can be recorded as N c (z). The inputs of the dual-path style generation network are z1 and z2, where z2 passes through the sampling network N c Then get N c (z2), then the random artistic portrait generated by the dual-path style generation network can be recorded as G(z1,N c (z2),w).

[0167] In the disclosed embodiments, on the one hand, the network framework can transfer learning on a small data set to generate high-definition images, thereby effectively reducing the problem of learning the mapping relationship between real face images I and artistic portraits S when training the network, and requiring a large amount of training data and only being applicable to small-sized images. On the other hand, by constructing an external style control module to introduce control of the external style, the two-way style generation network can simultaneously control the internal style and the external style, which is more suitable for processing sample-based face stylization tasks, thus avoiding the problem that StyleGAN can only render the overall style of a single field and is difficult to render the specified sample style. On the other hand, the progressive transfer learning method robustly guides the two-way style generation network to learn the image distribution of artistic portraits through targeted network initialization, in-field training, and inter-field training, which can avoid the problem that the transfer learning technology fine-tunes StyleGAN on the target data set and is not applicable to the two-way style generation network with a changed network architecture.

[0168] The present disclosure further provides an image processing method, such as Figure 5 As shown, the method includes:

[0169] Step S501, obtaining a style image and a content image to be fused;

[0170] The method can be implemented as a computer executable program, which is embodied as a client on the user side. When the user selects two images as the style image and content image to be fused, for example, the user selects two images from an album, the client receives the user's operation of selecting images, that is, the client determines the style image and content image to be fused with the two selected images. Alternatively, the user selects a content image, but the user does not need to select a style image, for example, the user can select a style style, such as the Impressionist style, and the device can obtain a Van Gogh style image from the local or server.

[0171] Step S502, extracting the content image to obtain an internal intermediate style code of the content image;

[0172] During implementation, the extracting of the content image to obtain the external intermediate style coding of the content image includes: using an encoder to extract the external style coding of the content image; and using a mapping network to extract the external style coding of the content image to obtain the external intermediate style coding of the content image.

[0173] Step S503, extracting the style image to obtain an external intermediate style code of the style image;

[0174] During implementation, the extracting of the style image to obtain the external intermediate style code of the style image includes: using an encoder to extract the external style code of the style image; using a mapping network to extract the external style code of the style image to obtain the external intermediate style code of the style image.

[0175] exist Figure 4 In the above, the input of the mapping network is the style code of the image; in the implementation process, after the user inputs the content image I and the style image S, the encoder encodes the content image I and the style image S respectively to obtain the style code of the content image I and the style code of the style image S, and then the mapping network converts the two into the internal intermediate style code of the content image I and the external intermediate style code of the style image S respectively.

[0176] Step S504: generating a fused image based on the internal intermediate style coding of the content image and the external intermediate style coding of the style image.

[0177] In some embodiments, generating a fused image based on the internal intermediate style coding of the content image and the external intermediate style coding of the style image includes: determining the structural style coding of the style image based on a part of the external intermediate style coding of the style image; extracting the color style coding of the style image based on another part of the external intermediate style coding of the style image; generating the fused image based on the structural style coding and the color style coding of the style image, and the internal intermediate style coding of the content image.

[0178] In some embodiments, each of said outer intermediate style codes comprises a sequence of N element features;

[0179] The determining the structural style code of the style image based on a part of the external intermediate style code of the style image comprises: determining the structural style code of the style image based on first k element features in the external intermediate style code of the style image;

[0180] The extracting the color style coding of the style image based on another part of the external intermediate style coding of the style image comprises: extracting the color style coding of the style image based on the last (Nk) element features of the external intermediate style coding of the style image.

[0181] The method provided by the embodiment of the present disclosure includes multiple user usage scenarios: 1) Customizing artistic avatars for users: This technology is used to establish a mapping relationship between portraits and artistic portraits such as cartoons / comics / caricatures, supporting users to generate creative avatars of various styles based on their own photos, and supporting artistic face rendering for online meetings. 2) Artistic portrait data generation: This technology can be used to generate random artistic portraits, and the generated image data can be used for the production of graphic products such as avatar design, anime posters and advertisements, and can also be used for subsequent deep learning and analysis.

[0182] Based on the foregoing embodiments, the embodiments of the present disclosure provide a training device for an image generation model, which includes various modules, each module includes various units, and each unit includes various sub-units, which can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0183] The image generation model includes a trained first generator and an external style control module to be trained; the first generator is pre-trained on a real sample set and can generate a real image based on the internal style coding; Fig. 6A As shown, the device 610 includes:

[0184] A first acquisition module 611, used to acquire a style sample set and the real sample set;

[0185] A first training module 612, configured to train the external style control module based on the real sample set so that the first fused image generated by the image generation model can reconstruct the second fused image generated by the first generator;

[0186] The second training module 613 is used to train the image generation model based on the style sample set and the randomly sampled internal style encoding after the training of the external style control module is completed, until the first convergence condition is met.

[0187] In some embodiments, the first training module includes: a first generation unit, used to generate each second fused image based on the real sample set using the first generator; a second generation unit, used to generate each first fused image based on the real sample set using the image generation model; wherein the first fused image and the second fused image are encoded based on the same pair of real images in the real sample set; a first determination unit, used to determine a first reconstruction loss between each first fused image and the corresponding second fused image; and a first training unit, used to iteratively train the external style control module based on the first reconstruction loss until a second convergence condition is met.

[0188] In some embodiments, the first generation unit includes: a first determination subunit, used to respectively determine the image coding of each real sample in the real sample set to obtain a real image coding set; a first acquisition subunit, used to obtain each pair of real image coding from the real image coding set; a first generation subunit, used to use the first generator to generate corresponding second fused images based on each pair of the real image coding; correspondingly, the second generation unit, used to use the image generation model to generate corresponding first fused images based on each pair of the real image coding.

[0189] In some embodiments, the first generator includes a first mapping network and a generation network; the first generation subunit is used to: obtain the intermediate style code of each real image code in each pair of the real image codes based on the first mapping network; each of the intermediate style codes includes N element feature sequences; wherein N is an integer greater than or equal to 2; for each pair of the real image codes, extract the first k element features and the last (Nk) element features from the two intermediate style codes and fuse them to obtain a fused intermediate style code; k is an integer greater than or equal to 1 and less than N; and use the generation network to generate a corresponding second fused image based on each of the fused intermediate style codes.

[0190] In some embodiments, the device also includes: a setting module, which is used to set the value of k to 7, 6, and 5 respectively when the first generator is StyleGAN; for each value of k, using the first generation subunit to implement: obtaining the intermediate style code based on the first mapping network, obtaining the fused intermediate style code, and obtaining the second fused image based on the generation network.

[0191] In some embodiments, the second generating unit includes: a second generating subunit, used to generate a corresponding real image based on the second real image code in each pair of the real image code pairs using the first generator; wherein the second real image code is the real image code corresponding to the last (Nk) element features extracted; an encoding subunit, used to encode each generated real image using an encoder for each pair of the real image codes to obtain a corresponding external style code; a third generating subunit, used to generate a corresponding first fused image based on the first real image code and the corresponding external style code in each pair of the real image code pairs using the image generation model to be trained; wherein the first real image code is the real image code corresponding to the first k element features extracted.

[0192] In some embodiments, the second training module includes: a first training unit, which is used to iteratively train the image generation model based on the style sample set and the randomly sampled internal style code to obtain each style image generated by the image generation model; a second determination unit, which is used to determine and accumulate the adversarial loss, the second reconstruction loss, the external style loss and the internal style loss based on each generated style image, and the style sample and the internal style code corresponding to each style image to obtain a cumulative sum; wherein: the adversarial loss is the adversarial loss of the first generator, the second reconstruction loss is the loss of the generated style image relative to the corresponding style sample, the external style loss is the loss of the generated style image relative to the corresponding style sample, and the internal style loss is the loss of the generated style image relative to the corresponding internal style code; a third determination unit, which is used to determine that the training of the image generation model is completed when the cumulative sum satisfies the first convergence condition.

[0193] In some embodiments, the first training unit includes: a second determining subunit, used to determine the external style code and the internal style code of each style sample; a second obtaining subunit, used to use the internal style code and the external style code of each style sample as inputs of the first generator and the external style control module, respectively, to obtain the first target style image generated by the image generation model; use the randomly sampled internal style code and the external style code of each style sample as inputs of the first generator and the external style control module, respectively, to obtain the second target style image generated by the image generation model;

[0194] The second determination unit includes: a third determination subunit, used to determine the second reconstruction loss based on each of the first target style images and the corresponding style samples; a fourth determination subunit, used to determine the external style loss based on each of the second target style images and the corresponding style samples; a fifth determination subunit, used to determine the internal style loss based on each of the second target style images and the corresponding random real images; wherein the random real images are generated by the first generator based on the internal style encoding of the random sampling.

[0195] In some embodiments, the second determining subunit is used to: respectively encode each of the style samples using an encoder to obtain an external style code of the corresponding style sample; and respectively de-stylize each of the style samples to obtain an internal style code of each style sample.

[0196] In some embodiments, the external style control module includes a second mapping network with the same structure as the first mapping network, an external structural style control module, an external color style control module and a residual module; the acquisition subunit is used to use the first mapping network to obtain the internal intermediate style code of each style sample based on the internal style code of each style sample, and to obtain the internal intermediate style code of the random sample based on the internal style code of the random sample; use the second mapping network to obtain the external intermediate style code of each external style code; each of the external intermediate style codes includes N element feature sequences; use the external structural style control module to obtain the external intermediate style code of each style sample based on the external intermediate style code of each style sample The method comprises the steps of: determining a structural style code of each style sample based on a part of the grid code; extracting a color style code of each style sample based on another part of the external intermediate style code of each style sample by using the external color style control module; performing structural style adjustment on a target image generated by the generation network based on the structural style code of each style sample by using the residual module; generating a first target style image based on the structural style code and the color style code of each style sample and the internal intermediate style code of each style sample by using the generation network; and generating a second target style image based on the structural style code and the color style code of each style sample and the internal intermediate style code of the random sampling.

[0197] Based on the foregoing embodiments, the embodiments of the present disclosure provide an image processing device, which includes various modules, each module includes various units, and can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0198] Figure 6B A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure is shown in FIG. Figure 6B As shown, the device 620 includes:

[0199] A first acquisition module 621, used to acquire a style image and a content image to be fused;

[0200] A first extraction module 622, configured to extract the content image to obtain an internal intermediate style code of the content image;

[0201] A second extraction module 623, configured to extract the style image to obtain an external intermediate style code of the style image;

[0202] The generating module 624 is configured to generate a fused image based on the internal intermediate style coding of the content image and the external intermediate style coding of the style image.

[0203] In some embodiments, the generation module includes: a fourth determination unit, used to determine the structural style coding of the style image based on a part of the external intermediate style coding of the style image; a fifth determination unit, used to extract the color style coding of the style image based on another part of the external intermediate style coding of the style image; and a third generation unit, used to generate the fused image based on the structural style coding and the color style coding of the style image, and the internal intermediate style coding of the content image.

[0204] In some embodiments, each of the external intermediate style codes includes a sequence of N element features; the fourth determination unit is used to determine the structural style code of the style image based on the first k element features in the external intermediate style code of the style image; and the fifth determination unit is used to extract the color style code of the style image based on the last (Nk) element features of the external intermediate style code of the style image.

[0205] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present disclosure, please refer to the description of the method embodiment of the present disclosure for understanding.

[0206] It should be noted that in the embodiments of the present disclosure, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present disclosure is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a device to execute all or part of the methods of each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present disclosure are not limited to any combination of hardware and software for any target.

[0207] Figure 7 A hardware entity diagram of a computer device provided in an embodiment of the present disclosure, such as Figure 7 As shown, the hardware entity of the computer device 700 includes: a processor 701 and a memory 702, wherein the memory 702 stores a computer program that can be run on the processor 701, and the processor 701 implements the steps in the method of any of the above embodiments when executing the program. In some implementations, the computer device 700 may be the device described in any of the above embodiments.

[0208] The memory 702 stores computer programs that can be run on the processor. The memory 702 is configured to store instructions and applications executable by the processor 701. It can also cache data to be processed or processed by the processor 701 and various modules in the computer device 700 (for example, image data). This can be achieved through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0209] When the processor 701 executes the program, the steps of any of the above methods are implemented. The processor 701 generally controls the overall operation of the computer device 700.

[0210] An embodiment of the present disclosure provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the method in any of the above embodiments.

[0211] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present disclosure, please refer to the description of the method embodiments of the present disclosure for understanding.

[0212] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, which is not specifically limited in the embodiments of the present disclosure.

[0213] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM) and the like; it can also be various computer devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0214] It should be understood that the "one embodiment" or "an embodiment" or "an embodiment of the present disclosure" or "the aforementioned embodiments" or "some embodiments" mentioned throughout the specification means that the target features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" or "in an embodiment" or "an embodiment of the present disclosure" or "the aforementioned embodiments" or "some embodiments" appearing in various places throughout the specification do not necessarily refer to the same embodiment. In addition, the features, structures or characteristics of these targets may be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the above-mentioned embodiments of the present disclosure are for description only and do not represent the advantages and disadvantages of the embodiments.

[0215] Unless otherwise specified, a computer device performs any step in the embodiments of the present disclosure, and the processor of the computer device may perform the step. Unless otherwise specified, the embodiments of the present disclosure do not limit the order in which the computer device performs the following steps. In addition, the methods used to process data in different embodiments may be the same method or different methods. It should also be noted that any step in the embodiments of the present disclosure can be independently executed by a computer device, that is, when a computer device executes any step in the above embodiments, it may not rely on the execution of other steps.

[0216] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0217] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, the functional units in the various embodiments of the present disclosure may all be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0218] The methods disclosed in several method embodiments provided by the present disclosure can be arbitrarily combined without conflict to obtain new method embodiments. The features disclosed in several product embodiments provided by the present disclosure can be arbitrarily combined without conflict to obtain new product embodiments. The features disclosed in several method or device embodiments provided by the present disclosure can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0219] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0220] Alternatively, if the above-mentioned integrated unit of the present disclosure is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server cluster, a server, a tablet computer) to execute all or part of the methods described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0221] In the embodiments of the present disclosure, the descriptions of the same steps and the same contents in different embodiments can refer to each other. In the embodiments of the present disclosure, the term "and" does not affect the sequence of the steps.

[0222] The above is only an embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A training method for an image generation model, characterized in that: The image generation model includes a trained first generator and an external style control module to be trained; The first generator is pre-trained on a real sample set and is capable of generating a real image based on an internal style code for describing facial features; The method comprises: Acquire a style sample set and the real sample set; Training the external style control module based on the real sample set so that the first fused image generated by the image generation model can reconstruct the second fused image generated by the first generator; After the training of the external style control module is completed, the image generation model is trained based on the style sample set and the randomly sampled internal style code until a first convergence condition is met; The first generator is StyleGAN, which includes a first mapping network and a generation network; the external style control module includes a second mapping network with the same structure as the first mapping network, an external structural style control module, an external color style control module and a residual module; The second mapping network is used to map the external style codes of each style sample respectively to obtain corresponding external intermediate style codes; the external style code is obtained by encoding the style image and is used to describe the artistic style; The external structural style control module is used to determine the structural style code of each style sample based on a part of the external intermediate style code of each style sample; The external color style control module is used to extract the color style code of each style sample based on another part of the external intermediate style code of each style sample; The residual module is used to perform structural style adjustment on the target image generated by the generation network based on the structural style encoding of each style sample.

2. The method according to claim 1, characterized in that The training of the external style control module based on the real sample set so that the first fused image generated by the image generation model can reconstruct the second fused image of the first generator includes: Using the first generator, generating each second fused image based on the real sample set; Using the image generation model, generating first fused images based on the real sample set; wherein the first fused image and the second fused image are encoded based on the same pair of real images in the real sample set; determining a first reconstruction loss between each of the first fused images and the corresponding second fused image; The external style control module is iteratively trained based on the first reconstruction loss until a second convergence condition is met.

3. The method according to claim 2, characterized in that The step of using the first generator to generate each second fused image based on the real sample set includes: Determine the image code of each real sample in the real sample set respectively to obtain a real image code set; Acquire each pair of real image codes from the real image code set; Using the first generator, generating corresponding second fused images based on encoding of each pair of the real images; Correspondingly, the using the image generation model to generate each first fused image based on the real sample set includes: The image generation model is used to generate corresponding first fused images based on the encoding of each pair of real images.

4. The method according to claim 3, characterized in that The using the first generator to generate corresponding second fused images based on encoding of the real images comprises: Based on the first mapping network, respectively obtain the intermediate style codes of each pair of the real image codes; each of the intermediate style codes includes N element feature sequences; wherein N is an integer greater than or equal to 2; For each pair of the real image codes, extract the first k element features and the last (Nk) element features from the two intermediate style codes respectively and fuse them to obtain a fused intermediate style code; k is an integer greater than or equal to 1 and less than N; The generation network is used to generate a corresponding second fused image based on each of the fused intermediate style codes.

5. The method according to claim 4, characterized in that The method further comprises: When the first generator is StyleGAN, the values ​​of k are set to 7, 6, and 5 respectively; For each value of k, the intermediate style code is obtained based on the first mapping network, a fused intermediate style code is obtained, and a second fused image is obtained based on the generation network.

6. The method according to claim 4, characterized in that The using the image generation model to generate corresponding first fused images based on encoding of each pair of real images includes: Generate a corresponding real image using the first generator based on the second real image code in each pair of the real image code pairs; wherein the second real image code is a real image code corresponding to the extracted last (Nk) element features; For each pair of real image encodings, an encoder is used to encode each generated real image to obtain a corresponding external style code for describing an artistic style; Using the image generation model to be trained, based on the first real image code and the corresponding external style code in each pair of the real image code pairs, a corresponding first fused image is generated; wherein the first real image code is the real image code corresponding to the first k element features extracted.

7. The method according to any one of claims 1 to 6, characterized in that: The step of training the image generation model based on the style sample set and the randomly sampled internal style encoding until the first convergence condition is met includes: Iteratively training the image generation model based on the style sample set and the randomly sampled internal style code to obtain each style image generated by the image generation model; Based on each generated style image, and the style sample and internal style code corresponding to each style image, respectively determine the adversarial loss, the second reconstruction loss, the external style loss and the internal style loss, and accumulate the adversarial loss, the second reconstruction loss, the external style loss and the internal style loss to obtain the cumulative sum; wherein: the adversarial loss is the adversarial loss of the first generator, the second reconstruction loss is the loss of the generated style image relative to the corresponding style sample, the external style loss is the loss of the generated style image relative to the corresponding style sample, and the internal style loss is the loss of the generated style image relative to the corresponding internal style code; When the accumulated sum satisfies the first convergence condition, the training of the image generation model is completed.

8. The method according to claim 7, characterized in that The iterative training of the image generation model based on the style sample set and the randomly sampled internal style coding to obtain each style image generated by the image generation model includes: Determine an external style code and an internal style code of each style sample; wherein the external style code and the internal style code are used to describe an artistic style and a facial feature, respectively; Using the internal style code and the external style code of each of the style samples as inputs of the first generator and the external style control module, respectively, to obtain a first target style image generated by the image generation model; Using the randomly sampled internal style code and the external style code of each of the style samples as inputs of the first generator and the external style control module, respectively, to obtain a second target style image generated by the image generation model; The second reconstruction loss, the external style loss and the internal style loss are determined based on the generated style images, and the style samples and internal style codes corresponding to each of the style images, respectively, including: determining the second reconstruction loss based on each of the first target style images and the corresponding style samples; determining the external style loss based on each of the second target style images and the corresponding style samples; determining the internal style loss based on each of the second target style images and the corresponding random real image; The random real image is generated by the first generator based on the internal style encoding of the random sampling.

9. The method according to claim 8, characterized in that The determining of the external style code and the internal style code of each style sample includes: Using an encoder to encode each of the style samples respectively to obtain an external style code of the corresponding style sample; After de-stylizing each of the style samples, the internal style code of each style sample is obtained.

10. The method according to claim 9, characterized in that After de-stylizing each of the style samples, the internal style code of each style sample is obtained, including: Determine an optimized style code of each of the style samples, wherein the optimized style code can be used to reconstruct the corresponding style sample through the trained second generator; Using the trained first generator to reconstruct each of the optimized style codes respectively to obtain a reconstructed style sample; wherein the first generator and the second generator are trained on a real image dataset and a style image dataset respectively; The encoder is used to encode each of the reconstructed style samples to obtain an internal style code of the corresponding style sample.

11. The method according to claim 10, characterized in that The method comprises: Obtain a first generator pre-trained on a real sample set; The first generator is trained on the style sample set to obtain the second generator.

12. The method according to any one of claims 8 to 11, characterized in that The internal style code and the external style code of each of the style samples are used as inputs of the first generator and the external style control module respectively to obtain a first target style image generated by the image generation model; Taking the randomly sampled internal style code and the external style code of each of the style samples as inputs of the first generator and the external style control module respectively, obtaining a second target style image generated by the image generation model, including: Using the first mapping network, respectively based on the internal style code of each of the style samples, correspondingly obtain the internal intermediate style code of each of the style samples, and based on the internal style code of the random samples, correspondingly obtain the internal intermediate style code of the random samples; Using the second mapping network, respectively obtaining the external intermediate style codes of each of the external style codes; each of the external intermediate style codes includes N element feature sequences; Determining the structural style code of each style sample based on a portion of the external intermediate style code of each style sample by using the external structural style control module; Using the external color style control module, extracting the color style code of each style sample based on another part of the external intermediate style code of each style sample; Using the residual module, based on the structural style encoding of each style sample, the target image generated by the generation network is adjusted in structural style; Using the generative network, a first target style image is generated based on the structural style coding and the color style coding of each style sample, and the internal intermediate style coding of each style sample; and a second target style image is generated based on the structural style coding and the color style coding of each style sample, and the internal intermediate style coding of the random sampling.

13. An image processing method, characterized in that: Applied to the image generation model described in any one of 1 to 12, the method comprises: Obtaining a style image and a content image to be fused; Extracting the content image to obtain an internal intermediate style code of the content image; Extracting the style image to obtain an external intermediate style code of the style image; A fused image is generated based on the internal intermediate style coding of the content image and the external intermediate style coding of the style image.

14. The method according to claim 13, characterized in that The generating a fused image based on the internal intermediate style coding of the content image and the external intermediate style coding of the style image comprises: determining a structural style code of the style image based on a portion of an outer intermediate style code of the style image; extracting a color style code of the style image based on another portion of the outer intermediate style code of the style image; The fused image is generated based on the structural style coding and the color style coding of the style image and the internal intermediate style coding of the content image.

15. The method according to claim 14, characterized in that Each of the external intermediate style codes comprises N element feature sequences; The determining the structural style code of the style image based on a part of the external intermediate style code of the style image comprises: determining the structural style code of the style image based on first k element features in the external intermediate style code of the style image; The extracting the color style coding of the style image based on another part of the external intermediate style coding of the style image comprises: extracting the color style coding of the style image based on the last (Nk) element features of the external intermediate style coding of the style image.

16. A training device for an image generation model, characterized in that: The image generation model includes a trained first generator and an external style control module to be trained; The first generator is pre-trained on a real sample set and is capable of generating a real image based on an internal style code for describing facial features; The device comprises: A first acquisition module, used to acquire a style sample set and the real sample set; A first training module, used for training the external style control module based on the real sample set, so that the first fused image generated by the image generation model can reconstruct the second fused image generated by the first generator; A second training module is used to train the image generation model based on the style sample set and the randomly sampled internal style code after the training of the external style control module is completed, until a first convergence condition is met; The first generator is StyleGAN, which includes a first mapping network and a generation network; the external style control module includes a second mapping network with the same structure as the first mapping network, an external structural style control module, an external color style control module and a residual module; The second mapping network is used to respectively obtain the external intermediate style codes of each of the external style codes; The external structural style control module is used to determine the structural style code of each style sample based on a part of the external intermediate style code of each style sample; The external color style control module is used to extract the color style code of each style sample based on another part of the external intermediate style code of each style sample; The residual module is used to perform structural style adjustment on the target image generated by the generation network based on the structural style encoding of each style sample.

17. An image processing device, characterized in that: Applied to any one of the trained image generation models 1 to 12, the device comprises: A first acquisition module, used to acquire a style image and a content image to be fused; A first extraction module, configured to extract the content image to obtain an internal intermediate style code of the content image; A second extraction module, used to extract the style image to obtain an external intermediate style code of the style image; A generating module is used to generate a fused image based on the internal intermediate style coding of the content image and the external intermediate style coding of the style image.

18. A computer device, characterized in that: include: Memory and processor, The memory stores a computer program executable on the processor. When the processor executes the computer program, the processor implements the steps of the method described in any one of claims 1 to 12, or implements the steps of the method described in any one of claims 13 to 15.

19. A non-volatile computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the method described in any one of claims 1 to 12, or to implement the steps in the method described in any one of claims 13 to 15.

Images