A text-guided neural portrait painting method based on latent coding

Abstract

The present invention relates to a text-guided portrait neural painting method based on latent coding. The method comprises the following steps: S1. Obtaining a face image and corresponding style description text, calculating the latent space code, calculating the global style loss and the block style loss, and obtaining a style synthesis model; S2. Inputting the face image and the corresponding style description text into a cross-modal multi-level mapping network. The three-level mapping structure of the cross-modal multi-level mapping network processes text features and information of different dimensions of the image latent coding respectively, and the cross-modal multi-level mapping network is trained to obtain a cross-modal multi-level mapping model; S3. Obtaining an actual image and actual style description text, inputting the actual image and actual style description text into the cross-modal multi-level mapping model and the style synthesis model, and obtaining an actual style painting result. Compared with the existing technology, the present invention has the advantages of improving the matching degree between the image output by the stylization model and the stylized description.

Description

Technical Field

[0001] The present invention relates to the field of deep learning image conversion, and in particular to a text-guided portrait neural painting method based on latent coding. Background Art

[0002] Neural painting aims to redraw an input image in a target style using a neural network, generating an image with the same visual content as the input image but in the target style. Traditionally, the target style is expressed using a single image as a reference, enabling the neural network to learn the target style features in the image domain and then fuse them with the content image to produce a neural painting in the corresponding style. However, researchers have found that a single image cannot accurately convey the required style information and lacks flexibility, leading them to propose using text as a new expression method.

[0003] Image-based neural painting methods can be roughly divided into two categories. The first category involves image-based optimization: the input image itself is iteratively optimized until the output image achieves the desired target style. These methods require computational processing on a single image, which is time-consuming and limited in general applicability. The second category involves model-based optimization: a model is trained to generate neural painting in a specific style for any input image. These methods improve the versatility and flexibility of neural painting. However, both approaches are limited by image guidance and cannot flexibly achieve a wider range of neural painting.

[0004] Text-based neural painting methods use text to describe the desired style, allowing for more flexible expression of stylistic features. A challenge with this approach is that text and images occupy completely different feature domains, making it difficult to integrate the style described in the text with the content of the image. With the advancement of deep learning technology, researchers have developed numerous text-guided neural painting methods, most of which are based on loss-constrained model training using the cross-modal CLIP model. Some researchers have proposed directly using the original CLIP to calculate the degree of image-text correspondence as a measure of model effectiveness, but this has not yielded promising results. Subsequently, researchers have proposed using CLIP's text and image encoders to encode the input and output of the two modalities separately, then calculating cosine similarity to encourage image translation along the same direction of change as the text, achieving promising results. This approach has become a commonly used loss-constrained method in text-guided methods. Recently, researchers have implemented text-guided image translation using various neural network architectures. However, these methods often produce neural paintings with significant artifacts that destroy the original image structure, or fail to accurately capture the style information in the text, resulting in results that differ significantly from the text description. Summary of the Invention

[0005] The purpose of the present invention is to provide a text-guided portrait neural painting method based on implicit coding in order to improve the matching degree between the image output by the stylized model and the stylized description.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A text-guided portrait neural painting method based on latent coding, the method comprising the following steps:

[0008] S1. Obtain a face image and the corresponding style description text, input the face image into the style synthesis network of StyleGAN2, pass through the encoder of the style synthesis network to obtain the latent space code, and input the latent space code into each serial sub-module of the style synthesis network. The input of each sub-module is the latent space code, the output of the previous sub-module, and random noise. The Mod module and Demod module within each sub-module mix the input of the sub-module together to generate the output of the sub-module. Based on the output of the last sub-module and the style description text, the global style loss, block style loss, content loss, and total variational loss are calculated, and the style synthesis network is trained to obtain a style synthesis model.

[0009] S2. Input the face image and the corresponding style description text into the cross-modal multi-level mapping network, and pass them through the text encoder and image encoder of the cross-modal multi-level mapping network respectively to obtain the increased-dimensional text features and image latent codes. The three-level mapping structure of the cross-modal multi-level mapping network processes the different dimensional information of the increased-dimensional text features and image latent codes respectively, and concatenates the mapping results through a convolutional layer to obtain the residual, which is then added to the image latent code to obtain the optimized latent code. The optimized latent code is input into the style synthesis model, and the block enhancement style loss, content loss, style loss and total variation loss are calculated based on the output image of the style synthesis model. The cross-modal multi-level mapping network is trained to obtain a cross-modal multi-level mapping model.

[0010] S3. Obtain the actual image and the actual style description text, input the actual image and the actual style description text into the cross-modal multi-level mapping model and the style synthesis model, and obtain the actual style painting result.

[0011] Furthermore, the Mod module and the Demod module mix the inputs of the submodules to generate the outputs of the submodules in the following specific process:

[0012]

[0013] o′ i+1 =Conv(MDM(ω i+1,1 ,l j ),o i )+B+b i1

[0014] o i+1 =Conv(MDM(ω i+1,2 ,l j+1 ),Upsample(o′ i+1 ))+B+b i2

[0015] Among them, MDM represents the calculation of the Mod module and the Demod module, ω′ represents the convolutional layer Conv weight after scaling adjustment, and s j Representation and latent space encoding component l j The relevant scaling factor, ω is the weight of the convolution layer Conv; Conv represents a convolution layer with a convolution kernel size of 3×3, ∈ represents a very small amount to avoid zero division errors, o ′ i+1 represents the intermediate output feature processed by the first convolutional layer of the submodule, ω i+1,1 Represents the weight of the first convolutional layer of the current submodule, o i represents the output of the previous submodule, B represents random noise, b i1 、b i2 represents two different learnable bias values ​​of the i-th submodule, w i+1,2 Represents the weight of the second convolutional layer of the current submodule, l j 、l j+1 represents the hidden coding components of the jth layer and the j+1th layer, w i+1,2 Represents the weight of the second convolutional layer of the current submodule, o i+1 Indicates the output of the current layer submodule, Upsample is 2 times upsampling.

[0016] Furthermore, the global style loss is:

[0017] ΔT=E T (T tgt )-E T (T ori )

[0018] ΔI=E I (I out )-E I (I in )

[0019]

[0020] Among them, T tgt is the style description text corresponding to the output of the style synthesis network, T ori is the style description text corresponding to the face image, E T (), E I() is the text encoder and image encoder, I out is the output of the style synthesis network, I in is a face image, ΔT represents the text error, ΔI represents the image error, l glob represents the global style loss.

[0021] Furthermore, the calculation process of the block style loss is as follows: the output image of the style synthesis network is randomly divided into multiple blocks, and for each block of the image, the cosine similarity in the cross-modal domain is calculated to obtain the block style loss.

[0022] Furthermore, the dimension-enhanced text features are:

[0023] f=concat(E T (T tgt ),18)

[0024] Among them, f represents the text feature after dimension increase, and concat(,18) means copying the input tensor 18 times to increase the dimension.

[0025] Furthermore, the optimized implicit code is:

[0026] I opt =I+αconv(concat(f ′ ,I′))

[0027] Among them, I opt Represents optimized hidden coding, α represents the sum weight of the residual, conv represents the convolutional layer, concat represents tensor cascade, f′ represents the image tensor, and I′ represents the text tensor.

[0028] Furthermore, the calculation process of the image tensor is:

[0029] The image hidden code is input into the cross-modal multi-level mapping network, which includes a text mapping module M T and image mapping module M I , the mapping module includes 3 mappers, the image hidden code input image mapping module M I , image mapping module M I The outputs of the three mappers are concatenated to obtain the image tensor;

[0030] The calculation process of the text tensor is:

[0031] Input the dimension-increased text feature f into the text mapping module M of the cross-modal multi-level mapping network T , text mapping module M T The outputs of the three mappers are concatenated to get the text tensor.

[0032] Furthermore, the block-enhanced style loss is:

[0033]

[0034]

[0035]

[0036]

[0037] Among them, I out Represents the output of the style synthesis network, Crop represents random blocking, and Aug represents the enhancement selected from random coloring, random translation, random resizing, and random cropping. Represents the block image after random block and random enhancement of the output of the style synthesis network, I in is the face image, ΔT represents the text error, ΔI i represents the block image error, represents the enhanced style loss of the i-th block, R(x,τ) represents a threshold function that outputs 0 when the input x is greater than τ and outputs x otherwise, l patch represents the block-wise enhanced style loss.

[0038] Furthermore, the content loss and style loss are:

[0039] The L2 distance between the output image of the style synthesis model and the input face image at different layers of the feature extraction network VGG-19.

[0040] Furthermore, the encoder of the style synthesis network is an e4e encoder.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The multi-level mapping network proposed in this paper effectively improves the neural painting effect and training stability by introducing text features into the input and using block-enhanced style loss. In combination with the optimization of the latent code of text features, the text features are introduced as the input of the multi-level mapping network, and the prediction residual directly optimizes the image latent code, achieving an improvement in the neural painting effect at the level of character structure, which can significantly enhance the stylization effect. At the same time, the present invention uses block-enhanced style loss to make the model more resistant to interference and generate results that are more consistent with the text description, improving the degree of match between the image output by the stylized model and the stylized description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of the present invention. DETAILED DESCRIPTION

[0044] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0045] The present invention proposes a text-guided portrait neural painting method based on latent coding, which is used to convert the input image into the corresponding neural painting style described by the input text, so as to achieve more convenient and flexible image stylization conversion. The method is divided into two optimization stages: first, in the first stage, we optimize a pre-trained stylized neural model, and constrain the training process in combination with the image-text multimodal model. This method can quickly give high-quality portrait neural painting results that meet the text description for any portrait without the need to retrain the network using a data set; secondly, in the second stage, we propose to use a cross-modal multi-level mapping network to incorporate text features as model input, and optimize the latent coding of the portrait, thereby further improving the neural stylization effect. In addition, we also propose a progressive mapping network training method, and introduce a new block-enhanced style loss to guide the network optimization process. After a large number of experiments, it has been verified that this method is superior to the existing text-guided neural painting method. The flowchart of the present invention is as follows Figure 1 shown.

[0046] This aspect includes the following steps:

[0047] S1. Obtain a face image and the corresponding style description text, input the face image into the style synthesis network of StyleGAN2, pass through the encoder of the style synthesis network to obtain the latent space code, and input the latent space code into each serial sub-module of the style synthesis network. The input of each sub-module is the latent space code, the output of the previous sub-module, and random noise. The Mod module and Demod module within each sub-module mix the input of the sub-module together to generate the output of the sub-module. Based on the output of the last sub-module and the style description text, the global style loss, block style loss, content loss, and total variational loss are calculated, and the style synthesis network is trained to obtain a style synthesis model.

[0048] S2. Input the face image and the corresponding style description text into the cross-modal multi-level mapping network, and pass them through the text encoder and image encoder of the cross-modal multi-level mapping network respectively to obtain the increased-dimensional text features and image latent codes. The three-level mapping structure of the cross-modal multi-level mapping network processes the different dimensional information of the increased-dimensional text features and image latent codes respectively, and concatenates the mapping results through a convolutional layer to obtain the residual, which is then added to the image latent code to obtain the optimized latent code. The optimized latent code is input into the style synthesis model, and the block enhancement style loss, content loss, style loss and total variation loss are calculated based on the output image of the style synthesis model. The cross-modal multi-level mapping network is trained to obtain a cross-modal multi-level mapping model.

[0049] S3. Obtain the actual image and the actual style description text, input the actual image and the actual style description text into the cross-modal multi-level mapping model and the style synthesis model, and obtain the actual style painting result.

[0050] In S1, the stylized neural model is optimized for multimodal graphics and text. The specific steps are as follows:

[0051] 1.1) First, the input image is converted into latent space codes with the same format using the encoder (e4e). Each code is a tensor of size (18,512). This process also removes several detailed features to avoid interfering with the optimization process of neural painting.

[0052] 1.2) The style model uses the style synthesis network of StyleGAN2, which is composed of 9 sub-modules with similar structures connected in series. Each sub-module uses two one-dimensional components of the latent code, namely two (1,512) tensors, and the low-resolution sub-module result of the previous layer as input, and outputs a result of equal or higher resolution.

[0053] 1.3) The style model is trained using global style loss, block-wise style loss, image content loss, and total variational loss. The global style loss encodes the input text and image using corresponding encoders and calculates cosine similarity in the cross-modal domain to constrain the global style loss. The block-wise style loss calculates cosine similarity in the cross-modal domain for randomly partitioned images, guiding the style model to generate neural paintings that are consistent with the text description. The content loss constrains model training by calculating the L2 distance between the content features of the output image and the input image. The total variational loss makes the generated image smoother by minimizing the gradient in pixel space.

[0054] In S2, the latent encoding optimization is performed in combination with text features. The specific steps are:

[0055] 2.1) Use the corresponding encoders to convert the style description text and input image into text features and latent space encodings. Upscale the text features so that their tensors have the same size as the image latent code.

[0056] 2.2) Design a cross-modal multi-level mapping network. This mapping network consists of two parts: a text mapping module and an image mapping module. Each mapping module has three feature mappers responsible for processing different dimensions of input features. The output tensors of the two mapping networks are cascaded and fused using a convolutional layer to predict a residual, which is then summed and optimized with the latent code.

[0057] 2.3) Use a progressive training method to train the multi-level mapping network. First, use a large learning rate to make the mapping network converge quickly to the optimal solution. Then reduce the learning rate to more finely approximate the optimal solution, thereby improving model training stability and stylization effect.

[0058] 2.4) The mapping network is trained using a block-enhanced style loss, content loss, style loss, and total variational loss. The block-enhanced style loss randomly blocks and enhances the block image, then calculates cross-modal cosine similarity to constrain the style consistency of the output and the input text. The content loss and style loss respectively calculate the L2 distance between the features of the original portrait and the output image at different layers of the feature extraction network. The total variational loss makes the generated image smoother by minimizing the gradient in pixel space.

[0059] In the process of optimizing the hidden code of text features, text features are introduced as the input of the multi-level mapping network, and the prediction residual is used to directly optimize the image hidden code, thereby improving the neural painting effect at the level of character structure.

[0060] A progressive training method divides the training of the multi-level mapping network into two simple stages. First, a large learning rate is used to quickly reach the optimal solution, and then a small learning rate is used for further optimization, which improves training stability and stylization effects.

[0061] During the multimodal optimization of images and text in the stylized neural model, a block-wise style loss is used to constrain the model training process. This loss first randomly divides the image into blocks and enhances each block using a random enhancement method (four in total). Then, cosine similarity is calculated in the cross-modal domain, making the stylization process more stable and improving the model's stylization effect.

[0062] The model of this invention comprises a stylized network based on the StyleGAN2 model, a cross-modal multi-level mapping network, a block-enhanced style loss, and a progressive training method for portrait neural painting. The method first performs efficient multi-modal optimization of images and text based on the pre-trained StyleGAN2 model to obtain a stylized network. This stage can initially generate high-quality portrait neural paintings that match the text style description. Then, using the cross-modal multi-level mapping network, the latent code corresponding to the input image is optimized using text features, further improving the neural painting results.

[0063] In S1 of the present invention, the style model adopts the StyleGAN2 structure, which is composed of 9 similar submodules connected in series. According to the input latent code, the corresponding image is generated. Each submodule has three inputs, namely the output from the submodule in the previous layer connected in series. i , the component l of the image hidden code j,j+1 And random noise B, the three kinds of information are mixed together through the Mod and Demod modules to generate the module result. The process of each sub-module can be expressed as follows:

[0064]

[0065] o′ i+1 =Conv(MOM(w i+1,1 ,l j ),p i )+B+b i1

[0066] o i+1 =Conv(MDM(w i+1,2 ,l j+1 ),Upsample(o′ i+1 ))+B+b i2

[0067] Where MDM represents the calculation of Mod and Demod modules, s j is the same as the latent coding component l j The relevant scaling factor, ω, is the weight of the convolution layer Conv; Conv represents a convolution layer with a convolution kernel size of 3×3, and Upsample is a 2x upsampling. The first submodule does not have an upsampling layer, but instead uses a fixed constant. Its output resolution is 4×4, and the resolution doubles with each submodule until a high-definition result of 1024×1024 resolution is obtained. The image latent code l is obtained by the image encoder (e4e) and is an 18×512 tensor, so each component l j is a 1×512 tensor.

[0068] The global style loss and block style loss described in step 1 are obtained by calculating the cosine similarity using the cross-modal model CLIP to encourage the image to find the optimal solution for neural painting along the direction of text change. For each input image I in , assuming the initial text T ori = "a photo", the output of the style model is I out , the corresponding input style text is T tgt , text encoder E T () and image encoder E I () can be expressed as follows: global style loss l glob :

[0069] ΔT=E T (T tgt )-E T (T ori )

[0070] ΔI=E I (I out )-E I (I in )

[0071]

[0072] The block loss is obtained by calculating the average loss value after randomly dividing the image into multiple blocks based on the above method.

[0073] The cross-modal multi-level mapping network in S2 consists of two modules: the text mapping module M T and image mapping module M I , the two modules have the same structure but process different input features, namely text features f and image latent codes l. The acquisition of text features can be expressed as follows:

[0074] f=concat(E T (T tgt ),18)

[0075] Where concat(,18) means to copy the input tensor 18 times to increase the dimension, so as to obtain the same size as the image hidden code. Each module has 3 mappers It is used to process feature information at three different levels: low, medium, and high. Among them, the low-level mapper and the middle-level mapper each process the feature tensor input from the (4,512) size, while the high-level mapper processes the remaining feature tensor input of the (10,512) size. The three mappers map the input to output tensors of the same size and concatenate them in sequence to obtain the mapping tensors f′, I′. The process of obtaining the residual based on the two mapping tensors to optimize the original latent code is as follows:

[0076] I opt =I+αconv(concat(f′,I′))

[0077] Where conv represents the convolution layer, α represents the residual sum weight, concat represents the tensor cascade, and the optimized hidden code I opt The final portrait neural painting result can be obtained through the style model generation in step 1.

[0078] S2's progressive training method is a strategy designed to improve training stability and efficiency. In the initial training phase of a cross-modal multi-level mapping network, a higher learning rate is used to quickly bring the mapping network close to the optimal solution, followed by a lower learning rate to more accurately approximate the optimal solution.

[0079] S2's block-wise augmented style loss is designed to improve model training stability. Using the original block-wise style loss can easily generate images with low correlation to the text. The block-wise augmented style loss randomly partitions the output image into blocks and then performs one of four random augmentations: random coloring, random translation, random resizing, and random cropping. The augmented style loss is then calculated using the block-wise augmented style loss formula. This ensures that attacks against the block-wise augmented style loss require simultaneous attacks on four different augmented data types, improving training stability.

[0080] This paper proposes a text-guided portrait neural painting model based on latent coding. This neural painting model can generate high-quality painting results that conform to the text style description for any portrait, and is highly efficient and does not require the use of a dataset for training. The multi-level mapping network proposed in this paper can effectively improve the neural painting effect and training stability by introducing text features into the input and enhancing the style loss in a block-by-block manner. The proposed progressive training strategy effectively improves training efficiency. In comparison with four existing text-guided neural painting methods, the method proposed in this paper has significant advantages in various qualitative and quantitative indicators.

[0081] Advantages of neural painting effects: The method of the present invention can more accurately capture the style information in the text, and fuse it with the input content image to generate high-quality neural painting results. In a visual comparison, under the premise of the same text as input, the present invention can generate neural paintings that are closer to the style described by the text, and will not generate artifacts to destroy the original structure of the image. In the comparison of qualitative indicators, the present invention has achieved obvious advantages in the two important indicators of SSIM and PSNR. At the same time, the method of the present invention performs rapid two-stage optimization on a single image to obtain high-quality results, which is more efficient than the existing methods that require the use of data sets for training.

[0082] Advantages of cross-modal multi-level mapping networks: Traditional neural painting methods that use images as style references only extract style information from image features, which makes these methods less flexible in describing style. Although text-guided neural painting methods use text to describe style requirements more flexibly, they are limited to using text features to calculate loss training models. The method of the present invention uses text features to calculate residuals to optimize image latent codes, which effectively improves the effect of neural painting. Experiments show that the stylization effect can be significantly improved by directly optimizing the latent codes using text features through the progressive training method of the present invention.

[0083] Advantages of the Block-Enhanced Style Loss: To improve the stability of the training process and achieve higher-quality results, this paper uses the Block-Enhanced Style Loss as one of the training losses in step 2. Ablation experiments have shown that this method has stronger anti-interference capabilities than the simple Block-Enhanced Style Loss and can produce results that are more consistent with the text description.

[0084] The following is a practical experiment:

[0085] The style model in step 1 of this invention consists of nine similar submodules, each of which receives three outputs: the result from the previous layer (the initial layer is a fixed constant), the image latent code component, and random noise. Each submodule includes a Mod, Demod, and a convolutional layer. The final output of the model is a high-definition neural painting image with a resolution of 1024×1024.

[0086] In the actual experiment, the steps are as follows:

[0087] S100: First, the input image is converted into the latent code input required by the style model through a latent encoder e4e. The different dimensional components of the latent code serve as the input of different sub-modules of the style model.

[0088] S110: The style model adopts the StyleGAN2 structure, which contains 9 similar sub-modules. Each sub-module is based on the results of the previous low-resolution module, and the latent encoded feature information is integrated to obtain higher-resolution results. At the same time, random noise is added to generate fine details.

[0089] S120: Calculate the content loss between the image result of the style model and the input image to encourage content preservation; calculate the global and block style loss between the image result and the input style text description to encourage the result to be more in line with the required style; finally, calculate the total variational loss to enhance the image smoothness.

[0090] See Figure 1The multi-level mapping network in step 2 of the present invention consists of two parts: a text mapping module and an image mapping module. This method introduces text features into the network to optimize the image latent coding, further improving the neural painting effect.

[0091] S130: The text description of the style and the input image are encoded by the text encoder E T () and image encoder E I () is encoded to obtain text features f and image latent codes l, and the text features are copied and dimensionally upgraded.

[0092] S140: The three-level mapping structure of the text mapping module and the image mapping module processes the different dimensional information of the upgraded text feature f and the image latent code l respectively, and the mapping results are cascaded and passed through a convolution layer with a convolution kernel of 3×3. The residual is added to the original latent code to obtain the optimized latent code l opt .

[0093] S150: Generate l using the optimized style model opt The corresponding latent code optimization results are then combined with the style text description to calculate a block-wise augmentation style loss to guide the training of the multi-level mapping network. This loss first randomly partitions the output image into blocks, then augments it using random coloring, random translation, random resizing, or random cropping. The cosine similarity between the style text description and the block-wise augmented image is then calculated in the cross-modal domain. After training the multi-level mapping network, the final optimization results are obtained.

[0094] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A text-guided portrait neural painting method based on latent coding, characterized by: The method comprises the following steps: S1. Obtain a face image and the corresponding style description text, input the face image into the style synthesis network of StyleGAN2, pass through the encoder of the style synthesis network to obtain the latent space code, and input the latent space code into each serial sub-module of the style synthesis network. The input of each sub-module is the latent space code, the output of the previous sub-module, and random noise. The Mod module and Demod module within each sub-module mix the input of the sub-module together to generate the output of the sub-module. Based on the output of the last sub-module and the style description text, the global style loss, block style loss, content loss, and total variational loss are calculated, and the style synthesis network is trained to obtain a style synthesis model. S2. Input the face image and the corresponding style description text into the cross-modal multi-level mapping network, and pass them through the text encoder and image encoder of the cross-modal multi-level mapping network respectively to obtain the increased-dimensional text features and image latent codes. The three-level mapping structure of the cross-modal multi-level mapping network processes the different dimensional information of the increased-dimensional text features and image latent codes respectively, and concatenates the mapping results through a convolutional layer to obtain the residual, which is then added to the image latent code to obtain the optimized latent code. The optimized latent code is input into the style synthesis model, and the block enhancement style loss, content loss, style loss and total variation loss are calculated based on the output image of the style synthesis model. The cross-modal multi-level mapping network is trained to obtain a cross-modal multi-level mapping model. S3. Obtain the actual image and the actual style description text, input the actual image and the actual style description text into the cross-modal multi-level mapping model and the style synthesis model, and obtain the actual style painting result.

2. The text-guided portrait neural painting method based on latent coding according to claim 1 is characterized in that: The specific process of the Mod module and the Demod module mixing the input of the submodule to generate the output of the submodule is as follows: o′ i+1 =Conv(MDM(ω i+1,1 ,l j ),o i )+B+b i1 Oh i+1 =Conv(MDM(ω i+1,2 ,l j+1 ),Upsample(o i ′ +1 ))+B+b i2 Among them, MDM represents the calculation of the Mod module and the Demod module, ω′ represents the convolutional layer Conv weight after scaling adjustment, and s j Representation and latent space encoding component l j The relevant scaling factor, ω is the weight of the convolution layer Conv; Conv represents a convolution layer with a convolution kernel size of 3×3, ∈ represents a very small amount to avoid zero division errors, o′ i+1 represents the intermediate output feature processed by the first convolutional layer of the submodule, ω i+1,1 Represents the weight of the first convolutional layer of the current submodule, o i represents the output of the previous submodule, B represents random noise, b i1 、b i2 represents two different learnable bias values ​​of the i-th submodule, ω i+1,2 Represents the weight of the second convolutional layer of the current submodule, l j 、l j+1 represents the hidden coding components of the jth layer and the j+1th layer, ω i+1,2 Represents the weight of the second convolutional layer of the current submodule, o i+1 Indicates the output of the current layer submodule, Upsample is 2 times upsampling.

3. The text-guided portrait neural painting method based on latent coding according to claim 1 is characterized in that: The global style loss is: ΔT=E T (T tgt )-E T (T ori ) ΔI=E I (I out )-HAVE BEEN I (I in ) Among them, T tgt is the style description text corresponding to the output of the style synthesis network, T ori is the style description text corresponding to the face image, E T (), E I () is the text encoder and image encoder, I out is the output of the style synthesis network, I in is a face image, ΔT represents the text error, ΔI represents the image error, l glob represents the global style loss.

4. The text-guided portrait neural painting method based on latent coding according to claim 3 is characterized in that: The calculation process of the block style loss is as follows: the output image of the style synthesis network is randomly divided into multiple blocks, and for each block of the image, the cosine similarity in the cross-modal domain is calculated to obtain the block style loss.

5. The text-guided portrait neural painting method based on latent coding according to claim 1 is characterized in that: The text features of the dimension increase are: f=concat(E T (T tgt ),18) Among them, f represents the text feature after dimension increase, and concat(,18) means copying the input tensor 18 times to increase the dimension.

6. The text-guided portrait neural painting method based on latent coding according to claim 5, characterized in that: The optimized implicit code is: I opt =I+αconv(concat(f′,I′)) Among them, I opt Represents optimized hidden coding, α represents the sum weight of the residual, conv represents the convolutional layer, concat represents tensor cascade, f′ represents the image tensor, and I′ represents the text tensor.

7. The text-guided portrait neural painting method based on latent coding according to claim 6 is characterized in that: The calculation process of image tensor is: The image hidden code is input into the cross-modal multi-level mapping network, which includes a text mapping module M T and image mapping module M I , the mapping module includes 3 mappers, the image hidden code input image mapping module M I , image mapping module M I The outputs of the three mappers are concatenated to obtain the image tensor; The calculation process of the text tensor is: Input the dimension-increased text feature f into the text mapping module M of the cross-modal multi-level mapping network T , text mapping module M T The outputs of the three mappers are concatenated to get the text tensor.

8. The text-guided portrait neural painting method based on latent coding according to claim 1, characterized in that: The block-enhanced style loss is: Among them, O oit Represents the output of the style synthesis network, Crop represents random blocking, and Aug represents the enhancement selected from random coloring, random translation, random resizing, and random cropping. Represents the block image after random block and random enhancement of the output of the style synthesis network, I in is the face image, ΔT represents the text error, ΔI i represents the block image error, represents the enhanced style loss of the i-th block, R(x,τ) represents a threshold function that outputs 0 when the input x is greater than τ and outputs x otherwise, l patch represents the block-wise enhanced style loss.

9. The text-guided portrait neural painting method based on latent coding according to claim 8, characterized in that: The content loss and style loss are: The L2 distance between the output image of the style synthesis model and the input face image at different layers of the feature extraction network VGG-19.

10. The text-guided portrait neural painting method based on latent coding according to claim 1, characterized in that: The encoder of the style synthesis network is an e4e encoder.

Images