Text-guided portrait nerve drawing method based on implicit coding

Abstract

The invention relates to a text-guided portrait nerve drawing method based on implicit coding, and the method comprises the following steps: S1, obtaining a face image and a corresponding style description text, calculating an implicit space code, calculating global style loss and block style loss, and obtaining a style synthesis model; s2, inputting the face image and the corresponding style description text into a cross-modal multi-level mapping network, processing different dimension information of text features and image hidden codes by a three-level mapping structure of the cross-modal multi-level mapping network, and training the cross-modal multi-level mapping network to obtain a cross-modal multi-level mapping model; and S3, acquiring an actual image and an actual style description text, and inputting the actual image and the actual style description text into the cross-modal multilevel mapping model and the style synthesis model to obtain an actual style drawing result. Compared with the prior art, the method has the advantages of improving the matching degree of the image output by the stylized model and the stylized description and the like.

Description

Technical Field

[0001] This invention relates to the field of deep learning image transformation, and in particular to a text-guided portrait neural painting method based on implicit coding. Background Technology

[0002] Neural painting aims to redraw an input image using a target style through neural networks, generating an image with the same content as the input image but in the target style. The traditional method of representing the target style uses an image as a reference, allowing the neural network to learn the target style features in the image domain and fuse them with the content image to obtain the corresponding style of the neural painting. However, researchers have found that a single image is insufficient to accurately represent the required style information and lacks flexibility, thus proposing the use of text as a new representation method.

[0003] Image-based neural painting methods can be broadly categorized into two types. The first type involves image-based optimization: iteratively optimizing the input image until the output image achieves the desired style. This type of method requires computational processing on a single image, resulting in a lengthy process and limited general applicability. The second type involves model-based optimization: training a model to generate neural paintings with a specific style from any input image. This type of method improves the general applicability and flexibility of neural painting. However, both types of methods are limited by image guidance, hindering the flexible implementation of more diverse neural paintings.

[0004] Text-based neural painting methods use text to describe the desired style, allowing for more flexible expression of stylistic features. The challenge lies in the fact that text and images reside in completely different feature domains, making it difficult to fuse the style described in the text with the content of the image. With the development of deep learning technology, researchers have developed many text-guided neural painting methods, most of which are based on loss-constrained training using the cross-modal model CLIP. Some researchers proposed directly using the original CLIP to calculate the similarity between the image and text to measure model performance, but this did not yield good results. Subsequently, researchers proposed using CLIP's text and image encoders to encode the input and output of the two modalities separately, then calculating cosine similarity to encourage the image to migrate along the same direction of change as the text, achieving good results. This method has become a commonly used loss constraint method in text-guided methods. Recently, researchers have implemented text-guided image transformation based on different neural network architectures, but these methods suffer from problems such as easily generating neural paintings with severe artifacts that destroy the original image structure, or failing to accurately capture stylistic information in the text, resulting in results that are significantly different from the text description. Summary of the Invention

[0005] The purpose of this invention is to provide a text-guided portrait neural painting method based on implicit coding to improve the matching degree between the image output by the stylization model and the stylization description.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A text-guided portrait neural painting method based on implicit coding, comprising the following steps:

[0008] S1. Obtain a face image and the corresponding style description text. Input the face image into the StyleGAN2 style synthesis network. After passing through the encoder of the style synthesis network, the latent space code is obtained. The latent space code is input into each concatenated sub-module of the style synthesis network. The input of each sub-module is the latent space code, the output of the previous sub-module, and random noise. The Mod and Demod modules inside each sub-module mix the input of the sub-module together to generate the output of the sub-module. Based on the output of the last sub-module and the style description text, calculate the global style loss, block style loss, content loss, and total variational loss. Train the style synthesis network to obtain the style synthesis model.

[0009] S2. Input the face image and the corresponding style description text into the cross-modal multilevel mapping network. The text encoder and image encoder of the cross-modal multilevel mapping network are respectively passed to obtain the upgraded text features and image latent code. The three-level mapping structure of the cross-modal multilevel mapping network processes the upgraded text features and image latent code of different dimensions. The mapping results are concatenated and passed through a convolutional layer. The residual is then added to the image latent code to obtain the optimized latent code. The optimized latent code is input into the style synthesis model. Based on the output image of the style synthesis model, the block-enhanced style loss, content loss, style loss and total variational loss are calculated. The cross-modal multilevel mapping network is trained to obtain the cross-modal multilevel mapping model.

[0010] S3. Obtain the actual image and actual style description text, input the actual image and actual style description text into the cross-modal multi-level mapping model and style synthesis model to obtain the actual style painting result.

[0011] Furthermore, the specific process by which the Mod and Demod modules combine the inputs of a submodule to generate its output is as follows:

[0012]

[0013] o′ i+1 =Conv(MDM(ω) i+1,1 ,l j ),o i )+B+b i1

[0014] o i+1 =Conv(MDM(ω) i+1,2 ,l j+1 ),Upsample(o′ i+1 ))+B+b i2

[0015] Where MDM represents the computation of the Mod and Demod modules, ω′ represents the Conv weights of the convolutional layer after scaling adjustment, and s j Represents the latent space coding component l j The relevant scaling factor, ω is the weight of the convolutional layer Conv; Conv represents a convolutional layer with a kernel size of 3×3, ∈ represents a minimal quantity to avoid division by zero error, o ′ i+1 ω represents the intermediate output feature processed by the first convolutional layer of the submodule. i+1,1 This represents the weights of the first convolutional layer in the current submodule, o i This represents the output of the previous submodule, where B represents random noise. i1 b i2 w represents two distinct learnable bias values ​​for the i-th submodule. i+1,2 Indicates the weights of the second convolutional layer in the current submodule, l j l j+1 w represents the implicit coding components at layer j and layer (j+1). i+1,2 This represents the weights of the second convolutional layer in the current submodule, o. i+1 This indicates the output of the current layer submodule, with Upsample being a factor of 2.

[0016] Furthermore, the global style loss is:

[0017] ΔT=E T (T tgt )-E T (T ori )

[0018] ΔI=E I (I out )-E I (I in )

[0019]

[0020] Among them, T tgt T is the style description text corresponding to the output of the style synthesis network. ori E is the style description text corresponding to a face image. T (), E I() represents a text encoder and an image encoder, I out For the output of the style synthesis network, I in For a face image, ΔT represents text error, ΔI represents image error, and l glob This indicates the global style loss.

[0021] Furthermore, the calculation process of the block style loss is as follows: the output image of the style synthesis network is randomly divided into multiple blocks, and for each block, the cosine similarity in the cross-modal domain is calculated to obtain the block style loss.

[0022] Furthermore, the enhanced text features are:

[0023] f = concat(E) T (T tgt ),18)

[0024] Where f represents the text feature after dimensionality increase, and concat(,18) means copying the input tensor 18 times to increase its dimensionality.

[0025] Furthermore, the implicit coding is optimized as follows:

[0026] I opt =I+αconv(concat(f ′ ,I′))

[0027] Among them, I opt denoted as 'optimize implicit coding', α represents the summation weight of residuals, conv represents a convolutional layer, concat represents tensor concatenation, f′ represents an image tensor, and I′ represents a text tensor.

[0028] Furthermore, the calculation process for image tensors is as follows:

[0029] The image implicit code is input into a cross-modal multilevel mapping network, which includes a text mapping module M. T and image mapping module M I The mapping module includes three mappers: an image implicit encoding input image mapping module M. I Image mapping module M I The image tensor is obtained by concatenating the outputs of the three mappers;

[0030] The calculation process for text tensors is as follows:

[0031] The upgraded text features f are input into the text mapping module M of the cross-modal multilevel mapping network. T Text mapping module M T The outputs of the three mappers are concatenated to obtain the text tensor.

[0032] Furthermore, the block-based style enhancement loss is:

[0033]

[0034]

[0035]

[0036]

[0037] Among them, I out This represents the output of the style synthesis network. Crop indicates random block division, and Aug indicates that one of the following enhancement methods is selected: random coloring, random translation, random resizing, and random cropping. I represents the image after random partitioning and random enhancement of the output of the style synthesis network. in For a face image, ΔT represents the text error, and ΔI represents the text error. i This indicates the error in the segmented image. Let R(x,τ) represent the style enhancement loss for the i-th block, and let R(x,τ) represent a threshold function that outputs 0 when the input x is greater than τ, and outputs x otherwise. patch This represents the style loss due to block enhancement.

[0038] Furthermore, the content loss and style loss are as follows:

[0039] The L2 distance between the output image and the input face image of the style synthesis model at different layers of the VGG-19 feature extraction network.

[0040] Furthermore, the encoder of the style synthesis network is an e4e encoder.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The multi-level mapping network proposed in this invention effectively improves neural painting effects and training stability by incorporating text features into the input and using block-based style enhancement loss. By integrating text features into the implicit coding optimization process, text features are used as input to the multi-level mapping network, and the prediction residuals directly optimize the image implicit coding, achieving improved neural painting effects at the structural level of the figure, thus significantly enhancing stylization results. Simultaneously, this invention uses block-based style enhancement loss, making the model more robust to interference and generating results that better match the text description, improving the matching degree between the stylization model's output image and the stylized description. Attached Figure Description

[0043] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0044] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0045] This invention proposes a text-guided portrait neural painting method based on implicit coding, used to convert input images into neural painting styles corresponding to the input text description, achieving more convenient and flexible image stylization conversion. The method consists of two optimization stages: First, in the first stage, a pre-trained stylization neural model is optimized, and the training process is constrained by a multimodal image-text model. This method can quickly provide high-quality portrait neural painting results that conform to the text description for any portrait without needing to retrain the network using a dataset. Second, in the second stage, we propose using a cross-modal multi-level mapping network to incorporate text features as model input and optimize the implicit coding of the portrait, thereby further improving the neural stylization effect. Furthermore, we propose a progressive mapping network training method and introduce a novel block-enhanced style loss to guide the network optimization process. Extensive experimental verification shows that this method outperforms existing text-guided neural painting methods. The flowchart of this invention is shown below. Figure 1 As shown.

[0046] This aspect includes the following steps:

[0047] S1. Obtain a face image and the corresponding style description text. Input the face image into the StyleGAN2 style synthesis network. After passing through the encoder of the style synthesis network, the latent space code is obtained. The latent space code is input into each concatenated sub-module of the style synthesis network. The input of each sub-module is the latent space code, the output of the previous sub-module, and random noise. The Mod and Demod modules inside each sub-module mix the input of the sub-module together to generate the output of the sub-module. Based on the output of the last sub-module and the style description text, calculate the global style loss, block style loss, content loss, and total variational loss. Train the style synthesis network to obtain the style synthesis model.

[0048] S2. Input the face image and the corresponding style description text into the cross-modal multilevel mapping network. The text encoder and image encoder of the cross-modal multilevel mapping network are respectively passed to obtain the upgraded text features and image latent code. The three-level mapping structure of the cross-modal multilevel mapping network processes the upgraded text features and image latent code of different dimensions. The mapping results are concatenated and passed through a convolutional layer. The residual is then added to the image latent code to obtain the optimized latent code. The optimized latent code is input into the style synthesis model. Based on the output image of the style synthesis model, the block-enhanced style loss, content loss, style loss and total variational loss are calculated. The cross-modal multilevel mapping network is trained to obtain the cross-modal multilevel mapping model.

[0049] S3. Obtain the actual image and actual style description text, input the actual image and actual style description text into the cross-modal multi-level mapping model and style synthesis model to obtain the actual style painting result.

[0050] In S1, the stylized neural model undergoes multimodal optimization of text and images. The specific steps are as follows:

[0051] 1.1) First, the input image is converted into a latent space code with the same format using an encoder (e4e). Each code is a tensor of size (18,512). This process removes some detail features to avoid interfering with the optimization process of neural drawing.

[0052] 1.2) The style model adopts the StyleGAN2 style synthesis network, which consists of nine structurally similar sub-modules connected in series. Each sub-module uses two one-dimensional components of the implicit encoding, namely two (1,512) tensors, as well as the results of the lower-resolution sub-module of the previous layer as input, and outputs results of equal or higher resolution.

[0053] 1.3) The style model is trained using global style loss, patch style loss, image content loss, and total variational loss. Global style loss encodes the input text and image using their respective encoders and calculates cosine similarity across modal domains to constrain the global style loss. Patch style loss calculates cosine similarity across modal domains for randomly segmented images, guiding the style model to generate neural paintings that match the text description. Content loss constrains model training by calculating the L2 distance between the content features of the output and input images. Total variational loss minimizes gradients in pixel space to make the generated image smoother.

[0054] In S2, implicit coding optimization is performed by incorporating text features. The specific steps are as follows:

[0055] 2.1) Convert the style description text and input image into text features and latent space encoding using the corresponding encoders. Upscale the text features so that their tensors have the same size as the image latent encoding.

[0056] 2.2) Design a cross-modal multi-level mapping network. The mapping network consists of two parts: a text mapping module and an image mapping module. Each mapping module has three feature mappers responsible for processing different dimensions of the input features. The output tensors of the two mapping networks are concatenated and then fused using a convolutional layer to predict a residual, which is then summed and optimized with the implicit coding.

[0057] 2.3) Use a progressive training method to train the multi-level mapping network. First, use a large learning rate to make the mapping network converge quickly to the vicinity of the optimal solution. Then, reduce the learning rate to more finely approximate the optimal solution, thereby improving the model training stability and stylization effect.

[0058] 2.4) The mapping network is trained using block-enhanced style loss, content loss, style loss, and total variational loss. Block-enhanced style loss randomly divides and enhances the image, then calculates cross-modal cosine similarity to constrain the style consistency between the output and the input text; content loss and style loss calculate the L2 distance between features in different layers of the feature extraction network of the original portrait and the output image, respectively; total variational loss makes the generated image smoother by minimizing the gradient in pixel space.

[0059] In the process of optimizing implicit coding by combining text features, text features are introduced as input to a multi-level mapping network. The prediction residuals directly optimize the image implicit coding, thereby improving the neural painting effect at the structural level of the figure.

[0060] The progressive training method divides the training of the multi-level mapping network into two simple stages: first, it quickly reaches the vicinity of the optimal solution with a large learning rate, and then it further optimizes with a small learning rate, thereby improving training stability and stylization effect.

[0061] In the process of image-text multimodal optimization of the stylized neural model, the block style loss is used to constrain the model training process. This loss first randomly divides the image into blocks and then uses a random enhancement method (4 in total) to enhance each block. Then, cosine similarity is calculated in the cross-modal domain, which makes the stylization process more stable and improves the stylization effect of the model.

[0062] The model of this invention comprises a stylization network based on the StyleGAN2 model, a cross-modal multi-level mapping network, a block-based style loss, and a progressive training method for portrait neural painting. This method first performs high-efficiency multimodal optimization based on a pre-trained StyleGAN2 model to obtain a stylization network. This stage can initially generate high-quality portrait neural paintings that conform to the text style description. Then, using the cross-modal multi-level mapping network, the implicit coding corresponding to the input image is optimized using text features, further improving the neural painting results.

[0063] In S1 of this invention, the style model adopts a StyleGAN2 structure, which consists of nine similar sub-modules connected in series. Based on the implicit encoding of the input, it generates the corresponding image. Each sub-module has three inputs: the output from the connected previous layer sub-module. i Image hidden coding components l j,j+1 In addition to random noise B, the Mod and Demod modules combine the three types of information to generate the module result. The process of each sub-module can be represented as follows:

[0064]

[0065] o′ i+1 =Conv(MOM(w i+1,1 ,l j ),p i )+B+b i1

[0066] o i+1 =Conv(MDM(w i+1,2 ,l j+1 ),Upsample(o′ i+1 ))+B+b i2

[0067] Where MDM represents the calculation of the Mod and Demod modules, s j It is related to the hidden coding component l j The relevant scaling factor, ω, represents the weight of the convolutional layer Conv; Conv represents a convolutional layer with a kernel size of 3×3, and Upsample is a 2x upsampling. The first submodule does not have an upsampling layer, but instead uses a fixed constant, and its output resolution is 4×4. The resolution doubles with each subsequent submodule until a high-resolution result of 1024×1024 is obtained. The image latent code l is obtained by the image encoder (e4e) and is an 18×512 tensor, therefore each component l j It is a 1×512 tensor.

[0068] The global style loss and block style loss mentioned in step 1 are obtained by calculating cosine similarity using the cross-modal model CLIP to encourage the image to find the optimal solution for neural painting along the direction of text variation. For each input image I in , for which the initial text T is assumed ori ="a photo", the output of the style model is I out The corresponding input style text is T. tgt Text Encoder E T () and image encoder E I The global style loss can be represented as follows: glob :

[0069] ΔT=E T (T tgt )-E T (T ori )

[0070] ΔI=E I (I out )-E I (I in )

[0071]

[0072] The block loss is obtained by calculating the average value of the loss after randomly dividing the image into multiple blocks, based on the method described above.

[0073] The cross-modal multilevel mapping network in S2 consists of two modules: the text mapping module M. T and image mapping module M I The two modules have the same structure but process different input features: text features f and image implicit coding l. The acquisition of text features can be represented as follows:

[0074] f = concat(E) T (T tgt ),18)

[0075] Here, `concat(,18)` means copying the input tensor 18 times to increase its dimensionality, thus obtaining the same size as the image implicit code. Each module has 3 mappers. This is used to process feature information at three different levels: low, medium, and high. The low-level and medium-level mappers each process the input from a feature tensor of size (4, 512), while the high-level mapper processes the remaining feature tensor input of size (10, 512). The three mappers map the input to output tensors of the same size and concatenate them sequentially to obtain the mapped tensors f′, I′. The process of obtaining the residual based on the two mapped tensors to optimize the original hidden code is as follows:

[0076] I opt =I+αconv(concat(f′,I′))

[0077] Where conv represents a convolutional layer, α represents the summation weights of the residuals, concat represents tensor concatenation, and the optimized implicit code I... opt The final portrait neural painting result can be obtained by generating the style model in step 1.

[0078] S2's progressive training method is a strategy to improve training stability and efficiency. In the initial stage of training the cross-modal multi-level mapping network, a higher learning rate is used to allow the mapping network to quickly reach the vicinity of the optimal solution, and then a lower learning rate is used to more accurately approximate the optimal solution.

[0079] S2's block-based style augmentation loss is a method to improve model training stability. Using the original block-based style loss easily generates image results with low relevance to the text. The block-based style augmentation loss randomly divides the output image into blocks and then performs one of four random augmentations: random coloring, random translation, random resizing, and random cropping, before calculating the style loss according to the block-based style loss formula. This approach requires attacking four different augmented data simultaneously, improving training stability.

[0080] This invention proposes a text-guided portrait neural painting model based on implicit coding. This model can generate high-quality paintings that conform to the style description of the text for any portrait, and it is highly efficient, requiring no dataset for training. The multi-level mapping network proposed in this invention effectively improves the neural painting effect and training stability by incorporating text features into the input and using block-based style loss. The proposed progressive training strategy effectively improves training efficiency. In comparison with four existing text-guided neural painting methods, the method proposed in this invention has significant advantages in various qualitative and quantitative indicators.

[0081] Advantages of the Neural Painting Results: The method of this invention can more accurately capture style information in text and fuse it with the input content image to generate high-quality neural painting results. In visualization comparisons, given the same text as input, this invention can generate neural paintings that more closely resemble the style described by the text, without generating artifacts that disrupt the original structure of the image. In qualitative metric comparisons, this invention achieves significant advantages in two important metrics: SSIM and PSNR. Furthermore, the method of this invention performs rapid two-stage optimization on a single image to obtain high-quality results, which is more efficient than existing methods that require training with datasets.

[0082] Advantages of cross-modal multi-level mapping networks: Traditional neural painting methods that use images as style references only extract style information from image features, resulting in limited flexibility in describing style. While text-guided neural painting methods use text for more flexible style description, they are limited to using text features to calculate loss and train the model. The method of this invention uses text features to calculate residuals to optimize the image latent coding, effectively improving the performance of neural painting. Experiments show that by using text features to directly optimize the latent coding through the progressive training method of this invention, the stylization effect can be significantly improved.

[0083] Advantages of Block-Based Style Loss: To improve the stability of the training process and obtain higher quality results, this invention uses block-based style loss as one of the training losses in step 2. Ablation experiments have verified that this method has stronger anti-interference capabilities compared to simple block-based style loss and can generate results that better match the text description.

[0084] The actual experiment will be conducted below:

[0085] The style model in step 1 of this invention comprises nine similar sub-modules, each accepting three outputs: the result from the concatenated previous layer (the initial layer is a fixed constant), the image's implicit coding components, and random noise. Each sub-module contains a Mod, a Demod, and a convolutional layer. The final output of this model is a high-resolution neural painting image with a resolution of 1024×1024.

[0086] In the actual experiment, the steps performed were as follows:

[0087] S100: First, the input image is converted into the implicit encoding input required by the style model through an implicit encoder e4e. The different dimensional components of the implicit encoding are used as the inputs of different sub-modules of the style model.

[0088] S110: The style model adopts the StyleGAN2 structure, which contains 9 similar sub-modules. Each sub-module is based on the result of the previous low-resolution module, and the implicitly encoded feature information is incorporated to obtain a higher resolution result. At the same time, random noise is added to generate finer details.

[0089] S120: Calculate the content loss between the image result of the obtained style model and the input image to encourage content preservation; calculate the global and block style losses between the image and the input style text description to encourage the result to better match the desired style; finally, calculate the total variational loss to enhance image smoothness.

[0090] See Figure 1The multi-level mapping network in step 2 of this invention consists of two parts: a text mapping module and an image mapping module. This method introduces text features into the network to optimize image implicit coding, further improving the neural painting effect.

[0091] S130: The style text description and input image are processed by the text encoder E. T () and image encoder E I The text features f and the image latent code l are encoded using (), and the text features are then copied and upgraded.

[0092] S140: A three-level mapping structure consisting of a text mapping module and an image mapping module processes the upgraded text features f and image implicit codes l from different dimensions. The mapping results are concatenated and passed through a 3×3 convolutional layer. The residual is then summed with the original implicit code to obtain the optimized implicit code l. opt .

[0093] S150: Generate l using an optimized style model opt The corresponding implicit coding optimization results are then used to calculate a block-enhanced style loss with the style text description to guide the training of the multi-level mapping network. This loss first randomly blocks the output image, then selects one of random coloring, random translation, random resizing, or random cropping for enhancement, and finally calculates the cosine similarity between the style description text and the block-enhanced image across the modal domain. After training, the multi-level mapping network yields the final optimized result.

[0094] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A text-guided portrait neural painting method based on implicit coding, characterized in that: The method comprises the following steps: S1. Obtain a face image and a corresponding style description text, input the face image into the style synthesis network of StyleGAN2, pass through the encoder of the style synthesis network to obtain a latent space code, and input the latent space code into each serially connected submodule of the style synthesis network. The input of each submodule is the latent space code, the output of the previous submodule and random noise. The Mod module and Demod module inside each submodule mix the input of the submodule together to generate the output of the submodule. The global style loss, block style loss, content loss and total variation loss are calculated based on the output of the last submodule and the style description text, and the style synthesis network is trained to obtain a style synthesis model. S2. Input the face image and the corresponding style description text into the cross-modal multi-level mapping network, and pass through the text encoder and image encoder of the cross-modal multi-level mapping network respectively to obtain the upgraded text features and image latent codes. The three-level mapping structure of the cross-modal multi-level mapping network processes the upgraded text features and image latent codes of different dimensions respectively, and concatenates the mapping results through a convolutional layer to obtain the residual, which is then added to the image latent code to obtain the optimized latent code. The optimized latent code is input into the style synthesis model, and the block enhancement style loss, content loss, style loss and total variation loss are calculated based on the output image of the style synthesis model. The cross-modal multi-level mapping network is trained to obtain the cross-modal multi-level mapping model. S3. Obtain the actual image and the actual style description text, input the actual image and the actual style description text into the cross-modal multi-level mapping model and the style synthesis model, and obtain the actual style painting result.

2. The text-guided portrait neural painting method based on implicit coding according to claim 1, characterized in that: The specific process of the Mod module and the Demod module mixing the input of the submodule together to generate the output of the submodule is: o′ i+1 =Conv(MDM(ω i+1,1 ,l j ),o i )+B+b i1 Oh i+1 =Conv(MDM(ω i+1,2 ,l j+1 ),Upsample(o i ′ +1 ))+B+b i2 Among them, MDM represents the calculation of Mod module and Demod module, ω′ represents the weight of the convolution layer Conv after scaling adjustment, and s j Representation and latent space encoding component l j The relevant scaling factor, ω is the weight of the convolution layer Conv; Conv represents a convolution layer with a convolution kernel size of 3×3, ∈ represents a very small amount to avoid zero division errors, o′ i+1 represents the intermediate output feature processed by the first convolutional layer of the submodule, ω i+1,1 represents the weight of the first convolutional layer of the current submodule, o i represents the output of the previous submodule, B represents random noise, and b i1 , b i2 represents two different learnable bias values ​​of the i-th submodule, ω i+1,2 represents the weight of the second convolutional layer of the current submodule, l j , l j+1 represents the hidden coding components of the jth and j+1th layers, ω i+1,2 represents the weight of the second convolutional layer of the current submodule, o i+1 Represents the output of the submodule of the current layer, and Upsample is 2 times upsampling.

3. The text-guided portrait neural painting method based on implicit coding according to claim 1, characterized in that: The global style loss is: ΔT=E T (T tgt )-E T (T ori ) ΔI=E I (I out )-HAVE BEEN I (I in ) Among them, T tgt is the style description text corresponding to the output of the style synthesis network, T ori is the style description text corresponding to the face image, E T (), E I () is the text encoder and image encoder, I out is the output of the style synthesis network, I in is a face image, ΔT represents the text error, ΔI represents the image error, and l glob represents the global style loss.

4. The text-guided portrait neural painting method based on implicit coding according to claim 3, characterized in that: The calculation process of the block style loss is: the output image of the style synthesis network is randomly divided into blocks multiple times, and for each block of the image, the cosine similarity in the cross-modal domain is calculated to obtain the block style loss.

5. The text-guided portrait neural painting method based on implicit coding according to claim 1, characterized in that: The text features of the dimension increase are: f=concat(E T (T tgt ),18) Among them, f represents the text feature after dimension increase, and concat(,18) means copying the input tensor 18 times to increase the dimension.

6. The text-guided portrait neural painting method based on implicit coding according to claim 5, characterized in that: The optimized implicit code is: I opt =I+αconv(concat(f′,I′)) Among them, I opt Represents optimized hidden coding, α represents the sum weight of the residual, conv represents convolutional layer, concat represents tensor cascade, f′ represents image tensor, and I′ represents text tensor.

7. The text-guided portrait neural painting method based on implicit coding according to claim 6, characterized in that: The calculation process of the image tensor is: The image hidden code is input into the cross-modal multi-level mapping network, which includes a text mapping module M T and image mapping module M I The mapping module includes three mappers, the image hidden code input image mapping module M I , image mapping module M I The outputs of the three mappers are concatenated to obtain the image tensor; The calculation process of the text tensor is: Input the dimension-increased text feature f into the text mapping module M of the cross-modal multi-level mapping network T , text mapping module M T The outputs of the three mappers are concatenated to get the text tensor.

8. The text-guided portrait neural painting method based on implicit coding according to claim 1, characterized in that: The block-enhanced style loss is: Among them, O oit represents the output of the style synthesis network, Crop represents random blocking, and Aug represents the enhancement selected from random coloring, random translation, random resizing, and random cropping. I represents the block image after random block and random enhancement of the output of the style synthesis network. in is the face image, ΔT represents the text error, ΔI i represents the block image error, represents the enhanced style loss of the i-th block, R(x,τ) represents a threshold function that outputs 0 when the input x is greater than τ and outputs x otherwise, l patch represents the block-enhanced style loss.

9. The text-guided portrait neural painting method based on implicit coding according to claim 8, characterized in that: The content loss and style loss are: The L2 distance between the output image of the style synthesis model and the input face image at different layers of the feature extraction network VGG-19.

10. The text-guided portrait neural painting method based on latent coding according to claim 1, characterized in that: The encoder of the style synthesis network is an e4e encoder.

Images