A high-quality portrait cartoon automatic generation method based on text control

Abstract

The application discloses a high-quality portrait cartoon automatic generation method based on text control. For a given real face photo, the algorithm can automatically create a lifelike and high-quality portrait cartoon, and can edit the generated portrait cartoon according to simple text information, so as to synthesize a portrait cartoon work conforming to the text description without affecting other features of the cartoon image. The main contributions of the application include two aspects. One is to propose a new high-quality portrait cartoon automatic generation method, which performs geometric exaggeration on the intermediate feature layer of StyleGAN, and proposes to use the intermediate feature map-based adversarial loss to supervise the deformation module. The other is to propose an attention mask-based portrait cartoon editing method, which improves the feature extraction capability of the CLIP model for cartoon images through model fine-tuning, introduces a mask-based attention mechanism to encourage face editing to focus on the text area, and proposes a mask initialization strategy to realize decoupled and controllable portrait cartoon editing based on text guidance.

Description

Technical Field

[0001] This invention belongs to the field of computer applications, specifically relating to a method for automatically generating high-quality portrait cartoons based on text control. Background Technology

[0002] Portrait caricature is a visually compelling art form that highlights a person's physical features to showcase their individual characteristics, thereby expressing their personality and reflecting the artist's attitude and emotions. Portrait caricatures are diverse in style, employing various techniques and styles such as oil painting, watercolor, and sketching to achieve rich artistic effects. Therefore, to create a portrait caricature, the artist needs to capture the subject's prominent facial features, exaggerate them, and imbue them with a cartoonish artistic style. Consequently, portrait caricature has a high creative threshold; it is difficult for ordinary people without professional training to create one independently. Furthermore, even skilled cartoonists require several hours or even days to complete a single portrait caricature. The development of the new media era has promoted a new wave of widespread dissemination and application of portrait caricature. Automated portrait caricature creation has reduced the learning and production costs of caricature creation, allowing ordinary people without artistic backgrounds to participate in caricature creation, thus possessing broad market value and application prospects.

[0003] Early cartoon generation methods mostly relied on user interaction or predefined deformation rules. These methods either depended on interactive deformation techniques or performed geometric deformations of faces according to specific or learned rules. Such methods were limited to a single exaggerated form and simple texture style, making it difficult to produce attractive portrait cartoons. In recent years, with the rapid development of deep neural networks in computer vision, image-to-image translation models based on generative adversarial networks (GANs) have matured and achieved remarkable results. Models based on GANs, such as CycleGAN, can be trained even with incomplete pairing of source and destination domain samples and complete cross-domain image translation tasks. However, due to the significant differences in facial structure between the photo domain and the cartoon domain, directly using these unpaired image translation algorithms to generate portrait cartoons of faces does not produce visually satisfactory results. Recent methods for automatically generating portrait cartoons break down the task into two sub-tasks: first, performing texture style conversion from a face photograph to a cartoon image; and second, exaggerating the photograph geometrically to highlight the features of the person. Specialized network structures are introduced to exaggerate and deform the face, such as CariGANs, WarpGAN, and StyleCariGAN. However, CariGANs handle texture transformation and geometric transformation models separately, making end-to-end training impossible. Furthermore, it requires pre-annotating facial landmarks in the training set of photos and comics, a very difficult task. WarpGAN learns control points on images to exaggerate facial features and performs texture style conversion based on an encoder-decoder structure. Limited by the generation effects of traditional GAN ​​models, it struggles to meticulously depict facial details, resulting in less realistic and vivid outputs. StyleCariGAN was the first to propose adding a deformation module to StyleGAN to complete comic generation tasks. Thanks to StyleGAN's powerful generation capabilities, it can produce visually high-quality comics. However, StyleCariGAN constructs two cross-domain transformation models: photo-to-comic and comic-to-photo. This allows the cycle consistency loss to be used as a supervisory signal to train the deformation network, thus introducing more network parameters. Under this constraint, the deformation module produces very little deformation, resulting in an insignificant deformation effect.

[0004] However, while these automatic portrait caricature generation methods can achieve grotesque and humorous visual effects, their capture and exaggeration of key features rely entirely on the autonomous extraction of deep learning models. Limited by the model's loss function and training methods, the "key features" they focus on often differ from the subject's inherent characteristics, making it difficult to achieve the desired goals. Furthermore, artistic creation based on deep learning models relies solely on historical knowledge and experience; the AI ​​model itself lacks true consciousness, making it difficult to satisfy the emotional, attitudinal, and sentimental characteristics of portrait caricatures. Therefore, providing people's creative intentions in the form of natural language descriptions to deep learning models, enabling them to generate artworks that meet the semantic requirements of natural language, is an effective and feasible approach. In this text-controlled portrait model, humans only need to fully utilize their imagination, simply express their creative needs, and specify the direction and degree of exaggeration. The deep learning model can then generate the portrait caricature based on human instructions, simplifying the originally complex and tedious manual creation process. While a series of text-guided image generation and editing methods already exist, their effectiveness in portrait caricature generation remains unexplored. CLIP is a multimodal pre-trained model that can process images and text simultaneously. Models such as StyleCLIP and StyleGAN-NANA use CLIP to complete image editing and cross-domain image conversion tasks. However, due to the significant differences between the real image domain and the portrait cartoon domain, there is still much room for improvement in their performance on portrait cartoon editing. Summary of the Invention

[0005] Purpose of the invention: This invention addresses the task of automatically generating controllable portrait cartoons by proposing a text-controlled method for generating high-quality portrait cartoons automatically.

[0006] Technical solution: A text-based method for automatically generating portrait cartoons, comprising at least the following steps:

[0007] Step 1: Cropping and aligning the face photos and portrait cartoons in the dataset based on facial landmarks;

[0008] Step 2: Train a photo StyleGAN model using a large number of face photos, and then fine-tune the photo StyleGAN model using cartoon images to obtain a cartoon StyleGAN model.

[0009] Step 3: Swap the layers of the photo StyleGAN model and the comic StyleGAN model to obtain a hybrid StyleGAN model, where the low-resolution layers come from the photo StyleGAN model and the high-resolution layers come from the comic StyleGAN model.

[0010] Step 4: For the original feature map obtained from the low-resolution layer, use a multilayer perceptron to predict N control points and the corresponding two-dimensional coordinates of the control points, and perform a warp operation on the original feature map to obtain the deformed feature map.

[0011] Step 5: For the input face photo, use GANinversion to obtain the hidden encoding of the real photo in the w+ space;

[0012] Step 6: Use the sampled hidden code and the deformed feature map as input to the comic styleGAN model, train the hybrid styleGAN model, and save the model parameters after the model converges.

[0013] Step 7: During training, further constraints are applied using feature map-based adversarial loss.

[0014] Step 8: For the input control text, perform comic editing using CLIP loss, learn through iterative optimization, and guide the editing to focus on the specified area of ​​the text based on the attention mask;

[0015] Step 9: Perform random cropping and enhancement on the output comic image, calculate the CLIP distance between each cropped block of the comic image and the control text, and obtain the initial attention mask;

[0016] Step 10: Fuse the results before and after editing based on the initial attention mask, and update the implicit coding and attention mask by combining CLIP Loss and other losses.

[0017] In a further embodiment, the specific steps of step four are as follows:

[0018] In the hierarchical generation of the hybrid StyleGAN model, the first four layers are geometric structure information layers that control the face. Based on the geometric structure information layers, the geometric features of the face are deformed and exaggerated to obtain intermediate feature maps.

[0019] The source coordinates of N control points and the position offset of each control point are predicted on the intermediate feature map using a multilayer perceptron, and the target coordinates of the control points are obtained. Based on the source coordinates and target coordinates of the control points, the source coordinates of each pixel in the feature map are calculated using thin plate spline interpolation, and the deformed feature map is obtained using bilinear interpolation.

[0020] In a further embodiment, step seven employs the following form of adversarial loss based on feature maps for further constraints:

[0021]

[0022]

[0023] Where w+ represents the hidden code sampled from the w+ space, and c represents the deformation coefficient sampled from the uniform distribution. This represents the first four layers of the ImageStyleGAN. This represents the first four levels of the StyleGAN comic, f c The feature map represents the corresponding level generated by StyleGAN, Warp(·) represents the deformation module, and D feature The logical value representing the feature map discriminator is used to classify the input feature map as... The probability of outputting the feature map;

[0024] use Update generator G, using For discriminator D feature Update.

[0025] In a further embodiment, step eight uses an iterative optimization approach to learn an attention mask, and the specific steps for guiding editing to focus on a specified region of text based on the attention mask are as follows:

[0026] The attention mask is multiplied by the corresponding position of each channel of the image to create the regions that need attention, plus the regions that do not need attention:

[0027]

[0028] Where m represents the current attention mask, and w represents the initial hidden mask. edited G(w) represents the implicit encoding after editing, and G(w) represents the generator output obtained by taking the implicit encoding w as input. edited ) represents the generator implicitly encoded with w edited The generated result, taken as input, shows that ⊙ represents multiplication at corresponding positions. The output image after attention masking;

[0029] During testing, the optimized attention map is further processed, retaining only the pixel positions with probability values ​​greater than the threshold τ, and setting the rest to 0.

[0030] In a further embodiment, a normalized loss is used to adjust the attention mask. Update:

[0031]

[0032] Where H and W are the width and height of the attention mask, respectively, and m ij The pixel value of the attention mask at position (i,j).

[0033] In a further embodiment, the attention mask is reasonably initialized, a sufficient number of fixed-size patches are randomly cropped from the initial portrait cartoon image, and image enhancement is performed on them. Then, their CLIP loss at the patch level is calculated:

[0034] ΔT=E T (t sty )-E T (t src )

[0035]

[0036]

[0037]

[0038] Among them, E T E represents the CLIP model text encoder. I The image encoder representing the CLIP model, t src t represents the text description corresponding to the initial comic. sty This represents the input text description, where ΔT represents the distance between the input text and the original text in the CLIP feature space, and I... c Represents the initial comic image. Let represent the i-th patch cropped from the comic image, aug(·) denotes the image enhancement operation, and ΔI represent the distance between each patch and the original image in the CLIP feature space. τ represents the CLIP loss between the i-th patch and the input text, τ represents the specified decision threshold, and R(·,·) represents the threshold function, which selects regions that are more relevant to the input text information as the initial value of the attention mask.

[0039] Beneficial Effects: Compared with existing technologies, the beneficial effects of this invention are as follows: 1. It proposes an automatic portrait cartoon generation method based on feature-based geometric deformation. Control points are learned in the intermediate feature layer for geometric deformation. Compared with traditional GAN-based automatic portrait cartoon generation methods, the generated cartoons are more vivid and realistic. Compared with existing automatic portrait cartoon generation methods using StyleGAN as the generation baseline, the exaggeration effect is significant. The network structure is simple, the number of parameters is small, and the model is easy to train. 2. It proposes a portrait cartoon editing method based on attention masks. The CLIP model, specifically designed for fine-tuning cartoon images, controls the cartoon editing process. Combined with attention masks, it achieves decoupled editing of text-related regions, avoiding influence on other irrelevant regions. Compared with existing image editing methods, it achieves better decoupled editing results. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the portrait cartoon generation algorithm model based on feature graphic transformation of the present invention;

[0041] Figure 2 This is a schematic diagram of the geometric deformation module structure in this invention;

[0042] Figure 3 This is a schematic diagram of the portrait cartoon editing method based on attention mask in this invention;

[0043] Figure 4 This is a schematic diagram of attention mask initialization in this invention. Detailed Implementation

[0044] The present invention will now be described in further detail with reference to the accompanying drawings.

[0045] This invention designs a text-controlled automatic portrait caricature generation method, which mainly includes two modules: a feature-map-based automatic portrait caricature generation method and a CLIP-based fine-tuning portrait caricature editing method. The feature-map-based automatic portrait caricature generation method utilizes the hierarchical generation characteristics of StyleGAN, extracting control points from the intermediate feature maps that primarily control the geometric features of the face, and predicting their offsets to exaggerate facial geometric features. During training, an adversarial loss based on feature maps is designed to further constrain the deformation module. The attention-mask-based portrait caricature editing method, after fine-tuning, obtains a CLIP model capable of extracting caricature features and introduces an attention mask to ensure that image editing only occurs in areas of user interest, guaranteeing that other image features remain unchanged. A mask initialization method beneficial for attention mask optimization is also introduced. In this way, for a given photograph, this invention can generate diverse portrait caricature images with exaggerated effects and perform decoupled and controllable image editing of the initially generated caricature based on the features described in the text.

[0046] 1. Portrait Cartoon Generation Based on Feature Graphic Variation

[0047] For pre-trained photo StyleGAN and cartoon StyleGAN models, directly swapping their hierarchical generation layers to create a hybrid StyleGAN model can generate images that both preserve facial structure and can also perform geometric exaggeration directly on the image. However, this can easily cause geometric artifacts. Exaggerating in the intermediate feature layers of the generator network avoids this problem. The detailed structure of the geometric deformation module is as follows... Figure 3As shown, considering the hierarchical generation characteristics of the StyleGAN model, since the facial geometric features of the generated image are mainly controlled by the coarse-grained layers of StyleGAN (i.e., the first four layers of the photo StyleGAN), the feature map obtained from it contains rich geometric shape information of the original photo. This makes it easy to learn the prominent features of the person and exaggerate them. Using a multi-layer fully connected network, the source and destination coordinates of sparse control points are learned from the original feature map. Utilizing the positional mapping relationship between these sets, the sampling positions of the intermediate feature map after deformation are obtained on the feature map, and geometric transformation is completed using bilinear interpolation. The geometrically transformed feature map is then used as input to the feature maps of subsequent layers of StyleGAN, thus enabling the layer-mixed StyleGAN model to possess the ability to exaggerate geometrically, enhancing the expressiveness of the output portrait cartoon.

[0048] Furthermore, the deformation module is further constrained by the adversarial loss based on feature maps, so that the deformed feature maps are still in the feature space of StyleGAN. The image adversarial loss, person identity loss, and face attribute loss are used as supervision information to participate in the training process of the deformation module.

[0049]

[0050]

[0051] Where w+ represents the hidden code sampled from the W+ space, and c represents the deformation coefficient sampled from the uniform distribution. This represents the first four layers of the ImageStyleGAN. This represents the first four levels of the StyleGAN comic, f c The feature map represents the corresponding level generated by StyleGAN, Warp(·) represents the deformation module, and D feature The logical value representing the feature map discriminator is used to classify the input feature map as... The probability of outputting the feature map;

[0052] use Update generator G, using For discriminator D feature Update.

[0053] 2. Attention Mask-Based Portrait Cartoon Editing

[0054] A common approach to image editing using the CLIP model is to use a CLIP text encoder and a CLIP image encoder to calculate text and image features respectively, and then calculate the cosine distance between them. The implicit coding of the input generator is then optimized based on CLIP backpropagation until a cartoon image matching the text description is obtained. Unlike ordinary image editing tasks, due to the significant gap between the cartoon image domain and the photographic domain, the CLIP model has limited feature extraction capabilities for cartoon images. Directly using the above method for portrait cartoon editing does not achieve satisfactory results. Image editing based on CLIP also faces the challenge of untangling editing, i.e., editing the target attributes required by the text while minimizing the impact on other attributes in the image. In some cases, untangling occurs naturally without explicit enforcement, but in cartoon editing, untangling editing is more challenging. Therefore, this invention, on the one hand, obtains a cartoon CLIP model with strong cartoon feature extraction capabilities through fine-tuning, and uses this cartoon CLIP model for portrait cartoon editing; on the other hand, it introduces a cartoon attention mask to focus the editing operation on the target area in the text description, avoiding impact on the image features of other areas.

[0055] To address the aforementioned issues, this invention, on one hand, fine-tunes a comic CLIP model with strong comic feature extraction capabilities, and then utilizes this model for portrait comic editing. Comic-text pairs are constructed using the WebCariA dataset, and appropriate descriptive text labels are added to each portrait comic in the dataset. Since each character has similar facial attributes, comic images with the same identity share the same text labels. N character identities are randomly selected from all characters, and one comic image is randomly selected from each identity. For these N comic-text pairs, the CLIP model is fine-tuned using cross-entropy loss, thereby improving the CLIP model's feature extraction capability on comic images.

[0056] To achieve decoupled attribute editing, an attention mask is used to focus editing on the regions of interest, optimizing the attention mask while performing implicit encoding. To edit only text-related target regions in the image, an attention mask map is used. This is a single-channel map of the same size as the output image, where each pixel represents a probability value between 0 and 1. The attention mask is multiplied by the corresponding position in each channel of the image to create the regions of interest, plus the regions of no interest.

[0057]

[0058] Where m represents the current attention mask, and w represents the initial hidden mask. edited G(w) represents the implicit encoding after editing, and G(w) represents the generator output obtained by taking the implicit encoding w as input.edited ) represents the generator implicitly encoded with w edited The generated result, taken as input, shows that ⊙ represents multiplication at corresponding positions. This represents the output image after the attention masking is applied. During testing, the optimized attention map is further processed, retaining only the pixel positions with a probability value greater than the threshold τ, and setting the rest to 0.

[0059] To encourage the mask to focus its attention on more compact regions relevant to the text description, a normalized loss is used to update the attention mask:

[0060]

[0061] Where H and W are the width and height of the attention mask, respectively, and m ij The pixel value of the attention mask at position (i,j).

[0062] To ensure regional coherence in the generated attention map mask, a total variational loss function is used:

[0063]

[0064] Furthermore, to facilitate the learning of the attention mask, it needs to be properly initialized. Specifically, a sufficient number of fixed-size patches are randomly cropped from the initial portrait cartoon image, and image augmentation is performed on them. Then, their CLIP loss at the patch level is calculated.

[0065] ΔT=E T (t sty )-E T (t src )

[0066]

[0067]

[0068]

[0069] Among them, E T E represents the CLIP model text encoder. I The image encoder representing the CLIP model, t src t represents the text description corresponding to the initial comic. sty This represents the input text description, where ΔT represents the distance between the input text and the original text in the CLIP feature space, and I... c Represents the initial comic image. Let represent the i-th patch cropped from the comic image, aug(·) denotes the image enhancement operation, and ΔI represent the distance between each patch and the original image in the CLIP feature space. τ represents the CLIP loss between the i-th patch and the input text, τ represents the specified decision threshold, and R(·,·) represents the threshold function, which selects regions that are more relevant to the input text information as the initial value of the attention mask.

[0070] Example 1

[0071] First, such as Figure 1 and Figure 2 As shown, this invention provides a method for automatically generating portrait cartoons based on feature-based graphic transformations. The main steps in the model training phase are as follows:

[0072] (1) Align the photograph and the portrait cartoon based on the pre-annotated facial key points so that the positions of important visual features are shared between the photograph and the cartoon images;

[0073] (2) Train the photo StyleGAN model using a large-scale face photo dataset, and fine-tune it using a set of cartoon images to obtain the cartoon StyleGAN model;

[0074] (3) By swapping the parameters of these two StyleGAN models, a layer hybrid model was created, whose coarse-grained layer parameters (4) 2 ~32 2 (From photo StyleGAN, fine-grained layer parameters (32)) 2 ~256 2 (This is from the comic StyleGAN;)

[0075] (4) For the original feature map obtained from the low-resolution layer, use a multilayer perceptron to predict N control points and the two-dimensional coordinates of the corresponding offset of the control points, and perform a warp operation on the original feature map to obtain the deformed feature map.

[0076] (5) For the input face photo, use GAN inversion to obtain the real photo. Hidden coding in space;

[0077] (6) The sampled style code and the deformed feature map are used together as input to the comic styleGAN to train the deformation module;

[0078] (7) An adversarial loss based on feature maps is proposed, which combines adversarial loss based on cartoon images, identity loss and attribute consistency loss to constrain the geometric deformation results generated by the deformation module.

[0079] (8) During training, keep the parameters of the StyleGAN model with layer mixing unchanged, and only update the parameters of the multilayer perceptron in the deformation module.

[0080] During the testing phase, the implicit encoding of the input photo is obtained by utilizing GANInversion-related work. The style encoding can be specified directly or obtained through random sampling. These are used as input to HybridStyleGAN, and the model automatically completes the processing of geometric exaggeration deformation and texture stylization, outputting the generated portrait cartoon.

[0081] Secondly, such as Figure 3 and Figure 4 As shown, based on the provided automatic portrait cartoon generation method, this invention provides a portrait cartoon editing method based on attention masks, enabling users to control the generated portrait cartoon results using simple text information. The portrait cartoon image is edited using an iterative optimization approach:

[0082] (1) Construct a pair of comic-text datasets and fine-tune the CLIP model using cross-entropy loss to enhance the CLIP model's ability to extract comic image features;

[0083] (2) Based on the hidden encoding of the input image, the initial portrait cartoon is generated through the portrait cartoon generation model;

[0084] (3) Save the initial hidden code w, and at the same time save the edited hidden code w edited Initialize to w;

[0085] (4) Randomly crop and enhance the initial comic image, calculate the CLIP distance between each image block and the input text, and determine its size relative to the threshold. If it is less than the threshold, the relevance is directly set to 0, thus obtaining the initial attention mask m0.

[0086] (5) The results before and after editing are fused based on the initial attention mask. The attention mask m and the hidden code w are then fused based on CLIP loss, attention mask normalization loss, and total variational loss function. edited Update;

[0087] (6) Repeat step (5) until a portrait cartoon result that satisfies the text description is obtained.

Claims

1. A method for automatically generating portrait cartoons based on text control, characterized in that, At least the following steps are included: Step 1: Cropping and aligning the face photos and portrait cartoons in the dataset based on facial landmarks; Step 2: Train a photo StyleGAN model using a large number of face photos, and then fine-tune the photo StyleGAN model using cartoon images to obtain a cartoon StyleGAN model. Step 3: Swap the layers of the photo StyleGAN model and the comic StyleGAN model to obtain a hybrid StyleGAN model, where the low-resolution layers come from the photo StyleGAN model and the high-resolution layers come from the comic StyleGAN model. Step 4: For the original feature map obtained from the low-resolution layer, use a multilayer perceptron to predict N control points and the corresponding two-dimensional coordinates of the control points, and perform a warp operation on the original feature map to obtain the deformed feature map. Step 5: For the input face photo, use GAN inversion to obtain the real photo. Hidden coding in space; Step 6: Use the sampled hidden code and the deformed feature map as input to the comic styleGAN model, train the hybrid styleGAN model, and save the model parameters after the model converges. Step 7: During training, further constraints are applied using feature map-based adversarial loss. Step 8: For the input control text, perform comic editing using CLIP loss, learn through iterative optimization, and guide the editing to focus on the specified area of ​​the text based on the attention mask; Step 9: Perform random cropping and enhancement on the output comic image, calculate the CLIP distance between each cropped block of the comic image and the control text, and obtain the initial attention mask; Step 10: Fuse the results before and after editing based on the initial attention mask, and update the implicit coding and attention mask by combining CLIP Loss and other losses.

2. The method for automatically generating portrait cartoons based on text control according to claim 1, characterized in that, The specific steps of step four are as follows: In the hierarchical generation of the hybrid StyleGAN model, the first four layers are geometric structure information layers that control the face. Based on the geometric structure information layers, the geometric features of the face are deformed and exaggerated to obtain intermediate feature maps. The source coordinates of N control points and the position offset of each control point are predicted on the intermediate feature map using a multilayer perceptron, and the target coordinates of the control points are obtained. Based on the source coordinates and target coordinates of the control points, the source coordinates of each pixel in the feature map are calculated using thin plate spline interpolation, and the deformed feature map is obtained using bilinear interpolation.

3. The method for automatically generating portrait cartoons based on text control according to claim 1, characterized in that, In step seven, the following form of feature map-based adversarial loss is used for further constraints: Where w+ represents from The implicit coding of samples in space, where c represents the deformation coefficient sampled from a uniform distribution. This represents the first four layers of the ImageStyleGAN. This represents the first four levels of the StyleGAN comic, f c The feature map represents the corresponding level generated by StyleGAN, Warp(·) represents the deformation module, and D feature The logical value representing the feature map discriminator is used to classify the input feature map as... The probability of outputting the feature map; use Update generator G, using For discriminator D feature Update.

4. The method for automatically generating portrait cartoons based on text control according to claim 1, characterized in that, In step eight, an attention mask is learned using an iterative optimization method. The specific steps for guiding editing to focus on a specified area of ​​the text based on the attention mask are as follows: The attention mask is multiplied by the corresponding position of each channel of the image to create the regions that need attention, plus the regions that do not need attention: Where m represents the current attention mask, and w represents the initial hidden mask. edited G(w) represents the implicit encoding after editing, and G(w) represents the generator output obtained by taking the implicit encoding w as input. edited ) represents the generator implicitly encoded with w edited The generated result, taken as input, shows that ⊙ represents multiplication at corresponding positions. The output image after attention masking; During testing, the optimized attention map is further processed, retaining only the pixel positions with probability values ​​greater than the threshold τ, and setting the rest to 0.

5. The method for automatically generating portrait cartoons based on text control according to claim 1, characterized in that, Using normalized loss for attention masking Update: Where H and W are the width and height of the attention mask, respectively, and m ij This represents the pixel value of the attention mask at position (i,j).

6. The method for automatically generating portrait cartoons based on text control according to claim 1, characterized in that, After properly initializing the attention mask, a sufficient number of fixed-size patches are randomly cropped from the initial portrait cartoon image, and image enhancement is performed on them. Then, their CLIP loss at the patch level is calculated: ΔT=E T (t sty )-E T (t src ) Among them, E T E represents the CLIP model text encoder. I The image encoder representing the CLIP model, t src t represents the text description corresponding to the initial comic. sty This represents the input text description, where ΔT represents the distance between the input text and the original text in the CLIP feature space, and I... c Represents the initial comic image. Let represent the i-th patch cropped from the comic image, aug(·) denotes the image enhancement operation, and ΔI represent the distance between each patch and the original image in the CLIP feature space. τ represents the CLIP loss between the i-th patch and the input text, τ represents the specified decision threshold, and R(·,·) represents the threshold function, which selects regions that are more relevant to the input text information as the initial value of the attention mask.

Images