Figure cartoon generation method and device assisted by fine control style migration mask

Abstract

The invention provides a fine control style migration mask assisted character cartoon generation method and device, and the method comprises the steps: 1, mask generation: generating different masks for the face and background of an original character image I, and setting style intensities for different regions; step 2, face processing: using a VToonify model GvToonify, retaining face details and applying a specified animation style to carry out style migration on the face to generate a stylized face image Iface; step 3, background processing: using a diffusion model Gdiffusion to carry out stylized generation on a background region, and generating a style background Ibg; according to the method, figures and backgrounds in one picture are generated respectively, key points are generated through mark removal by means of the masks, so that various models can only exert skilled parts of themselves in one picture, splicing is carried out at last, and finally, the face and the background are fused into a complete image # imgabs0 #. Therefore, the overall composition of the picture is optimal in multiple aspects.

Description

Technical Field

[0001] The present application relates to the field of image processing and computing, and in particular to a method and device for generating character comics with the assistance of finely controlled style transfer masks. Background Art

[0002] In recent years, with the popularity of social media and short video platforms, the demand for creative editing of photos has increased, and image style transfer technology has received widespread attention in multiple fields due to its innovative applications. This technology can produce visually appealing results while maintaining the essential characteristics of the original content in style transfer. Among them, comic-style portraits are widely used in creative industries such as social media avatars, filmmaking, and entertainment advertising. With the advancement of deep learning technology, it has become possible for models to automatically generate high-quality comic-style portraits from real-life photos. Many methods have been developed for image style transfer, most of which have relied on generative adversarial networks (GANs) for a long time. In 2019, NVIDIA proposed StyleGAN, an image generation technology based on generative adversarial networks, which has the ability to generate high-quality and highly realistic images. Researchers found that StyleGAN can be well applied to comic style transfer technology, which once again promoted the research on image comicization from a new perspective. In 2021, Yang et al. proposed DualStyleGAN, a facial comic style transfer model based on StyleGAN, which can hierarchically model color and complex structural styles, so that the style of the reference sample can be accurately reproduced. In 2022, Yang et al. proposed a new fully convolutional VToonify framework, which implemented set-based and example-based portrait style transfer based on Toonify and DualStyleGAN, which is currently the best style transfer model in GAN.

[0003] The development of diffusion models provides a new research direction for image style transfer. In order to further improve the flexibility of style transfer, in 2023, Li et al. proposed DiffStyler. DiffStyler successfully copes with the difference between cross-modal input and output using a double diffusion architecture. It naturally incorporates user-provided text guidance in the gradual diffusion process, ensuring the accuracy and intuitiveness of style transfer. DiffStyler also optimizes the reverse denoising process and strengthens the retention of the original image structure information in the final result.

[0004] The success of the above existing generative models often depends on the guiding techniques they use, such as discriminators, generators, image denoising, etc. They all provide effective mechanisms to ensure the authenticity and diversity of the generated images. However, these single methods cannot guarantee the generation effect while being faithful to the original image, and these methods also have many details lost in the process of style transfer, such as facial distortion and background blur. Specifically, the VToonify model is the best in terms of faithfulness to the original image, but the effect is not satisfactory in terms of background and color richness, while the DiffStyler model is very good in terms of background and color richness and the naturalness of the overall composition, and it often deviates from the original image for its own performance, making it deviate from the original content of the original image. . Summary of the invention

[0005] In view of this, the present application provides a method and device for generating character comics with the assistance of fine-grained control of style transfer masks, so that multiple generation models can each play their own advantages and achieve the best stylization effect with global consistency.

[0006] To achieve the above objectives, the technical solutions adopted in this application are as follows:

[0007] Fine-grained control of style transfer mask-assisted character caricature generation method, including:

[0008] Step 1: Mask generation: Generate different masks for the face and background of the original person image I, and set style strength for different areas. Style strength refers to the strength of the stylized effect applied to the target image during the style transfer process.

[0009] Step 2: Face processing: Using VToonify model G vtoonify , retain facial details and apply the specified anime style to transfer the face to generate a stylized facial image I face ;

[0010] Step 3: Background processing: using diffusion model G diffusion Generate style background I by stylizing the background area bg ;

[0011] Step 4: Mask fusion: Use mask to fuse the face and background into a complete image

[0012] Furthermore, the mask can be automatically generated using FastSAM, Mask or R-CNN tools.

[0013] Furthermore, the diffusion model is a DiffStyler double diffusion architecture.

[0014] Furthermore, the mask M of the facial region face and the background area mask M bg Satisfy formula M face +M bg =1.

[0015] Furthermore, the image fused by mask in step 4 It is expressed as:

[0016]

[0017] Among them, ⊙ represents the pixel-by-pixel multiplication operation, style face is the style code of the face, s face is the style intensity of the face, prompt bg is the cue word used by the diffusion model to generate the background, s bg is the style strength of the background.

[0018] A character caricature generation device assisted by fine-grained style transfer mask control, including:

[0019] The mask generation unit is used for mask generation: generating different masks for the face and background of the original person image I, and setting style strength for different regions. The style strength refers to the strength of the stylization effect applied to the target image during the style transfer process.

[0020] Face processing unit for face processing: using VToonify model G vtoonify , retain facial details and apply the specified anime style to transfer the face to generate a stylized facial image I face ;

[0021] Background processing unit for using the diffusion model G diffusion Stylize the background area to generate a free and abstract style background I bg ;

[0022] Mask fusion unit, used to fuse the face and background into a complete image through masking

[0023] Furthermore, the mask can be automatically generated using FastSAM, Mask or R-CNN tools.

[0024] Furthermore, the diffusion model is a DiffStyler double diffusion architecture.

[0025] Furthermore, the mask M of the facial region face and the background area mask M bg Satisfy formula M face +Mbg =1.

[0026] Furthermore, the image fused by mask in the mask fusion unit It is expressed as:

[0027]

[0028] Among them, ⊙ represents the pixel-by-pixel multiplication operation, style face is the style code of the face, s face is the style intensity of the face, prompt bg is the cue word used by the diffusion model to generate the background, s bg is the style strength of the background.

[0029] Compared with the prior art, the beneficial effects of this application are:

[0030] 1. This application generates the characters and background in a picture separately, and uses masks to mark the key points of generation, so that multiple models can only play their own strengths in a picture, and then splice them at the end, so that the overall composition of the picture is optimized in multiple aspects;

[0031] 2. This application uses masks to achieve fine region division, so that the style transfer effects of the face and background can be adjusted independently, thereby improving the quality and diversity of image generation;

[0032] 3. Through the organic combination of masks, style intensity control and models, this application can provide more delicate and flexible effects when processing complex images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 Flowchart of the character caricature generation method assisted by mask for fine control of style transfer;

[0035] Figure 2 Schematic diagram of a character caricature generation method assisted by mask for fine control of style transfer;

[0036] Figure 3 Comparison chart of the effects of character caricature generation methods assisted by fine-grained control of style transfer masks;

[0037] Figure 4Structural block diagram of a character caricature generation device assisted by fine-grained control of style transfer masks. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0039] like Figure 1 and 2 As shown, the character caricature generation method assisted by fine-grained control of style transfer mask includes:

[0040] S101: Mask generation: generating different masks for the face and background of the original person image I, and setting style strength for different regions. The style strength refers to the strength of the stylized effect applied to the target image during the style transfer process.

[0041] The style strength determines the saliency of the style features in the generated image, that is, to what extent the details of the original image are retained while adding the features of the target style. The higher the style strength, the closer the generated image will be to the target style; the lower the style strength, the more likely the generated image will retain the features of the original image.

[0042] Style strength is represented by a continuous variable, a floating point number between 0 and 1.

[0043] Lower values ​​(such as 0.2): The generated images tend to retain the original details and only add a small amount of style features.

[0044] Higher values ​​(such as 0.8): The generated image is closer to the target style, and the original details are largely covered.

[0045] The specific implementation is controlled by the following technologies:

[0046] Weight adjustment: Adjust the weight ratio of style loss to content loss in the style transfer algorithm.

[0047] Content loss: retain the original image details.

[0048] Style loss: applies target style features.

[0049] In style transfer, adjusting the weight ratio of content loss and style loss can be achieved by the following formula. Let the input image be I content , the generated target image is I generated , the target style image is I style , the optimization goal is to minimize the total loss function L total , the total loss formula is:

[0050] L total =α·L content +β·L style

[0051] Among them, α is the weight of content loss, β is the weight of style loss, and L content is the content loss, i.e. retaining the structure and features of the original image, L style is the style loss, that is, applying the target style features; by adjusting the values ​​of α and β, the balance between content and style can be controlled. For example, increasing α will focus more on preserving the details of the original content, while increasing β will focus more on applying the target style.

[0052] Content loss formula:

[0053] The content loss is used to compare the feature similarity of the generated target image and the input image at a specific convolutional layer and is defined as the Euclidean distance between the two feature maps:

[0054]

[0055] in, Indicates the target image I generated on the convolution layer l generated The activation value of the feature map; Indicates that on the convolution layer l, the input image I content The activation value of the feature map; i and j represent the pixel positions of the feature map.

[0056] The style loss is measured by comparing the feature statistics (Gram matrix) of the generated target image and the target style image. The Gram matrix is ​​used to capture the covariance information of the feature map and is defined as follows:

[0057]

[0058] Style loss formula:

[0059]

[0060] in, Represents the generated target image I generated Gram matrix; Represents the target style image I style Gram matrix; N l Represents the number of channels in convolutional layer l; M l Represents the feature map size of convolutional layer l.

[0061] Style Strength Adjustment:

[0062] In the formula, the style strength is controlled directly by adjusting the weights α and β:

[0063] Style strength = β / (α+β)

[0064] The higher the style strength, the greater the proportion of β, and the generated image is more stylized; the lower the style strength, the greater the proportion of α, and the generated image retains more original details. In this way, by adjusting the weight ratio β / α, refined style transfer control can be achieved.

[0065] The values ​​of weights α and β should meet the actual task requirements.

[0066] A practical approach to experimental conditioning:

[0067] Initial value:

[0068] Set α=1, β=1, that is, the initial weights of content loss and style loss are equal, and observe the generated results;

[0069] According to the image effect, gradually increase or decrease the ratio of α and β until the target requirement is met;

[0070] Regional regulation:

[0071] Set α and β separately for different image regions (such as face and background);

[0072] For example:

[0073] Facial area:

[0074] α=10,β=2 (focus on details)

[0075] Background area:

[0076] α=2,β=10 (more stylized).

[0077] This application generates different masks for the face and background. The masks can be automatically generated using tools such as FastSAM, Mask, R-CNN, or manually annotated. The mask M of the face area face and the background area mask M bg Satisfy formula M face +M bg =1.

[0078] Specifically, the mask map is usually the same size as the original image and is a binary or Boolean image. The area marked as 1 (or True) indicates the part that needs to be stylized, while the area marked as 0 (or False) is not affected. When applying operations (such as filtering, edge detection, image enhancement, etc.), the mask can limit the operation to only specific areas.

[0079] As a preferred implementation, the FastSAM tool is used to automatically generate masks; FastSAM can quickly generate semantic segmentation masks, is suitable for real-time scenarios, has high speed, and is compatible with large-scale images.

[0080] Through mask technology, images can be divided into different regions and different style strengths can be assigned to each region. This method allows for more flexible control of style transfer effects. The use of masks ensures that key details and feature areas are preserved during the stylization process, and also helps achieve independent control of different regions, allowing multiple generative models to play their respective advantages.

[0081] By setting different style strengths for the face and background, the style effect can be adjusted more flexibly. The stylization degree of the VToonify model used for the face can be adjusted according to user needs, such as retaining more facial features or adding more anime style, while the background part can generate a more free or abstract effect based on the prompt words and diffusion model. By controlling the style strength of the region, the generated image can be more personalized and meet the needs.

[0082] S102: Face Processing: Using VToonify Model G vtoonify , retain facial details and apply the specified anime style to transfer the face to generate a stylized facial image I face ;

[0083] like Figure 2 As shown, VToonify model G vtoonify It consists of a content extraction module E, a style extraction module Es and a generator G. After features are extracted by E and Es, they are input into G together to generate a stylized image.

[0084] S103: Background processing: using diffusion model G diffusion Stylize the background area to generate a free and abstract style background I bg ;

[0085] Specifically, the diffusion model is a DiffStyler double diffusion architecture. Figure 2 ,The diffusion model learns style features by adding noise Zr and removing noise Zr.

[0086] VToonify mainly processes facial details, while the diffusion model is applied to the background area. This regional processing method allows more flexible control of different parts of the image, ensuring that the stylized effect is both delicate and balanced, with both details and artistic sense.

[0087] By achieving fine region division through masks, the style transfer effects of the face and background can be adjusted independently, thereby improving the quality and diversity of image generation.

[0088] S104: Mask fusion: fusion of face and background into a complete image through masking

[0089] The mask is used to guide the image fusion, ensuring that the style transfer effect of different regions can be smoothly transitioned. The application of the mask allows each region to define its style and processing method separately, avoiding the abruptness and sense of fragmentation of the generated results. Compared with the traditional style transfer method, the image generated by the method of this application appears more coherent and coordinated in terms of style. In addition, the data smoothing technology can also be used to blend the face and background well to obtain the best effect.

[0090] Image after mask fusion It is expressed as:

[0091]

[0092] Among them, ⊙ represents the pixel-by-pixel multiplication operation, style face is the style code of the face, s face is the style intensity of the face, prompt bg is the cue word used by the diffusion model to generate the background, s bg is the style strength of the background.

[0093] Figure 3 This is a comparison chart of the effects of the character comic generation method assisted by fine-grained control of style transfer masks. In the figure, Content represents the original input image, Style represents the target style image, Ours represents the image processed by the application method, VToonify represents the image processed by the VToonify model, Diffstyler represents the image processed by the diffusion model, and IEContrAST is called Artistic Style Transfer with Internal-external Learning and Contrastive Learning. It is an artistic style transfer method that uses internal and external contrast learning. This method solves the artifact problems such as color inconsistency and pattern repetition in stylized images. However, in pictures with characters, blurred faces still appear, reflecting the instability of the model.

[0094] from Figure 3It can be seen that the details of the facial area of ​​the character cartoon generated by the method of the present application are retained and enhanced by VToonify, while the background area generates a more free stylized effect through the diffusion model. This combination can achieve higher quality generation of stylized images, producing a stylized effect that is both beautiful and meets the style requirements, and has both details and artistic sense.

[0095] Furthermore, the method of the present application has high flexibility and scalability. By adjusting the range and accuracy of the mask, it can be extended to more scenes. For example, not only can the face and background be controlled, but it can also be further extended to different areas such as hair, eyes, clothes, etc. to achieve more complex style combinations. Specifically, the FastSAM target detection model can recognize different areas in the picture and mark them with different colors, so that in the migration process, the pixel values ​​corresponding to the colors can be used to achieve masking. For example, the target detection model detects eyes, hair, tops, pants, and mountains, and marks them with five colors: red, yellow, blue, green, and purple. Then, to stylize the hair, we mark the yellow area as (255, 255, 255), and the remaining areas are marked as (0, 0, 0) for stylization, and so on, all separations can be achieved.

[0096] At the same time, the flexible control of style intensity also allows the method of this application to adapt to different application scenarios. Compared with the traditional single style transfer method, the method of this application provides more operational dimensions, allowing users to control the style effect of each area according to specific needs, and combine multiple areas and multiple models. This means that not only can a single image area be processed, but also local style transfer of multiple parts can be achieved by setting different target prompts. This flexibility is rare in existing models.

[0097] like Figure 4 As shown, the present application also provides a character comics generation device assisted by finely controlling style transfer mask, including:

[0098] The mask generation unit 410 is used for mask generation: generating different masks for the face and background of the initial person image I, and setting style strength for different regions. The style strength refers to the strength of the stylized effect applied to the target image during the style transfer process;

[0099] The face processing unit 420 is used for face processing: using the VToonify model G vtoonify , retain facial details and apply the specified anime style to transfer the face to generate a stylized facial image I face ;

[0100] The background processing unit 430 is used to use the diffusion model G diffusionStylize the background area to generate a free and abstract style background I bg ;

[0101] The mask fusion unit 440 is used to fuse the face and the background into a complete image through the mask.

[0102] Furthermore, the mask can be automatically generated using FastSAM, Mask or R-CNN tools.

[0103] Furthermore, the diffusion model is a DiffStyler double diffusion architecture.

[0104] Furthermore, the mask M of the facial region face and the background area mask M bg Satisfy formula M face +M bg =1.

[0105] Furthermore, the mask fusion unit 440 fuses the image by mask It is expressed as:

[0106]

[0107] Among them, ⊙ represents the pixel-by-pixel multiplication operation, style face is the style code of the face, s face is the style intensity of the face, prompt bg is the cue word used by the diffusion model to generate the background, s bg is the style strength of the background.

[0108] As for the device embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units, such as distributed on a server and a client. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0109] This application generates the characters and background in a picture separately, and uses masks to mark the generation focus, so that multiple models can only play their own strengths in a picture, and finally splice them together, so that the overall composition of the picture is optimized in multiple aspects.

[0110] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A character caricature generation method assisted by finely controlled style transfer mask, characterized in that: include: Step 1: Mask generation: Generate different masks for the face and background of the original person image I, and set style strength for different areas. Style strength refers to the strength of the stylized effect applied to the target image during the style transfer process. Step 2: Face processing: Using VToonify model G vtoonify , retain facial details and apply the specified anime style to transfer the face to generate a stylized facial image I face ; Step 3: Background processing: using diffusion model G diffusion Generate style background I by stylizing the background area bg ; Step 4: Mask fusion: Use mask to fuse the face and background into a complete image 2. The method for generating comic characters assisted by fine-grained style transfer mask according to claim 1, characterized in that: The mask can be automatically generated using FastSAM, Mask or R-CNN tools.

3. The method for generating comic characters assisted by fine-grained style transfer mask according to claim 1 or 2, characterized in that: The diffusion model is a DiffStyler double diffusion architecture.

4. The method for generating comic characters assisted by fine-grained style transfer mask according to claim 3, characterized in that: The mask M of the face region face and the background area mask M bg Satisfy formula M face +M bg =1.

5. The method for generating comic characters assisted by fine-grained style transfer mask according to claim 4, characterized in that: The image fused by mask in step 4 It is expressed as: Among them, ⊙ represents the pixel-by-pixel multiplication operation, style face is the style code of the face, s face is the style intensity of the face, prompt bg is the cue word used by the diffusion model to generate the background, s bg is the style strength of the background.

6. A character caricature generation device assisted by finely controlling style transfer mask, characterized in that: include: The mask generation unit is used for mask generation: generating different masks for the face and background of the original person image I, and setting style strength for different regions. The style strength refers to the strength of the stylization effect applied to the target image during the style transfer process. Face processing unit for face processing: using VToonify model G vtoonify , retain facial details and apply the specified anime style to transfer the face to generate a stylized facial image I face ; Background processing unit for using the diffusion model G diffusion Stylize the background area to generate a free and abstract style background I bg ; Mask fusion unit, used to fuse the face and background into a complete image through masking 7. The character caricature generation device assisted by fine control style transfer mask according to claim 6, characterized in that: The mask can be automatically generated using FastSAM, Mask or R-CNN tools.

8. The character comics generation device assisted by fine control style transfer mask according to claim 6 or 7, characterized in that: The diffusion model is a DiffStyler double diffusion architecture.

9. The character comics generation device assisted by fine control style transfer mask according to claim 8, characterized in that: The mask M of the face region face and the background area mask M bg Satisfy formula M face +M bg =1.

10. The character comics generation device assisted by fine control style transfer mask according to claim 9, characterized in that: Image after mask fusion in the mask fusion unit It is expressed as: Among them, ⊙ represents the pixel-by-pixel multiplication operation, style face is the style code of the face, s face is the style intensity of the face, prompt bg is the cue word used by the diffusion model to generate the background, s bg is the style strength of the background.

Images