Multi-style picture book generation method based on diffusion model

Abstract

The invention discloses a multi-style picture book generation method based on a diffusion model. The method comprises the following steps: S1, constructing a data set; s2, constructing a picture book generation model; the preprocessing module encodes the picture book image and the story text to obtain text embedding, role mask and role image; s3, importing a style reference image to a style consistency module, and obtaining style feature embedding; s4, importing text embedding, role masks and role images to a role consistency module, and obtaining role embedding and layout embedding; s5, importing the picture book image, performing style feature embedding, role embedding and layout embedding into a Unet module to perform picture book image noise prediction, and obtaining a predicted picture book image; and S6, constructing a loss function. According to the method, a diffusion model and an image style migration algorithm in deep learning are combined, a coherent picture book scene with various artistic styles can be generated according to the input reference image, and the method can be widely applied to the fields of animation production, image creative design, digital content generation and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for generating a multi-style picture book based on a diffusion model. Background Art

[0002] Currently, manually creating story picture books with different artistic styles is time-consuming and labor-intensive, and requires artists to have high painting skills and style control capabilities. With the rapid development of deep learning technology, the capabilities of computer vision and generative models have been greatly improved, especially in the fields of image generation and style transfer, where breakthrough progress has been made.

[0003] Existing image generation techniques based on diffusion models are already able to generate high-quality single images based on text prompts. However, these techniques face some challenges when applied to generating complex and coherent multi-frame story picture books. The main problem is that the characters and backgrounds lack consistency between different frames, resulting in incoherent changes in character images and scenes, which affects the coherence of the story and the fluency of the narrative.

[0004] Furthermore, while existing style transfer techniques are able to convert images into a specific artistic style, they can typically only handle the transfer of a single style and lack the flexibility and consistency required to generate images in multiple styles in consecutive picture books.

[0005] In the story picture book generation task, it is necessary to ensure that the generated characters maintain consistency in appearance and features in the previous and next frames to avoid image changes or distortion. At the same time, it is necessary to ensure that the overall style of the generated results is unified and the background transition is smooth and natural to maintain the sense of space and narrative fluency. Therefore, the existing technology still needs to be improved in the following aspects: it is impossible to efficiently and flexibly generate continuous picture book images of various artistic styles; it is difficult to ensure the coherence and consistency of the generated images in the picture book narrative; the existing image style transfer algorithms are mostly limited to a single scene and do not have the ability to dynamically transfer multiple styles. Summary of the invention

[0006] In order to solve these problems, the present invention proposes a method for generating multi-style picture books based on a diffusion model, including the following steps: Step S1: construct a picture book dataset, which includes a number of picture book images and corresponding story texts; construct a style reference dataset, which includes a number of style reference images; Step S2: construct a picture book generation model based on a diffusion model, which includes a preprocessing module, a style consistency module, a role consistency module, and a Unet module based on an attention mechanism; the preprocessing module encodes the picture book image and the corresponding story text in step S1 to obtain text embedding, role mask, and role image; Step S3: import the style reference image of step S1 into the style consistency module to obtain style feature embedding; Step S4: import the text embedding, role mask and role image of step S2 into the role consistency module to obtain the role embedding and layout embedding; Step S5: import the picture book image in step S1, the style feature embedding in step S3, the character embedding and the layout embedding in step S4 into the attention block in the Unet module based on the attention mechanism to perform picture book image noise prediction and obtain the predicted picture book image; Step S6: Construct a loss function and minimize the loss function to optimize the parameters of the picture book generation model.

[0007] Furthermore, the preprocessing module in step S1 includes a CLIP encoder and an image segmentation model GSA; Step S1 is specifically as follows: Step S11: Encode the story text through the CLIP encoder to obtain the text embedding with preset data dimensions , expressed as: ; in, represents regularization, represents a multilayer perceptron, n represents the number of internal operations performed by the multilayer perceptron, represents the self-attention operation, Indicates the word segmentation operation on the text. represents the position embedding code, Represents the input story text; Step S12: Use the image segmentation model GSA to segment the picture book image and obtain the character mask and character images , expressed as: ; in, Indicates picture book images; represents the selection function, which is used to obtain the mask information of a specific character in the picture book image. represents the image segmentation module, Represents dot product.

[0008] Furthermore, step S3 is specifically as follows: Step S31: For the input style reference image To process, specifically: Call the large language model of images and text to generate the reference image of the style The text semantic content of the style reference image is then encoded using the image encoder and text encoder of the CLIP encoder. Encode the corresponding text semantic content to obtain the style reference image The image encoding and text encoding are mapped to the same semantic space, and the text encoding is subtracted from the image encoding to obtain the style feature embedding without the text semantic content. , expressed as: ; in, represents the text encoder of the CLIP encoder, Representing a large language model that is multimodal, represents the image encoder of the CLIP encoder, represents a multi-layer perceptron; Step S32: embedding style features Further processing is performed to obtain the style features, which are expressed as: ; in, represents a learnable embedding, represents the self-attention operation, represents the cross attention operation, represents the fully connected layer, Indicates style characteristics.

[0009] Furthermore, the role consistency module includes a resampling module and a layout embedding module. Step S4 is specifically as follows: Step S41: Transform the character image and role mask Input the role consistency module to obtain the resampled embedding corresponding to each role, and then perform cross-attention calculation through MLP mapping and diffusion model intermediate noise to obtain the role embedding , expressed as: ; in, represents the resampled embedding, represents the intermediate noise of the diffusion model, represents role embedding, represents the cross attention mechanism, represents the resampling operation, represents a multi-layer perceptron; Step S42: Input text embedding and resampled embedding Go to the layout control module for processing and obtain the layout embedding , expressed as: ; in, represents the fully connected layer, and noise represents the input noise.

[0010] Further, the attention mechanism-based Unet module of step S5 includes several attention blocks for guiding image generation; Step S5 is specifically as follows: ; in, represents the predicted picture book image, Represents the Unet module, represents the picture book image after adding noise, Indicates the number of noise adding steps.

[0011] Furthermore, the loss function in step S6 It is expressed as: ; in, , , Represents the weight coefficient of different losses, , , They are the loss of the diffusion model, the loss of role consistency, and the loss of style consistency.

[0012] The positive and progressive effects of the present invention are:

[0013] The present invention proposes an efficient multi-style picture book generation method based on the diffusion model. The method can generate story picture books of any artistic style according to style reference images, which greatly simplifies the creation process and improves the quality and style diversity of the generated picture book images. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flowchart of the steps of a method for generating multi-style picture books based on a diffusion model of the present invention. DETAILED DESCRIPTION

[0015] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0016] Reference Figure 1, a method for generating multi-style picture books based on a diffusion model, in one example, step S1: constructing a picture book dataset, the picture book dataset includes a number of picture book images and corresponding story texts; constructing a style reference dataset, the style reference dataset includes a number of style reference images; Step S2: construct a picture book generation model based on a diffusion model, which includes a preprocessing module, a style consistency module, a role consistency module, and a Unet module based on an attention mechanism; the preprocessing module encodes the picture book image and the corresponding story text in step S1 to obtain text embedding, role mask, and role image; Step S3: import the style reference image of step S1 into the style consistency module to obtain style feature embedding; Step S4: import the text embedding, role mask and role image of step S2 into the role consistency module to obtain the role embedding and layout embedding; Step S5: import the picture book image in step S1, the style feature embedding in step S3, the character embedding and the layout embedding in step S4 into the attention block in the Unet module based on the attention mechanism to perform picture book image noise prediction and obtain the predicted picture book image; Step S6: Construct a loss function and minimize the loss function to optimize the parameters of the picture book generation model.

[0017] Further, in one example, the preprocessing module in step S1 includes a CLIP encoder and an image segmentation model GSA; Step S1 is specifically as follows: Step S11: Encode the story text through the CLIP encoder to obtain the text embedding with preset data dimensions , expressed as: ; in, represents regularization, represents a multilayer perceptron, n represents the number of internal operations performed by the multilayer perceptron, represents the self-attention operation, Indicates the word segmentation operation on the text. represents the position embedding code, Represents the input story text; Step S12: Use the image segmentation model GSA to segment the picture book image and obtain the character mask and character images , expressed as: ; in, Indicates picture book images; represents the selection function, which is used to obtain the mask information of a specific character in the picture book image. represents the image segmentation module, Represents dot product.

[0018] Furthermore, in one example, step S3 is specifically: Step S31: For the input style reference image To process, specifically: Call the large language model of images and text to generate the reference image of the style The text semantic content of the style reference image is then encoded using the image encoder and text encoder of the CLIP encoder. Encode the corresponding text semantic content to obtain the style reference image The image encoding and text encoding are mapped to the same semantic space, and the text encoding is subtracted from the image encoding to obtain the style feature embedding without the text semantic content. , expressed as: ; in, represents the text encoder of the CLIP encoder, Representing a large language model that is multimodal, represents the image encoder of the CLIP encoder, represents a multi-layer perceptron; Step S32: embedding style features Further processing is performed to obtain the style features, which are expressed as: ; in, represents a learnable embedding, represents the self-attention operation, represents the cross attention operation, represents the fully connected layer, Indicates style characteristics.

[0019] Further, in one example, the role consistency module includes a resampling module and a layout embedding module. Step S4 is specifically as follows: Step S41: Transform the character image and role mask Input the role consistency module to obtain the resampled embedding corresponding to each role, and then perform cross-attention calculation through MLP mapping and diffusion model intermediate noise to obtain the role embedding , expressed as: ; in, represents the resampled embedding, represents the intermediate noise of the diffusion model, represents role embedding, represents the cross attention mechanism, represents the resampling operation, represents a multi-layer perceptron; Step S42: Input text embedding and resampled embedding Go to the layout control module for processing and obtain the layout embedding , expressed as: ; in, represents the fully connected layer, and noise represents the input noise.

[0020] Further, in one example, the attention mechanism-based Unet module of step S5 includes a number of attention blocks for guiding image generation; Step S5 is specifically as follows: ; in, represents the predicted picture book image, Represents the Unet module, represents the picture book image after adding noise, Indicates the number of noise adding steps.

[0021] Furthermore, in one example, the loss function in step S6 is It is expressed as: ; in, , , Represents the weight coefficient of different losses, , , They are the loss of the diffusion model, the loss of role consistency, and the loss of style consistency.

[0022] The present invention is described in detail above in conjunction with the embodiments of the drawings, and those skilled in the art can make various variations of the present invention according to the above description. Therefore, certain details in the embodiments should not constitute limitations on the present invention, and the present invention shall be protected by the scope defined by the attached claims.

Claims

1. A method for generating multi-style picture books based on a diffusion model, characterized in that: The following steps are involved: Step S1: construct a picture book dataset, which includes a number of picture book images and corresponding story texts; construct a style reference dataset, which includes a number of style reference images; Step S2: construct a picture book generation model based on a diffusion model, which includes a preprocessing module, a style consistency module, a role consistency module, and a Unet module based on an attention mechanism; the preprocessing module encodes the picture book image and the corresponding story text in step S1 to obtain text embedding, role mask, and role image; Step S3: import the style reference image of step S1 into the style consistency module to obtain style feature embedding; Step S4: import the text embedding, role mask and role image of step S2 into the role consistency module to obtain the role embedding and layout embedding; Step S5: import the picture book image in step S1, the style feature embedding in step S3, the character embedding and the layout embedding in step S4 into the attention block in the Unet module based on the attention mechanism to perform picture book image noise prediction and obtain the predicted picture book image; Step S6: Construct a loss function and minimize the loss function to optimize the parameters of the picture book generation model.

2. The method for generating a multi-style picture book based on a diffusion model as claimed in claim 1, characterized in that: The preprocessing module in step S1 includes the CLIP encoder and the image segmentation model GSA, specifically: Step S11: Encode the story text using the CLIP encoder to obtain text embedding with preset data dimensions , expressed as: ； in, represents regularization, represents a multilayer perceptron, n represents the number of internal operations performed by the multilayer perceptron, represents the self-attention operation, Indicates word segmentation operation on text. represents the position embedding code, Represents the input story text; Step S12: Use the image segmentation model GSA to segment the picture book image and obtain the character mask and character images , expressed as: ； in, Indicates picture book images; represents the selection function, which is used to obtain the mask information of a specific character in the picture book image. represents the image segmentation module, Represents dot product.

3. The method for generating a multi-style picture book based on a diffusion model as claimed in claim 2, characterized in that: Step S3 is specifically as follows: Step S31: For the input style reference image To process, specifically: Call the large language model of images and text to generate the reference image of the style The text semantic content of the style reference image is then encoded using the image encoder and text encoder of the CLIP encoder. Encode the corresponding text semantic content to obtain the style reference image The image encoding and text encoding are mapped to the same semantic space, and the text encoding is subtracted from the image encoding to obtain the style feature embedding without the text semantic content. , expressed as: ； in, represents the text encoder of the CLIP encoder, Representing a large language model that is multimodal, represents the image encoder of the CLIP encoder, represents a multi-layer perceptron; Step S32: embedding style features Further processing is performed to obtain the style features, which are expressed as: ； in, represents a learnable embedding, represents the self-attention operation, represents the cross attention operation, represents the fully connected layer, Indicates style characteristics.

4. The method for generating a multi-style picture book based on a diffusion model as claimed in claim 3, characterized in that: The role consistency module includes a resampling module and a layout embedding module. Step S4 is specifically as follows: Step S41: Transform the character image and role mask Input the role consistency module to obtain the resampled embedding corresponding to each role, and then perform cross-attention calculation through MLP mapping and diffusion model intermediate noise to obtain the role embedding , expressed as: ； in, represents the resampled embedding, represents the intermediate noise of the diffusion model, represents role embedding, represents the cross attention mechanism, represents the resampling operation, represents a multi-layer perceptron; Step S42: Input text embedding and resampled embedding Go to the layout control module for processing and obtain the layout embedding , expressed as: ； in, represents the fully connected layer, and noise represents the input noise.

5. The method for generating a multi-style picture book based on a diffusion model as claimed in claim 4, characterized in that: The attention-based Unet module of step S5 includes several attention blocks for guiding image generation; Step S5 is specifically as follows: ； in, represents the predicted picture book image, Represents the Unet module, represents the picture book image after adding noise, Indicates the number of noise adding steps.

6. The method for generating a multi-style picture book based on a diffusion model as claimed in claim 1, characterized in that: Loss function in step S6 It is expressed as: ； in, , , Represents the weight coefficient of different losses, , , They are the loss of the diffusion model, the loss of role consistency, and the loss of style consistency.

Images