Generation method and system of story cartoon with consistency, and terminal

Abstract

The invention discloses a method, a system and a terminal for generating a comic with a consistent story, and the method comprises the steps: determining a target comic prompt word, carrying out the comic story generation processing and the split processing, and obtaining a plurality of split prompt words and a plurality of role prompt words; obtaining a preset diffusion model, and inputting each split prompt word and the plurality of role prompt words into the preset diffusion model to obtain a cross attention graph; performing mask segmentation processing on the cross attention map to obtain a role segmentation mask; constructing an expanded attention graph according to the cross attention graph and the role segmentation mask, and performing mask assignment processing and attention weight distribution processing to obtain an isolated self-attention graph; and performing cross attention feature extraction processing and weighted fusion processing on the isolated self-attention graph to obtain fusion features, and performing iterative diffusion denoising processing on the fusion features to obtain a target cartoon image with a consistent story. The visual quality of automatic cartoon generation can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, system, terminal and computer-readable storage medium for generating a consistent story comic. Background Art

[0002] As a form of visual narrative, comics have a wide audience. However, traditional comic creation methods rely on manual drawing, which is not only time-consuming and laborious, but also requires extremely high skills from artists. With the rapid development of artificial intelligence, automated comic generation technology has gradually become a research hotspot. Automated comic generation technology has made some progress in improving the efficiency of comic generation, but there are still many challenges.

[0003] Existing technologies have difficulty maintaining the consistency of character images between different comic frames. This consistency is the key to telling a coherent story. However, since the generative model lacks cross-frame correlation information when processing multi-frame images, the image of the character in different frames may be inconsistent. When multiple characters appear in the same frame at the same time, the model may not be able to effectively distinguish and maintain the unique characteristics of each character, resulting in confusion of character features and loss of details in the generated image, which seriously affects the overall quality of the comic.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a method, system, terminal and computer-readable storage medium for generating consistent story comics, aiming to solve the problem that the automated comic generation model in the prior art lacks cross-frame correlation information when processing multiple frame images. When multiple characters appear in the same frame at the same time, the model cannot effectively distinguish and maintain the unique characteristics of each character, resulting in the generated comic image having confused character features and lost details, which seriously affects the overall quality of the comic.

[0006] To achieve the above object, the present invention provides a method for generating a comic book with a consistent story, the method comprising the following steps:

[0007] Determine a target comic prompt, and perform comic story generation processing and storyboard processing according to the target comic prompt to obtain a plurality of storyboard prompt words and a plurality of character prompt words corresponding to each storyboard prompt word;

[0008] Obtaining a preset diffusion model, inputting each of the storyboard prompt words and the corresponding plurality of character prompt words into the preset diffusion model to obtain a cross-attention map;

[0009] Performing segmentation mask processing on the cross attention map to obtain a character segmentation mask;

[0010] Constructing an extended attention map according to the cross attention map and the role segmentation mask, and performing mask assignment processing and attention weight allocation processing on the extended attention map to obtain an isolated self-attention map;

[0011] The isolated self-attention map is subjected to cross-attention feature extraction processing to obtain scene cross-attention features and character cross-attention features, the scene cross-attention features and the character cross-attention features are weighted fused according to the character segmentation mask to obtain fused features, and the fused features are iteratively diffused denoising processed according to the preset diffusion model to obtain a target comic image with a consistent story.

[0012] Optionally, the method for generating a comic with a consistent story, wherein the step of determining a target comic prompt, and performing comic story generation processing and storyboard processing according to the target comic prompt to obtain a plurality of storyboard prompt words and a plurality of character prompt words corresponding to each storyboard prompt word, specifically comprises:

[0013] Determine a target comic prompt and a preset macro model, input the target comic prompt into the preset macro model, and perform comic story generation processing on the target comic prompt through the preset macro model to obtain a target comic story;

[0014] The target comic story is storyboarded to obtain a plurality of storyboard prompt words and a plurality of character prompt words corresponding to each storyboard prompt word.

[0015] Optionally, in the method for generating a consistent story comic, the preset diffusion model includes an original branch and an extended branch; the original branch includes a self-attention mechanism module and a cross-attention module; the extended branch includes an ID isolation self-attention mechanism module and an ID isolation cross-attention module;

[0016] The step of obtaining a preset diffusion model and inputting each of the storyboard prompt words and the corresponding plurality of role prompt words into the preset diffusion model to obtain a cross attention map specifically includes:

[0017] Obtaining a preset diffusion model, inputting each of the storyboard prompt words and the corresponding multiple character prompt words into the preset diffusion model to obtain multiple input feature maps;

[0018] The multiple input feature maps are sequentially input into the self-attention mechanism module and the cross-attention module of the original branch to obtain a cross-attention map.

[0019] Optionally, the method for generating a comic with a consistent story, wherein the step of performing segmentation mask processing on the cross attention map to obtain a character segmentation mask specifically includes:

[0020] The maximum inter-class variance method is used to calculate the inter-class variance of the cross-attention map, and the optimal segmentation threshold is calculated according to the inter-class variance to obtain the character segmentation mask;

[0021] Among them, the expression of the between-class variance is:

[0022]

[0023] Where T is the segmentation threshold, is the inter-class variance, ω1(T) is the weight of the first segmentation area, ω2(T) is the weight of the second segmentation area, μ1(T) is the mean of the first segmentation area, and μ2(T) is the mean of the second segmentation area;

[0024] Among them, the expression of the optimal segmentation threshold is:

[0025]

[0026] Among them, T * is the optimal segmentation threshold, Solve for the optimal segmentation threshold.

[0027] Optionally, the method for generating a consistent story comic, wherein the step of constructing an extended attention map according to the cross attention map and the character segmentation mask, and performing mask assignment processing and attention weight allocation processing on the extended attention map to obtain an isolated self-attention map, specifically includes:

[0028] Inputting the plurality of input feature maps, the cross attention map, and the role segmentation mask into the ID isolation self-attention mechanism module of the extended branch to obtain a target role name;

[0029] Obtaining a preset role database, and searching the preset role database according to the target role name;

[0030] If the search is successful, the target role feature corresponding to the target role name is obtained through the preset role database, and an extended attention graph is constructed according to the target role name and the target role feature;

[0031] The extended attention map is subjected to mask assignment processing and attention weight allocation processing according to the cross attention map to obtain an isolated self-attention map.

[0032] Optionally, the method for generating a comic book with a consistent story, wherein the step of obtaining a preset character database and searching the preset character database according to the target character name, further comprises:

[0033] If the search fails, the screening role feature corresponding to the target role name is calculated according to the role segmentation mask, and the target role name and the corresponding screening role feature are stored in the preset role database.

[0034] Optionally, the method for generating a comic with a consistent story, wherein the isolated self-attention map is subjected to cross-attention feature extraction processing to obtain scene cross-attention features and character cross-attention features, the scene cross-attention features and the character cross-attention features are subjected to weighted fusion processing according to the character segmentation mask to obtain fused features, and the fused features are subjected to iterative diffusion denoising processing according to the preset diffusion model to obtain a target comic image with a consistent story, specifically includes:

[0035] Inputting the isolated self-attention map into the ID isolation cross attention module of the extended branch, performing cross attention feature extraction processing on the isolated self-attention map through the ID isolation cross attention module to obtain scene cross attention features and character cross attention features;

[0036] Performing weighted fusion processing on the scene cross-attention feature and the character cross-attention feature according to the character segmentation mask to obtain a fusion feature;

[0037] The fused features are input into the diffusion model again, and the fused features are iteratively subjected to diffusion denoising through the diffusion model to obtain a target comic image with a consistent story.

[0038] In addition, to achieve the above-mentioned purpose, the present invention also provides a system for generating a comic book with a consistent story, wherein the system for generating a comic book with a consistent story comprises:

[0039] A storyboard processing module is used to determine a target comic prompt word, and perform comic story generation processing and storyboard processing according to the target comic prompt word to obtain a plurality of storyboard prompt words and a plurality of character prompt words corresponding to each storyboard prompt word;

[0040] A cross-attention map generation module is used to obtain a preset diffusion model, input each of the storyboard prompt words and the corresponding multiple role prompt words into the preset diffusion model to obtain a cross-attention map;

[0041] A segmentation mask processing module, used to perform segmentation mask processing on the cross attention map to obtain a character segmentation mask;

[0042] An isolated self-attention map generation module is used to construct an extended attention map according to the cross attention map and the role segmentation mask, and perform mask assignment processing and attention weight allocation processing on the extended attention map to obtain an isolated self-attention map;

[0043] The target comic image generation module is used to perform cross-attention feature extraction processing on the isolated self-attention map to obtain scene cross-attention features and character cross-attention features, perform weighted fusion processing on the scene cross-attention features and the character cross-attention features according to the character segmentation mask to obtain fusion features, and perform iterative diffusion denoising processing on the fusion features according to the preset diffusion model to obtain a target comic image with a consistent story.

[0044] In the present invention, a target comic prompt is determined, and a comic story generation process and a storyboard process are performed according to the target comic prompt to obtain a plurality of storyboard prompt words and a plurality of role prompt words corresponding to each storyboard prompt word; a preset diffusion model is obtained, and each of the storyboard prompt words and the corresponding plurality of role prompt words are input into the preset diffusion model to obtain a cross-attention map; a segmentation mask process is performed on the cross-attention map to obtain a role segmentation mask; an extended attention map is constructed according to the cross-attention map and the role segmentation mask, and a mask assignment process and an attention weight allocation process are performed on the extended attention map to obtain an isolated self-attention map; a cross-attention feature extraction process is performed on the isolated self-attention map to obtain a scene cross-attention feature and a role cross-attention feature, and a weighted fusion process is performed on the scene cross-attention feature and the role cross-attention feature according to the role segmentation mask to obtain a fusion feature, and an iterative diffusion denoising process is performed on the fusion feature according to the preset diffusion model to obtain a target comic image with a consistent story. The present invention realizes consistent story comic generation through a large language model and a diffusion model. By using a large model to realize storyboarding, modifying the structure of the diffusion model, adding ID isolation self-attention and ID isolation cross-attention, the identity consistency of multiple characters in the generated comic images is ensured, and the attribute mixing between multiple characters is reduced, which significantly improves the visual quality and story coherence of the generated comics. It not only improves the efficiency of automatic comic generation, but also expands its application scope, and is suitable for comic creation of various styles and plots. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of two-dimensional contour lines of a preferred embodiment of the method for generating a consistent story comic of the present invention;

[0046] Figure 2It is a schematic diagram of a preferred embodiment of the method for generating a consistent story comic of the present invention, in which a story, a storyboard prompt word and a role prompt word are generated through a large language model;

[0047] Figure 3 It is a schematic diagram of the overall structure of the diffusion model consistent comic generation of a preferred embodiment of the method for generating consistent story comics of the present invention;

[0048] Figure 4 It is a structural diagram of a preferred embodiment of the system for generating a consistent story comic of the present invention;

[0049] Figure 5 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] As a form of visual narrative, comics have a wide audience. However, traditional comic creation methods rely on manual drawing, which is not only time-consuming and laborious, but also requires extremely high skills from artists. With the rapid development of artificial intelligence, automated comic generation technology has gradually become a research hotspot. Currently, most automated comic generation technologies rely on generative adversarial networks (GANs) or pre-trained diffusion models. These methods have made some progress in improving the efficiency of comic generation, but there are still many challenges.

[0052] First, the storyboards of existing comics usually need to be manually set, which makes it difficult to achieve automatic generation. Secondly, existing methods have difficulty maintaining the consistency of character images between different comic frames. This consistency is the key to telling a coherent story, but because the generative model lacks cross-frame correlation information when processing multi-frame images, the image of the character in different frames may be inconsistent. This not only affects the coherence of the story, but also reduces the user's reading experience. Finally, when generating comics containing multiple characters, existing models often face the problem of character feature confusion. When multiple characters appear in the same frame at the same time, the model may not be able to effectively distinguish and maintain the unique characteristics of each character, resulting in confusion of character features and loss of details in the generated image. This problem is particularly prominent in plots involving complex scenes or interactions between multiple characters, which seriously affects the overall quality of the comics.

[0053] In addition, although some methods attempt to train models through specific datasets to allow them to remember common characters and backgrounds to achieve character consistency, the effectiveness of these methods is usually limited by the scale and diversity of the datasets and cannot meet the diverse needs in the open world. This makes it difficult for existing technologies to cope with comic generation tasks of different styles and plots in practical applications, limiting their potential for wide application.

[0054] To solve the above problems, the present invention proposes an innovative comic generation technology. By combining a large language model and a pre-trained diffusion model, the present invention can effectively solve the problems of inconsistent character images and confusion of character features, and significantly improve the visual quality and story coherence of the generated comics. This technology not only improves the efficiency of automatic comic generation, but also expands its application scope, and is suitable for comic creation of various styles and plots.

[0055] The present invention belongs to image generation technology, and the main difficulty it faces is how to generate a logically reasonable, story-telling and coherent comic with consistent character appearance. The present invention uses a large language model to assist in storyboarding and the self-attention and cross-attention mechanisms in the text-generated image diffusion model to achieve consistency generation. The present invention proposes a technical framework for automatically generating story comics with consistent appearance. The framework mainly includes two parts: using a large language model to process the story and using an improved text-generated image diffusion model to generate consistent comics. First, the large language model divides the story into multiple storyboards and generates storyboard prompts and character prompts. Secondly, the present invention makes two important improvements to the pre-trained text-generated image diffusion model: 1. After obtaining the object mask through the cross-attention mechanism, the self-attention mechanism is controlled so that the generated character refers more to the same character in the previous image, thereby improving the consistency of the character. 2. Character features and overall features are generated separately in the cross-attention mechanism, and these features are fused through the mask to avoid confusion of character attributes and further improve character consistency. In this way, the entire system can automatically generate story comics with consistent and coherent appearance.

[0056] The method for generating a consistent story comic according to a preferred embodiment of the present invention is as follows: Figure 1 As shown, the method for generating a comic with a consistent story includes the following steps:

[0057] Step S10, determining a target comic prompt word, and performing comic story generation processing and storyboard processing according to the target comic prompt word, to obtain a plurality of storyboard prompt words and a plurality of character prompt words corresponding to each storyboard prompt word.

[0058] Specifically, a target comic prompt and a preset large model are determined, the target comic prompt is input into the preset large model, and a comic story generation process is performed on the target comic prompt through the preset large model to obtain a target comic story; the target comic story is storyboarded to obtain a plurality of storyboard prompt words and a plurality of character prompt words corresponding to each storyboard prompt word.

[0059] like Figure 2 As shown, the present invention uses a large model to generate a story, and automatically generates storyboards and storyboard description words S = {S1, S2, ..., S N ] (a total of N storyboards) and the character description words C = {C1, C2, ..., C K ](K roles in total), character description words for each storyboard It may contain 0 or more roles, with n i represents the number of characters contained in the i-th storyboard, so there is a relationship n i =|C i | In addition, the present invention also supports storyboard division for a given story (i.e., no large model generation is required, but an existing story).

[0060] Step S20, obtaining a preset diffusion model, inputting each of the storyboard prompt words and the corresponding multiple role prompt words into the preset diffusion model to obtain a cross-attention map.

[0061] Specifically, a preset diffusion model is obtained, and each of the storyboard prompt words and the corresponding multiple character prompt words are input into the preset diffusion model to obtain multiple input feature maps; the multiple input feature maps are sequentially input into the self-attention mechanism module and the cross-attention module of the original branch to obtain a cross-attention map.

[0062] like Figure 3 As shown in a, the random noise Z is i The storyboard prompt word and the character prompt word C corresponding to the character contained in the storyboard shot i The present invention also needs to specify index i (represents the index of all characters in the i-th storyboard), New_index i (representing the index of the new character in the i-th storyboard) and Old_index i (The index of the old character in the i-th storyboard).

[0063] The preset diffusion model includes an original branch and an extended branch; the original branch includes a self-attention mechanism module and a cross-attention module; the extended branch includes an ID isolation self-attention mechanism module and an ID isolation cross-attention module.

[0064] like Figure 3 As shown in b in the diffusion model, in the upsampling, a Transformer module contains a self-attention mechanism and a cross-attention mechanism (i.e. Figure 3 The original branch in b in ). In self-attention, the similarities between the features of the image are calculated, which is called the self-attention map, and the image features are weighted and updated according to the similarities. In cross-attention, the image features are calculated with the input text features (i.e., the target comic story in the present invention) The cross-attention map is then used to weight the text features as new image features, thereby guiding the generation in the direction of the text. The input I of the Transformer is a matrix (i.e., the input feature map in the present invention) with a shape of [h*w, dim], where h and w refer to the height and width of the image features, and dim refers to the feature latitude (similar to the RGB channel, but the latitude here is much greater than 3). The input I obtains the output O through the self-attention and cross-attention of the Transformer block. origin At the same time, the attention map of the corresponding role in the cross attention (i.e., the cross attention map in the present invention) A={A m}m∈index i , the attention map roughly reflects the position of the corresponding character in the comics.

[0065] Step S30: perform segmentation mask processing on the cross-attention map to obtain a character segmentation mask.

[0066] Specifically, the maximum inter-class variance method is used to calculate the inter-class variance of the cross-attention map, and the optimal segmentation threshold is calculated according to the inter-class variance to obtain the role segmentation mask; wherein the expression of the inter-class variance is: Where T is the segmentation threshold, is the inter-class variance, ω1(T) is the weight of the first segmentation region, ω2(T) is the weight of the second segmentation region, μ1(T) is the mean of the first segmentation region, and μ2(T) is the mean of the second segmentation region; the expression of the optimal segmentation threshold is: Among them, T * is the optimal segmentation threshold, Solve for the optimal segmentation threshold.

[0067] The present invention obtains the role segmentation mask M={M m}m∈index i The core idea of ​​the Otsu algorithm is to automatically determine the image segmentation threshold T by maximizing the inter-class variance. The Otsu algorithm needs to find a threshold that makes the inter-class variance Maximum, T * It is the optimal segmentation threshold, which is the number of attention graphs A that are greater than T. * The part is set to 1, which is less than T * The part of is set to 0, and the character segmentation mask M is obtained. m .

[0068] Due to the data difference problem when training the diffusion model, the diffusion model pays different attention to each word, and previous studies have shown that the attention map of cross attention has noise. The present invention finds that attention maps with weaker noise tend to capture objects more accurately, while attention maps with stronger noise usually have higher attention and capture objects inaccurately. Therefore, the present invention calculates the coefficient of variation of each attention map, and the one with a lower coefficient of variation has a higher priority. If different role masks overlap, the role mask with a higher priority is considered.

[0069] Step S40, constructing an extended attention map according to the cross-attention map and the role segmentation mask, and performing mask assignment processing and attention weight allocation processing on the extended attention map to obtain an isolated self-attention map.

[0070] Specifically, a plurality of the input feature maps, the cross-attention map and the role segmentation mask are input into the ID isolation self-attention mechanism module of the extended branch to obtain the target role name; a preset role database is obtained, and the preset role database is searched according to the target role name; if the search is successful, the target role feature corresponding to the target role name is obtained through the preset role database, and an extended attention map is constructed according to the target role name and the target role feature; the extended attention map is subjected to mask assignment processing and attention weight allocation processing according to the cross-attention map to obtain an isolated self-attention map.

[0071] Furthermore, if the search fails, the screening role feature corresponding to the target role name is calculated according to the role segmentation mask, and the target role name and the corresponding screening role feature are stored in the preset role database.

[0072] like Figure 3 As shown in b in the figure, the input feature map, the cross-attention map obtained in the cross-attention module, and the character segmentation mask are input again into the self-attention mechanism of the Transformer block (i.e. Figure 3 The extended branch part in b includes Figure 3 The ID in b in the isolated self-attention and Figure 3The ID in d isolates the self-attention, Figure 3 The d in the sentence is correct. Figure 3 In the self-attention mechanism, a database consisting of character names and character feature data pairs is saved. The first appearance of a character is determined by whether the character name appears in the database. If it is the first appearance (i.e., the character name does not exist in the database), the character feature obtained by the character segmentation mask and the corresponding character name are stored in the database. The screening process of the character features obtained by the character segmentation mask can be expressed as: F m =I⊙M m (m∈New_indexi), F m represents the character feature of character m, ⊙ represents the multiplication of the corresponding positions, and the formula represents the screening of input I according to the character segmentation mask. If it is not the first time (that is, the database contains this character name), the character feature is called out from the database as the K value and V value of the attention mechanism and spliced ​​behind the attention map. Therefore, the expression of Q, K, V is: Q = Proj_q (I);

[0073] K = Proj_k[Concat(I, F m )](m∈Old_index i );

[0074] V = Proj_v[Concat(I, F m )](m∈Old_indexi);

[0075] The expression of the extended attention graph A is: A = Q × K; where Proj_k is Figure 3 There is a mark in d, Proj_k is a linear transformation matrix multiplication, which is a linear layer in deep learning. Concat means concatenation. There are s features in input I, F m There are j features in , and after Concat, there are s+j features.

[0076] Through the above steps, the present invention obtains an extended attention graph A, whose shape is (h*w, h*w+∑ m n m ), where m∈Old_index i , and n m Represents the number of features stored in character m. And according to the cross attention map, the extended attention map A corresponding to the character is modified to obtain the isolated self-attention map, so that the character features can refer to the features in the database as much as possible.

[0077] The modification of the extended attention map A specifically includes two means: 1. Make the area other than this character unable to pay attention to this character to prevent feature confusion, which requires blocking part of the spliced ​​attention map. The present invention finds the position of the spliced ​​character feature begin: begin+n m , assign the corresponding position of the attention map through the mask: A[:, begin: begin+n m ]=M m .repeat(1, n m )(m∈Old_index i ), this is a pseudo code. Originally, it was a vector of length h*w. Repeat(1,n) is to copy this vector n times to become a matrix of (h*w,n). Repeat(n,1) is to become a matrix of (n,h*w). 2. Since the image background will also affect the appearance of the subject, it will cause the consistency of the appearance to weaken. Therefore, it is necessary to operate the attention map to reduce the attention of the character to the background. The cross attention map can be used to change the self-attention weight, which can be expressed as: A[M m ,:h*w)=A m .repeat(n m ,1)(m∈Old_index i ), these two operations effectively ensure the consistency of the generated characters.

[0078] Step S50, performing cross-attention feature extraction processing on the isolated self-attention map to obtain scene cross-attention features and character cross-attention features, performing weighted fusion processing on the scene cross-attention features and the character cross-attention features according to the character segmentation mask to obtain fused features, and performing iterative diffusion denoising processing on the fused features according to the preset diffusion model to obtain a target comic image with a consistent story.

[0079] Optionally, the isolated self-attention map is input into the ID isolation cross-attention module of the extended branch, and the ID isolation cross-attention module is used to perform cross-attention feature extraction on the isolated self-attention map to obtain scene cross-attention features and character cross-attention features; the scene cross-attention features and the character cross-attention features are weighted fused according to the character segmentation mask to obtain fused features; the fused features are input into the diffusion model again, and the fused features are iteratively diffused denoising processed through the diffusion model to obtain a target comic image with a consistent story.

[0080] like Figure 3 As shown in c, the present invention will Figure 3 The output O obtained by SAThe image features (i.e., the isolated self-attention map in the present invention) enter the cross-attention module, and the image features will be cross-attended with the storyboard prompt words and the character prompt words respectively to obtain multiple updated features. and (i.e., the scene cross-attention feature and the role cross-attention feature in the present invention). Then, it is fused with the role segmentation mask to achieve the purpose that the role is mainly controlled by the role prompt word, which can prevent feature confusion and enhance consistency. The formula can be expressed as: m refers to the index of the role. i Belongs to the set, ∪ is the union of the masks of all roles.

[0081] Furthermore, the fused feature map and the cross-attention map are fused by weighted processing to obtain the final output of the Transformer module (i.e., the target comic image in the present invention). The expression of weighted processing of the fused feature map and the cross-attention map is:

[0082] O=O orig ×λ+O ISO ×(1-λ), this weight is used to control the consistency strength. If the output weight in the extended model is high, the consistency is high. If the weight in the ID isolation cross attention is high, the consistency is slightly weaker. This parameter can be controlled manually and is usually set to 0.9.

[0083] Furthermore, in the present invention, after obtaining the fused features, the fused features need to be input into the diffusion model again to realize a multi-step diffusion process of the fused features (i.e., multi-step denoising). After multiple iterative denoising of the diffusion model, the target comic image with a consistent story is finally obtained.

[0084] The present invention uses a large language model and a diffusion model to achieve consistent story comic generation, uses a large model to achieve storyboarding, and modifies the structure of the diffusion model to add ID isolation self-attention and ID isolation cross-attention, thereby ensuring the identity consistency of multiple characters in the generated comic image and reducing the attribute mixing between multiple characters. In addition, the present invention can also achieve appearance consistency through other trained plug-ins, such as IP-Adapter.

[0085] Beneficial effects:

[0086] 1. The present invention utilizes the cross attention mechanism in the diffusion model to influence the self-attention map to enhance role consistency;

[0087] 2. The present invention uses the segmentation map obtained by the cross-attention mechanism in the diffusion model to fuse the global prompt words and the role prompt words to further enhance the role consistency and alleviate the problem of feature leakage between different roles.

[0088] Furthermore, if Figure 4 As shown, based on the above-mentioned method for generating a story comic with consistency, the present invention also provides a system for generating a story comic with consistency, wherein the system for generating a story comic with consistency includes:

[0089] A storyboard processing module 51 is used to determine a target comic prompt word, and perform comic story generation processing and storyboard processing according to the target comic prompt word to obtain a plurality of storyboard prompt words and a plurality of character prompt words corresponding to each storyboard prompt word;

[0090] A cross-attention map generating module 52 is used to obtain a preset diffusion model, input each of the storyboard prompt words and the corresponding multiple character prompt words into the preset diffusion model, and obtain a cross-attention map;

[0091] A segmentation mask processing module 53 is used to perform segmentation mask processing on the cross attention map to obtain a character segmentation mask;

[0092] An isolated self-attention map generation module 54 is used to construct an extended attention map according to the cross attention map and the role segmentation mask, and perform mask assignment processing and attention weight allocation processing on the extended attention map to obtain an isolated self-attention map;

[0093] The target comic image generation module 55 is used to perform cross-attention feature extraction processing on the isolated self-attention map to obtain scene cross-attention features and character cross-attention features, perform weighted fusion processing on the scene cross-attention features and the character cross-attention features according to the character segmentation mask to obtain fusion features, and perform iterative diffusion denoising processing on the fusion features according to the preset diffusion model to obtain a target comic image with a consistent story.

[0094] Furthermore, if Figure 5 As shown, based on the above-mentioned method and system for generating a consistent story comic, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 5 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0095] The memory 20 may be an internal storage unit of the terminal in some embodiments, such as a hard disk or memory of the terminal. The memory 20 may also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed in the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a generation program 40 with a consistent story comic is stored on the memory 20, and the generation program 40 with a consistent story comic can be executed by the processor 10, thereby realizing the generation method of a consistent story comic in the present application.

[0096] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the method for generating a consistent story comic.

[0097] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0098] In one embodiment, when the processor 10 executes the generating program 40 of the consistent story comic in the memory 20, the following steps are implemented:

[0099] Determine a target comic prompt, and perform comic story generation processing and storyboard processing according to the target comic prompt to obtain a plurality of storyboard prompt words and a plurality of character prompt words corresponding to each storyboard prompt word;

[0100] Obtaining a preset diffusion model, inputting each of the storyboard prompt words and the corresponding plurality of character prompt words into the preset diffusion model to obtain a cross-attention map;

[0101] Performing segmentation mask processing on the cross attention map to obtain a character segmentation mask;

[0102] Constructing an extended attention map according to the cross attention map and the role segmentation mask, and performing mask assignment processing and attention weight allocation processing on the extended attention map to obtain an isolated self-attention map;

[0103] The isolated self-attention map is subjected to cross-attention feature extraction processing to obtain scene cross-attention features and character cross-attention features, the scene cross-attention features and the character cross-attention features are weighted fused according to the character segmentation mask to obtain fused features, and the fused features are iteratively diffused denoising processed according to the preset diffusion model to obtain a target comic image with a consistent story.

[0104] The step of determining a target comic prompt, and performing comic story generation processing and storyboard processing according to the target comic prompt to obtain a plurality of storyboard prompt words and a plurality of character prompt words corresponding to each storyboard prompt word specifically includes:

[0105] Determine a target comic prompt and a preset macro model, input the target comic prompt into the preset macro model, and perform comic story generation processing on the target comic prompt through the preset macro model to obtain a target comic story;

[0106] The target comic story is storyboarded to obtain a plurality of storyboard prompt words and a plurality of character prompt words corresponding to each storyboard prompt word.

[0107] The preset diffusion model includes an original branch and an extended branch; the original branch includes a self-attention mechanism module and a cross-attention module; the extended branch includes an ID isolation self-attention mechanism module and an ID isolation cross-attention module;

[0108] The step of obtaining a preset diffusion model and inputting each of the storyboard prompt words and the corresponding plurality of role prompt words into the preset diffusion model to obtain a cross attention map specifically includes:

[0109] Obtaining a preset diffusion model, inputting each of the storyboard prompt words and the corresponding multiple character prompt words into the preset diffusion model to obtain multiple input feature maps;

[0110] The multiple input feature maps are sequentially input into the self-attention mechanism module and the cross-attention module of the original branch to obtain a cross-attention map.

[0111] The step of performing segmentation mask processing on the cross attention map to obtain a character segmentation mask specifically includes:

[0112] The maximum inter-class variance method is used to calculate the inter-class variance of the cross-attention map, and the optimal segmentation threshold is calculated according to the inter-class variance to obtain the character segmentation mask;

[0113] Among them, the expression of the between-class variance is:

[0114]

[0115] Where T is the segmentation threshold, is the inter-class variance, ω1(T) is the weight of the first segmentation area, ω2(T) is the weight of the second segmentation area, μ1(T) is the mean of the first segmentation area, and μ2(T) is the mean of the second segmentation area;

[0116] Among them, the expression of the optimal segmentation threshold is:

[0117]

[0118] Among them, T * is the optimal segmentation threshold, Solve for the optimal segmentation threshold.

[0119] The step of constructing an extended attention map according to the cross attention map and the role segmentation mask, and performing mask assignment processing and attention weight allocation processing on the extended attention map to obtain an isolated self-attention map specifically includes:

[0120] Inputting the plurality of input feature maps, the cross attention map, and the role segmentation mask into the ID isolation self-attention mechanism module of the extended branch to obtain a target role name;

[0121] Obtaining a preset role database, and searching the preset role database according to the target role name;

[0122] If the search is successful, the target role feature corresponding to the target role name is obtained through the preset role database, and an extended attention graph is constructed according to the target role name and the target role feature;

[0123] The extended attention map is subjected to mask assignment processing and attention weight allocation processing according to the cross attention map to obtain an isolated self-attention map.

[0124] The step of obtaining a preset role database and searching the preset role database according to the target role name may further include:

[0125] If the search fails, the screening role feature corresponding to the target role name is calculated according to the role segmentation mask, and the target role name and the corresponding screening role feature are stored in the preset role database.

[0126] Among them, the cross-attention feature extraction processing is performed on the isolated self-attention map to obtain scene cross-attention features and character cross-attention features, the scene cross-attention features and the character cross-attention features are weighted fused according to the character segmentation mask to obtain fused features, and the fused features are iteratively diffused and denoised according to the preset diffusion model to obtain a target comic image with a consistent story, specifically including:

[0127] Inputting the isolated self-attention map into the ID isolation cross attention module of the extended branch, performing cross attention feature extraction processing on the isolated self-attention map through the ID isolation cross attention module to obtain scene cross attention features and character cross attention features;

[0128] Performing weighted fusion processing on the scene cross-attention feature and the character cross-attention feature according to the character segmentation mask to obtain a fused feature;

[0129] The fused features are input into the diffusion model again, and the fused features are iteratively subjected to diffusion denoising through the diffusion model to obtain a target comic image with a consistent story.

[0130] In summary, the present invention provides a method, system and terminal for generating a comic with a consistent story, the method comprising: determining a target comic prompt, and performing comic story generation processing and storyboard processing according to the target comic prompt to obtain a plurality of storyboard prompt words and a plurality of role prompt words corresponding to each storyboard prompt word; obtaining a preset diffusion model, inputting each of the storyboard prompt words and the corresponding plurality of role prompt words into the preset diffusion model to obtain a cross-attention map; performing segmentation mask processing on the cross-attention map to obtain a role segmentation mask; constructing an extended attention map according to the cross-attention map and the role segmentation mask, and performing mask assignment processing and attention weight allocation processing on the extended attention map to obtain an isolated self-attention map; performing cross-attention feature extraction processing on the isolated self-attention map to obtain a scene cross-attention feature and a role cross-attention feature, performing weighted fusion processing on the scene cross-attention feature and the role cross-attention feature according to the role segmentation mask to obtain a fusion feature, and performing iterative diffusion denoising processing on the fusion feature according to the preset diffusion model to obtain a target comic image with a consistent story. The present invention realizes consistent story comic generation through a large language model and a diffusion model. By using a large model to realize storyboarding, modifying the structure of the diffusion model, adding ID isolation self-attention and ID isolation cross-attention, the identity consistency of multiple characters in the generated comic images is ensured, and the attribute mixing between multiple characters is reduced, which significantly improves the visual quality and story coherence of the generated comics. It not only improves the efficiency of automatic comic generation, but also expands its application scope, and is suitable for comic creation of various styles and plots.

[0131] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0132] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0133] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for generating a comic strip with a consistent story, characterized in that: The method for generating a comic strip with a consistent story includes: Determine a target comic prompt, and perform comic story generation processing and storyboard processing according to the target comic prompt to obtain a plurality of storyboard prompt words and a plurality of character prompt words corresponding to each storyboard prompt word; Obtaining a preset diffusion model, inputting each of the storyboard prompt words and the corresponding plurality of character prompt words into the preset diffusion model to obtain a cross-attention map; Performing segmentation mask processing on the cross attention map to obtain a character segmentation mask; Constructing an extended attention map according to the cross attention map and the role segmentation mask, and performing mask assignment processing and attention weight allocation processing on the extended attention map to obtain an isolated self-attention map; The isolated self-attention map is subjected to cross-attention feature extraction processing to obtain scene cross-attention features and character cross-attention features, the scene cross-attention features and the character cross-attention features are weighted fused according to the character segmentation mask to obtain fused features, and the fused features are iteratively diffused denoising processed according to the preset diffusion model to obtain a target comic image with a consistent story.

2. The method for generating a comic strip with a consistent story according to claim 1, characterized in that: The step of determining a target comic prompt word, and performing comic story generation processing and storyboard processing according to the target comic prompt word to obtain a plurality of storyboard prompt words and a plurality of character prompt words corresponding to each storyboard prompt word specifically includes: Determine a target comic prompt and a preset macro model, input the target comic prompt into the preset macro model, and perform comic story generation processing on the target comic prompt through the preset macro model to obtain a target comic story; The target comic story is storyboarded to obtain a plurality of storyboard prompt words and a plurality of character prompt words corresponding to each storyboard prompt word.

3. The method for generating a comic strip with a consistent story according to claim 2, characterized in that: The preset diffusion model includes an original branch and an extended branch; the original branch includes a self-attention mechanism module and a cross-attention module; the extended branch includes an ID isolation self-attention mechanism module and an ID isolation cross-attention module; The step of obtaining a preset diffusion model and inputting each of the storyboard prompt words and the corresponding plurality of role prompt words into the preset diffusion model to obtain a cross attention map specifically includes: Obtaining a preset diffusion model, inputting each of the storyboard prompt words and the corresponding multiple character prompt words into the preset diffusion model to obtain multiple input feature maps; The multiple input feature maps are sequentially input into the self-attention mechanism module and the cross-attention module of the original branch to obtain a cross-attention map.

4. The method for generating a comic strip with a consistent story according to claim 1, characterized in that: The segmentation mask processing is performed on the cross attention map to obtain the role segmentation mask, specifically including: The maximum inter-class variance method is used to calculate the inter-class variance of the cross-attention map, and the optimal segmentation threshold is calculated according to the inter-class variance to obtain the character segmentation mask; Among them, the expression of the between-class variance is: Where T is the segmentation threshold, is the inter-class variance, ω1(T) is the weight of the first segmentation area, ω2(T) is the weight of the second segmentation area, μ1(T) is the mean of the first segmentation area, and μ2(T) is the mean of the second segmentation area; Among them, the expression of the optimal segmentation threshold is: Among them, T * is the optimal segmentation threshold, Solve for the optimal segmentation threshold.

5. The method for generating a comic strip with a consistent story according to claim 3, characterized in that: The step of constructing an extended attention map according to the cross attention map and the role segmentation mask, and performing mask assignment processing and attention weight allocation processing on the extended attention map to obtain an isolated self-attention map specifically includes: Inputting the plurality of input feature maps, the cross attention map, and the role segmentation mask into the ID isolation self-attention mechanism module of the extended branch to obtain a target role name; Obtaining a preset role database, and searching the preset role database according to the target role name; If the search is successful, the target role feature corresponding to the target role name is obtained through the preset role database, and an extended attention graph is constructed according to the target role name and the target role feature; The extended attention map is subjected to mask assignment processing and attention weight allocation processing according to the cross attention map to obtain an isolated self-attention map.

6. The method for generating a comic strip with a consistent story according to claim 5, characterized in that: The step of obtaining a preset role database and searching the preset role database according to the target role name further includes: If the search fails, the screening role feature corresponding to the target role name is calculated according to the role segmentation mask, and the target role name and the corresponding screening role feature are stored in the preset role database.

7. The method for generating a comic strip with a consistent story according to claim 3, characterized in that: The cross-attention feature extraction process is performed on the isolated self-attention map to obtain a scene cross-attention feature and a character cross-attention feature, the scene cross-attention feature and the character cross-attention feature are weightedly fused according to the character segmentation mask to obtain a fused feature, and the fused feature is iteratively diffused and denoised according to the preset diffusion model to obtain a target comic image with a consistent story, specifically including: Inputting the isolated self-attention map into the ID isolation cross attention module of the extended branch, performing cross attention feature extraction processing on the isolated self-attention map through the ID isolation cross attention module to obtain scene cross attention features and character cross attention features; Performing weighted fusion processing on the scene cross-attention feature and the character cross-attention feature according to the character segmentation mask to obtain a fusion feature; The fused features are input into the diffusion model again, and the fused features are iteratively subjected to diffusion denoising through the diffusion model to obtain a target comic image with a consistent story.

8. A system for generating comic strips with consistent stories, characterized in that: The generation system of the consistent story comics comprises: A storyboard processing module is used to determine a target comic prompt word, and perform comic story generation processing and storyboard processing according to the target comic prompt word to obtain a plurality of storyboard prompt words and a plurality of character prompt words corresponding to each storyboard prompt word; A cross-attention map generation module is used to obtain a preset diffusion model, input each of the storyboard prompt words and the corresponding multiple role prompt words into the preset diffusion model to obtain a cross-attention map; A segmentation mask processing module, used to perform segmentation mask processing on the cross attention map to obtain a character segmentation mask; An isolated self-attention map generation module is used to construct an extended attention map according to the cross attention map and the role segmentation mask, and perform mask assignment processing and attention weight allocation processing on the extended attention map to obtain an isolated self-attention map; The target comic image generation module is used to perform cross-attention feature extraction processing on the isolated self-attention map to obtain scene cross-attention features and character cross-attention features, perform weighted fusion processing on the scene cross-attention features and the character cross-attention features according to the character segmentation mask to obtain fusion features, and perform iterative diffusion denoising processing on the fusion features according to the preset diffusion model to obtain a target comic image with a consistent story.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a generation program for consistent story comics stored in the memory and executable on the processor. When the generation program for consistent story comics is executed by the processor, the steps of the generation method for consistent story comics as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program for generating a consistent story comic, and when the program for generating a consistent story comic is executed by a processor, the steps of the method for generating a consistent story comic as described in any one of claims 1-7 are implemented.

Images