A method, system and terminal for generating a consistent story comic

Abstract

The application discloses a generation method, system and terminal of a consistent story comic, and the method comprises the following steps: determining a target comic prompt, performing comic story generation processing and shot processing to obtain a plurality of shot prompt words and a plurality of character prompt words; acquiring a preset diffusion model, inputting each shot prompt word and the plurality of character prompt words into the preset diffusion model to obtain a cross attention graph; performing segmentation mask processing on the cross attention graph to obtain a character segmentation mask; constructing an extended attention graph according to the cross attention graph and the character segmentation mask, and performing mask assignment processing and attention weight distribution processing to obtain an isolated self-attention graph; performing cross attention feature extraction processing and weighted fusion processing on the isolated self-attention graph to obtain fusion features, and performing iterative diffusion denoising processing on the fusion features to obtain a target comic image with consistency. The application can effectively improve the visual quality of automatic comic generation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a generation method, system, terminal and computer readable storage medium of a consistent story comic. BACKGROUND

[0002] As a visual narrative form, comics have a wide audience. However, the traditional comic creation method relies on manual drawing, which is not only time-consuming and laborious, but also requires high skills of artists. With the rapid development of artificial intelligence, automated comic generation technology has gradually become a research hotspot. Automated comic generation technology has made certain progress in improving comic generation efficiency, but still has many challenges.

[0003] The prior art is difficult to maintain the consistency of the image of the character between different comic frames. This consistency is the key to telling a coherent story. However, due to the lack of cross-frame correlation information when the generation model processes multiple frames of images, the image of the character in different frames may be inconsistent. When multiple characters appear in the same frame, the model may not effectively distinguish and maintain the unique features of each character, resulting in confusion of character features and loss of details in the generated image, which seriously affects the overall quality of the comic.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] The main purpose of the present application is to provide a generation method, system, terminal and computer readable storage medium of a consistent story comic, which aims to solve the problem that the automated comic generation model in the prior art lacks cross-frame correlation information when processing multiple frames of images, and when multiple characters appear in the same frame, the model cannot effectively distinguish and maintain the unique features of each character, resulting in confusion of character features and loss of details in the generated comic image, which seriously affects the overall quality of the comic.

[0006] To achieve the above purpose, the present application provides a generation method of a consistent story comic, which comprises the following steps:

[0007] determining a target comic prompt, and performing comic story generation processing and shot processing according to the target comic prompt to obtain multiple shot prompts and multiple character prompts corresponding to each shot prompt;

[0008] acquiring a preset diffusion model, inputting each shot prompt and the corresponding multiple character prompts into the preset diffusion model to obtain a cross-attention map;

[0009] performing segmentation mask processing on the cross-attention map to obtain a character segmentation mask;

[0010] According to the cross attention map and the role segmentation mask, an extended attention map is constructed, and the extended attention map is subjected to mask assignment processing and attention weight distribution processing to obtain an isolated self-attention map;

[0011] The isolated self-attention map is subjected to cross attention feature extraction processing to obtain scene cross attention features and role cross attention features, the scene cross attention features and the role cross attention features are subjected to weighted fusion processing according to the role segmentation mask to obtain fusion features, and the fusion features are subjected to iterative diffusion denoising processing according to the preset diffusion model to obtain a target comic image with a consistent story.

[0012] Optionally, the method for generating a comic with a consistent story, wherein the target comic prompt is determined, and comic story generation processing and shot processing are performed according to the target comic prompt to obtain a plurality of shot prompts and a plurality of role prompts corresponding to each shot prompt, specifically comprising:

[0013] The target comic prompt and a preset large model are determined, the target comic prompt is input into the preset large model, and comic story generation processing is performed on the target comic prompt by the preset large model to obtain a target comic story;

[0014] The target comic story is subjected to shot processing to obtain a plurality of shot prompts and a plurality of role prompts corresponding to each shot prompt.

[0015] Optionally, the method for generating a comic with a consistent story, wherein the preset diffusion model comprises an original branch and an extended branch; the original branch comprises a self-attention mechanism module and a cross-attention module; and the extended branch comprises an ID isolated self-attention mechanism module and an ID isolated cross-attention module.

[0016] The preset diffusion model is obtained, each shot prompt and a plurality of corresponding role prompts are input into the preset diffusion model to obtain a cross-attention map, specifically comprising:

[0017] The preset diffusion model is obtained, each shot prompt and a plurality of corresponding role prompts are input into the preset diffusion model to obtain a plurality of input feature maps;

[0018] The plurality of input feature maps are sequentially input into the self-attention mechanism module and the cross-attention module of the original branch to obtain a cross-attention map.

[0019] Optionally, the method for generating a consistent story comic, wherein the cross-attention map is subjected to segmentation mask processing to obtain a character segmentation mask, specifically comprising:

[0020] The inter-class variance of the cross-attention map is calculated by using the maximum inter-class variance method, and the optimal segmentation threshold is calculated according to the inter-class variance to obtain the character segmentation mask.

[0021] The expression of the inter-class variance is:

[0022]

[0023] wherein T is the segmentation threshold, is the inter-class variance, ω1(T) is the first segmentation region weight, ω2(T) is the second segmentation region weight, μ1(T) is the first segmentation region mean, and μ2(T) is the second segmentation region mean.

[0024] The expression of the optimal segmentation threshold is:

[0025]

[0026] wherein T * is the optimal segmentation threshold, is the solution of the optimal segmentation threshold.

[0027] Optionally, the method for generating a consistent story comic, wherein the extended attention map is constructed according to the cross-attention map and the character segmentation mask, and the extended attention map is subjected to mask assignment processing and attention weight distribution processing to obtain an isolated self-attention map, specifically comprising:

[0028] The input feature map, the cross-attention map, and the character segmentation mask are input into the ID isolated self-attention mechanism module of the extended branch to obtain a target character name.

[0029] A preset character database is obtained, and the target character name is searched in the preset character database.

[0030] If the search is successful, the target character feature corresponding to the target character name is obtained from the preset character database, and the extended attention map is constructed according to the target character name and the target character feature.

[0031] The extended attention map is subjected to mask assignment processing and attention weight distribution processing according to the cross-attention map to obtain an isolated self-attention map.

[0032] Optionally, the generation method of the comic with a consistent story, wherein the preset role database is obtained, and the target role name is searched in the preset role database, and then the method further comprises:

[0033] If the search fails, the filtering role feature corresponding to the target role name is calculated according to the role segmentation mask, and the target role name and the corresponding filtering role feature are stored in the preset role database.

[0034] Optionally, the generation method of the comic with a consistent story, wherein the cross-attention feature extraction processing is performed on the isolated self-attention graph to obtain scene cross-attention features and role cross-attention features, the scene cross-attention features and the role cross-attention features are weighted and fused according to the role segmentation mask to obtain fusion features, and the fusion features are iteratively diffused and denoised according to the preset diffusion model to obtain a target comic image with a consistent story, and the method specifically comprises:

[0035] The isolated self-attention graph is input into the ID isolated cross-attention module of the extended branch, the cross-attention feature extraction processing is performed on the isolated self-attention graph through the ID isolated cross-attention module to obtain scene cross-attention features and role cross-attention features;

[0036] The scene cross-attention features and the role cross-attention features are weighted and fused according to the role segmentation mask to obtain fusion features;

[0037] The fusion features are input into the diffusion model again, and the fusion features are iteratively diffused and denoised through the diffusion model to obtain a target comic image with a consistent story.

[0038] In addition, to achieve the above-mentioned purpose, the application further provides a generation system of a comic with a consistent story, wherein the generation system of the comic with a consistent story comprises:

[0039] A shot processing module is configured to determine a target comic prompt, and perform comic story generation processing and shot processing according to the target comic prompt to obtain a plurality of shot prompts and a plurality of role prompts corresponding to each shot prompt;

[0040] A cross-attention graph generation module is configured to obtain a preset diffusion model, input each shot prompt and the corresponding plurality of role prompts into the preset diffusion model, and obtain a cross-attention graph;

[0041] A segmentation mask processing module is configured to perform segmentation mask processing on the cross-attention graph to obtain a role segmentation mask;

[0042] The isolated self-attention map generation module is configured to construct an extended attention map according to the cross-attention map and the role segmentation mask, and perform mask assignment processing and attention weight distribution processing on the extended attention map to obtain an isolated self-attention map.

[0043] The target comic image generation module is configured to perform cross-attention feature extraction processing on the isolated self-attention map to obtain scene cross-attention features and role cross-attention features, perform weighted fusion processing on the scene cross-attention features and the role cross-attention features according to the role segmentation mask to obtain fusion features, and perform iterative diffusion denoising processing on the fusion features according to the preset diffusion model to obtain a target comic image with a consistent story.

[0044] In the present application, the target comic prompt is determined, and comic story generation processing and shot processing are performed according to the target comic prompt to obtain a plurality of shot prompt words and a plurality of role prompt words corresponding to each shot prompt word; a preset diffusion model is obtained, each shot prompt word and the corresponding plurality of role prompt words are input into the preset diffusion model to obtain a cross-attention map; the cross-attention map is subjected to segmentation mask processing to obtain a role segmentation mask; an extended attention map is constructed according to the cross-attention map and the role segmentation mask, and mask assignment processing and attention weight distribution processing are performed on the extended attention map to obtain an isolated self-attention map; cross-attention feature extraction processing is performed on the isolated self-attention map to obtain scene cross-attention features and role cross-attention features, weighted fusion processing is performed on the scene cross-attention features and the role cross-attention features according to the role segmentation mask to obtain fusion features, and iterative diffusion denoising processing is performed on the fusion features according to the preset diffusion model to obtain a target comic image with a consistent story. The present application realizes consistent story comic generation through a large language model and a diffusion model, realizes shot through the use of a large model, modifies the structure of the diffusion model, adds ID isolated self-attention and ID isolated cross-attention, ensures the multi-role identity consistency of the generated comic image, and reduces the attribute mixing between multiple roles, significantly improving the visual quality and story coherence of the generated comic. Not only does it improve the efficiency of automatic comic generation, but it also expands its application range and is suitable for comic creation of various styles and plots. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a schematic diagram of a two-dimensional contour line of a preferred embodiment of the generation method of the comic with a consistent story of the present application;

[0046] Figure 2is a schematic diagram of generating a story, a shot prompt word, and a character prompt word by a large language model, which is a preferred embodiment of the generation method of the consistent story comic of the present application;

[0047] Figure 3 is a schematic diagram of a diffusion model consistent comic generation overall structure, which is a preferred embodiment of the generation method of the consistent story comic of the present application;

[0048] Figure 4 is a structure diagram of a preferred embodiment of the generation system of the consistent story comic of the present application;

[0049] Figure 5 is a structure diagram of a preferred embodiment of the terminal of the present application. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present application clearer and more explicit, the present application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0051] As a visual narrative form, comics have a wide audience. However, the traditional comic creation method relies on manual drawing, which not only takes time and effort, but also requires high artistic skills. With the rapid development of artificial intelligence, automated comic generation technology has gradually become a research hotspot. Currently, most automated comic generation technologies rely on generative adversarial networks (GANs) or pre-trained diffusion models. These methods have made some progress in improving comic generation efficiency, but still face many challenges.

[0052] Firstly, the shot of the existing comic usually needs to be set manually, making it difficult to achieve automated generation. Secondly, the existing methods have difficulty in maintaining the consistency of the character image between different comic frames, which is crucial for telling a coherent story. However, due to the lack of cross-frame correlation information when generating multiple images, the character image may not be consistent in different frames. This not only affects the coherence of the story, but also reduces the user's reading experience. Finally, when generating a comic containing multiple characters, the existing model often faces the problem of character feature confusion. When multiple characters appear in the same frame, the model may not be able to effectively distinguish and maintain the unique features of each character, resulting in confusion of character features and loss of details in the generated image. This problem is particularly prominent in complex scenes or multiple character interactions, seriously affecting the overall quality of the comic.

[0053] In addition, although some methods attempt to train the model to remember common characters and backgrounds through specific data sets to achieve character consistency, the effectiveness of these methods is often limited by the size and diversity of the data set, which cannot meet the diverse needs in open worlds, making it difficult for existing technologies to generate comics of different styles and different plots in practical applications, limiting their potential for widespread application.

[0054] To solve the above problems, the present application proposes an innovative comic generation technology. By combining a large language model and a pre-trained diffusion model, the present application can effectively solve the problem of inconsistent character image and character feature confusion, and significantly improve the visual quality and story coherence of generated comics. This technology not only improves the efficiency of automated comic generation, but also expands its application range, suitable for comic creation of various styles and plots.

[0055] The present application belongs to the field of image generation technology, and the main difficulty it faces is how to generate a coherent comic with logical and story-telling characters. The present application uses a large language model to assist in shot division and self-attention and cross-attention mechanisms in the text-to-image diffusion model to achieve consistent generation. The present application proposes a technical framework for automatically generating story comics with consistent appearance. The framework mainly includes two parts: story processing using a large language model and generating consistent comics using an improved text-to-image diffusion model. First, the large language model divides the story into multiple shots and generates shot cues and character cues. Second, the present application makes two important improvements to the pre-trained text-to-image diffusion model: 1. After obtaining the object mask through cross-attention mechanism, control the self-attention mechanism to make the generated characters more refer to the same character in the previous image, thereby improving the consistency of the characters. 2. Generate character features and overall features in the cross-attention mechanism, and fuse these features through the mask to avoid confusion of character attributes and further improve the consistency of the characters. In this way, the entire system can automatically generate coherent story comics with consistent appearance.

[0056] The generation method of the consistent story comic according to the preferred embodiment of the present application, as shown in Figure 1 The generation method of the consistent story comic includes the following steps:

[0057] Step S10, determine the target comic prompt, and perform comic story generation processing and shot division processing according to the target comic prompt, to obtain multiple shot cues and multiple character cues corresponding to each shot cue.

[0058] Specifically, the target comic prompt and a preset large model are determined. The target comic prompt is input into the preset large model, and the target comic prompt is processed by the preset large model to generate a comic story, thereby obtaining the target comic story. The target comic story is then processed by storyboarding to obtain multiple storyboard prompts and multiple character prompts corresponding to each storyboard prompt.

[0059] like Figure 2 As shown, this invention utilizes a large model to generate a story and automatically generates storyboards and storyboard descriptors S = {S1, S2, ..., S...}. N (A total of N storyboards) and character descriptions for each character in the story C = {C1, C2, ..., C...} K (There are K characters in total), character descriptions for each storyboard. It may contain zero or more of these characters, represented by n. i This represents the number of characters in the i-th storyboard, hence the relation n i =|C i Additionally, this invention also supports storyboarding for a given story (i.e., an existing story that does not require large model generation).

[0060] Step S20: Obtain a preset diffusion model. Input each of the storyboard prompt words and the corresponding multiple character prompt words into the preset diffusion model to obtain a cross-attention map.

[0061] Specifically, a preset diffusion model is obtained, and each of the storyboard prompt words and the corresponding multiple character prompt words are input into the preset diffusion model to obtain multiple input feature maps; the multiple input feature maps are sequentially input into the self-attention mechanism module and the cross-attention module of the original branch to obtain a cross-attention map.

[0062] like Figure 3 As shown in a, the i-th segment of the random noise Z is... i The storyboard cues and the character cues corresponding to the characters included in that storyboard shot C i They are fed into the diffusion model together. This invention also requires specifying an index here. i (Represents the index of all characters in the i-th storyboard), New_index i (representing the index of the new character in the i-th storyboard) and Old_index i (Index of the old character in the i-th storyboard).

[0063] The preset diffusion model includes an original branch and an extended branch; the original branch includes a self-attention mechanism module and a cross-attention module; the extended branch includes an ID-isolated self-attention mechanism module and an ID-isolated cross-attention module.

[0064] like Figure 3 As shown in b, in the upsampling of the diffusion model, a Transformer module contains a self-attention mechanism and a cross-attention mechanism (i.e., ... Figure 3 (The original branch in b in the original text). In self-attention, the similarity between image features is calculated, called a self-attention map, and the image features are updated with weights based on the similarity. In cross-attention, the image features are compared with the input text features (i.e., the target comic story in this invention) to calculate a cross-attention map, and then the attention map is used as weights to weight the text features as new image features, thereby guiding the generation in the direction indicated by the text. The input I of the Transformer is a matrix (i.e., the input feature map in this invention), with the shape [h*w, dim], where h and w are the height and width of the image features, and dim is the feature dimension (similar to an RGB channel, but the dimension here is much greater than 3). The input I obtains the output O through the self-attention and cross-attention of the Transformer block. origin At the same time, the attention map of the corresponding role in the cross-attention (i.e., the cross-attention map in this invention) A = {A m}m∈index i This attention map roughly reflects the corresponding character's position in the comic.

[0065] Step S30: Perform segmentation masking on the cross attention map to obtain the role segmentation mask.

[0066] Specifically, the inter-class variance of the cross-attention map is calculated using the maximum inter-class variance method, and the optimal segmentation threshold is calculated based on the inter-class variance to obtain the role segmentation mask; wherein, the expression for the inter-class variance is: Where T is the segmentation threshold. Let ω1(T) be the inter-class variance, ω2(T) be the weight of the first segmentation region, ω2(T) be the weight of the second segmentation region, μ1(T) be the mean of the first segmentation region, and μ2(T) be the mean of the second segmentation region; where the expression for the optimal segmentation threshold is: Among them, T * To achieve the optimal segmentation threshold, Solve for the optimal segmentation threshold.

[0067] This invention obtains the role segmentation mask M = {M} for each attention map in A (i.e., the cross-attention map) using the Otsu algorithm (maximum inter-class variance method). m}m∈index i The core idea of ​​the Otsu algorithm is to automatically determine the segmentation threshold T of an image by maximizing the inter-class variance. The Otsu algorithm needs to find a threshold that minimizes the inter-class variance. Maximum, T * The optimal segmentation threshold is the value greater than T in attention map A. * Set part of it to 1, less than T * Setting a portion of the mask to 0 yields the character segmentation mask M. m .

[0068] Due to data discrepancies during the training of the diffusion model, the diffusion model exhibits varying attention levels for each word. Furthermore, previous research has shown that attention maps with cross-attention contain noise. This invention discovers that attention maps with weaker noise tend to capture objects more accurately, while attention maps with stronger noise typically show higher attention levels but are less accurate in capturing objects. Therefore, this invention calculates the coefficient of variation for each attention map, prioritizing those with lower coefficients. If different role masks overlap, the higher-priority role mask is considered.

[0069] Step S40: Construct an extended attention graph based on the cross attention graph and the role segmentation mask, and perform mask assignment and attention weight allocation on the extended attention graph to obtain an isolated self-attention graph.

[0070] Specifically, multiple input feature maps, the cross-attention map, and the role segmentation mask are input into the ID isolation self-attention mechanism module of the extended branch to obtain the target role name; a preset role database is obtained, and a search is performed in the preset role database according to the target role name; if the search is successful, the target role features corresponding to the target role name are obtained through the preset role database, and an extended attention map is constructed according to the target role name and the target role features; the extended attention map is processed by mask assignment and attention weight allocation according to the cross-attention map to obtain the isolation self-attention map.

[0071] Furthermore, if the search fails, the filter character features corresponding to the target character name are calculated based on the character segmentation mask, and the target character name and the corresponding filter character features are stored in the preset character database.

[0072] like Figure 3 As shown in b, the self-attention mechanism of the Transformer block is to input the input feature map, the cross-attention map obtained in the cross-attention module, and the role segmentation mask again. Figure 3 The extended branch in b includes Figure 3 ID isolation self-attention in b and Figure 3ID isolation self-attention in d in Figure 3 d in is to Figure 3 refine the process of ID isolation self-attention in b). In the self-attention mechanism, a database composed of role names and role feature data pairs is saved, and whether the role appears for the first time is determined by indexing whether the role name appears in the database. If it is the first time (that is, the database does not contain this role name), the role feature obtained by the role segmentation mask and the corresponding role name are stored in the database to form a data pair. The filtering process of the role feature obtained by the role segmentation mask can be represented as: F m = I o M m (m e New_indexi), F m represents the role feature of role m, o represents multiplication of corresponding positions, and the formula represents filtering input I according to the role segmentation mask. If it is not the first time (that is, the database contains the role name), the role feature is called from the database as the K value and V value of the attention mechanism and is spliced behind the attention graph, so that the expressions of Q, K and V are: Q = Proj_q(I);

[0073] K = Proj_k[Concat(I, F m )] (m e Old_index i );

[0074] V = Proj_v[Concat(I, F m )] (m e Old_indexi);

[0075] The expression of the extended attention graph A is: A = Q x K; wherein, Proj_k has an identifier in d in Figure 3 , and Proj_k is a linear transformation matrix multiplication, which is a linear layer in deep learning. Concat represents splicing, and there are s features in input I and j features in F m , so that there are s+j features after splicing.

[0076] Through the above steps, the extended attention graph A is obtained, which has a shape of (h*w, h*w+∑ m n m ), wherein m e Old_index i , and n m represents the number of features stored by the role m. And the extended attention graph A corresponding to the role is modified according to the cross-attention graph to obtain the isolated self-attention graph, so as to achieve the purpose that the role feature refers to the features in the database as much as possible.

[0077] The modification to the extended attention map A specifically involves two methods: 1. Preventing regions other than this character from focusing on this character, thus preventing feature confusion, requires partially occluding the stitched attention map. This invention finds the position of the stitched character features: begin: begin+n m The attention map is assigned values ​​at the corresponding positions using a mask: A[:,begin:begin+n m ] = M m .repeat(1, n) m (m∈Old_index) i This is pseudocode. Originally, it's a vector of length h*w. `repeat(1, n)` copies this vector n times to create a matrix of length (h*w, n), and `repeat(n, 1)` creates a matrix of length (n, h*w). 2. Since the image background also affects the appearance of the subject, leading to reduced appearance consistency, it's necessary to manipulate the attention map to reduce the focus of the subject on the background. A cross-attention map can be used to change the self-attention weights, which can be represented as: A[M m ,:h*w)=A m .repeat(n m ,1)(m∈Old_index i These two operations effectively ensure the consistency of generated roles.

[0078] Step S50: Perform cross-attention feature extraction processing on the isolated self-attention map to obtain scene cross-attention features and character cross-attention features. Perform weighted fusion processing on the scene cross-attention features and the character cross-attention features according to the character segmentation mask to obtain fused features. Perform iterative diffusion denoising processing on the fused features according to the preset diffusion model to obtain a target comic image with a consistent story.

[0079] Optionally, the isolated self-attention map is input to the ID-isolated cross-attention module of the extended branch, and the ID-isolated cross-attention module performs cross-attention feature extraction processing on the isolated self-attention map to obtain scene cross-attention features and character cross-attention features; the scene cross-attention features and character cross-attention features are weighted and fused according to the character segmentation mask to obtain fused features; the fused features are input again to the diffusion model, and the diffusion model performs iterative diffusion denoising processing on the fused features to obtain a target comic image with a consistent story.

[0080] like Figure 3 As shown in c, the present invention will Figure 3 The output O obtained by d in SAThe picture features enter the cross attention module, and cross attention is performed between the picture features and the shot prompt words and the character prompt words to obtain multiple updated features and The scene cross attention features and the character cross attention features in the application) are fused with the character segmentation mask to achieve the purpose that the characters are mainly controlled by the character prompt words, and the feature confusion is prevented and the consistency is enhanced, and the formula can be expressed as: m is the index of the character, index i belongs to the set, and the mask of all characters is taken as a union.

[0081] Further, the fused feature map and the cross attention map are obtained by weighting processing to obtain the output of the Transformer module (i.e., the target comic image in the application). The expression of the weighting processing of the fused feature map and the cross attention map is:

[0082] O = O orig x lambda + O ISO x (1-lambda), the weight is used to control the consistency strength, if the output weight of the extended model is high, the consistency is high, the weight in the ID isolation cross attention is high, and the consistency is slightly weak, the parameter can be artificially controlled, and is usually set to 0.9.

[0083] Further, after obtaining the fused features, the fused features need to be input into the diffusion model again to realize the multi-step diffusion process of the fused features (i.e., multi-step denoising), and after the multi-step iterative denoising of the diffusion model, the target comic image with consistent stories is finally obtained.

[0084] The application uses a large language model and a diffusion model to realize the generation of a consistent story comic, uses a large model to realize shot, and modifies the structure of the diffusion model to increase the ID isolation self-attention and the ID isolation cross attention, so that the multi-character identity consistency of the generated comic image is ensured, and the attribute mixing between the multi-roles is reduced. In addition, the application can also realize appearance consistency through other trained plug-ins, such as IP-Adapter (IP adapter).

[0085] Advantages:

[0086] 1. The application uses the cross attention mechanism in the diffusion model to affect the self-attention map to enhance the character consistency.

[0087] 2. The application uses the segmentation map obtained by the cross attention mechanism in the diffusion model to fuse the global prompt words and the character prompt words to further enhance the character consistency and alleviate the feature leakage problem between different characters.

[0088] Further, as shown in Figure 4 Based on the above generation method of the comic with a consistent story, the present application also correspondingly provides a generation system of the comic with a consistent story, wherein the generation system of the comic with a consistent story comprises:

[0089] The shot processing module 51 is configured to determine a target comic prompt, and perform comic story generation processing and shot processing according to the target comic prompt, to obtain a plurality of shot prompts and a plurality of role prompts corresponding to each shot prompt.

[0090] The cross-attention map generation module 52 is configured to obtain a preset diffusion model, input each shot prompt and the plurality of role prompts corresponding thereto into the preset diffusion model, and obtain a cross-attention map.

[0091] The segmentation mask processing module 53 is configured to perform segmentation mask processing on the cross-attention map, to obtain a role segmentation mask.

[0092] The isolated self-attention map generation module 54 is configured to construct an extended attention map according to the cross-attention map and the role segmentation mask, and perform mask assignment processing and attention weight distribution processing on the extended attention map, to obtain an isolated self-attention map.

[0093] The target comic image generation module 55 is configured to perform cross-attention feature extraction processing on the isolated self-attention map, to obtain scene cross-attention features and role cross-attention features, perform weighted fusion processing on the scene cross-attention features and the role cross-attention features according to the role segmentation mask, to obtain fused features, and perform iterative diffusion denoising processing on the fused features according to the preset diffusion model, to obtain a target comic image with a consistent story.

[0094] Further, as shown in Figure 5 Based on the above generation method of the comic with a consistent story and the system thereof, the present application also correspondingly provides a terminal, which comprises a processor 10, a memory 20 and a display 30. Figure 5 Only part of the components of the terminal are shown, but it should be understood that all the shown components are not required to be implemented, and more or fewer components can be alternatively implemented.

[0095] The memory 20 can be an internal storage unit of the terminal in some embodiments, such as a hard disk or a memory of the terminal. The memory 20 can also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Further, the memory 20 can include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store application software and various data installed on the terminal, such as program codes of the terminal, etc. The memory 20 can also be used to temporarily store data that has been output or will be output. In an embodiment, the memory 20 stores a generation program 40 of a consistent story comic, which can be executed by the processor 10 to implement the generation method of a consistent story comic in the present application.

[0096] The processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, which is used to run program codes or process data stored in the memory 20, such as to execute the generation method of a consistent story comic, etc.

[0097] The display 30 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 30 is used to display information of the terminal and to display a visualized user interface. The components 10-30 of the terminal communicate with each other through a system bus.

[0098] In an embodiment, the following steps are implemented when the processor 10 executes the generation program 40 of a consistent story comic in the memory 20:

[0099] determining a target comic prompt, and performing comic story generation processing and shot processing according to the target comic prompt to obtain a plurality of shot prompts and a plurality of role prompts corresponding to each shot prompt;

[0100] obtaining a preset diffusion model, inputting each shot prompt and the plurality of role prompts corresponding thereto into the preset diffusion model to obtain a cross-attention graph;

[0101] performing mask processing on the cross-attention graph to obtain a role mask;

[0102] According to the cross attention graph and the role segmentation mask, an extended attention graph is constructed, and the extended attention graph is subjected to mask assignment processing and attention weight distribution processing to obtain an isolated self-attention graph;

[0103] The isolated self-attention graph is subjected to cross-attention feature extraction processing to obtain scene cross-attention features and role cross-attention features, the scene cross-attention features and the role cross-attention features are subjected to weighted fusion processing according to the role segmentation mask to obtain fusion features, and the fusion features are subjected to iterative diffusion denoising processing according to the preset diffusion model to obtain a target comic image with consistent stories.

[0104] The determination of the target comic prompt and the comic story generation processing and the shot processing according to the target comic prompt to obtain a plurality of shot prompt words and a plurality of role prompt words corresponding to each shot prompt word specifically include:

[0105] The target comic prompt and the preset large model are determined, the target comic prompt is input into the preset large model, and the target comic prompt is subjected to comic story generation processing by the preset large model to obtain a target comic story.

[0106] The target comic story is subjected to shot processing to obtain a plurality of shot prompt words and a plurality of role prompt words corresponding to each shot prompt word.

[0107] The preset diffusion model includes an original branch and an extended branch; the original branch includes a self-attention mechanism module and a cross-attention module; and the extended branch includes an ID isolated self-attention mechanism module and an ID isolated cross-attention module.

[0108] The preset diffusion model is obtained, each shot prompt word and the corresponding plurality of role prompt words are input into the preset diffusion model to obtain a cross-attention graph, and the method specifically includes:

[0109] The preset diffusion model is obtained, each shot prompt word and the corresponding plurality of role prompt words are input into the preset diffusion model to obtain a plurality of input feature maps.

[0110] The plurality of input feature maps are sequentially input into the self-attention mechanism module and the cross-attention module of the original branch to obtain a cross-attention graph.

[0111] The cross-attention graph is subjected to segmentation mask processing to obtain a role segmentation mask, and the method specifically includes:

[0112] An inter-class variance of the cross-attention map is calculated by using the maximum inter-class variance method, and an optimal segmentation threshold is calculated according to the inter-class variance, so as to obtain a role segmentation mask;

[0113] The expression of the inter-class variance is as follows:

[0114]

[0115] The expression of the inter-class variance is as follows: The expression of the inter-class variance is as follows: ω1(T) is a first segmentation region weight, ω2(T) is a second segmentation region weight, μ1(T) is a first segmentation region mean, and μ2(T) is a second segmentation region mean.

[0116] The expression of the optimal segmentation threshold is as follows:

[0117]

[0118] The expression of the optimal segmentation threshold is as follows: * The expression of the optimal segmentation threshold is as follows: The expression of the optimal segmentation threshold is as follows.

[0119] The expression of the optimal segmentation threshold is as follows.

[0120] The input feature map, the cross-attention map and the role segmentation mask are input into the ID isolated self-attention mechanism module of the expansion branch, so as to obtain a target role name.

[0121] A preset role database is obtained, and the target role name is searched in the preset role database.

[0122] If the search is successful, a target role feature corresponding to the target role name is obtained from the preset role database, and an expansion attention map is constructed according to the target role name and the target role feature.

[0123] The expansion attention map is processed by mask assignment and attention weight distribution according to the cross-attention map, so as to obtain an isolated self-attention map.

[0124] The preset role database is obtained, and the target role name is searched in the preset role database.

[0125] If the search fails, a screening role feature corresponding to the target role name is calculated according to the role segmentation mask, and the target role name and the corresponding screening role feature are stored in the preset role database.

[0126] The cross-attention feature extraction processing is performed on the isolated self-attention graph to obtain scene cross-attention features and role cross-attention features, the scene cross-attention features and the role cross-attention features are weighted and fused according to the role segmentation mask to obtain fusion features, and the fusion features are iteratively diffused and denoised according to the preset diffusion model to obtain the target comic image with a consistent story, and the specific operations include the following steps.

[0127] The isolated self-attention graph is input into the ID isolated cross-attention module of the extended branch, and the cross-attention feature extraction processing is performed on the isolated self-attention graph by the ID isolated cross-attention module to obtain scene cross-attention features and role cross-attention features.

[0128] The scene cross-attention features and the role cross-attention features are weighted and fused according to the role segmentation mask to obtain fusion features.

[0129] The fusion features are input into the diffusion model again, and the fusion features are iteratively diffused and denoised by the diffusion model to obtain the target comic image with a consistent story.

[0130] In summary, the present application provides a generation method, system and terminal of a consistent story comic, the method comprising: determining a target comic prompt, and performing comic story generation processing and shot processing according to the target comic prompt, to obtain a plurality of shot prompts and a plurality of role prompts corresponding to each shot prompt; obtaining a preset diffusion model, inputting each shot prompt and the corresponding plurality of role prompts into the preset diffusion model, to obtain a cross attention graph; performing segmentation mask processing on the cross attention graph, to obtain a role segmentation mask; constructing an extended attention graph according to the cross attention graph and the role segmentation mask, and performing mask assignment processing and attention weight distribution processing on the extended attention graph, to obtain an isolated self-attention graph; performing cross attention feature extraction processing on the isolated self-attention graph, to obtain scene cross attention features and role cross attention features, performing weighted fusion processing on the scene cross attention features and the role cross attention features according to the role segmentation mask, to obtain fused features, and performing iterative diffusion denoising processing on the fused features according to the preset diffusion model, to obtain a target comic image with a consistent story. The present application realizes consistent story comic generation through a large language model and a diffusion model, realizes shot through a large model, modifies the structure of the diffusion model, adds ID isolated self-attention and ID isolated cross attention, ensures the multi-role identity consistency of the generated comic image, reduces the attribute mixing between multiple roles, and significantly improves the visual quality and story coherence of the generated comic. Not only does the present application improve the efficiency of automatic comic generation, but also expands the application range of the automatic comic generation, and is suitable for comic creation of various styles and plots.

[0131] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusions, so that processes, methods, articles or terminals including a series of elements not only include those elements, but also include other elements not explicitly listed, or include elements inherent to such processes, methods, articles or terminals. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of another identical element in the process, method, article or terminal including the element.

[0132] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable computer-readable storage medium, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a magnetic disc, an optical disc, etc.

[0133] It is to be understood that the application is not limited to the examples described above, which can be modified or adapted in several ways by those skilled in the art without departing from the scope of the present application, as defined by the appended claims.

Claims

1. A method for generating a consistent story comic, characterized by, The generation method of the comic with a consistent story comprises the following steps: determining a target comic prompt, and performing comic story generation processing and shot processing according to the target comic prompt to obtain a plurality of shot prompts and a plurality of character prompts corresponding to each shot prompt; obtaining a preset diffusion model, inputting each shot prompt and the corresponding plurality of character prompts into the preset diffusion model to obtain a cross-attention graph; performing segmentation mask processing on the cross-attention graph to obtain a character segmentation mask; constructing an extended attention graph according to the cross-attention graph and the character segmentation mask, and performing mask assignment processing and attention weight distribution processing on the extended attention graph to obtain an isolated self-attention graph; performing cross-attention feature extraction processing on the isolated self-attention graph to obtain scene cross-attention features and character cross-attention features, performing weighted fusion processing on the scene cross-attention features and the character cross-attention features according to the character segmentation mask to obtain fused features, and performing iterative diffusion denoising processing on the fused features according to the preset diffusion model to obtain a target comic image with a consistent story.

2. The method of generating a consistent story comic according to claim 1, wherein, The determination of the target comic prompt and the performance of the comic story generation processing and the shot processing according to the target comic prompt to obtain the plurality of shot prompts and the plurality of character prompts corresponding to each shot prompt specifically comprises the following steps: determining a target comic prompt and a preset large model, inputting the target comic prompt into the preset large model, and performing comic story generation processing on the target comic prompt by the preset large model to obtain a target comic story; performing shot processing on the target comic story to obtain a plurality of shot prompts and a plurality of character prompts corresponding to each shot prompt.

3. The method of generating a consistent story comic according to claim 2, wherein, The preset diffusion model comprises an original branch and an extended branch; the original branch comprises a self-attention mechanism module and a cross-attention module; and the extended branch comprises an ID isolated self-attention mechanism module and an ID isolated cross-attention module. The obtaining of the preset diffusion model and the inputting of each shot prompt and the corresponding plurality of character prompts into the preset diffusion model to obtain a cross-attention graph specifically comprises the following steps: obtaining a preset diffusion model, inputting each shot prompt and the corresponding plurality of character prompts into the preset diffusion model to obtain a plurality of input feature maps; inputting the plurality of input feature maps into the self-attention mechanism module and the cross-attention module of the original branch in sequence to obtain a cross-attention graph.

4. The method of generating a consistent story comic according to claim 1, wherein, The segmentation mask processing on the cross-attention graph to obtain a character segmentation mask specifically comprises the following steps: calculating the inter-class variance of the cross-attention graph by using the maximum inter-class variance method, and performing optimal segmentation threshold calculation according to the inter-class variance to obtain a character segmentation mask; wherein the expression of the inter-class variance is: wherein T is a segmentation threshold, is the inter-class variance, ω1(T) is the first segmentation region weight, ω2(T) is the second segmentation region weight, μ1(T) is the first segmentation region mean, and μ2(T) is the second segmentation region mean. wherein the expression of the optimal segmentation threshold is: where T * is the optimal segmentation threshold, is the optimal segmentation threshold.

5. The method of generating a consistent story comic according to claim 3, wherein, The cross-attention graph and the role segmentation mask are used to construct an extended attention graph, and the extended attention graph is subjected to mask assignment processing and attention weight distribution processing to obtain an isolated self-attention graph, specifically including: The input feature map, the cross-attention graph and the role segmentation mask are input into the ID isolated self-attention mechanism module of the extended branch to obtain a target role name. A preset role database is obtained, and the target role name is searched in the preset role database. If the search is successful, the target role feature corresponding to the target role name is obtained from the preset role database, and the target role name and the target role feature are used to construct an extended attention graph. The cross-attention graph is used to perform mask assignment processing and attention weight distribution processing on the extended attention graph to obtain an isolated self-attention graph.

6. The method of generating a consistent story comic according to claim 5, wherein, The preset role database is obtained, and the target role name is searched in the preset role database. If the search fails, the target role feature corresponding to the target role name is calculated according to the role segmentation mask, and the target role name and the corresponding screening role feature are stored in the preset role database.

7. The method of generating a consistent story comic according to claim 3, wherein, The isolated self-attention graph is input into the ID isolated cross-attention module of the extended branch, and the isolated self-attention graph is subjected to cross-attention feature extraction processing by the ID isolated cross-attention module to obtain scene cross-attention features and role cross-attention features. The scene cross-attention features and the role cross-attention features are subjected to weighted fusion processing according to the role segmentation mask to obtain fusion features. The fusion features are input into the diffusion model again, and the fusion features are subjected to iterative diffusion denoising processing by the diffusion model to obtain a target comic image with a consistent story. The comic generation system with a consistent story includes:

8. A system for generating a consistent story comic, characterized by A shot processing module is configured to determine a target comic prompt and perform comic story generation processing and shot processing according to the target comic prompt to obtain a plurality of shot prompts and a plurality of role prompts corresponding to each shot prompt. A cross-attention graph generation module is configured to obtain a preset diffusion model, input each shot prompt and the corresponding plurality of role prompts into the preset diffusion model, and obtain a cross-attention graph. A segmentation mask processing module is configured to perform segmentation mask processing on the cross-attention graph to obtain a role segmentation mask. ​ The isolated self-attention map generation module is configured to construct an extended attention map according to the cross-attention map and the role segmentation mask, perform mask assignment processing and attention weight distribution processing on the extended attention map, and obtain an isolated self-attention map. The target comic image generation module is configured to perform cross-attention feature extraction processing on the isolated self-attention map, obtain scene cross-attention features and role cross-attention features, perform weighted fusion processing on the scene cross-attention features and the role cross-attention features according to the role segmentation mask, obtain fusion features, and perform iterative diffusion denoising processing on the fusion features according to the preset diffusion model, to obtain a target comic image with a consistent story.

9. A terminal, characterized by comprising: The terminal comprises a memory, a processor, and a program for generating a comic with a consistent story stored on the memory and capable of running on the processor. When the program for generating a comic with a consistent story is executed by the processor, the steps of the method for generating a comic with a consistent story according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program for generating a comic with a consistent story. When the program for generating a comic with a consistent story is executed by the processor, the steps of the method for generating a comic with a consistent story according to any one of claims 1-7 are implemented.

Images