Automatic cartoon generation method and system based on text graph model

Abstract

The embodiment of the invention provides an automatic cartoon generation method and system based on a text graph model, and the method can analyze the semantic information of descriptors from an input script through a multi-text encoder after the input script and posture control parameters are obtained. And generating composition data according to the semantic information. Wherein the composition data comprises a role template generated through role description word solidification and a sub-mirror composition parameter generated based on a sub-mirror cue word. And setting a space structure through a conditional generative adversarial network in combination with the posture control parameters, and generating a cartoon image according to the space structure and the composition data. According to the method, role consistency control can be realized based on the role template, the consistency of roles in multiple sub-mirrors in the cartoon image is improved, dynamic sub-mirror generation is realized in combination with the conditional generative adversarial network, and the image generation accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an automated comic generation method and system based on a text-based graphic model. Background Technology

[0002] Comics, as a form of visual art, rely on manual drawing for their style and character design, which is time-consuming and prone to stylistic shifts. Furthermore, some comics may include multiple panels, and these panels need to be repeatedly adjusted during the drawing process, limiting the expressive power of movement.

[0003] To improve the efficiency of comic drawing, artificial intelligence (AI) models can be used. These AI models can generate image models from pre-trained text. During the comic drawing process, users can write comic scripts based on the storyline, design characters and scenes, and then input the scripts into the AI ​​model so that it can generate comic images.

[0004] However, since most AI models are general text-to-image generation models, the consistency of style and characters among the generated comic images is poor. Furthermore, the high rendering error rate when rendering images based on comic script text leads to severe distortion of details in the comic images; for example, details such as the hands and faces of characters are severely distorted. Therefore, text-to-image generation models have low accuracy in generating comic images. Summary of the Invention

[0005] In view of this, embodiments of this application provide an automated comic generation method and system based on a text-to-image model to solve the problem of low accuracy of image content generated by the text-to-image model when generating comic images.

[0006] According to one aspect of this application, an automated comic generation method based on a text-based image model is provided, the method comprising:

[0007] The input script and posture control parameters are obtained. The input script is text data including multiple descriptive words. The descriptive words include character descriptive words and storyboard prompts. The posture control parameters include at least one of posture control parameters extracted from the input script and posture control parameters input in real time.

[0008] The semantic information of the descriptive words is parsed from the input script using a multi-text encoder;

[0009] The semantic information is used to generate composition data, which includes a character template generated by solidifying the character description words and storyboard composition parameters generated based on the storyboard prompt words.

[0010] The spatial structure of the mapping data is set by using a conditional generative adversarial network in conjunction with the posture control parameters.

[0011] Cartoon images are generated according to the spatial structure and the composition data.

[0012] In some embodiments, a multi-text encoder is used to parse the semantic information of the descriptive words from the input script, including:

[0013] The pre-trained text-to-graph model is invoked. The text-to-graph model includes a role consistency engine and a dynamic graph composition engine. The role consistency engine is built into the multi-text encoder.

[0014] Extract the expected result data from the input script;

[0015] The generation parameters of the text image model are set according to the expected result data. The generation parameters include one or more combinations of image resolution, generation steps, guidance ratio, and random seed.

[0016] The input script is input into the text graph model to perform semantic parsing on the input script using the multi-text encoder of the role consistency engine;

[0017] Obtain the semantic information output by the multi-text encoder.

[0018] In some embodiments, a multi-text encoder is used to parse the semantic information of the descriptive words from the input script, including:

[0019] The input script is segmented to obtain a keyword set, which includes multiple keywords;

[0020] The keyword set is input into the multi-text encoder, which includes a general semantic encoder, a long text optimization encoder, and a multimodal encoder.

[0021] Obtain the semantic information output by the multi-text encoder for the keyword set. The semantic information includes core semantics, related semantics, and cross-modal understanding semantics. The core semantics are the semantic information obtained by the general semantic encoder by parsing the keywords. The related semantics are the semantic information obtained by the long text optimization encoder by capturing contextual information. The cross-modal understanding semantics are the semantic information obtained by the multi-modal encoder by parsing long text.

[0022] In some embodiments, generating graph data based on the semantic information includes:

[0023] Extract the character description words from the input script;

[0024] Based on the semantic information, obtain the semantic information associated with the character description words;

[0025] Generate a character template, the character template including the character descriptive words and the character semantic information associated with the character descriptive words;

[0026] The character template is stored in the target path, which includes the installation root directory of the Wenshengtu model.

[0027] In some embodiments, generating graph data based on the semantic information includes:

[0028] The character template is loaded via the target path;

[0029] Parse the character descriptive terms and the character semantic information from the character template;

[0030] Combine the semantic information of the input script to generate fused input data, wherein the fused input data is the semantic fusion result of the character description words in the input script and the character description words in the character template;

[0031] The fused input data is input into the text-based graph model to generate the composition data through the text-based graph model.

[0032] In some embodiments, generating graph data based on the semantic information includes:

[0033] Based on the semantic information, the scene prompt words are extracted from the input script;

[0034] Based on the semantics of the scene prompts, scene features are extracted, and these scene features are used to characterize the logical order and positional relationship between multiple scene regions.

[0035] A baseline storyboard region is determined according to the aforementioned storyboard features; the baseline storyboard region is the storyboard region whose storyboard features satisfy a preset baseline condition.

[0036] Using the aforementioned baseline storyboard area as a reference, storyboard composition parameters for multiple storyboard areas are set.

[0037] In some embodiments, the spatial structure of the mapping data is set by a conditional generative adversarial network in conjunction with pose control parameters, including:

[0038] The pose control parameters and the composition data are input into the conditional generative adversarial network, so that the conditional generative adversarial network generates an intermediate image with the pose control parameters as a conditional constraint and the composition data as an input.

[0039] Calculate the adversarial loss based on the intermediate image and the posture control parameters;

[0040] The spatial structure of the mapping data is set according to the adversarial loss.

[0041] In some embodiments, generating a cartoon image according to the spatial structure and the composition data includes:

[0042] Obtain the storyboard image output by the conditional generative adversarial network, wherein the storyboard image is an image generated by the conditional generative adversarial network when the adversarial loss is less than or equal to a preset loss threshold;

[0043] Layout parameters are generated based on the composition data;

[0044] The storyboard image is laid out according to the layout parameters to generate the comic image.

[0045] In some embodiments, setting the spatial structure of the mapping data according to the adversarial loss includes:

[0046] The mapping data after setting the spatial structure is input into the conditional generative adversarial network to obtain the regenerated image output by the conditional generative adversarial network;

[0047] Record the number of times the regenerated image is generated;

[0048] If the number of generation times is greater than or equal to a preset threshold, a prompt message is generated, and a posture control map input by the user based on the prompt message is obtained;

[0049] The pose control graph is set as a conditional constraint for the conditional generative adversarial network.

[0050] According to another aspect of this application, an automated comic generation system based on a text-based graphic model is provided, the system comprising:

[0051] An input module is used to acquire an input script and posture control parameters. The input script is text data including multiple descriptive words. The descriptive words include character description words and storyboard prompts. The posture control parameters include at least one of posture control parameters extracted from the input script and posture control parameters input in real time.

[0052] A semantic parsing module is used to parse the semantic information of the descriptive words from the input script using a multi-text encoder;

[0053] The composition module is used to generate composition data based on the semantic information. The composition data includes a character template generated by solidifying the character description words, and storyboard composition parameters generated based on the storyboard prompt words.

[0054] A spatial structure module is used to set the spatial structure of the mapping data by combining the posture control parameters with a conditional generative adversarial network.

[0055] The output module is used to generate cartoon images according to the spatial structure and the composition data.

[0056] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described automated comic generation method based on the text-based graphic model.

[0057] According to another aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described automated comic generation method based on the text-based graphic model.

[0058] By employing the above technical solutions, this application provides an automated comic generation method and system based on a text-based image model. The method, after acquiring an input script and pose control parameters, uses a multi-text encoder to parse the semantic information of descriptive words from the input script. Then, it generates composition data based on the semantic information. The composition data includes character templates generated by fixing character descriptive words and storyboard composition parameters generated based on storyboard prompts. A conditional generative adversarial network (GAN) is then used, combined with pose control parameters, to set the spatial structure of the composition data, and to generate comic images according to the spatial structure and composition data. This method can achieve character consistency control based on character templates, improving the consistency of characters in multiple storyboards within a comic image, and, combined with a GAN, achieve dynamic storyboard generation, improving image generation accuracy. The method can also shorten the generation time of a single-page comic image and reduce hardware requirements during the comic generation process.

[0059] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0060] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0061] Figure 1 This is a schematic diagram of the text image model provided in the embodiments of this application;

[0062] Figure 2 A schematic diagram of the automated comic generation method based on the text-based image model provided in this application embodiment;

[0063] Figure 3 This is a schematic diagram of the system architecture of the text-based graph model provided in the embodiments of this application;

[0064] Figure 4 A flowchart of the comic generation stage provided in this application embodiment;

[0065] Figure 5 A schematic diagram of the image generation process using a conditional generative adversarial network provided in the embodiments of this application;

[0066] Figure 6 A schematic diagram illustrating the process of adding dialogue text to a storyboard image, provided as an embodiment of this application.

[0067] Figure 7 This is a schematic diagram of the structure of an automated comic generation system based on a text-based graphic model provided in an embodiment of this application. Detailed Implementation

[0068] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0069] In this application embodiment, Text-to-Image Generation Models (CTEM) is a deep learning model that generates corresponding images from natural language descriptions. CTEM can align text semantics with visual features across modalities, enabling end-to-end or staged generation of text-to-image mappings.

[0070] For example, in some embodiments of this application, the texturing image model can be a StableDiffusion3Medium (SD3 Medium) model. The SD3 Medium model has 2 billion parameters, is small in size, and is suitable for efficient operation on the graphics processing units (GPUs) of consumer computers (PCs) and laptops without performance degradation due to hardware limitations. The SD3 Medium model can generate high-quality images with realistic details, vibrant colors, and natural lighting. It excels in the representation of hand and facial details and supports flexible output in various styles, requiring no fine-tuning and generating stylized images such as anime and thick-painted images simply by using prompts.

[0071] The SD3 Medium model is also capable of understanding complex natural language cues, including spatial reasoning, compositional elements, gestures, and style descriptions. Through the Diffusion Transformer architecture, the SD3 Medium model reduces errors in spelling, kerning, letter formation, and spacing in images. The SD3 Medium model can be trained on a dataset of 1 billion images and fine-tuned on 30 million high-quality images, enabling it to understand and generate a wide range of visual styles and concepts.

[0072] like Figure 1 As shown, in the process of generating images using a text-based image model, users can write input text containing detailed prompts and then input this text into the model. The model can then generate images that meet the requirements of the input text prompts. For example, to generate a comic, a user can write a comic script based on a storyline and design characters and scenes. The written comic script is then input into a pre-trained text-based image model, allowing the model to generate comic images according to the script.

[0073] Because most AI models are general-purpose text-to-image generation models, the consistency of style and characters among the generated comic images is poor. Furthermore, the high rendering error rate when rendering images based on comic script text leads to severe distortion of details in the comic images; for example, details such as the hands and faces of characters are severely distorted. Therefore, text-to-image generation models have low accuracy in generating comic images.

[0074] To address the issue of low accuracy in generating comic images using text-to-image models, this application provides an automated comic generation method based on a text-to-image model in some embodiments. This method can be applied to electronic devices with data processing capabilities. These electronic devices include, but are not limited to, computers, servers, mobile terminals, smart wearable devices, and industrial control machines. For ease of description, this application uses an electronic device as the execution subject of the method. It should be understood that the method can also be applied to other types of execution subjects, which are not illustrated in all embodiments of this application. Figure 2 As shown, the method includes:

[0075] S101. Obtain the input script and posture control parameters.

[0076] To automatically generate comic images, electronic devices can acquire an input script. This input script is text data written by the user based on the comic's content. The input script can be entered through a specific interactive interface. For example, the electronic device can install a comic generation application that executes a text-to-image model. By running the comic generation application, the electronic device can present an interactive interface. This interface can include text input controls such as text input boxes. After selecting a text input control with the focus cursor, the user can input text data via keyboard, voice input, or other methods to form the input script.

[0077] The input script can also be obtained by reading specific text data files. That is, the user can upload a script file to the electronic device, which can then read the text content within the script file to obtain the input script. In some embodiments, the electronic device can support multiple types of file data, such as text, image, audio, and video script files. When the input script is completed via a script file, the electronic device can read the format of the script file uploaded by the user and invoke a data reading tool suitable for the current script file format, thereby converting the non-text script file into a text type to meet the input requirements of the text-to-graph model.

[0078] For example, the user interface of a comic generation application may include a file upload control. When a user uploads an image containing the text "Character AA, playing baseball with Character BB on a baseball field, Scene 1: Close-up of Character AA's face, Scene 2: Close-up of Character BB's face, Scene 3: Character AA swings the bat, Scene 4: Baseball in flight...", the electronic device can use the uploaded file format to call an Optical Character Recognition (OCR) tool. The OCR tool will then perform text recognition on the image to obtain the input script.

[0079] The input script may include multiple descriptive terms. These descriptive terms describe the content and specifications of the generated comic book image. In some embodiments, the descriptive terms may include character descriptive terms and storyboard prompts. Character descriptive terms describe the characters and their various characteristics contained in the comic book image. Character descriptive terms may include the character's identity information, such as name, title, or code name. They may also include the character's appearance and behavior information, such as pink hair, blue pants, and playing baseball.

[0080] Storyboard cues are used to describe multiple features of different plot stages in a comic strip image. Storyboard cues can be represented by keywords with meanings related to location, scene, and angle. For example, when a user writes a comic strip script containing keywords such as "rainy night" and "overhead shot," it means that the current input script includes storyboard cues.

[0081] Since comic book images can be composed of multiple static images arranged according to the plot, they can include multiple storyboard images. Each storyboard image corresponds to a set of keywords indicating location, scene, angle, etc. For example, each page of comic book images can include eight storyboard images. The first storyboard image in these eight storyboard images can correspond to the keyword set of "rainy night, overhead shot," the second storyboard image to the keyword set of "rainy night, level shot," the third storyboard image to the keyword set of "night, close-up of face," and so on. By constraining the storyboard scenes of multiple storyboard images, different comic book combinations can be formed to create a complete page of comic book images.

[0082] After acquiring the input script (or simultaneously), the electronic device can also acquire pose control parameters. These parameters control the specific poses and spatial structure of the characters in each panel of the comic. Pose control parameters can be image-based or text-based. Image-based parameters can be user-drawn pose sketches, serving as a reference for character poses during image generation to generate comic characters with poses identical or similar to those depicted in the sketches. Text-based parameters describe the specific poses of the comic characters, serving as a constraint to emphasize the character's state in the corresponding panel.

[0083] The posture control parameters may include at least one of posture control parameters extracted from the input script and posture control parameters input in real time. That is, the posture control parameters can be extracted from the input script; the electronic device can perform semantic analysis on the input script and extract the parameters used for posture control from the semantic analysis results. When the input script contains keywords that characterize a character's actions, position, and posture, the electronic device can analyze the semantics and context of the keywords to extract the keywords used for posture control.

[0084] For example, when the input script includes text such as "Scene 1: Character AA stands under a tree and looks up at the treetop," the electronic device can perform semantic analysis on the input script to determine the keywords in the input script that represent the character's actions, position, and posture, such as "standing," "under a tree," and "looking up." These keywords can then be used as posture control parameters to constrain the character's posture in the scene 1 image.

[0085] In some embodiments, the input script may further include image content, i.e., the input script is a graphic script that combines text and images. The electronic device can then extract the image content from the input script and perform image recognition on the image content to determine whether the image content is used for gesture control. When the image content is used for gesture control, it can be used as a gesture control parameter.

[0086] For example, a user-written comic script might include the phrase "In Act 2, character BB is in a rainy night scene according to the pose and position shown in Figure CC," and a simple pose diagram named "Figure CC" could be inserted near this text. In this case, the electronic device can extract Figure CC from the comic script as a pose control parameter.

[0087] Pose control parameters can also be input by the user in real time during the comic generation process. That is, the interactive interface of the comic generation application can also include parameter input controls for inputting pose control parameters. To accommodate both text and image types of pose control parameters, the parameter input controls in the interactive interface can be either text input boxes or image upload controls. Users can input pose control parameters through the parameter input controls, allowing the electronic device to obtain these parameters.

[0088] For example, after a comic generation application generates a page of comic, if a user is not satisfied with the pose of a character in a particular panel, they can upload a simplified pose diagram via parameter input controls, along with the text description "Character AA in Act 3 is drawn according to the pose in this diagram," to indicate the scope of application of the pose diagram. After uploading the pose diagram and accompanying text, the comic generation application can then obtain the pose control parameters through the parameter input controls.

[0089] S102. Use a multi-text encoder to parse the semantic information of the descriptive words from the input script;

[0090] After obtaining the input script, the electronic device can perform semantic recognition on the input script based on the text-to-image model. To do this, the electronic device needs to first deploy the text-to-image model. Specifically, in some embodiments, the electronic device can first download the model file by accessing the model download address. Then, according to the operational requirements of the text-to-image model, it installs the model dependencies to obtain a stable operating environment for the model. Finally, by loading the text-to-image model, the electronic device can perform image generation based on the loaded model.

[0091] For example, when the Wensheng graph model is based on SD3 Medium, the electronic device can download the model file by accessing the HuggingFace Stable Diffusion 3Medium model page. When using a cloud platform like OpenBayes, the model can be directly cloned and loaded onto the platform. After downloading the model file, the electronic device can install dependencies, namely Python and necessary libraries such as torch and diffusers. If using ComfyUI, the ComfyUI workflow visualization tool also needs to be downloaded and launched. The model is then loaded into the GPU runtime environment using `pipe=StableDiffusion3Pipeline.from_pretrained("stabilityai / stable-diffusion-3-medium", torch_dtype=torch.float16).to("cuda")`. If using ComfyUI, the workflow configuration file, such as `sd3_medium_example_workflow_basic.json`, needs to be loaded, and the model path adjusted.

[0092] like Figure 3 As shown, in some embodiments, the text-based image model may include a character consistency engine and a dynamic composition engine. The character consistency engine is used to maintain consistency in the appearance, features, and style of characters across multiple images. The character consistency engine incorporates a multi-text encoder. It can embed the generated images into a high-dimensional semantic space using feature extractors such as DINOv2, and then use clustering algorithms such as K-MEANS++ to cluster these embeddings, selecting the most cohesive cluster as the input for consistency extraction, thus achieving feature extraction and comparison.

[0093] The role consistency engine, based on clustering results, iteratively optimizes the model using Low-Rank Adaptation (LoRA) technology to fine-tune the model and extract more consistent role features. This process continues iteratively until the generated images reach a preset consistency threshold in terms of features. The role consistency engine can also utilize text prompts to generate an initial image library and extract role features—the role consistency term—from images describing consistent identities through personalized training methods such as textual inversion.

[0094] The dynamic composition engine can automatically generate a reasonable image layout based on the input text description or creative requirements. It achieves layout planning through semantic understanding, using natural language processing technology to understand the scene, objects, and relationships in the text description, and then planning the image layout accordingly. Furthermore, during the generation process, it can adjust the composition in real time based on dynamic changes such as the actions of characters and scene variations to ensure image balance. It can also combine multimodal information such as text and images for multimodal fusion to generate compositions that better match the creative intent.

[0095] After loading and deploying the text graph model, the electronic device can invoke the text graph model and utilize the multitext encoder within the text graph model to parse the semantic information of the descriptive words from the input script. Therefore, in some embodiments, when the electronic device performs the parsing of the semantic information of the descriptive words from the input script using the multitext encoder, it can first invoke a pre-trained text graph model and extract expected result data from the input script. Then, the generation parameters of the text graph model are set according to the expected result data. These generation parameters include one or more combinations of image resolution, generation steps, guidance ratio, and random seed. The input script is then input into the text graph model to perform semantic parsing on the input script using the multitext encoder of the role consistency engine, thereby obtaining the semantic information output by the multitext encoder.

[0096] For example, after loading the text-based image model, the electronic device can set generation parameters. These parameters can include text prompts, which are descriptive text in the input script. Negative prompts can also be set, indicating unwanted input content. Generation parameters can also include other types of parameters, such as image resolution, generation steps, guidance scale, and random seed. Image resolution sets the width and height of the generated image, such as 1024×1024. Generation steps refer to the number of iterations the model performs during image generation; for example, 20-50 steps can be set. The guidance scale (CFG Scale) controls the influence of the text prompts on the generated image; for example, a guidance scale of around 5.0 can be set. The random seed is a parameter that controls the initial random state of the generation process, determining the starting point for the model's image generation. For example, the random seed can be set to a fixed value such as 42 to generate identical images, or it can be set to -1 for random generation. After setting the generation parameters, the electronic device inputs the script into the text graph model, so that the multi-text encoder of the role consistency engine can perform semantic parsing on the input script to obtain the semantic information output by the multi-text encoder.

[0097] To obtain more comprehensive semantic information, a multitext encoder can work collaboratively with multiple encoders of different types to parse text prompts. Therefore, in some embodiments, the multitext encoder of a role consistency engine can specifically include a general semantic encoder, a long text optimization encoder, and a multimodal encoder. Correspondingly, when an electronic device parses the semantic information of descriptive words from an input script using a multitext encoder, it can first perform word segmentation on the input script to obtain a keyword set including multiple keywords. Then, the keyword set is input into the multitext encoder to obtain the semantic information output by the multitext encoder for the keyword set.

[0098] Since a multi-text encoder includes a general semantic encoder, a long-text optimized encoder, and a multimodal encoder, an electronic device can input a set of keywords into the general semantic encoder, the long-text optimized encoder, and the multimodal encoder respectively. When obtaining the semantic information output by the multi-text encoder for the set of keywords, it can acquire core semantics, related semantics, and cross-modal understanding semantics respectively. Specifically, core semantics is the semantic information obtained by the general semantic encoder through parsing keywords; related semantics is the semantic information obtained by the long-text optimized encoder through capturing contextual information; and cross-modal understanding semantics is the semantic information obtained by the multimodal encoder through parsing long text.

[0099] For example, when the text-generated graph model is a Stable Diffusion 3 (SD3) model, the multi-text encoder can include the general semantic encoder CLIP-G, the long-text optimized encoder CLIP-L, and the multimodal encoder T5-XXL. CLIP-G, as a general semantic encoder, is suitable for short text prompts and can quickly extract core semantics. CLIP-L, as a long-text optimized encoder, is suitable for complex descriptions and can capture contextual relationships and details. T5-XXL, as an enhanced multimodal encoder, is suitable for very long texts and complex logic, and can handle technical terms and cross-modal understanding.

[0100] After loading the SD3 model file, the electronic device can use `sd3_medium_incl_clips_t5xxlfp8.safetensors` to load CLIP-G, CLIP-L, and T5-XXL encoders. This combines the general semantics of CLIP-G, the contextual details of CLIP-L, and the complex logic of T5-XXL to obtain positive prompt words. The processing weights of the CLIP-G, CLIP-L, and T5-XXL encoders are adjusted to control the global semantic suppression strength, contextual negation, and logical negation. Combined with the input positive prompt words, CLIP-G, CLIP-L, and T5-XXL can output semantic vectors, i.e., semantic information.

[0101] S103. Generate graph data based on the semantic information.

[0102] After generating semantic information, the electronic device can generate composition data based on the semantic information. This composition data may include character templates generated by solidifying the character descriptors. The character templates are image template libraries solidified based on the character descriptors corresponding to the input script. These templates can be used as templates in subsequent image generation processes of the text-based image model to ensure consistency of characters across generated images.

[0103] To generate a character template, in some embodiments, when the electronic device generates composition data based on the semantic information, it can first extract the character descriptors from the input script and, based on the semantic information, obtain the semantic information associated with the character descriptors. Then, it generates a character template including the character descriptors and the associated semantic information, and stores the character template in a target path, which includes the installation root directory of the text-based graph model.

[0104] For example, when using the Stable Diffusion 3 model, electronic devices can generate template libraries by embedding prompts to efficiently produce high-quality images. Specifically, after acquiring the input script and parsing the prompts from it, the electronic device can store the parsed prompts as templates. Since the prompts include character descriptions, the constructed templates can also be called character templates. The saved character templates are stored in the styles.csv file, located in the StableDiffusion installation root directory.

[0105] After building the role templates, electronic devices can also edit and expand them in response to user interactions. For example, an electronic device can expand the template library by editing the styles.csv file; that is, it can copy the styles.csv file and open it with an application such as Notepad. Then, it can add new templates in the format of name, prompt, negative_prompt, and save the file to obtain the new role template, which will then take effect and run in subsequent text-to-image processes.

[0106] After generating and storing the character template, the electronic device can generate multiple storyboard images with a consistent character style based on the character template. Specifically, in some embodiments, when the electronic device generates composition data based on the semantic information, it can also load the character template through the target path and parse the character descriptors and semantic information from the character template. This is then combined with the semantic information of the input script to generate fused input data. The fused input data is the semantic fusion result of the character descriptors in the input script and the character descriptors in the character template. The fused input data is then input into a text-based graph model to generate the composition data.

[0107] For example, to enable a text-based image model to generate images using character templates, the electronic device can select a saved character template based on a set target path and load the corresponding prompts into the input box. Users can also enter additional descriptions in the prompt input box, allowing the electronic device to combine these descriptions with the template prompts to generate composition data.

[0108] During the process of generating image composition data, electronic devices can also optimize prompts and character templates based on the image generation results. This means fine-tuning the descriptive or negative prompts in the prompts according to the generated image effect. Additionally, tools like MagicPrompt can be used to automatically generate prompts, or high-quality prompts can be generated using large language models like ChatGPT.

[0109] Furthermore, electronic devices can provide candidate words during the generation of image composition data via plugins to speed up the input of prompts and improve image generation efficiency. For example, electronic devices can install plugins such as Tagcomplete to provide candidate words when inputting prompts, thus accelerating the input process. Plugins like Prompt-all-in-one can also be used to support direct input of Chinese keywords and automatically translate them into English. Tools such as PromptPerfect and StableDiffusion Prompt Generator can also be used to optimize prompts.

[0110] The composition data may also include storyboard composition parameters generated based on the storyboard prompts. The storyboard composition parameters may include the scene graph corresponding to each storyboard image, as well as composition parameters used to set the spatial relationships and actions of characters and targets in the scene graph.

[0111] To generate storyboard composition parameters, in some embodiments, when the electronic device generates composition data based on the semantic information, it can extract the storyboard prompts from the input script based on the semantic information, and extract storyboard features according to the semantics of the storyboard prompts. These storyboard features characterize the logical order and positional relationship between multiple storyboard regions. A baseline storyboard region is then determined according to the storyboard features, and storyboard composition parameters for multiple storyboard regions are set based on this baseline storyboard region.

[0112] The baseline storyboard region is the storyboard region whose storyboard features satisfy preset baseline conditions. The preset baseline conditions can be set based on region proportions or generation order. For example, the baseline storyboard region is the area containing the storyboard image that occupies the largest proportion of the current page's comic image. Alternatively, the baseline storyboard region can be the area containing the first storyboard image generated on the current page's comic image. After setting the baseline storyboard region, the text-based graphic model then sets other storyboard regions sequentially according to the storyboard prompts, following logical order and positional relationships, to obtain storyboard composition parameters.

[0113] S104. Using a conditional generative adversarial network, the spatial structure of the mapping data is set in conjunction with the posture control parameters.

[0114] After generating the mapping data, the electronic device can also use a conditional generative adversarial network to set the spatial structure of the mapping data in conjunction with posture control parameters. That is, for example... Figure 4 As shown, by injecting pose control parameters, electronic devices can combine the Conditional Generative Adversarial Network (ControlNet) to control the pose of characters and the perspective relationship of the scene in each storyboard image, thereby optimizing the spatial structure between contents in the storyboard image.

[0115] To establish the spatial structure, in some embodiments, when an electronic device executes a conditional generative adversarial network (GAN) to set the spatial structure of the mapping data in conjunction with posture control parameters, the posture control parameters and the mapping data can be input into the GAN. This causes the GAN to generate an intermediate image with the posture control parameters as a conditional constraint and the mapping data as input. Then, an adversarial loss is calculated based on the intermediate image and the posture control parameters, and the spatial structure of the mapping data is set according to the adversarial loss.

[0116] For example, when generating cartoon images using Stable Diffusion 3 and ControlNet, you can first install and configure ControlNet. After installing the Stable Diffusion WebUI and ControlNet plugin, download and install the OpenPose model for pose control. Then, based on the acquired pose control parameters, prepare a pose reference image. This involves selecting or creating a pose reference image within the pose control parameters; the reference image can be a real photograph or a hand-drawn skeleton diagram. Next, control the pose by configuring ControlNet. Specifically, enable ControlNet in the Stable Diffusion WebUI's cartoon image interface, select OpenPose as the preprocessor, and choose the corresponding control_v11p_sd15_openpose model. Upload the pose reference image and enable the corresponding ControlNet unit for precise pose matching.

[0117] When controlling the perspective effect of a scene is required, electronic devices can also use a Depth model as a second ControlNet. By selecting or uploading a depth map as a reference in the pose control parameters, choosing Depth as the preprocessor, and selecting the corresponding depth model, the balance between pose and perspective can be ensured by adjusting the weights of the ControlNet.

[0118] During the process of setting up the spatial structure, the electronic device can also optimize and fine-tune the model based on the generation results of each storyboard image. That is, it can fine-tune the pose reference image or prompt words based on the generation results to obtain better results.

[0119] Therefore, such as Figure 5As shown, in some embodiments, when an electronic device sets the spatial structure of the mapping data according to the adversarial loss, it can input the mapping data after setting the spatial structure into the conditional generative adversarial network (GAN) to obtain a regenerated image output by the GAN, and then record the number of times the regenerated image is generated. The number of generation is compared with a preset threshold. If the number of generation is greater than or equal to the preset threshold, a prompt message is generated, and a posture control map input by the user based on the prompt message is obtained. This posture control map is then set as a conditional constraint of the GAN to reset the spatial structure.

[0120] For example, after generating an intermediate image, an electronic device can evaluate its generation quality based on adversarial loss. If the generated image does not meet preset requirements, the composition data with the spatial structure set can be input into ControlNet, and ControlNet can regenerate the intermediate image according to the composition data with the spatial structure set—that is, regenerate the image. After regenerating the intermediate image, the generation quality is evaluated again. If the generation quality meets preset requirements, the intermediate image can be output as a storyboard image in the final comic. If the generation quality still does not meet preset requirements, the spatial structure can be reset, and the composition data with the spatial structure set can be input into ControlNet again for image generation, until the generation quality meets preset requirements.

[0121] To improve generation efficiency and reduce iteration count, the electronic device can record the number of times the regenerated image is generated, N, and obtain a preset threshold N0, such as 20 times. When the number of generation N reaches the threshold N0, a prompt message can be generated, such as "Please enter a simplified pose diagram of character AA in the second act." By displaying the prompt message in the interactive interface, the user is prompted to input a pose control diagram. The pose control diagram input by the user based on the prompt message is then obtained, and the pose control diagram is set as a conditional constraint of ControlNet to reset the spatial structure. The intermediate image is then generated again according to the composition data of the reset spatial structure, and the iteration is performed according to the method provided in the above embodiment to ensure that the generation effect meets the preset requirements.

[0122] S105. Generate a cartoon image according to the spatial structure and the composition data.

[0123] After setting the spatial structure, the electronic device can generate comic images according to the control structure and composition data. That is, the text-based image model generates storyboard images sequentially based on the input prompts and the character descriptions and scene details corresponding to the composition data, combined with the poses and scene perspective effects of the reference images. The generated storyboard images are then laid out to obtain comic images.

[0124] To generate comic book images, in some embodiments, when generating comic book images according to the spatial structure and the composition data, the electronic device may first acquire the storyboard images output by the conditional generative adversarial network (GAN). The storyboard images are generated by the GAN when the adversarial loss is less than or equal to a preset loss threshold. Then, layout parameters are generated based on the composition data, and the storyboard images are laid out according to the layout parameters to generate the comic book image.

[0125] For example, after setting the spatial structure of the composition data, an electronic device can generate images using ControlNet and calculate the adversarial loss during image generation. When the adversarial loss is greater than a preset loss threshold, it indicates that the currently generated image has not yet achieved the desired effect, so adversarial training can continue using ControlNet. Conversely, when the adversarial loss is less than or equal to the preset loss threshold, it indicates that the currently generated image has achieved the desired effect, and therefore the generated image can be used as a storyboard image.

[0126] Based on the layout parameters generated from the composition data, that is, according to the logical order and positional relationship between the storyboard images in the composition data, the storyboard images are arranged to generate a comic image containing multiple storyboard images. For example, when the input script includes content such as "Act 1 storyboard: close-up of character AA's face, Act 2 storyboard: close-up of character BB's face, Act 3 storyboard: character AA swings the bat to hit the ball, Act 4 storyboard: baseball in flight...", ControlNet can generate multiple storyboard images sequentially according to the input script, with each storyboard image representing Act 1, Act 2, and so on. Then, following the layout of 8 storyboards per page, the multiple storyboard images are arranged to generate a comic image.

[0127] By applying the technical solutions of the above embodiments, the automated comic generation method based on the text-based image model described in the above embodiments can achieve character consistency control based on character templates, improve the consistency of characters in multiple panels of a comic image, and combine conditional generative adversarial networks to achieve dynamic panel generation, thereby improving the accuracy of image generation. The method can also shorten the generation time of a single-page comic image and reduce the hardware requirements in the comic generation process.

[0128] In some embodiments, as a refinement and extension of the specific implementation of the above embodiments, and in order to fully illustrate the specific implementation process of this embodiment, some embodiments of this application also provide an automated comic generation method based on a text-based image model, such as... Figure 6 As shown, this method, based on the automated comic generation method based on the text-to-image model described in the above embodiments, further integrates a text rendering module into the text-to-image model to achieve error-free dialogue bubbles. Therefore, the method also includes:

[0129] S201. Read the dialogue text from the input script;

[0130] S202. Identify the dialogue region from the storyboard image based on the dialogue text;

[0131] S203. Add a dialog bubble to the dialog area and set the display parameters of the dialog text according to the dialog bubble;

[0132] S204. Add the dialogue text to the storyboard image according to the display parameters.

[0133] By applying the technical solutions of the above embodiments, the method can identify dialogue regions from storyboard images based on the dialogue text, add dialogue bubbles in the dialogue regions, and set display parameters for the dialogue text according to the dialogue bubbles, thereby adding dialogue text according to the display parameters. The method integrates a text rendering module into the comic image model to achieve error-free dialogue bubbles, complete adaptive dialogue text rendering, and improve the generation quality of comic images.

[0134] In some embodiments, as a specific implementation of the automated comic generation method based on the text-to-image model described in the above embodiments, some embodiments of this application also provide an automated comic generation system based on the text-to-image model, such as... Figure 7 As shown, the system includes:

[0135] An input module is used to acquire an input script and posture control parameters. The input script is text data including multiple descriptive words. The descriptive words include character description words and storyboard prompts. The posture control parameters include at least one of posture control parameters extracted from the input script and posture control parameters input in real time.

[0136] A semantic parsing module is used to parse the semantic information of the descriptive words from the input script using a multi-text encoder;

[0137] The composition module is used to generate composition data based on the semantic information. The composition data includes a character template generated by solidifying the character description words, and storyboard composition parameters generated based on the storyboard prompt words.

[0138] A spatial structure module is used to set the spatial structure of the mapping data by combining the posture control parameters with a conditional generative adversarial network.

[0139] The output module is used to generate cartoon images according to the spatial structure and the composition data.

[0140] By applying the technical solutions of the above embodiments, the automated comic generation system based on the text-to-image model provided in the above embodiments can, after the input module obtains the input script and posture control parameters, the semantic parsing module uses a multi-text encoder to parse the semantic information of descriptive words from the input script. Then, the composition module generates composition data based on the semantic information. The composition data includes character templates generated by fixing character descriptive words, and storyboard composition parameters generated based on storyboard prompts. Then, the spatial structure module sets the spatial structure of the composition data based on a conditional generative adversarial network and the posture control parameters, and the output module generates comic images according to the spatial structure and composition data. The system can achieve character consistency control based on character templates, improving the consistency of characters in multiple storyboards in a comic image, and achieve dynamic storyboard generation by combining a conditional generative adversarial network, improving the accuracy of image generation.

[0141] It should be noted that other corresponding descriptions of the functional units involved in the automated comic generation system based on the text-to-image model provided in the embodiments of this application can be found in the corresponding descriptions in the automated comic generation method based on the text-to-image model provided in the above embodiments, and will not be repeated here.

[0142] This application also provides a computer device, specifically a personal computer, server, network device, etc. The computer device includes a bus, processor, memory, and communication interface, and may also include input / output interfaces and a display device. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores location information. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0143] Those skilled in the art will understand that the structure of the computer device described above is only a partial structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. A specific computer device may include more or fewer components, or combine certain components, or have different component arrangements.

[0144] In one embodiment, a computer-readable storage medium is also provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0145] In one embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0147] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0148] Any references to memory, database, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc.

[0149] Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take many forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0150] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processors involved in the embodiments provided in this application may be, but are not limited to, general-purpose processors, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc.

[0151] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An automated comic generation method based on a text-based image model, characterized in that, The method includes: The input script and posture control parameters are obtained. The input script is text data including multiple descriptive words. The descriptive words include character descriptive words and storyboard prompts. The posture control parameters include at least one of posture control parameters extracted from the input script and posture control parameters input in real time. The semantic information of the descriptive words is parsed from the input script using a multi-text encoder; The semantic information is used to generate composition data, which includes a character template generated by solidifying the character description words and storyboard composition parameters generated based on the storyboard prompt words. The spatial structure of the mapping data is set by using a conditional generative adversarial network in conjunction with the posture control parameters. Cartoon images are generated according to the spatial structure and the composition data.

2. The method according to claim 1, characterized in that, Parsing the semantic information of the descriptive words from the input script using a multi-text encoder includes: The pre-trained text-to-graph model is invoked. The text-to-graph model includes a role consistency engine and a dynamic graph composition engine. The role consistency engine is built into the multi-text encoder. Extract the expected result data from the input script; The generation parameters of the text image model are set according to the expected result data. The generation parameters include one or more combinations of image resolution, generation steps, guidance ratio, and random seed. The input script is input into the text graph model to perform semantic parsing on the input script using the multi-text encoder of the role consistency engine; Obtain the semantic information output by the multi-text encoder.

3. The method according to claim 1, characterized in that, Parsing the semantic information of the descriptive words from the input script using a multi-text encoder includes: The input script is segmented to obtain a keyword set, which includes multiple keywords; The keyword set is input into the multi-text encoder, which includes a general semantic encoder, a long text optimization encoder, and a multimodal encoder. Obtain the semantic information output by the multi-text encoder for the keyword set. The semantic information includes core semantics, related semantics, and cross-modal understanding semantics. The core semantics are the semantic information obtained by the general semantic encoder by parsing the keywords. The related semantics are the semantic information obtained by the long text optimization encoder by capturing contextual information. The cross-modal understanding semantics are the semantic information obtained by the multi-modal encoder by parsing long text.

4. The method according to claim 1, characterized in that, Generate graph data based on the semantic information, including: Extract the character description words from the input script; Based on the semantic information, obtain the semantic information associated with the character description words; Generate a character template, the character template including the character descriptive words and the character semantic information associated with the character descriptive words; The character template is stored in the target path, which includes the installation root directory of the Wenshengtu model.

5. The method according to claim 4, characterized in that, Generate graph data based on the semantic information, including: The character template is loaded via the target path; Parse the character descriptive terms and the character semantic information from the character template; Combine the semantic information of the input script to generate fused input data, wherein the fused input data is the semantic fusion result of the character description words in the input script and the character description words in the character template; The fused input data is input into the text-based graph model to generate the composition data through the text-based graph model.

6. The method according to claim 1, characterized in that, Generate graph data based on the semantic information, including: Based on the semantic information, the scene prompt words are extracted from the input script; Based on the semantics of the scene prompts, scene features are extracted, and these scene features are used to characterize the logical order and positional relationship between multiple scene regions. A baseline storyboard region is determined according to the aforementioned storyboard features; the baseline storyboard region is the storyboard region whose storyboard features satisfy a preset baseline condition. Using the aforementioned baseline storyboard area as a reference, storyboard composition parameters for multiple storyboard areas are set.

7. The method according to claim 1, characterized in that, By using a conditional generative adversarial network and combining pose control parameters, the spatial structure of the mapping data is set, including: The pose control parameters and the composition data are input into the conditional generative adversarial network, so that the conditional generative adversarial network generates an intermediate image with the pose control parameters as a conditional constraint and the composition data as an input. Calculate the adversarial loss based on the intermediate image and the posture control parameters; The spatial structure of the mapping data is set according to the adversarial loss.

8. The method according to claim 7, characterized in that, Generating a comic image according to the spatial structure and the composition data includes: Obtain the storyboard image output by the conditional generative adversarial network, wherein the storyboard image is an image generated by the conditional generative adversarial network when the adversarial loss is less than or equal to a preset loss threshold; Layout parameters are generated based on the composition data; The storyboard image is laid out according to the layout parameters to generate the comic image.

9. The method according to claim 7, characterized in that, The spatial structure of the mapping data is set according to the adversarial loss, including: The mapping data after setting the spatial structure is input into the conditional generative adversarial network to obtain the regenerated image output by the conditional generative adversarial network; Record the number of times the regenerated image is generated; If the number of generation times is greater than or equal to a preset threshold, a prompt message is generated, and a posture control map input by the user based on the prompt message is obtained; The pose control graph is set as a conditional constraint for the conditional generative adversarial network.

10. An automated comic generation system based on a text-to-image model, characterized in that, The system includes: An input module is used to acquire an input script and posture control parameters. The input script is text data including multiple descriptive words. The descriptive words include character description words and storyboard prompts. The posture control parameters include at least one of posture control parameters extracted from the input script and posture control parameters input in real time. A semantic parsing module is used to parse the semantic information of the descriptive words from the input script using a multi-text encoder; A composition module is used to generate composition data based on the semantic information. The composition data includes a character template generated by solidifying the character description words, and storyboard composition parameters generated based on the storyboard prompt words. A spatial structure module is used to set the spatial structure of the mapping data by combining the posture control parameters with a conditional generative adversarial network. The output module is used to generate cartoon images according to the spatial structure and the composition data.

Images