An AI picture book generation method based on a multi-modal large model

Abstract

The application discloses an AI picture book generation method based on a multi-modal large model, comprising the following steps: S1, deploying a multi-modal large model with text and image understanding and generation capabilities; S2, receiving and analyzing picture book story guide information input by a user; S3, driving the model to generate corresponding picture book text content through theme, style and other guide information input by the user; S4, splitting the picture book text into shots, and generating picture book shot picture content matched with the text through the model; and S5, structurally combining the picture book text and the picture, performing multi-dimensional quality evaluation and controllable adjustment on the generated picture book content, and outputting a final work meeting user requirements. The AI picture book generation method based on the multi-modal large model solves the problems of long time consumption and poor controllability in picture book story creation, and provides a complete solution for intelligent and efficient one-stop AI picture book generation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and digital content creation technology, specifically to an AI picture book generation method based on a multimodal large model. Background Technology

[0002] Picture books, as a form of content combining text and illustrations, have wide applications in children's education, storytelling, and artistic creation. Traditional picture book creation relies on close collaboration between professional writers and illustrators, a time-consuming and costly process that demands high levels of comprehensive ability from creators. Currently, with the rapid development of artificial intelligence technology in digital content creation, AI-based image and text generation methods are gaining attention. For example, an AI-based dynamic image and text story generation method (CN118262010A) receives user-input story data, uses ChatGPT for story continuation, and combines a Stable Diffusion model to generate picture book pages. It further supports features such as multi-ending story generation, converting user-uploaded photos into anime style, and adding Unity-based interactive elements. This method, to a certain extent, automates picture book generation and lowers the creative threshold.

[0003] However, existing technologies still have the following shortcomings: 1. Fragmented text and image generation process: In this method, the story text is generated by ChatGPT and the picture book pages are generated by StableDiffusion. The two are independent modules and lack deep cross-modal fusion and collaborative optimization, which may lead to fragmentation in the narrative logic, emotional expression and stylistic consistency of the text and image content.

[0004] 2. Storyboard design and content matching rely on manual intervention: Although it supports the generation of multi-page picture books, it has not achieved automated storyboard splitting and structured alignment of text and image content. Users still need to make a lot of manual adjustments and screenings to the generated results.

[0005] 3. Inadequate quality inspection and redrawing mechanism: Although a redrawing mechanism is mentioned, the degree of automation is low, and a closed-loop generation-quality inspection-redrawing mechanism has not been formed, making it difficult to guarantee the consistency and high quality of the generated content.

[0006] Therefore, there is an urgent need for an intelligent picture book generation solution that can deeply integrate text and image generation capabilities to automate the entire process from story conception to finalized illustrations. Summary of the Invention

[0007] To overcome the shortcomings of fragmented workflows and inconsistent quality in existing picture book creation technologies, this invention provides a method for automated and intelligent picture book content generation by combining a multimodal large-scale model with natural language processing and computer vision. This method deploys a multimodal large-scale model with powerful cross-modal understanding and generation capabilities to transform simple user guidance information into structurally complete picture book text. It automatically performs scene segmentation and image generation with text-image matching, and finally outputs high-quality, stylistically consistent picture book works through structured merging and multi-dimensional evaluation and adjustment. This invention achieves intelligent and streamlined picture book creation, significantly improving creation efficiency and work quality, while ensuring user guidance and control throughout the process.

[0008] The objective of this invention is achieved through the following technical solutions.

[0009] An AI-based picture book generation method based on a multimodal large model includes the following steps: S1. Receive and parse the picture book guidance information input by the user; S2. Generate a structured draft of the picture book text based on the picture book's guiding information; S3. Based on the picture book text draft, the storyboard text unit is divided into multiple storyboard text units. Then, for each storyboard text unit, a corresponding image generation prompt is constructed, and a storyboard image is generated based on the prompt. S4. Arrange the generated storyboard text units and storyboard images in a structured manner according to the storyboard order to form a preliminary picture book; S5. Subsequently, a quality assessment is conducted on the initial picture book, and users review the assessment results and make adjustments accordingly. S6. After making adjustments, export the final revised work.

[0010] Furthermore, in step S1, the picture book guidance information is not limited to the story theme, art style, character settings, target audience age, plot keywords, and educational objectives.

[0011] Furthermore, in step S2, the process of generating the picture book text draft adopts a phased iterative generation mechanism: The first stage is the story outline generation stage: based on the picture book guidance information in step S1, a preliminary story outline is generated through a conditional prompt generation mechanism; The second stage is the pagination expansion stage: After confirming the story outline, the outline content is expanded and generated according to the pagination rule of 1-2 sentences per page, forming a complete picture book text draft.

[0012] Furthermore, the conditional prompt generation mechanism is based on the conditional probability prediction principle of an autoregressive language model, generating a story summary under the constraints of topic, style, and other factors.

[0013] Furthermore, in step S3, the prompt word consists of three parts: Fixed components: These are derived from the style parameters determined in step S1, including unified art style descriptors and core character appearance descriptors; Variables: Scene and action descriptions in the current storyboard text; Quality constraint section: Descriptive terms used to constrain image quality.

[0014] Furthermore, in step S5, the quality assessment uses a visual question-answering model to evaluate the relevance of images and text, uses image feature comparison to evaluate character consistency, and uses a text analysis model to evaluate story coherence.

[0015] Furthermore, the adjustment in step S5 is based on the evaluation results, and the picture book content is iteratively optimized by modifying text prompts, adjusting image generation parameters, or performing local regeneration.

[0016] Furthermore, when generating text in step S3 and generating images in step S4, the same core style Lora and character feature description words are used to ensure the consistency of the overall work style with the character.

[0017] This invention also provides a system for implementing the aforementioned AI picture book generation method based on a multimodal large model, comprising: The text generation submodule is used to generate a preliminary story outline based on the picture book's guiding information and through a conditional prompt generation mechanism, and to perform the text generation task.

[0018] The image generation submodule is used to generate the corresponding storyboard image for each storyboard text unit.

[0019] The automatic evaluation module is used to assess the relevance of images and text, and to evaluate character consistency and story coherence by comparing image features.

[0020] The unified control and scheduling module is used to drive the text generation submodule or image generation submodule in the multimodal large model to locally regenerate or fine-tune the parameters of the specified storyboard text unit or storyboard image based on user feedback.

[0021] A computer device according to the present invention includes a memory and a processor, the memory being electrically connected to the processor, the memory storing a computer program, which, when executed by the processor, causes the processor to implement the method described herein.

[0022] The present invention provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor implements the method described herein.

[0023] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. One-stop efficient generation: This invention integrates text creation, storyboard design, image generation and layout into a coherent process, driven by a multimodal large model, realizing full automation from creative input to finished product output, greatly shortening the creation cycle.

[0024] 2. Cross-modal consistency guarantee: Through unified multimodal large model for collaborative generation of text and images, and by adopting shared style and character descriptors, a high degree of consistency in storyline, visual content, and artistic style is effectively ensured across multiple pages of picture books.

[0025] 3. Balance between intelligence and controllability: While using AI for automated creation, the system deeply integrates user creative guidance and aesthetic judgment into the generation process through multi-dimensional quality assessment and feedback-based iterative adjustment mechanisms. This achieves an efficient collaborative model of "AI-led generation and human oversight and control," solving the pain point of uncontrollable traditional AI-generated content.

[0026] 4. High-quality and flexible output: Through structured storyboarding and targeted image generation strategies, each illustration is guaranteed to accurately match the corresponding textual plot. The final output supports multiple formats, adapting to different application scenarios such as screen reading and physical printing. Attached Figure Description

[0027] Figure 1 This is a flowchart of an AI picture book generation method based on a multimodal large model, according to an embodiment of the present invention.

[0028] Figure 2 This is a flowchart illustrating the generation of a draft illustration text for an embodiment of the present invention.

[0029] Figure 3 This is a flowchart of the scene splitting and image generation process according to an embodiment of the present invention. Detailed Implementation

[0030] This section will describe in detail specific embodiments of the present invention. Preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the drawings is to supplement the textual description with graphics, so that people can intuitively and vividly understand the overall technical solution of the present invention, but they should not be construed as limiting the scope of protection of the present invention.

[0031] like Figure 1 As shown in one embodiment of the present invention, an AI picture book generation method based on a multimodal large model is proposed. This method involves constructing and deploying a core engine for a picture book generation system, and deploying a multimodal pre-trained large model with cross-modal understanding and generation capabilities. The multimodal pre-trained large model is hereinafter referred to as the multimodal large model. The multimodal large model is used to perform subsequent steps such as text generation, image generation, image-text consistency evaluation, and feedback-based regeneration tasks. Specifically, it includes the following steps: S1. Receive and parse the picture book guidance information input by the user. The user inputs the picture book guidance information through a graphical user interface or API interface. The picture book guidance information includes, but is not limited to: story theme, art style, character settings, etc.

[0032] S2. Generate a structured picture book text draft.

[0033] S3. Storyboard Breakdown and Image Generation. The picture book text draft generated in step S2 is broken down into 12-18 storyboard text units. Then, for each storyboard text unit, a corresponding image generation prompt is constructed, and a storyboard image is generated based on the prompt.

[0034] S4. Text and image compositing, quality assessment, and controllable adjustments. The storyboard text units and storyboard images generated in step S3 are arranged in a structured layout according to the storyboard sequence to form a preliminary picture book; S5. Subsequently, a multi-dimensional quality assessment is conducted on the initial picture book. This assessment includes evaluations of story coherence, text-image relevance, visual appeal, stylistic consistency, and content safety. For example, it checks whether the text and images are related, whether characters remain consistent across different pages, and whether the story's logic is coherent. Users can review the assessment results and make multiple rounds of adjustments. These controllable adjustments are based on the assessment results and involve iterative optimization of the picture book content by modifying text prompts, adjusting image generation parameters, or performing partial regeneration. For instance, this might involve rewriting text on a page or regenerating an image.

[0035] S6. Once you are satisfied with the adjustments, export the final revised work as an editable digital document format or a printable publication file format, such as PDF, EPUB, or a printable high-resolution image set.

[0036] As a preferred embodiment, the multimodal large model can use an existing multimodal pre-trained large model with cross-modal understanding and generation capabilities as its core engine, or it can be implemented using a combined collaborative architecture. The combined collaborative architecture includes a text generation submodule and an image generation submodule, and tasks are distributed through a unified control and scheduling module.

[0037] In one embodiment, the picture book guidance information may include, in addition to the story theme, art style, and character settings, the target audience's age, plot keywords, and educational objectives.

[0038] In one embodiment, such as Figure 2 As shown, the process of generating the picture book text draft is a phased iterative generation mechanism, including a synopsis generation phase and a page expansion phase.

[0039] The first stage is the story outline generation stage. Based on the picture book guidance information from step S1, the multimodal big model generates a preliminary story outline through a conditional prompt generation mechanism. The conditional prompt generation mechanism is based on the conditional probability prediction principle of an autoregressive language model, generating a story outline under given themes, styles, and other constraints.

[0040] The second stage is the pagination expansion stage. After confirming the story outline, the outline is expanded according to the pagination rule of 1-2 sentences per page to generate a complete draft of the picture book text.

[0041] like Figure 3 As shown, in step S3, the splitting criteria include: punctuation marks, paragraph structure, narrative logic, plot turning points, scene changes, and character action changes. The generation of each split mirror image is guided by its corresponding text description.

[0042] In one embodiment, the prompt word consists of three parts: 1. Fixed Components: These are derived from the style parameters determined in step S1, including unified art style descriptors and core character appearance descriptors. For example, unified art style descriptors could be "watercolor style, soft fairytale tones, healing illustration, flat vector illustration," etc.; core character appearance descriptors must include the character's core visual features, such as "a seven-year-old boy with blond curly hair, wearing red overalls and round-framed glasses" or "an orange cat with fluffy fur, black stripes, and a blue bow around its neck." The introduction of these fixed descriptors ensures that the large model remains anchored to the global style control parameters when generating different paginations.

[0043] 2. Variables: Scene and action descriptions in the current storyboard text. This part is dynamically extracted and translated into visual instructions by the multimodal large model based on the specific storyboard text. For example, if the current storyboard text is "Xiaoming is chasing a glowing butterfly in the forest," the corresponding variable prompts could be constructed as: "The little boy is running in a sunny, dense forest, reaching out to try to catch a butterfly that emits golden light; in terms of camera language, dynamic snapshots and medium shots are used, motion blur is used to enhance the narrative tension and dynamic feeling of running, and the Tyndall effect is combined to represent the light and shadow layers in the forest."

[0044] 3. Quality Constraints: Descriptive terms used to constrain image quality, such as "high-resolution illustration," "rich in detail," and "suitable for children's picture books."

[0045] The unified style descriptors and character feature descriptors in the fixed section are derived from the picture book guidance information in step S1 or randomly generated, serving as global style control parameters. These parameters are maintained consistently throughout the generation of all storyboard images to ensure style and character consistency. Subsequently, a multimodal large model is invoked to generate corresponding storyboard images for each storyboard text unit.

[0046] Furthermore, when generating text in step S3 and generating images in step S4, the same core style Lora and character feature description words are used to ensure the consistency of the overall work style with the character.

[0047] In step S5, the quality assessment uses a visual question-answering model to evaluate the relevance of the image and text, uses image feature comparison to evaluate the consistency of the characters, and uses a text analysis model to evaluate the coherence of the story. It should be noted that the evaluation models used in this invention are all existing mature models that are available for reference. Specifically: (1) The visual question-answering model used to evaluate the relevance of the image and text can be the existing CLIP (Contrastive Language-Image Pre-training) or BLIP model, which determines the matching degree by calculating the cosine similarity between the feature vectors of the generated image and the storyboard text; (2) The image feature comparison technology used to evaluate the consistency of the characters can be the existing ReID (Target Re-identification) model or the ViT-based extraction of local features of the characters and calculation of feature distance; (3) The text analysis model used to evaluate the coherence of the story can be the existing BERT or GPT series natural language processing pre-trained models.

[0048] After the evaluation is completed, a preview with images and text is displayed to the user through the user evaluation interface, along with intuitive annotation tools. These annotation tools are existing graphical user interface interactive tools in the field, such as front-end brush smearing tools, rectangular selection tools, polygonal lasso tools, or image mask generation tools developed based on HTML5 Canvas. Users can directly select flawed or unsatisfactory parts on the preview screen, and annotate and comment on unsatisfactory pages or elements, just like using existing open-source image annotation tools such as LabelMe and CVAT. User feedback is used as new input conditions to drive the text generation submodule or image generation submodule in the multimodal large model to locally regenerate or fine-tune parameters of specified storyboard text units or storyboard images until the preset quality standards are met.

[0049] This invention utilizes a powerful multimodal model as its core to construct an intelligent, integrated picture book generation pipeline from creative input to high-quality finished product output. It supports real-time feedback and adjustments to text outlines and storyboards during multiple rounds of interaction, achieving a dynamically controllable creative process. This not only significantly lowers the barrier to entry and time cost for professional picture book creation but also ensures the personalization and controllability of the generated works by embedding a user feedback loop, providing an efficient content production tool for education, entertainment, personalized publishing, and other fields.

[0050] The system for implementing the AI ​​picture book generation method based on a multimodal large model includes: The text generation submodule is used to generate a preliminary story outline based on the picture book's guiding information and through a conditional prompt generation mechanism, and to perform the text generation task.

[0051] The image generation submodule is used to generate the corresponding storyboard image for each storyboard text unit.

[0052] The automatic evaluation module is used to assess the relevance of images and text, and to evaluate character consistency and story coherence by comparing image features.

[0053] The unified control and scheduling module is used to drive the text generation submodule or image generation submodule in the multimodal large model to locally regenerate or fine-tune the parameters of the specified storyboard text unit or storyboard image based on user feedback.

[0054] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, enabling those skilled in the art to better understand and utilize the invention.

Claims

1. An AI-based picture book generation method based on a multimodal large model, characterized in that, Includes the following steps: S1. Receive and parse the picture book guidance information input by the user; S2. Generate a structured draft of the picture book text based on the picture book's guiding information; S3. Based on the picture book text draft, the storyboard text unit is divided into multiple storyboard text units. Then, for each storyboard text unit, a corresponding image generation prompt is constructed, and a storyboard image is generated based on the prompt. S4. Arrange the generated storyboard text units and storyboard images in a structured manner according to the storyboard order to form a preliminary picture book; S5. Subsequently, a quality assessment is conducted on the initial picture book, and users review the assessment results and make adjustments accordingly. S6. After making adjustments, export the final revised work.

2. The AI ​​picture book generation method based on a multimodal large model according to claim 1, characterized in that: In step S1, the picture book guidance information is not limited to the story theme, art style, character settings, target audience age, plot keywords, and educational objectives.

3. The AI ​​picture book generation method based on a multimodal large model according to claim 1, characterized in that: In step S2, the process of generating the picture book text draft is a phased iterative generation mechanism: The first stage is the story outline generation stage: based on the picture book guidance information in step S1, a preliminary story outline is generated through a conditional prompt generation mechanism; The second stage is the pagination expansion stage: After confirming the story outline, the outline content is expanded and generated according to the pagination rule of 1-2 sentences per page, forming a complete picture book text draft.

4. The AI ​​picture book generation method based on a multimodal large model according to claim 3, characterized in that: The conditional prompt generation mechanism is based on the conditional probability prediction principle of autoregressive language models, and generates a story summary under the constraints of theme, style and other factors.

5. The AI ​​picture book generation method based on a multimodal large model according to claim 1, characterized in that: In step S3, the prompt consists of three parts: Fixed components: These are derived from the style parameters determined in step S1, including unified art style descriptors and core character appearance descriptors; Variables: Scene and action descriptions in the current storyboard text; Quality constraint section: Descriptive terms used to constrain image quality.

6. The AI ​​picture book generation method based on a multimodal large model according to claim 1, characterized in that: In step S5, the quality assessment uses a visual question-answering model to evaluate the relevance of images and text, uses image feature comparison to evaluate character consistency, and uses a text analysis model to evaluate story coherence.

7. The AI ​​picture book generation method based on a multimodal large model according to claim 1, characterized in that: The adjustments in step S5 are based on the evaluation results. The content of the picture book is iteratively optimized by modifying text prompts, adjusting image generation parameters, or performing local regeneration.

8. The AI ​​picture book generation method based on a multimodal large model according to claim 1, characterized in that: When generating text in step S3 and generating images in step S4, the same core style Lora and character feature description words are used to ensure the consistency of the overall work style with the character.

9. A computer device comprising a memory and a processor, the memory being electrically connected to the processor, the memory storing a computer program, characterized in that: When the computer program is executed by the processor, it causes the processor to implement the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor implements the method as described in any one of claims 1 to 8.

Images