An AI split-screen picture generation method, an electronic device, and a storage medium

Abstract

The application relates to the technical field of artificial intelligence content generation, and provides an AI split-screen picture generation method, an electronic device and a storage medium. The method comprises the following steps: inputting long text content; extracting information of the long text, wherein the information comprises role information, scene information and event information; generating intermediate visual materials according to the extracted information, wherein the visual materials comprise reference pictures and text descriptions; constructing group picture prompt words according to the intermediate visual materials; and generating group picture pictures containing multiple screens by an image generation model at one time according to the group picture prompt words and the reference pictures. The application can significantly improve the consistency of the front and rear roles and scenes, improve the event-level context perception, reduce the number of image generation API calls, and effectively improve the consistency constraint scalability.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence content generation technology, and in particular to an AI storyboard image generation method, electronic device, and storage medium. Background Technology

[0002] Artificial Intelligence Generated Content (AIGC) is a form of content generated by artificial intelligence technology, often used to generate comics from text.

[0003] The existing animation generation process is usually as follows: In the text parsing stage, the original text of the novel or script is split into sentences or paragraphs; in the prompt word generation stage, a large language model generates prompt words for each split segment independently, and the prompt words mainly depend on the text semantics of the current segment; in the image generation stage, each prompt word is fed into the image generation model in turn to generate storyboard images one by one; in the result output, all storyboard images are arranged in order to form a storyboard list.

[0004] However, the above-mentioned existing technologies have the following obvious drawbacks and deficiencies: (1) Since each storyboard image is generated by an independent prompt word, the image generation model cannot know the appearance of the characters in other storyboards when called at different times, resulting in significant differences in the face, body shape, hairstyle and clothing of the same character in different storyboard images, which seriously damages the narrative continuity. (2) In different storyboard images of the same scene, visual elements such as light, color tone and architectural details cannot be kept consistent, making the same location appear as multiple different locations on the screen, reducing the sense of story coherence. (3) Since there is no common reference between the storyboards, the relative position, orientation and action of the characters in multi-character scenes frequently change irrationally between adjacent storyboards, violating the 180-degree axis rule of film narrative. (4) Each storyboard prompt word only contains the semantic information of the current segment, and cannot know the contextual information such as other storyboards in the same event, the preparation of the previous scene, and the generated visual materials. The large language model has extremely limited control over the overall visual style. Summary of the Invention

[0005] To help improve the inconsistency between characters and scenes in existing technologies, this application provides an AI storyboard image generation method, electronic device, and storage medium.

[0006] Firstly, this application provides an AI-generated storyboard image generation method, employing the following technical solution: Enter long text content; Extract information from long texts, including character information, scene information, and event information; Based on the extracted information, intermediate visual materials are generated, including reference images and text descriptions. Based on the intermediate visual materials, construct image prompts; Based on the prompts and reference images, the image generation model generates a series of images containing multiple scenes in one go.

[0007] By adopting the above technical solutions, the consistency of the video before and after can be effectively improved.

[0008] Secondly, this application provides an electronic device that adopts the following technical solution: An electronic device comprising: At least one processor; Memory; At least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, the at least one application being configured to: execute an AI storyboard image generation method.

[0009] Thirdly, this application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform any of the AI ​​storyboard image generation methods provided in the first aspect.

[0010] In summary, this application includes at least one of the following beneficial technical effects: Significantly improves the consistency of characters and scenes: By injecting standard reference images of characters into the image prompts in the form of tags, the image generation model always uses the same reference image as a constraint when generating all storyboards, ensuring that the same character's face, body shape, and clothing are highly consistent throughout the entire storyboard; by injecting standard reference images of scenes with tags, the continuity of scene lighting, color tone, and spatial layout in all storyboards of the same event is guaranteed. Event-level context awareness: The large language model can simultaneously perceive the narrative needs of all scenes within an event, thereby making overall shot composition, camera movement, and emotional arrangement, rather than generating them independently frame by frame. Reduce the number of image generation API calls: A single call generates a multi-frame image set, which reduces the number of API calls and improves overall generation efficiency compared to calling each frame independently; Consistency constraints are scalable: The reference graph labeling mechanism supports any number of roles and scenarios, and handles multiple names for the same role through an alias mapping mechanism, which has good scalability. Attached Figure Description

[0011] Figure 1This is a flowchart illustrating one embodiment of the AI ​​storyboard image generation method; Figure 2 This is a flowchart illustrating another embodiment of the AI ​​storyboard image generation method; Figure 3 This is an example of a set of images from one embodiment of the AI ​​storyboard image generation method; Figure 4 This is a schematic diagram of a module of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] To make the purpose, technical solution, and advantages of this application clearer, the following description is provided in conjunction with the appendix. Figure 1-4 The present application will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the application.

[0013] This application provides an AI storyboard image generation method for use in a novel-to-comic-drama system.

[0014] This embodiment uses the example of generating a comic book from a novel to illustrate the above method. In actual implementation, the above method can also be used to generate videos from other texts, and this embodiment does not limit this application.

[0015] Specific reference Figure 1 This invention proposes an AI-generated storyboard image generation method. Specifically, the method in this embodiment includes: S10: Enter long text content; S20: Extract information from long texts, including character information, scene information, and event information; S30: Generate intermediate visual materials based on the extracted information, wherein the visual materials include reference images and text descriptions; S40: Based on the intermediate visual materials, construct image prompts; S50: Based on the image prompts and reference images, the image generation model generates a set of images containing multiple scenes in one go.

[0016] Before generating storyboards, separate character and scene visual asset libraries are built. These assets are then injected into the storyboard prompts as reference image tags. All storyboards for the same event are then input into a large language model as a unified image set for standardized planning. Finally, a multi-frame image set is generated in one go. The image set showcases the multi-frame storyboard group, with multiple frames originating from the same event unit, and character appearance and scene lighting maintaining a high degree of consistency across the frames.

[0017] Reference Figure 2 and Figure 3As shown, in one example, this AI storyboard generation method includes the following steps: Character Information Extraction and Image Generation: This module extracts structured information for all characters appearing in the original novel, including fields such as character name, list of aliases, gender, age range, identity, and physical characteristics (e.g., distinguishing between human and non-human forms). Based on this information, it constructs descriptive keywords for the character's appearance and calls an image generation API (such as Seedream / Doubao) to generate a standard reference image for each character, obtaining the image URL (role_image_url) for each character. All aliases for the same character (e.g., "I" and "Queen") are mapped to the same reference image, ensuring a unique correspondence between character identities.

[0018] Scene information extraction and scene image generation module: This module extracts all scene information that appears in the original novel, and constructs scene description prompts according to the stage-like thinking (open space in the center of the scene, no temporary traces, and space for convenient photography), calls the image generation API to generate standard scene reference images, and obtains the image URL (scene_image_url) of each scene.

[0019] Event Information Extraction and Character Scene Assignment Module: This module breaks down the original novel text into several event units according to dimensions such as scene changes, core action transitions, changes in participating characters, and time jumps. It assigns a corresponding list of character names and scene names to each event unit, forming a structured event list. The fields include: event ID, scene name, list of participating character names, core description of the event, and original text fragments.

[0020] Image prompt word building module: Step 1: Referencing Graph Label Mapping For each event, establish a reference graph label mapping relationship according to the following rules: The scene reference image is fixedly mapped to the "Image 1 Event" label; The first image of a person involved in the event is mapped to the tag "Image 2 People"; The second participating role is tagged "Image 3 Person"; All of the character's aliases are mapped to the same tag (e.g., "I" and "Queen" are both mapped to "Character in Image 2"); To prevent accidental replacement of character name substrings, the mapping replacement is performed in descending order of character name length.

[0021] Step Two: Encapsulating Image Hints The following information is encapsulated into a single image-based prompt and input into the large language model: [Reference Image Material] Image 1 incident:<scene_image_url> (Scene reference image) People in Image 2:<role_image_url_1> (Reference image for character 1) People in Image 3:<role_image_url_2> (Reference image for character 2) [Event Information] Original content:<original_text> Event Description:<event_desc> Participating characters: Character 1 (character in image 2), Character 2 (character in image 3) Scene: Scene Name (Image 1 Event) [Storyboard Constraint Rules] Shot type rotation: wide shot → long shot → medium shot → close-up → extreme close-up loop Camera movement: Pushing in the camera to enhance emotion, pulling back the camera to express loneliness. Spatial axis: Adhere to the 180-degree rule, and cross the axis without transition is prohibited. Lighting and shadow: Select color temperature according to the mood of the scene (e.g., 2500K firelight → privacy and danger). [Generation Requirements] Based on the information above, please generate a set of image prompts containing N storyboard panels (N=3 to 9). Each frame corresponds to a storyboard shot, requiring smooth visual flow and accurate character appearance and reference. Figure 1 To Scenarios and References Figure 1 To, in the format: [Image 1 description | Image 2 description | ... | Image N description] Step 3: Reasoning using the large language model's thought chain After receiving the encapsulated information, the large language model reasones through a chain-of-thought approach, plans the visual narrative logic of the entire sequence of shots, including shot composition, camera movement design, character actions and emotional changes, lighting selection, etc., and outputs a set of descriptive prompts for the entire event unit.

[0022] Step 4: Enhance and optimize prompt words For each storyboard prompt word output by the large language model, the large language model is called again for enhancement and optimization. The optimization includes: replacing the third-person pronouns ("he" / "she") in the prompt words with specific character reference image labels, detecting conflicts between actions and shot descriptions, supplementing lighting and shadow details, and adapting to the 9:16 vertical screen composition ratio.

[0023] Image group generation: Input the image group prompts output in step four and all reference images (e.g., the URL of the event in image 1, the URL of the person in image 2, etc.) into the image generation API. The image generation model will generate an image group containing multiple grid scenes at once, such as a nine-grid storyboard where each grid corresponds to one storyboard scene.

[0024] In another embodiment of the invention, a sub-scene hierarchy is further introduced, whereby the same large scene is divided into several sub-scenes, such as "Courtyard - Daytime" and "Courtyard - Nighttime." Each sub-scene generates its own reference image. The sub-scene reference image replaces the main scene reference image with the label "Image 1 Sub-scene" when participating in the construction of image prompts, thereby achieving finer-grained scene consistency constraints than the main scene level. The difference from the main solution is that the scene reference image changes from one level to two levels (large scene + sub-scene). The prompts can simultaneously include both the large scene image and the sub-scene image, providing more precise scene constraints for the image generation model.

[0025] In another embodiment of the present invention, an additional event summary map is generated for each event. This summary map integrates the positional relationships and spatial layout of all characters in the event and is injected as an additional reference map (labeled "Image 0 Event Summary") into the group map prompt of the next event, thereby achieving consistent scene transitions across events and solving the problem of visual jumps when switching between adjacent events. The difference from the main solution is that a cross-event visual transmission mechanism is introduced, the reference map list is updated with an event summary map, and the group map prompt includes descriptive constraints on the ending state of the previous event.

[0026] In another embodiment of the present invention, during the image generation stage, multiple image generation APIs (such as Seedream, Doubao, and Wan API) can be used concurrently to process image generation requests for different events. Through concurrent scheduling using ThreadPoolExecutor, image sets for multiple events can be generated simultaneously. The generation results from different APIs are filtered for quality using white background detection (analyze_image), and the highest quality image is selected as the final output. The difference from the main solution is that the image generation stage is changed from serial to concurrent, and a quality filtering mechanism is introduced, making it suitable for large-scale batch storyboard generation scenarios.

[0027] Compared with the prior art, the present invention has the following beneficial effects: By injecting standard reference images of characters into the image group prompts in the form of "Image N Character", the image generation model always uses the same reference image as a constraint when generating all storyboard frames, ensuring a high degree of consistency in the face, body shape, and clothing of the same character throughout the entire set of frames. This significantly improves scene consistency: by injecting standard scene reference images with the tag "Image 1 Event", the continuity of scene lighting, color tone, and spatial layout across all storyboard frames within the same event is guaranteed. It also enhances event-level context awareness: by planning image groups on an event-by-event basis, the large language model can simultaneously perceive the narrative needs of all storyboard frames within that event, thereby making overall shot composition, camera movement, and emotional arrangement, rather than generating images frame by frame independently. Furthermore, it reduces the number of image generation API calls: generating a multi-frame image group with a single call reduces the number of API calls compared to frame-by-frame independent calls, improving overall generation efficiency. The reference image tagging mechanism supports any number of characters and scenes, and handles multiple names for the same character through an alias mapping mechanism, exhibiting good scalability.

[0028] The technical solution of this invention introduces a reference image labeling mapping mechanism: the character reference image and the scene reference image are mapped to the labels "Image N Character" and "Image 1 Event" respectively, and storyboard prompts are injected to realize a method for visual consistency constraints in the image generation stage.

[0029] The event-level image prompt encapsulation method uses events as the smallest unit, and encapsulates the original text content, event description, character reference image, scene reference image, and storyboard constraint rules into a single image prompt, which is then input into a large language model for overall planning.

[0030] The proposed method uses a large language model-based thinking chain for storyboard planning. This model is based on image prompts and uses thinking chain reasoning to plan the shot size, camera movement, emotion, and lighting arrangement of the entire storyboard in one go, and outputs a multi-panel storyboard description.

[0031] In this embodiment, the method for generating multi-panel storyboard images in a single call includes multi-panel storyboard prompts and references. Figure 1 It also provides a method to generate a series of images containing multiple frames by inputting an image generation API.

[0032] Furthermore, the unified mapping mechanism for role aliases in the embodiment maps multiple aliases of the same role to the same reference map label and replaces them in descending order of name length to avoid incorrect substring replacement.

[0033] It is understandable that the AI ​​storyboard image generation system, composed of the above modules, includes a character image generation module, a scene image generation module, an event extraction and allocation module, a group image prompt word construction module, a large language model inference module, a group image generation module, and their collaborative system architecture.

[0034] This application also provides an electronic device, such as... Figure 4 As shown, Figure 4 The illustrated electronic device 700 includes a processor 701 and a memory 703. The processor 701 and the memory 703 are connected, for example, via a bus 702. Optionally, the electronic device 700 may also include a transceiver 704. It should be noted that in practical applications, the transceiver 704 is not limited to one type, and the structure of this electronic device 700 does not constitute a limitation on the embodiments of this application.

[0035] Processor 701 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 701 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0036] Bus 702 may include a pathway for transmitting information between the aforementioned components. Bus 702 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 702 can be divided into address bus, data bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0037] The memory 703 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0038] The memory 703 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 701. The processor 701 is used to execute the application code stored in the memory 703 to implement the content shown in the foregoing method embodiments.

[0039] Electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, PDAs (personal digital assistants), and PADs (tablet computers), as well as fixed terminals such as digital TVs and desktop computers. Servers can also be included. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0040] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed in a computer, causes the computer to execute the AI ​​storyboard image generation method provided in the above embodiments.

[0041] It should be understood that although the steps in the flowcharts in the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise expressly stated herein, there is no strict order in which these steps are performed, and they may be performed in other orders.

[0042] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for generating AI-generated storyboard images, characterized in that, The method includes: S10: Enter long text content; S20: Extract information from long texts, including character information, scene information, and event information; S30: Generate intermediate visual materials based on the extracted information, wherein the visual materials include reference images and text descriptions; S40: Based on the intermediate visual materials, construct image prompts; S50: Based on the image prompts and reference images, the image generation model generates a set of images containing multiple scenes in one go.

2. The AI ​​storyboard image generation method according to claim 1, characterized in that, The process of creating image prompts includes: S41: Reference graph label mapping: For each event, a reference graph label mapping relationship is established according to preset rules; S42: Image prompt encapsulation: Visual materials are uniformly encapsulated into a single image prompt and input into the large language model; the visual materials include reference image materials, event information, storyboard constraints, and generation requirements. S43: After receiving the above encapsulated information, the large language model reasones through the thought chain, plans the visual narrative logic of the entire storyboard, and outputs a set of storyboard description prompts for the entire event unit. S44: Call the large language model again to enhance and optimize each scene prompt word.

3. The AI ​​storyboard image generation method according to claim 2, characterized in that, When encapsulating image prompts: Reference materials include: scene reference images and character reference images for each character; Event information includes: original text, event description, participating roles, and the setting; Storyboard constraints include: shot size rotation, camera movement, spatial axis, and lighting characteristics.

4. The AI ​​storyboard image generation method according to claim 3, characterized in that, The visual narrative logic of the entire storyboard includes shot arrangement, camera movement design, character actions and emotional changes, and lighting selection, outputting a set of storyboard description prompts for the entire event unit.

5. The AI ​​storyboard image generation method according to claim 1, characterized in that, Step S50 includes: inputting the output image group prompts and all reference images into the image generation API, and having the image generation model generate an image group containing multiple grid scenes at once, with each grid corresponding to a storyboard scene.

6. The AI ​​storyboard image generation method according to claim 2, characterized in that, The enhancements and optimizations in S44 include: replacing third-person pronouns in prompts with specific character reference image tags, detecting conflicts between actions and camera descriptions, adding lighting and shadow details, and adapting to landscape / portrait screen aspect ratios.

7. The AI ​​storyboard image generation method according to claim 1, characterized in that, It also includes setting up a sub-scene hierarchy, which involves splitting the same large scene into multiple sub-scenes, generating a separate reference image for each sub-scene, and having the sub-scene reference image replace the main scene reference image with a label to participate in the construction of group image prompts.

8. The AI ​​storyboard image generation method according to claim 1, characterized in that, Also includes: An additional event summary diagram is generated for each event. The event summary diagram includes the positional relationships and spatial layout of all characters in the event, and is injected as an additional reference diagram into the group diagram prompt of the next event.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; Memory; At least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, said at least one application being configured to: perform the AI ​​storyboard image generation method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed in the computer, the computer is instructed to perform the AI ​​storyboard image generation method according to any one of claims 1 to 8.

Images