Story visualization method, story visualization server, and program

Abstract

This invention provides a method and apparatus for visualizing a story as an image using a generative artificial intelligence model. [Solution] The story visualization method includes: a story acquisition step of acquiring a story; an explanation information extraction step of extracting scene explanation information and character explanation information by analyzing the acquired story; a prompt generation step of generating prompts for each major scene of the story based on the extracted scene explanation information and character explanation information; an image generation step of generating background images and foreground images based on the prompts for each major scene of the story; and a story image generation step of generating a story image relating to the major scenes of the story by integrating the generated background images and foreground images.

Description

Technical Field

[0001] The present invention relates to story visualization technology, and more specifically, to a method and apparatus for visualizing a story as an image using a generative artificial intelligence model.

Background Art

[0002] Webtoon is a story-telling based visual content. Webtoon writers create a storyboard for their works, and based on this, they draw and color each frame of the story manually. Since such a creative process is carried out manually within a limited time (deadline), a lot of effort is required from Webtoon writers. Therefore, various artificial intelligence (AI) technologies have been proposed to support the creative activities of Webtoon writers. Among them, there is an increasing interest in story visualization technology using generative artificial intelligence (AI) models. Story visualization technology is a technology that visualizes a story as an image using generative AI models such as the GAN (Generative Adversarial Network) model, the VAE (Variational AutoEncoder) model, and the diffusion model.

[0003] However, conventional story visualization technology has problems that the image generated using a generative AI model cannot directly represent the narrative structure of the story (for example, the three-act structure). In addition, conventional story visualization technology lacks sufficient semantic consistency and context consistency between the story and the generated image, and it is difficult to systematically extract character attributes and narrative flow from the story. Therefore, there is a need for story visualization to solve the conventional problems.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present invention aims to solve the aforementioned problems and other problems. Another object of the present invention is to provide a method and apparatus for extracting scene description information and character description information by analyzing the narrative structure and characters of a story, and for generating prompts necessary for image generation based on the extracted scene description information and character description information.

[0005] Another object of the present invention is to provide a method and apparatus for generating images that can represent the narrative structure of a story by generating background images and foreground images based on prompts for each major scene of the story, and integrating the generated background and foreground images in an optimal manner.

[0006] A further object of the present invention is to provide a method and apparatus for generating images that have semantic and contextual consistency with the story by generating background and foreground images based on prompts for each major scene of the story, and integrating the generated background and foreground images in an optimal manner. [Means for solving the problem]

[0007] To achieve the aforementioned or other objectives, according to one aspect of the present invention, a story visualization method is provided, comprising: a story acquisition step of acquiring a story; an explanation information extraction step of extracting scene explanation information and character explanation information by analyzing the acquired story; a prompt generation step of generating prompts for each major scene of the story based on the extracted scene explanation information and character explanation information; an image generation step of generating background images and foreground images based on the prompts for each major scene of the story; and a story image generation step of integrating the generated background images and foreground images to generate a story image relating to the major scenes of the story.

[0008] According to another aspect of the present invention, a story visualization server is provided, comprising one or more processors that execute a plurality of instructions for performing a plurality of operations, and one or more memories for storing the plurality of instructions, wherein the plurality of operations include a story acquisition operation for acquiring a story, an explanation information extraction operation for extracting scene explanation information and character explanation information by analyzing the acquired story, a prompt generation operation for generating prompts for each major scene of the story based on the extracted scene explanation information and character explanation information, an image generation operation for generating background images and foreground images based on the prompts for each major scene of the story, and a story image generation operation for generating a story image relating to the major scenes of the story by integrating the generated background images and foreground images.

[0009] According to yet another aspect of the present invention, a computer program stored on a computer-readable recording medium is provided so that a computer can perform the following steps: a story acquisition step of acquiring a story; an explanation information extraction step of extracting scene explanation information and character explanation information by analyzing the acquired story; a prompt generation step of generating prompts for each major scene of the story based on the extracted scene explanation information and character explanation information; an image generation step of generating background images and foreground images based on the prompts for each major scene of the story; and a story image generation step of integrating the generated background images and foreground images to generate a story image relating to the major scenes of the story. Furthermore, a program for causing a computer to perform the above steps is also provided.

[0010] It should be noted that the means of solving the aforementioned problems do not constitute a complete list of all the features of the present invention. Furthermore, the various features of the present invention, as well as their benefits and effects, can be understood in more detail by referring to the specific embodiments described later. [Effects of the Invention]

[0011] The effects of the story visualization method and apparatus according to the embodiment of the present invention are as follows.

[0012] According to at least one embodiment of the present invention, by analyzing and visualizing the main scenes and characters of a story based on a text generation model and an image generation model, it is possible to directly represent the narrative structure of the story and generate high-quality images that have semantic consistency and contextual consistency with respect to the story.

[0013] However, the effects that can be achieved by the story visualization method and apparatus according to the embodiments of the present invention are not limited to those described above. Other effects not mentioned above will be clearly understood by those with ordinary skill in the art to which the present invention belongs from the description below. [Brief explanation of the drawing]

[0014] [Figure 1] This is a diagram showing the configuration of a story visualization system according to one embodiment of the present invention. [Figure 2] This is a block diagram of the story visualization server configuration for one embodiment of the present invention. [Figure 3] This is an example diagram showing the prompts for each scene, generated through the story module. [Figure 4] This is an illustrative diagram showing multiple scene images generated through the image module. [Figure 5a] This is a reference diagram showing the operation of the scene extraction unit according to one embodiment of the present invention. [Figure 5b] This is a reference diagram showing the operation of the scene extraction unit according to one embodiment of the present invention. [Figure 6a] This is a reference diagram showing the operation of a character extraction unit according to one embodiment of the present invention. [Figure 6b] This is a reference diagram showing the operation of a character extraction unit according to one embodiment of the present invention. [Figure 7a] It is a reference diagram showing the operation of the prompt generation unit according to an embodiment of the present invention. [Figure 7b] It is a reference diagram showing the operation of the prompt generation unit according to an embodiment of the present invention. [Figure 8a] It is a reference diagram showing the operation of the prompt evaluation unit according to an embodiment of the present invention. [Figure 8b] It is a reference diagram showing the operation of the prompt evaluation unit according to an embodiment of the present invention. [Figure 9a] It is a reference diagram showing the operation of the scene element generation unit according to an embodiment of the present invention. [Figure 9b] It is a reference diagram showing the operation of the scene element generation unit according to an embodiment of the present invention. [Figure 10a] It is a reference diagram showing the operation of the position extraction unit according to an embodiment of the present invention. [Figure 10b] It is a reference diagram showing the operation of the position extraction unit according to an embodiment of the present invention. [Figure 11a] It is a reference diagram showing the operation of the scene element integration unit according to an embodiment of the present invention. [Figure 11b] It is a reference diagram showing the operation of the scene element integration unit according to an embodiment of the present invention. [Figure 11c] It is a reference diagram showing the operation of the scene element integration unit according to an embodiment of the present invention. [Figure 12] It is a flowchart showing a story visualization method according to an embodiment of the present invention. [Figure 13] It is a configuration block diagram of a computer device according to an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0015] The embodiments disclosed herein will now be described in detail with reference to the accompanying drawings, but identical or similar components will be given the same reference numeral, regardless of the drawing number, and redundant descriptions will be omitted. The suffixes “module” and “part” used in the following description with respect to components are given or mixed for the sole purpose of ease of specification preparation and do not have a distinct meaning or role in themselves. That is, the term “part” as used in the present invention means a software, FPGA or ASIC or other hardware component, and “part” plays a role. However, “part” is not limited to software or hardware. A “part” may be configured to reside on an addressable storage medium and may be configured to regenerate one or more processors. Thus, as an example, a “part” includes components such as software components, object-oriented software components, class components, and task components, and processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The components and the functions provided within each "part" can be combined into a smaller number of components and "parts," or they can be further separated into additional components and "parts."

[0016] Furthermore, in describing the embodiments disclosed herein, if it is determined that a specific description of related known technology would obscure the gist of the embodiments disclosed herein, such detailed description will be omitted. In addition, the accompanying drawings are merely for the purpose of facilitating the understanding of the embodiments disclosed herein, and should be understood not to limit the technical ideas disclosed herein, but to include all modifications, equivalents, or substitutes that fall within the concept and technical scope of the present invention.

[0017] This invention proposes a method and apparatus for analyzing the narrative structure and characters of a story to extract scene description information and character description information, and generating prompts necessary for image generation based on the extracted scene description information and character description information. Furthermore, this invention proposes a method and apparatus for generating background images and foreground images based on prompts for each major scene of a story, and integrating the generated background and foreground images in an optimal manner to generate an image that represents the narrative structure of the story. Furthermore, this invention proposes a method and apparatus for generating background images and foreground images based on prompts for each major scene of a story, and integrating the generated background and foreground images in an optimal manner to generate an image that has semantic consistency and contextual consistency with respect to the story. This invention is applicable to all content with a story, including webtoons, web videos, web novels, games, movies, dramas, and animations.

[0018] Various embodiments of the present invention will be described in detail below with reference to the drawings.

[0019] Figure 1 is a diagram showing the configuration of a story visualization system according to one embodiment of the present invention.

[0020] Referring to Figure 1, a story visualization system 10 according to one embodiment of the present invention may include a user terminal 100, a story visualization server 200, an AI server 300, and a communication network 400. The story visualization server 200 may be referred to as a story visualization device.

[0021] The user terminal 100 and the story visualization server 200 can be connected to each other via a communication network 400. The communication network 400 can include wired and wireless networks, and specifically, it can include various networks such as local area networks (LANs), metropolitan area networks (MANs), and wide area networks (WANs). The communication network 400 can also include the well-known World Wide Web (WWW). However, the communication network 400 according to the present invention is not limited to the networks listed above, and may also include at least one of a well-known wireless data network, a well-known telephone network, or a well-known wired / wireless television network.

[0022] The user terminal 100 can work in conjunction with the story visualization server 200 to provide a service that visualizes stories into images (hereinafter referred to as the "story visualization service"). In this case, the user terminal 100 can display a user interface (UI) for providing the story visualization service on its display unit.

[0023] The user terminal 100 can provide the story visualization server 200 with a story to be visualized (hereinafter referred to as the "target story") in response to user commands or other instructions. In this case, the story consists of text data or audio data, but is not necessarily limited to these. In this embodiment, the story will be described as text data as an example.

[0024] The user terminal 100 can receive intermediate results generated in the process of visualizing a story as an image from the story visualization server 200. The user terminal 100 can display the intermediate results received from the story visualization server 200 on the user interface (UI). The user can review the intermediate results displayed on the user interface (UI) and modify them according to the results of the review. If the user modifies the intermediate results, the user terminal 100 can send user feedback information regarding the intermediate results to the story visualization server 200. The story visualization server 200 can update the intermediate results based on the received user feedback information.

[0025] The user terminal 100 described herein may include, but is not limited to, a desktop computer, a laptop computer, a slate PC, a tablet PC, an ultrabook, a mobile phone, a smartphone, a wearable device, and the like.

[0026] The story visualization server 200 can work in conjunction with the AI ​​server 300 to provide story visualization services to the user terminal 100. The story visualization server 200 is a web server or a web application server, but is not necessarily limited to these.

[0027] The story visualization server 200 analyzes the story received from the user terminal 100 to extract scene description information and character description information, and can generate prompts necessary for image generation based on the extracted scene description information and character description information. At this time, the story visualization server 200 can analyze the narrative structure and characters of the story to extract major scenes, and can generate prompts for each of the extracted major scenes. These prompts may include background prompts and foreground prompts.

[0028] The story visualization server 200 generates background and foreground images based on prompts for each major scene of the story, and can integrate the generated background and foreground images in an optimal manner to generate multiple scene (story) images. At this time, the scene images can be generated so as to represent the narrative structure of the story and have semantic consistency and context consistency with the story.

[0029] The story visualization server 200 can store the generated multiple scene images in its storage. The story visualization server 200 can also provide the generated multiple scene images to the user terminal 100.

[0030] The AI ​​server 300 can construct multiple generative AI models. These multiple generative AI models may include text generation models and image generation models. As the text generation model, a Large Language Model (LLM) such as Chat-GPT or LLAMA can be used. As the image generation model, a Generative Adversarial Network (GAN) model, a Variational AutoEncoder (VAE) model, or a Diffusion Model (DM) can be used, with a Diffusion Model being more preferable.

[0031] The AI ​​server 300 can generate text data and / or image data based on prompts received from the story visualization server 200 using multiple generative AI models. The AI ​​server 300 can provide the generated text data and / or image data to the story visualization server 200.

[0032] On the other hand, while this embodiment illustrates that the AI ​​server 300 providing multiple generative AI models is constructed independently outside the story visualization server 200, it is not necessarily limited to this. Therefore, it will be obvious to those skilled in the art that the AI ​​server 300 can be constructed within the story visualization server 200.

[0033] Figure 2 is a block diagram of the configuration of a story visualization server according to one embodiment of the present invention.

[0034] Referring to Figure 2, a story visualization server 200 according to one embodiment of the present invention may include a story module 210 and an image module 220. The story module 210 may include a scene extraction unit 211, a character extraction unit 212, a prompt generation unit 213, and a prompt evaluation unit 214. The image module 220 may include a scene element generation unit 221, a position extraction unit 222, and a scene element integration unit 223. Since the components shown in Figure 2 are not essential for realizing the story visualization server, the story visualization server described herein may have more or fewer components than those listed above.

[0035] The story module 210 can extract scene description information and character description information by analyzing the main scenes and characters of the target story, and can generate prompts necessary for image generation based on the extracted scene description information and character description information. The story module 210 can provide the generated prompts to the image module 220.

[0036] More specifically, the scene extraction unit 211 can classify the narrative structure of the target story using a text generation model, and for each classified narrative structure, it can identify the main scenes and extract scene description information. Here, the scene description information may include information about the background, foreground, and interactions included in the main scenes of the target story. Furthermore, the narrative structure of the story can be a three-act structure, a five-act structure, or the like, and is not necessarily limited to these.

[0037] The scene extraction unit 211 can provide the extracted scene description information to the user terminal 100. This is so that the user can review the scene description information and correct any errors. The scene extraction unit 211 can update the scene description information based on user feedback information received from the user terminal 100. The scene extraction unit 211 can provide the updated scene description information to the prompt generation unit 213.

[0038] The character extraction unit 212 can analyze the main characters appearing in the target story using a text generation model and extract character description information based on this analysis. Here, the character description information may include character characteristic (attribute) information and relationship information included in the main scenes of the target story. The character characteristic information can be defined using predefined categories (e.g., gender, clothing, actions, location, etc.).

[0039] The character extraction unit 212 can provide the extracted character description information to the user terminal 100. This is so that the user can review the character description information and correct any errors. The character extraction unit 212 can update the character description information based on user feedback information received from the user terminal 100. The character extraction unit 212 can provide the updated character description information to the prompt generation unit 213.

[0040] The prompt generation unit 213 aggregates scene description information and character description information, and can generate prompts necessary for image generation based on the aggregated integrated description information. At this time, the prompt generation unit 213 can generate prompts for each major scene of the story using a text generation model. The prompts can be generated in a form optimized for input to the diffusion model.

[0041] For example, as shown in Figure 3, the prompt generation unit 213 can generate prompts for each act and scene of the story using the LLM model. Each prompt may include one background prompt and one or more foreground prompts. The foreground prompts can be generated for each character included in the scene. If there are N characters in a particular scene, then N foreground prompts can be generated.

[0042] The prompt generation unit 213 can provide prompts for each major scene of the target story to the prompt evaluation unit 214.

[0043] The prompt evaluation unit 214 can use a text generation model to calculate evaluation scores for prompts in each major scene of the target story. At this time, the prompt evaluation unit 214 can calculate evaluation scores by measuring the similarity between the prompts in each major scene of the story and the original text content of the story.

[0044] The prompt evaluation unit 214 can provide the user terminal 100 with information regarding the calculated evaluation score.

[0045] The image module 220 can generate background and foreground images for each main scene of the target story based on prompts for each scene, and then integrate the generated background and foreground images in an optimal manner to generate multiple scene images.

[0046] More specifically, the scene element generation unit 221 can obtain prompts for each major scene of the target story from the story module 210. The scene element generation unit 221 can generate one background image based on one background prompt using an image generation model. In addition, the scene element generation unit 221 can generate one or more foreground images based on one or more foreground prompts using an image generation model.

[0047] The scene element generation unit 221 can provide background images and foreground images related to the main scenes of the target story to the position extraction unit 222 and the scene element integration unit 223.

[0048] The position extraction unit 222 can use a text generation model to extract position information of characters included in each scene based on the background image and global prompts. Here, the global prompt may include one background prompt and one or more foreground prompts.

[0049] The position extraction unit 222 can provide the scene element integration unit 223 with character position information relating to the main scenes of the target story.

[0050] The scene element integration unit 223 can aggregate background and foreground images based on character position information to generate a stitched image. The scene element integration unit 223 can input the generated stitched image into an image generation model to generate the final scene image.

[0051] For example, as shown in Figure 4, the scene element integration unit 223 can generate high-quality scene images using a diffusion model. The diffusion model uses a Semantic Aware-Cross Attention (SA-CA) layer to represent the narrative structure of a story and can generate scene images that have semantic consistency and contextual consistency with respect to that story.

[0052] The scene element integration unit 223 can provide the scene images of the target story to the user terminal 100.

[0053] As described above, the story visualization server according to one embodiment of the present invention can generate high-quality images that accurately represent the narrative structure of a story and have semantic and contextual consistency with respect to the story, by analyzing and visualizing the main scenes and characters of the story based on a text generation model and an image generation model.

[0054] The components of story module 210 will be explained in more detail below.

[0055] Figures 5a and 5b are reference diagrams showing the operation of the scene extraction unit according to one embodiment of the present invention.

[0056] Referring to Figures 5a and 5b, the scene extraction unit 211 according to the present invention can acquire the target story from the user terminal 100 (S501).

[0057] The scene extraction unit 211 can generate a first LLM prompt that includes the target story and a first instruction. Here, the first instruction may be a command requesting the classification of the story's narrative structure.

[0058] The scene extraction unit 211 can send the generated first LLM prompt to the AI ​​server 300 (S503).

[0059] The AI ​​server 300 can input the first LLM prompt received from the scene extraction unit 211 into the LLM model to classify the narrative structure of the story (S504). For example, as shown in Figure 5b, the first LLM model 510 can analyze the narrative structure of the target story and classify it into three acts: the introduction (Act 1), the conflict (Act 2), and the resolution (Act 3).

[0060] The AI ​​server 300 can provide the scene extraction unit with information regarding the classified narrative structure (S505).

[0061] The scene extraction unit 211 can generate a second LLM prompt including text information separated for each narrative structure, predefined parameter information, and second instruction information (S506). Here, the parameter information refers to information to be extracted from the main scenes of the target story (e.g., background information, foreground information, etc.). The second instruction information may be an instruction requesting the extraction of a description of the main scenes of the target story.

[0062] The scene extraction unit 211 can send the generated second LLM prompt to the AI ​​server 300 (S507).

[0063] The AI ​​server 300 can input the second LLM prompt received from the scene extraction unit 211 into the LLM model to extract descriptive information about the main scenes of the target story (S508). For example, as shown in Figure 5b, the second LLM model 520 can grasp the main scenes for each narrative structure of the target story and extract multiple scene descriptive pieces of information. In this case, each scene descriptive piece of information may include information about the background, foreground, and interactions included in the main scenes of the target story.

[0064] The AI ​​server 300 can transmit scene description information of the target story to the scene extraction unit 211 (S509). The scene extraction unit 211 can store the scene description information received from the AI ​​server 300 in storage (not shown).

[0065] The scene extraction unit 211 can transmit scene description information of the target story to the user terminal 100 (S510). The user terminal 100 can display the scene description information received from the scene extraction unit 211 on its display unit. The user can review the scene description information displayed on the user terminal 100 and modify the scene description information according to the results of the review (S511).

[0066] If the user modifies the scene description information, the scene extraction unit 211 can receive user feedback information regarding the scene description information from the user terminal 100 (S512). Based on the user feedback information received from the user terminal 100, the scene extraction unit 211 can update the scene description information of the target story (S513).

[0067] The scene extraction unit 211 can provide the updated scene description information to the prompt generation unit 213.

[0068] Figures 6a and 6b are reference diagrams showing the operation of a character extraction unit according to one embodiment of the present invention.

[0069] Referring to Figures 6a and 6b, the character extraction unit 212 according to the present invention can acquire the target story from the user terminal 100 (S601).

[0070] The character extraction unit 212 can generate an LLM prompt that includes the target story, predefined category information, and instruction information (S602). Here, the category information refers to information that indicates the characteristics of the characters appearing in the target story (e.g., gender, clothing, actions, location, etc.). The instruction information may be an instruction that requests the extraction of a description of the main characters appearing in the story.

[0071] The character extraction unit 212 can send the generated LLM prompt to the AI ​​server 300 (S603).

[0072] The AI ​​server 300 can input the LLM prompt received from the character extraction unit 212 into the LLM model and extract descriptive information about the characters appearing in the target story (S604). For example, as shown in Figure 6b, the LLM model 610 can grasp the characteristics of the main characters appearing in the target story and extract multiple character descriptive pieces of information. At this time, each character descriptive piece of information may include characteristic information and relationship information of the characters included in the main scenes of the target story.

[0073] The AI ​​server 300 can transmit character description information for the target story to the character extraction unit 212 (S605). The character extraction unit 212 can store the character description information received from the AI ​​server 300 in its storage.

[0074] The character extraction unit 212 can transmit character description information for the target story to the user terminal 100 (S606). The user terminal 100 can display the character description information received from the character extraction unit 212 on its display unit. The user can review the character description information displayed on the user terminal 100 and modify the character description information according to the review results (S607).

[0075] If the user modifies the character description information, the character extraction unit 212 can receive user feedback information regarding the character description information from the user terminal 100 (S608). Based on the user feedback information received from the user terminal 100, the character extraction unit 212 can update the character description information for the target story (S609).

[0076] The character extraction unit 212 can provide the updated character description information to the prompt generation unit 213.

[0077] Figures 7a and 7b are reference diagrams showing the operation of a prompt generation unit according to one embodiment of the present invention.

[0078] Referring to Figures 7a and 7b, the prompt generation unit 213 according to the present invention can obtain scene description information of the target story from the scene extraction unit 211 and character description information of the target story from the character extraction unit 212 (S701).

[0079] The prompt generation unit 213 can aggregate the acquired scene description information and character description information to generate integrated description information (S702). At this time, the integrated description information can be generated for each major scene of the target story.

[0080] The prompt generation unit 213 can generate an LLM prompt that includes integrated explanatory information and instruction information (S703). Here, the instruction information may be an instruction requesting the generation of background prompts and foreground prompts necessary for image generation.

[0081] The prompt generation unit 213 can send the generated LLM prompt to the AI ​​server 300 (S704).

[0082] The AI ​​server 300 can input the LLM prompt received from the prompt generation unit 213 into the LLM model to generate background and foreground prompts related to the main scenes of the target story (S705). At this time, the background and foreground prompts may be text data. Furthermore, the number of foreground prompts can be increased to the number of characters included in the scene.

[0083] For example, as shown in Figure 7b, the LLM model 710 can understand the background and foreground of the main scenes of the target story and generate background and foreground prompts. For instance, the background prompt for a particular scene might be "A towering, fantastical beanstalk spiraling up into the clouds," and the foreground prompt for that scene might be "A young boy in green medieval clothing climbing the beanstalk."

[0084] The AI ​​server 300 can send prompt information for each major scene of the target story to the prompt generation unit 213 (S706). The prompt generation unit 213 can store the prompt information received from the AI ​​server 300 in storage (S707).

[0085] The prompt generation unit 213 can provide prompt information for each major scene of the target story to the prompt evaluation unit 214.

[0086] Figures 8a and 8b are reference diagrams showing the operation of the prompt evaluation unit according to one embodiment of the present invention.

[0087] Referring to Figures 8a and 8b, the prompt evaluation unit 214 according to the present invention can obtain prompt information for each major scene of the target story from the prompt generation unit 213 (S801).

[0088] The prompt evaluation unit 214 can generate LLM prompts that include text information for each narrative structure of the target story, prompt information for each major scene of the target story, and instruction information (S802). Here, the instruction information may be an instruction requesting the calculation of an evaluation score for each prompt in each major scene of the target story.

[0089] The prompt evaluation unit 214 can send the generated LLM prompt to the AI ​​server 300 (S803).

[0090] The AI ​​server 300 can input the LLM prompts received from the prompt evaluation unit 214 into the LLM model and calculate an evaluation score for each prompt in the main scenes of the target story (S804). For example, as shown in Figure 8b, the LLM model 810 can calculate an evaluation score by measuring the similarity between the prompts in each main scene of the story and the content of the original text of the story. This is to quantitatively confirm to what extent the prompts in each main scene of the story contain content that matches the original text of the story.

[0091] The AI ​​server 300 can transmit the calculated evaluation score information to the prompt evaluation unit 214 (S805). The prompt evaluation unit 214 can store the evaluation score information received from the AI ​​server 300 in its storage.

[0092] The prompt evaluation unit 214 can send evaluation scores for each prompt in the main scenes of the target story to the user terminal 100 (S806). The user terminal 100 can display the evaluation score information received from the prompt evaluation unit 214 on its display unit. The user can review the evaluation score information displayed on the user terminal 100 and modify the prompt information according to the results of the review (S807).

[0093] If a user modifies a specific prompt, the prompt evaluation unit 214 can receive user feedback information for that prompt from the user terminal 100 (S808). The prompt evaluation unit 214 can update the prompt information based on the user feedback information received from the user terminal 100 (S809). On the other hand, according to one embodiment of the present invention, the process of updating the prompt based on user feedback information can be omitted.

[0094] The prompt evaluation unit 214 can provide updated prompt information to the image module 220.

[0095] The components of the image module 220 will be described in more detail below.

[0096] Figures 9a and 9b are reference diagrams showing the operation of a scene element generation unit according to one embodiment of the present invention.

[0097] Referring to Figures 9a and 9b, the scene element generation unit 221 according to the present invention can obtain background and foreground prompts related to the main scenes of the target story from the story module 210 (S901).

[0098] The scene element generation unit 221 can generate a DM prompt that includes background and foreground prompt information, reference image information, and instruction information related to the main scenes of the target story (S902). Here, the instruction information may be a command requesting the generation of background and foreground images. The reference image information is a scene image previously generated with respect to the current time, and can be used to generate a scene image for the next time. The reference image information is optional in embodiments of the present invention.

[0099] The scene element generation unit 221 can send the generated DM prompt to the AI ​​server 300 (S903).

[0100] The AI ​​server 300 can input DM prompts received from the scene element generation unit 221 into the diffusion model to generate background and foreground images related to the main scenes of the target story (S904). For example, as shown in Figure 9b, the diffusion model 910 can generate one background image based on the input background prompt. The diffusion model 910 can also generate one or more foreground images based on one or more input foreground prompts.

[0101] The AI ​​server 300 can transmit background and foreground images related to the main scenes of the target story to the scene element generation unit 221 (S905). The scene element generation unit 221 can store the background and foreground images received from the AI ​​server 300 in its storage (S906).

[0102] The scene element generation unit 221 can provide the generated background image and foreground image to the position extraction unit 222 and the scene element integration unit 223.

[0103] Figures 10a and 10b are reference diagrams showing the operation of a position extraction unit according to one embodiment of the present invention.

[0104] Referring to Figures 10a and 10b, the position extraction unit 222 according to the present invention can obtain background images relating to the main scenes of the target story from the scene element generation unit 221 (S1001). The position extraction unit 222 can also obtain global prompts relating to the main scenes of the target story from the story module 210. Here, the global prompts may include one background prompt and one or more foreground prompts.

[0105] The position extraction unit 222 can generate an LLM prompt including background image information, global prompt information, and instruction information (S1002). Here, the instruction information may be an instruction requesting position information of a character included in the main scene of the target story. On the other hand, in another embodiment, the LLM prompt may further include foreground image information.

[0106] The position extraction unit 222 can send the generated LLM prompt to the AI ​​server 300 (S1003).

[0107] The AI ​​server 300 inputs the LLM prompt received from the position extraction unit 222 into the LLM model, and can extract positional information of characters included in the main scenes of the target story (S1004). For example, as shown in Figure 10b, the LLM model 1010 can analyze the semantic relationship between the background and foreground elements of each scene to detect where the character is located on the background image.

[0108] The AI ​​server 300 can transmit the extracted character position information to the position extraction unit 222 (S1005). The position extraction unit 222 can store the character position information received from the AI ​​server 300 in storage. The position extraction unit 222 can provide the character position information relating to the main scenes of the target story to the scene element integration unit 223.

[0109] Figures 11a to 11c are reference diagrams showing the operation of a scene element integration unit according to one embodiment of the present invention.

[0110] Referring to Figures 11a to 11c, the scene element integration unit 223 according to the present invention can obtain global prompts relating to the main scenes of the target story from the story module 210 (S1101).

[0111] The scene element integration unit 223 can obtain background and foreground images related to the main scenes of the target story from the scene element generation unit 221. In addition, the scene element integration unit 223 can obtain character position information related to the main scenes of the target story from the position extraction unit 222.

[0112] The scene element integration unit 223 can generate a stitched image by combining background and foreground images layer by layer based on the position information of the characters included in each scene (S1102). At this time, the stitched image can be generated for each major scene of the target story. Furthermore, the stitched image can be used as an image to be subjected to noise reduction.

[0113] The scene element integration unit 223 can generate a DM prompt that includes a stitched image, diffusion condition information, and instruction information relating to the main scene of the target story (S1103). Here, the diffusion condition information may include at least one of a background image, a foreground image, a global prompt, and character position information. The instruction information may be an instruction requesting the generation of a high-quality scene image based on the stitched image.

[0114] The scene element integration unit 223 can send the generated DM prompt to the AI ​​server 300 (S1104).

[0115] The AI ​​server 300 inputs the DM prompt received from the scene element integration unit 223 into the diffusion model to generate the final image for the main scenes of the target story (S1105).

[0116] For example, as shown in Figure 11b, the diffusion model 1110 can generate a high-quality scene image by sequentially removing noise contained in the input stitched image while referring to the input diffusion condition information. Also, as shown in Figure 11c, the diffusion model 1110 can use the SA-CA layer to represent the narrative structure of the target story and generate a scene image that has semantic consistency and contextual consistency with respect to the target story. For example, if the feature vector of the stitched image is set as the query vector and the feature vector of the diffusion condition information is set as the key vector and value vector, the diffusion model 1110 can perform cross-attention between the feature vector of the stitched image and the feature vector of the diffusion condition information. Furthermore, the diffusion model 1110 can maintain semantic consistency between text tokens and image tokens using the AdaIN layer.

[0117] The AI ​​server 300 can transmit the generated scene image to the scene element integration unit 223 (S1106). The scene element integration unit 223 can store the scene image received from the AI ​​server 300 in storage (S1107). The scene element integration unit 223 can provide the scene image related to the target story to the user terminal 100.

[0118] Figure 12 is a flowchart illustrating a story visualization method according to one embodiment of the present invention. The story visualization method according to this embodiment can be performed by a story visualization server 200. In the illustrated flowchart, the story visualization method is described in several steps, but at least some of the steps can be performed in a different order, combined with other steps, omitted, divided into detailed steps, or with the addition of one or more steps (not shown).

[0119] Referring to Figure 12, the story visualization server 200 according to the present invention can acquire the target story from the user terminal 100 (S1201).

[0120] The story visualization server 200 can use the LLM model to classify the narrative structure of the acquired target story based on the target story (S1202).

[0121] The story visualization server 200 can use an LLM model to extract scene description information for each major scene of the target story based on text information classified according to the narrative structure of the target story (S1203). At this time, the scene description information may include background information, foreground information, and interaction information included in the major scenes of the target story.

[0122] The story visualization server 200 can use the LLM model to extract character description information for each major scene of the acquired target story (S1204). At this time, the character description information may include characteristic information and relationship information of the characters included in the major scenes of the target story.

[0123] The story visualization server 200 can aggregate scene description information and character description information for each major scene of the target story to generate integrated description information. Using an LLM model, the story visualization server 200 can generate background prompts and foreground prompts for the major scenes of the target story based on the generated integrated description information (S1205).

[0124] The story visualization server 200 can use a diffusion model to generate background images and foreground images for the main scenes of the target story, based on background prompts and foreground prompts for the main scenes of the target story (S1206).

[0125] The story visualization server 200 can use the LLM model to extract character position information for the main scenes of the target story based on background images and global prompts related to the main scenes of the target story (S1207).

[0126] The story visualization server 200 can stitch background and foreground images together based on extracted character position information to generate stitched images of the main scenes of the target story (S1208). Here, image stitching refers to the technique of joining multiple images together to create a single image.

[0127] The story visualization server 200 can use a diffusion model to generate a final image of the main scenes of the target story based on stitched images and diffusion condition information of the main scenes of the target story (S1209). At this time, the diffusion model can use an SA-CA layer to represent the narrative structure of the target story and generate a final image that has semantic consistency and contextual consistency with respect to the target story.

[0128] The story visualization server 200 can store the final images of the main scenes of the target story in its storage. The story visualization server 200 can also provide the final images of the main scenes of the target story to the user terminal 100.

[0129] As described above, the story visualization method according to one embodiment of the present invention analyzes and visualizes the main scenes and characters of a story based on a text generation model and an image generation model, thereby representing the narrative structure of the story and generating high-quality images that have semantic consistency and contextual consistency within the story.

[0130] Figure 13 is a block diagram of the configuration of a computer device according to one embodiment of the present invention.

[0131] Referring to Figure 13, the computer device 1300 according to one embodiment of the present invention may include at least one processor 1310, a computer-readable storage medium 1320, and a communication bus 1330. The computer device 1300 can implement the story visualization server 200 described above, or the elements 211 to 223 that constitute the story visualization server 200. Furthermore, the computer device 1300 can implement the user terminal 100 or AI server 300 described above.

[0132] The processor 1310 can cause the computer device 1300 to operate according to the exemplary embodiments described above. For example, the processor 1310 can execute one or more programs 1325 stored in a computer-readable storage medium 1320. The one or more programs may include one or more computer-executable instruction words, which, when executed by the processor 131, can be configured to cause the computer device 1300 to operate according to the exemplary embodiments.

[0133] The computer-readable storage medium 1320 is configured to store computer-executable instructions or program code, program data and / or other appropriate forms of information. The program 1325 stored in the computer-readable storage medium 1320 includes a set of instructions that can be executed by the processor 1310. In one embodiment, the computer-readable storage medium 1320 may be memory (volatile memory such as random access memory, non-volatile memory, or an appropriate combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, other forms of storage media that can be accessed by the computer device 1300 and store the desired information, or an appropriate combination thereof.

[0134] The communication bus 1330 interconnects various other components of the computer device 1300, including the processor 1310 and the computer-readable storage medium 132.

[0135] Furthermore, the computer device 1300 may include one or more input / output interfaces 1340 that provide interfaces for one or more input / output devices 1350, and one or more network communication interfaces 1360. The input / output interfaces 1340 and the network communication interfaces 1360 are connected to the communication bus 1330.

[0136] The input / output device 1350 can be connected to other components of the computer device 1300 via the input / output interface 1340. An exemplary input / output device 1350 may include input devices such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), an audio or sound input device, various types of sensor devices and / or imaging devices, and / or output devices such as a display device, a printer, a speaker and / or a network card. The exemplary input / output device 1350 may be included inside the computer device 1300 as a component of the computer device 1300, or it may be connected to the computer device 1300 as a separate device distinct from the computer device 1300.

[0137] A story visualization server according to an embodiment of the present invention can be implemented by at least one computer device 1300, and a story visualization method according to an embodiment of the present invention can be performed by at least one computer device included in the story visualization server. In this case, a computer program according to one embodiment of the present invention can be installed and driven in the computer device, and the computer device can perform the story visualization method according to an embodiment of the present invention based on the control of the driven computer program. The above-described computer program can be coupled with the computer device to cause the computer to execute the story visualization method. Furthermore, the computer program can be stored on a computer-readable recording medium in order to be coupled with the computer device to cause the computer to execute the story visualization method.

[0138] The present invention, as described above, can be realized as computer-readable code on a medium on which a program is recorded. A computer-readable medium may continuously store programs executable by a computer, or it may temporarily store them for execution or download. Furthermore, the medium can be various recording or storage means in the form of a combination of one or more hardware components, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Other examples of media include recording media or storage media managed by app stores and other sites, servers, etc., that supply or distribute applications and various other software. Therefore, the above detailed description should not be interpreted restrictively in any way, but should be considered illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included within the scope of the present invention.

[0139] The present invention is not limited by the embodiments described above and the accompanying drawings. It will be apparent to those with ordinary skill in the art to which the present invention pertains that the components of the present invention can be substituted, modified, or changed without departing from the technical spirit of the present invention. [Explanation of symbols]

[0140] 10 Story Visualization System 100 user terminals 200 Story Visualization Server 300 AI servers 400 Communication Networks 210 Story Modules 220 Image Modules 1300 Computer equipment

Claims

1. A story visualization method performed by a computer device, The story acquisition step to retrieve the story, The process involves extracting descriptive information by analyzing the acquired story, and extracting scene description information and character description information. A prompt generation step that generates prompts for each major scene of the story based on extracted scene description information and character description information, Image generation step that generates background and foreground images based on prompts for each main scene of the aforementioned story, A story visualization method comprising: a story image generation step of generating a story image relating to the main scenes of the story by integrating generated background and foreground images.

2. The above-mentioned explanatory information extraction step is: The first step involves classifying the narrative structure of the acquired story using a large-scale language model, A story visualization method according to claim 1, comprising the steps of: using a second large language model to identify a main scene for each classified narrative structure; and extracting scene description information for each identified main scene.

3. The story visualization method according to claim 1, wherein the scene description information includes background information and foreground information included in the main scenes of the story.

4. The above-mentioned explanatory information extraction step is: The story visualization method according to claim 1, wherein character description information is extracted for each major scene of the story using a third large language model.

5. The story visualization method according to claim 1, wherein the character description information includes characteristic information and relationship information of characters included in the main scenes of the story.

6. The steps include providing the extracted scene description information and character description information to the user terminal, The story visualization method according to claim 1, further comprising the step of updating the scene description information and the character description information based on user feedback information received from the user terminal.

7. The prompt generation step is, The steps include: aggregating extracted scene description information and character description information to generate integrated description information; The story visualization method according to claim 1, further comprising the step of generating prompts for each major scene of the story based on the integrated explanatory information using a fourth large language model.

8. A step of calculating an evaluation score for the generated prompt using a fifth large language model, The story visualization method according to claim 1, further comprising the step of providing the calculated evaluation score to the user terminal.

9. The story visualization method according to claim 1, wherein the prompts include one background prompt and one or more foreground prompts.

10. The image generation step described above is: A step of generating the background image based on the background prompt using a first diffusion model, The story visualization method according to claim 9, comprising the step of generating the foreground image based on the foreground prompt using the first diffusion model.

11. The story visualization method according to claim 1, further comprising a character location information extraction step of extracting character location information of key scenes of the story based on the background image and the prompts using a sixth large language model.

12. The aforementioned story image generation step is: Based on the extracted character position information, the background image and foreground image are aggregated to generate a stitched image. The story visualization method according to claim 11, comprising the step of generating the story image based on the stitching image and diffusion condition information using a second diffusion model.

13. The story visualization method according to claim 12, wherein the diffusion condition information includes at least one of a background image, a foreground image, a prompt, and character position information.

14. The story visualization method according to claim 12, wherein the second diffusion model uses a Semantic Aware-Cross Attention (SA-CA) layer to represent the narrative structure of the story and generates a story image having semantic consistency and contextual consistency in the story.

15. The story visualization method according to claim 1, further comprising the step of providing the generated story image to a user terminal.

16. A program for causing a computer to execute the story visualization method described in any one of claims 1 to 15.

17. One or more processors that execute multiple instructions to perform multiple actions, A story visualization server including one or more memories for storing the plurality of instructions, The aforementioned multiple operations are, The story acquisition process for obtaining a story, By analyzing the acquired story, an information extraction operation is performed to extract scene description information and character description information. A prompt generation operation generates prompts for each major scene of the story based on extracted scene description information and character description information, An image generation operation that generates background and foreground images based on prompts for each major scene of the aforementioned story, A story visualization server, including a story image generation operation that generates story images relating to the main scenes of the story by integrating generated background and foreground images.

Images