System

Abstract

A system is provided.SOLUTION: A system, comprising: means for inputting a natural language sentence; means for dividing the natural language sentence into paragraphs; means for allowing a user to specify a particular character setting; means for invoking a generative model to generate cartoon frames for each paragraph; means for laying out a plurality of cartoon frames in an appropriate order; means for generating a final cartoon; and means for returning the generated cartoon to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Creating manga is a time-consuming and labor-intensive process, placing a significant burden on non-professionals and target audiences of all ages. To solve this problem, technology is needed that allows anyone to easily generate high-quality manga simply by entering text. Furthermore, a function that allows users to select from multiple character patterns when character settings are not specified is also required. Furthermore, a system that can efficiently reflect user requests for revisions to the generated manga is also needed. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system including: a means for inputting a natural language sentence; a means for dividing the natural language sentence into paragraphs; a means for allowing a user to specify specific character settings; a means for calling a generative model that generates manga panels for each paragraph; a means for laying out multiple manga panels in an appropriate order; a means for generating the final manga; and a means for returning the generated manga to the user. Furthermore, by providing a means for selecting multiple character patterns when character settings are not available and a means for accepting user requests for corrections to the generated manga, the system realizes an easy-to-use manga generation system that meets a variety of user needs.

[0006] A "natural language text" is a text written in a language that humans use on a daily basis, and is a text that is written freely without relying on a specific format.

[0007] A "paragraph" refers to a unit that makes up a sentence, and refers to a series of sentences that are organized based on a specific theme or topic.

[0008] "Character settings" refers to detailed information about people and characters who appear in manga, anime, etc., including their name, gender, appearance, personality, etc.

[0009] A "generative model" refers to a machine learning model or algorithm that has been trained to perform tasks such as natural language processing or image generation.

[0010] A "panel" refers to a frame or scene within a single screen in a manga, and is used to visually express the story.

[0011] "Layout" refers to the process of arranging manga panels in appropriate positions on a page and the results of that arrangement.

[0012] A "final manga" refers to a manga work in which all panels are properly laid out and the story is complete.

[0013] A "request for modification" refers to a user's request for some kind of change or modification to the generated comic.

[0014] "Character patterns" are multiple candidate characters that are provided to the user when there is no specific character setting, and their appearances and attributes are set in advance. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention relates to a system that automatically generates manga based on natural language text entered by a user. The system divides the input text into paragraphs, applies specific character settings, and then uses a generative AI model to create manga panels for each paragraph, laying them out appropriately to generate the final manga. In addition, if the user does not specify character settings, the system also has the ability to select from multiple character patterns and to accept requests for revisions to the generated manga.

[0037] System Configuration

[0038] Server Program Processing

[0039] Receiving and dividing text

[0040] The server first receives the natural language text sent by the user. Next, it divides the text into paragraphs using natural language processing technology. For example, if the text is "He confessed to her in the sunset," it will divide it into paragraphs based on specific grammatical rules and punctuation.

[0041] Check and apply character settings

[0042] If the user specifies a specific character setting, the server acquires the character's appearance and characteristics based on that information. If no setting is specified, the server presents the user with multiple pre-prepared character patterns and applies the selected pattern.

[0043] Invoking a generative AI model

[0044] For each paragraph, a generative AI model is invoked. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that. For example, from the paragraph "He confessed to her in the sunset," the model extracts the character settings of "he" and "she" as well as the situations of "sunset" and "confession," and draws a picture based on that.

[0045] Panel layout and final manga generation

[0046] The generated frames are arranged in order based on the provided template and laid out as pages. Then, they are combined into a multi-page manga to generate the final manga. This is then converted into image data or PDF format and prepared for delivery to the user.

[0047] Terminal Programming

[0048] Providing a user interface

[0049] The terminal provides an interface for users to input natural language sentences. Here, a text area is provided, and a sentence input form is provided. There are also character setting selection boxes and a form for inputting specific character settings.

[0050] Submitting data and viewing the generated results

[0051] After the user enters text and character settings and presses the send button, the device sends the data in JSON format to the server. When the server returns the generated manga, the device displays it to the user. An interactive viewer is provided to make it easy to browse the page.

[0052] User operation method

[0053] Entering text and character settings

[0054] The user accesses the system's interface and inputs the text of the novel or story. If necessary, the user can also set up the characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako."

[0055] Manga generation and review

[0056] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[0057] Correction request

[0058] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends the data back to the server. The server accepts the modification request and provides the user with a regenerated cartoon.

[0059] As described above, according to the embodiment of the present invention, a user can easily create a comic from a natural language text, and can check and modify the created comic.

[0060] The processing flow will be explained below.

[0061] Specific processing steps of the program

[0062] Server Processing

[0063] Step 1: Receiving data

[0064] The server receives the natural language sentence and character setting information sent from the terminal.

[0065] Step 2: Segmentation

[0066] The server uses natural language processing techniques to divide the received natural language text into paragraphs, for example, based on grammar rules and punctuation.

[0067] Step 3: Check your character settings

[0068] The server checks whether or not a character setting exists. If a setting exists, the server acquires that information. If not, it selects one of several pre-prepared character patterns.

[0069] Step 4: Invoke the generative AI model

[0070] The server calls a generative AI model for each paragraph. The model extracts key information from the paragraph and generates a comic panel. For example, it draws a scene based on the paragraph, "He confessed his love to her in the sunset."

[0071] Step 5: Creating frames

[0072] The generative AI model generates panels based on each paragraph, including character appearances and backgrounds.

[0073] Step 6: Panel layout

[0074] The server arranges the generated frames in order and lays them out as pages. The frames are arranged based on the layout.

[0075] Step 7: Generate the final cartoon

[0076] After all the paragraph panels have been laid out, the server generates the final comic, converts it into image data or PDF format, and prepares it for delivery back to the user.

[0077] Terminal handling

[0078] Step 8: Displaying the User Interface

[0079] The terminal displays an interface for inputting natural language text, providing text areas and a character configuration form.

[0080] Step 9: Send data

[0081] The text and character setting information entered by the user is sent to the server, packaged in JSON format, and sent via an HTTP request.

[0082] User operations

[0083] Step 10: Enter text

[0084] The user enters a natural language sentence into the text area.

[0085] Step 11: Character Setup

[0086] The user configures the character as desired, entering the character's name, gender, and appearance.

[0087] Step 12: Send

[0088] The user presses the send button to send the entered text and character settings to the server.

[0089] Step 13: Check the manga

[0090] The manga returned from the server can be viewed on the user's device. All pages can be viewed in an interactive viewer.

[0091] Step 14: Request a correction

[0092] The user requests modifications to the generated manga, inputs the specific changes, and sends the request back to the server.

[0093] By following the above steps, a user can easily create, check, and modify a manga based on natural language text.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] Traditionally, manga production has been done manually, requiring a great deal of time and effort. It also requires specialized skills and experience, making it difficult for ordinary users to turn their own stories into manga. Furthermore, corrections and changes take time, making it inefficient.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes means for inputting natural language sentences, means for dividing the natural language sentences into paragraphs, means for allowing the user to specify specific character settings, means for calling a generative model that generates comic panels for each paragraph, means for inputting prompt sentences to the generative model and generating comic panels based on key information, means for laying out multiple comic panels in an appropriate order, means for generating the final comic, and means for returning the generated comic to the user. This enables users to easily generate and edit comics based on natural language sentences without having specialized knowledge or skills.

[0099] A "natural language sentence" is a text entered by a user and written in a normal human language.

[0100] The "means for dividing into paragraphs" is a function that uses natural language processing technology to divide an input natural language sentence into multiple paragraphs based on grammatical rules and punctuation marks.

[0101] "Character settings" are information about the appearance and characteristics of the characters in the manga, specified by the user.

[0102] A "generative model" is an AI model that extracts key information from natural language sentences and generates manga panels based on that information.

[0103] A "prompt sentence" is an instruction sentence for inputting key information extracted from a natural language sentence to a generative model.

[0104] A "manga panel" is an individual image or frame that depicts a scene or situation in a manga.

[0105] The "layout means" is a function that arranges the generated comic panels in the appropriate order and position.

[0106] The "final manga" refers to the entire manga work that is completed by laying out the multiple manga panels that have been generated.

[0107] "Means for returning" is a function for sending the final comic created to the user.

[0108] This invention relates to a system that automatically generates manga based on natural language text entered by a user. The system divides the input text into paragraphs, applies specific character settings, and then uses a generative AI model to create manga panels for each paragraph, laying them out appropriately to generate the final manga. In addition, if the user does not specify character settings, the system also has the ability to select from multiple character patterns and to accept requests for corrections to the generated manga.

[0109] Server Program Processing

[0110] First, the server receives the natural language text sent by the user. This text is received in JSON format. Next, the server uses a Python natural language processing library (e.g., NLTK or spaCy) to divide the text into paragraphs. For example, if the server receives the text "He confessed to her in the sunset," it divides it into paragraphs based on grammar rules and punctuation.

[0111] Next, if the user specifies a specific character configuration, the server retrieves that information from a database that stores the character's appearance and characteristics. If no configuration is specified, the server presents the user with a list of character configurations and applies the pattern selected by the user.

[0112] Next, for each paragraph, the server calls a generative AI model (for example, OpenAI's GPT-4). The model extracts key information from the paragraph (characters, action, background) and generates manga panels based on that information. A prompt is created and input to the model. For example, for the paragraph "He confessed to her in the sunset," the prompt "The scene should include a situation where 'He confesses to her in the sunset'" is input to the generative AI model.

[0113] The generated comic panels are arranged in order based on a prepared template. They are laid out as pages using HTML5 and CSS3, and combined into a single multi-page comic. The final comic is converted into image data or PDF format, and prepared for delivery to the user. The conversion is performed using the Python ReportLab library.

[0114] Terminal Programming

[0115] The terminal provides an interface for the user to input natural language text. For example, it uses HTML and JavaScript to provide text areas, character setting selection boxes, and forms for entering specific character settings. When the user enters text and character settings and presses the submit button, the terminal sends the data to the server in JSON format.

[0116] Once the generated comic is sent back from the server, the device displays it to the user, allowing them to easily browse the page using an interactive viewer (e.g., React or Vue.js).

[0117] User operation method

[0118] The user accesses the system's interface and inputs the text of the novel or story. If necessary, the user can also set up the characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako."

[0119] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[0120] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends the data back to the server. The server accepts the modification request and provides the user with a regenerated cartoon.

[0121] In this way, according to the embodiment of the present invention, the user can easily convert natural language text into a cartoon, and can check and correct the generated cartoon.

[0122] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0123] Specific explanation of processing steps

[0124] Step 1: Enter and send natural language text

[0125] User Actions

[0126] The user accesses the system's interface and inputs a natural language sentence. For example, they can input a sentence such as "He confessed his love to her in the sunset," and set up the characters as needed. They can also specify the appearance and personality of "Taro" and "Hanako." When the user has completed input and pressed the send button, the device sends this data to the server in JSON format.

[0127] input

[0128] Natural language text and character settings.

[0129] output

[0130] The data is sent to the server in JSON format.

[0131] Step 2: Receiving text and dividing it into paragraphs

[0132] Server Operation

[0133] The server receives the JSON data sent from the device. The received data is stored in memory. Then, using Python's natural language processing library (NLTK or spaCy), the server divides the sentence into paragraphs based on grammar rules and punctuation. For example, the sentence "He confessed to her in the sunset" is divided into paragraphs such as "He confessed to her in the sunset" and "He confessed to her."

[0134] input

[0135] JSON data of natural language text sent from the device.

[0136] output

[0137] A list of separated paragraphs.

[0138] Step 3: Check and apply character settings

[0139] Server Operation

[0140] The server checks whether the user has specified a specific character setting. If so, it retrieves that information from a database, which stores the character's appearance and characteristics. If no setting is specified, it presents the user with a list of multiple character patterns and applies the pattern selected by the user.

[0141] input

[0142] User-specified character settings or multiple character patterns.

[0143] output

[0144] Applied character settings.

[0145] Step 4: Generate manga panels using a generative AI model

[0146] Server Operation

[0147] For each paragraph, the server calls a generative AI model (for example, OpenAI's GPT-4). A prompt is created and input to the generative AI model. The model extracts key information (characters, action, background) from the paragraph and generates manga panels based on that information. For example, for the paragraph "He confessed to her in the sunset," the prompt "The scene should include the situation where 'He confesses to her in the sunset'" is input to the generative AI model.

[0148] input

[0149] A prompt sentence created for each paragraph.

[0150] output

[0151] Generated comic panels.

[0152] Step 5: Layout of the comic panels

[0153] Server Operation

[0154] The generated comic panels are arranged in order based on a prepared template. The page is laid out using HTML5 and CSS3. The order and position of the panels are determined by the template.

[0155] input

[0156] Generated comic panels.

[0157] output

[0158] Layout of the manga page.

[0159] Step 6: Generate and return the final comic

[0160] Server Operation

[0161] The process combines multiple pages of comics into one to generate the final comic. The generated comic is then converted into image data or PDF format and prepared for delivery to the user. Specifically, the process converts to PDF format using the Python ReportLab library.

[0162] input

[0163] Layout of the manga page.

[0164] output

[0165] Final manga data (image data or PDF format).

[0166] Step 7: View and modify the generated results

[0167] Device behavior

[0168] The device receives the final comic data sent back from the server. It displays the results to the user using an interactive viewer (such as React or Vue.js). If the user requests corrections, they enter the corrections and send them back to the server. The server processes the regenerated comic and provides it to the user.

[0169] input

[0170] The final comic data returned from the server and the user's modification request.

[0171] output

[0172] Displayed cartoon and modified cartoon data.

[0173] In this way, each processing step proceeds with specific data inputs and outputs, ultimately providing the comic to the user.

[0174] (Application example 1)

[0175] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0176] Conventional manga generation systems using natural language processing have limited ways for users to view the generated manga, making it difficult to provide an interactive experience. In addition, there is a lack of means to visually present product information in a virtual space, which limits the user experience.

[0177] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0178] In this invention, the server includes means for inputting a natural language sentence, means for dividing the natural language sentence into paragraphs, means for allowing a user to specify specific character settings, means for calling a generative model that generates comic panels for each paragraph, means for laying out multiple comic panels in an appropriate order, means for generating a final comic, means for returning the generated comic to the user, means for displaying the generated comic using a head-mounted display, and means for the user to input natural language in a virtual space, thereby enabling the user to visually understand product information through an interactive experience.

[0179] "Natural language sentences" are text data entered by a user using a general language.

[0180] A "paragraph" refers to a coherent part of a natural language sentence.

[0181] "Character settings" is information about the characters and attributes that appear in the manga specified by the user.

[0182] A "generative model" is an algorithm or program that automatically generates manga panels from natural language text.

[0183] A "manga panel" is a small screen or frame in a manga that depicts part of a story using pictures and text.

[0184] "Layout" refers to arranging multiple comic panels in the appropriate order.

[0185] A "head-mounted display" is a display device that provides visual information when worn by a user.

[0186] "Virtual space" is a virtual three-dimensional space generated by a computer.

[0187] "Product information" is detailed data about products and services that interest users.

[0188] This invention relates to a system that automatically generates manga based on natural language text entered by a user and displays the content in a virtual space. The system uses natural language processing technology to divide the text into paragraphs and apply specific character settings. Next, a generative AI model is used to create manga panels for each paragraph, and the panels are appropriately laid out to generate the final manga. The generated manga is then displayed to the user via a head-mounted display.

[0189] System Configuration

[0190] Server Program Processing

[0191] Receiving and dividing text

[0192] The server first receives the natural language sentence sent by the user. This sentence is divided into paragraphs using natural language processing techniques (such as BERT or GPT-3). For example, if the sentence received is "He confessed to her in the sunset," it is divided into paragraphs based on specific grammatical rules and punctuation.

[0193] Check and apply character settings

[0194] If the user specifies a specific character setting, the server acquires the character's appearance and characteristics based on that information. If no setting is specified, the server presents the user with multiple pre-prepared character patterns and applies the selected pattern.

[0195] Invoking a generative AI model

[0196] For each paragraph, a generative AI model (such as DALL-E or Stable Diffusion) is called. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that. For example, from the paragraph "He confessed to her in the sunset," the character settings of "he" and "she" and the situations of "sunset" and "confession" are extracted, and a picture is drawn based on that.

[0197] Panel layout and final manga generation

[0198] The generated frames are arranged in order based on the provided template and laid out as pages. Then, they are combined into a multi-page manga to generate the final manga. This is then converted into image data or PDF format and prepared for delivery to the user.

[0199] Display in virtual space

[0200] The final manga is displayed in a virtual space using a head-mounted display, allowing users to interactively view the generated manga in the virtual space.

[0201] Terminal Programming

[0202] Providing a user interface

[0203] The terminal provides an interface for users to input natural language sentences. Here, a text area is provided, and a sentence input form is provided. There are also character setting selection boxes and a form for inputting specific character settings.

[0204] Submitting data and viewing the generated results

[0205] After the user inputs text and character settings and presses the send button, the device sends the data in JSON format to the server. The generated manga is then sent back from the server, and the device displays it in a virtual space via a head-mounted display.

[0206] User operation method

[0207] Entering text and character settings

[0208] The user accesses the system's interface and inputs the text of the novel or story. If necessary, the user can also set up characters. For example, the user can specify the appearance and personality of characters "A" and "B."

[0209] Manga generation and review

[0210] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is then sent back to the device from the server, where the user can view it through the head-mounted display. For example, if the user enters the text "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[0211] Correction request

[0212] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends the data back to the server. The server accepts the modification request and provides the user with a regenerated cartoon.

[0213] Examples and prompts

[0214] Specific examples

[0215] The user inputs the following natural language in the virtual space.

[0216] Input: "What is the history of this item?"

[0217] Result: The history and background related to the product is displayed in a virtual space in comic form.

[0218] Prompt Sentence Examples

[0219] Write a detailed scene description for a manga panel where a character named A explains the history of a particular product.

[0220] As described above, according to the embodiment of the present invention, a user can easily turn natural language text into a cartoon, and can check and modify the generated cartoon in a virtual space.

[0221] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0222] Step 1:

[0223] The user accesses the terminal's user interface and inputs natural language text. This interface provides a text area, a text input form, a selection box for character settings, and a form for inputting specific character settings. Input: Natural language text and character settings. Output: Data (natural language text, character settings) from the terminal to the server.

[0224] Step 2:

[0225] The server receives the natural language text sent from the device and divides it into paragraphs using natural language processing technology (e.g., BERT or GPT-3). Input: The natural language text received by the server. Output: The paragraphs after division.

[0226] Step 3:

[0227] The server checks the character settings specified by the user. If no character settings are specified, it presents multiple character patterns to the user and applies the selected pattern. Input: User's character settings or unspecified information. Output: Character setting information.

[0228] Step 4:

[0229] For each paragraph, the server invokes a generative AI model (e.g., DALL-E or Stable Diffusion) to extract key information (characters, actions, background, etc.) from the natural language text and generate a manga panel. Input: paragraph, character settings. Output: generated manga panel.

[0230] Step 5:

[0231] The server arranges the generated comic panels in order based on the provided template and lays them out as a page. This layout information becomes the final comic. Input: Generated comic panels. Output: Laid-out comic page.

[0232] Step 6:

[0233] The server converts the final manga into image data or PDF format and sends it back to the terminal. Input: Layout of manga page. Output: Manga in image data or PDF format.

[0234] Step 7:

[0235] The terminal displays the returned manga to the user. This display is done in a virtual space through a head-mounted display. The user can view the manga interactively. Input: Manga in image data or PDF format. Output: Manga displayed on the head-mounted display.

[0236] Step 8:

[0237] When a user requests corrections to the generated manga, they input the changes to a specific frame or character and send the data back to the server from their device. Input: User's correction request. Output: Correction data sent to the server.

[0238] Step 9:

[0239] The server accepts the modification request and returns the regenerated comic to the terminal. Input: Modified data. Output: Modified comic data.

[0240] Step 10:

[0241] The device then displays the revised manga to the user via the head-mounted display. Input: Revised manga data. Output: Revised manga displayed on the head-mounted display.

[0242] Through the above processing steps, the user can generate a manga based on natural language text and interactively check and modify its content in a virtual space.

[0243] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0244] This invention relates to a system that automatically generates manga based on natural language text entered by a user, and by combining it with an emotion engine, it is possible to create manga that reflects the user's emotions. The system configuration and specific program processing are described below.

[0245] System Configuration

[0246] Server Program Processing

[0247] Receiving and dividing text

[0248] The server first receives the natural language text sent by the user. Next, it divides the text into paragraphs using natural language processing technology. For example, if the text is "He confessed to her in the sunset," it will divide it into paragraphs based on specific grammatical rules and punctuation.

[0249] Check and apply character settings

[0250] If the user specifies a specific character setting, the server acquires the character's appearance and characteristics based on that information. If no setting is specified, the server presents the user with multiple pre-prepared character patterns and applies the selected pattern. At this time, the server uses an emotion engine to confirm the user's emotion and adjusts the character's facial expression and pose based on that emotion.

[0251] Invoking a generative AI model

[0252] For each paragraph, a generative AI model is invoked. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that information. For example, from the paragraph "He confessed to her in the sunset," the character settings of "he" and "she" and the situations of "sunset" and "confession" are extracted, and a picture is drawn based on those. The results of the emotion engine are used to adjust the presentation of the panels.

[0253] Panel layout and final manga generation

[0254] The generated frames are arranged in order based on the provided template and laid out as pages. Then, they are combined into a multi-page manga to generate the final manga. This is then converted into image data or PDF format and prepared for delivery to the user.

[0255] Terminal Programming

[0256] Providing a user interface

[0257] The terminal provides an interface for users to input natural language sentences. Here, a text area is provided and a sentence input form is provided. There are also character setting selection boxes and a form for inputting specific character settings. Furthermore, an interface is provided for recognizing the user's emotions using an emotion engine. For example, the user's emotions can be detected by facial expression recognition or voice analysis.

[0258] Submitting data and viewing the generated results

[0259] After the user enters text, character settings, and emotion information and presses the send button, the device sends the data in JSON format to the server. When the server returns the generated manga, the device displays it to the user. An interactive viewer is provided to make it easy to browse the page.

[0260] User operation method

[0261] Entering text and character settings

[0262] The user accesses the system's interface and inputs the text of a novel or story. If necessary, the user can also set up characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako." Furthermore, the user can input or detect their own emotions through the interface.

[0263] Manga generation and review

[0264] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[0265] Correction request

[0266] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends them back to the server. The server accepts the modification requests and provides the user with a regenerated cartoon.

[0267] Harnessing emotions

[0268] The emotion engine optimizes the expression of each panel of the manga based on the user's emotions recognized by the engine. For example, if joy is recognized, the character's expression will be changed to a smile, or other appropriate effect will be applied.

[0269] For example, if a user inputs the sentence "He confessed his feelings to her in the sunset," and the emotion engine recognizes the user's emotion as "tension," the character in the generated manga frame will have a tense expression and pose. In this way, real-time manga generation that reflects the user's emotions is realized.

[0270] As described above, according to the embodiment of the present invention, a user can easily create, check, and modify a comic that reflects emotions based on natural language text.

[0271] The processing flow will be explained below.

[0272] Specific processing steps of the program

[0273] Server Processing

[0274] Step 1: Receiving data

[0275] The server receives the natural language sentences, character setting information, and emotion information sent from the terminal.

[0276] Step 2: Segmentation

[0277] The server uses natural language processing technology to divide the received natural language sentence into paragraphs. For example, it divides the sentence "He confessed to her in the sunset" into paragraphs based on punctuation marks.

[0278] Step 3: Check your character settings

[0279] The server checks whether or not a character setting exists. If a setting exists, the server acquires that information. If not, the server presents multiple pre-prepared character patterns.

[0280] Step 4: Extracting emotional information

[0281] The server analyzes the received emotional information and extracts a specific emotional state (e.g., joy, sadness, tension, etc.) based on the results.

[0282] Step 5: Invoke the generative AI model

[0283] The server calls a generative AI model for each paragraph. The model extracts key information from the paragraph and generates manga panels taking into account emotional information. For example, a scene is drawn based on the paragraph "He confessed his love to her in the sunset," and the character's facial expression and pose are adjusted based on the emotional information.

[0284] Step 6: Creating frames

[0285] The generative AI model generates manga panels for each paragraph based on the extracted key information and emotional information. For example, if the emotional information is "tension," the model will make the character's facial expression tense.

[0286] Step 7: Panel layout

[0287] The server arranges the generated frames in the appropriate order and lays them out as pages, arranging the frames according to a specific storyboard.

[0288] Step 8: Generate the final cartoon

[0289] After all the paragraph panels have been laid out, the server generates the final comic, converts it into image data or PDF format, and prepares it for delivery back to the user.

[0290] Terminal handling

[0291] Step 9: Displaying the User Interface

[0292] The terminal displays an interface for inputting natural language sentences, providing a text area and an input box for character settings, as well as a form for recognizing the user's emotions using an emotion engine.

[0293] Step 10: Send data

[0294] The text, character settings, and emotion information entered by the user are sent to the server, packaged in JSON format and sent via an HTTP request.

[0295] User operations

[0296] Step 11: Enter text

[0297] The user inputs a natural language sentence into the text area, for example, "He confessed to her in the sunset."

[0298] Step 12: Character Setup

[0299] The user configures the character as desired, entering the character's name, gender, and appearance.

[0300] Step 13: Input or Recognize Emotions

[0301] Users input their emotions through the emotion engine, which then automatically detects emotions through facial expression recognition and voice analysis, for example.

[0302] Step 14: Send

[0303] The user presses the send button to send the entered text, character settings, and emotion information to the server.

[0304] Step 15: Check the manga

[0305] Check the manga returned from the server on your device. You can easily browse all pages in the interactive viewer. For example, you can confirm that the generated manga scene depicts a character expressing the emotion of "tension."

[0306] Step 16: Request a correction

[0307] The user requests modifications to the generated cartoon. The user inputs the specific changes and sends them back to the server. The server accepts the modification request and provides the regenerated cartoon to the user.

[0308] These steps allow users to use a system that uses natural language text and emotional information to easily generate, review, and revise more emotionally rich comics.

[0309] Example 2

[0310] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0311] In today's world, systems that automatically generate manga based on users' natural language input are required to meet diverse entertainment needs. However, current technology for manga generation systems that incorporate character expressions that reflect the user's emotions is insufficient. Furthermore, there is a lack of flexibility in the ability to request corrections to the generated manga and in character settings. A system that can resolve these issues is needed.

[0312] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0313] In this invention, the server includes means for inputting a natural language sentence, means for dividing the natural language sentence into paragraphs, means for allowing the user to specify a specific character setting, means for acquiring the character's appearance and characteristics based on the character setting, means for confirming the user's emotions using an emotion engine and adjusting the character's facial expression and pose, means for calling a generative model for generating manga panels for each paragraph, means for laying out multiple manga panels in an appropriate order, means for generating the final manga, and means for returning the generated manga to the user. This enables real-time manga generation that reflects the user's emotions and also realizes a flexible system that can respond to requests for revisions.

[0314] A "natural language sentence" is a sentence written in a language that a user uses on a daily basis.

[0315] A "paragraph" is a portion of text that uses line breaks and spaces to separate meanings or content within a sentence.

[0316] "Character settings" are setting information that allows a user to specify the appearance, personality, and characteristics of a specific character.

[0317] A "generative model" is an algorithm or program that automatically generates manga panels from input natural language text.

[0318] The "emotion engine" is a system that recognizes emotions from the user's facial expressions and voice, and adjusts the character's expressions based on that information.

[0319] A "manga panel" is a combination of pictures and text that separates scenes in a manga.

[0320] A "template" is a predetermined framework or format for determining the page layout of a manga.

[0321] The "final manga" is a manga work that is compiled into a single story by appropriately laying out the multiple frames that have been generated.

[0322] A "request for correction" is a request made by a user to notify the server of any corrections or changes that the user wishes to make to a work.

[0323] This invention relates to a system that automatically generates manga based on natural language text entered by a user, and by combining it with an emotion engine, it is possible to create manga that reflects the user's emotions. The system configuration and specific program processing are described below.

[0324] Server Program Processing

[0325] The server first receives the natural language text sent by the user. It then uses natural language processing technology to divide the text into paragraphs and invokes a generative AI model to extract key information (such as characters, actions, and background) from each paragraph. If the user has specified a specific character setting, the server obtains the character's appearance and characteristics based on that information, uses an emotion engine to confirm the user's emotions, and adjusts the character's facial expressions and poses based on those emotions.

[0326] For example, when processing the paragraph "He confessed his love to her in the sunset," the generative AI model extracts the character settings of "him" and "her" and the situations of "sunset" and "confession," and draws a picture based on these. The results of the emotion engine are used to adjust the presentation of the panels. The generated panels are then arranged in order based on the provided template and laid out as pages. The resulting multi-page manga is then compiled into a single page, generating the final manga. This is then converted into image data or PDF format, ready to be sent back to the user.

[0327] Terminal Programming

[0328] The terminal provides an interface for users to input natural language text. A text area is provided here, providing a text input form. Selection boxes for character settings and a form for inputting specific character settings are also provided. An interface is also provided for recognizing the user's emotions using an emotion engine. For example, the user's emotions are detected using facial expression recognition or voice analysis. After the user enters the text, character settings, and emotional information and presses the send button, the terminal sends this data in JSON format to the server. When the generated manga is returned from the server, the terminal displays it to the user. An interactive viewer is provided to make it easy to browse the page.

[0329] User operation method

[0330] The user accesses the system's interface and inputs the text of a novel or story. If necessary, the user can also set up characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako." Furthermore, the user can input or detect their own emotions through the interface.

[0331] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[0332] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends them back to the server. The server accepts the modification requests and provides the user with a regenerated cartoon.

[0333] The emotion engine optimizes the expression of each panel of the manga based on the user's emotions recognized by the engine. For example, if joy is recognized, the character's expression will be changed to a smile, or other appropriate effect will be applied.

[0334] For example, if a user inputs the sentence "He confessed his feelings to her in the sunset," and the emotion engine recognizes the user's emotion as "tension," the character in the generated manga frame will have a tense expression and pose. In this way, real-time manga generation that reflects the user's emotions is realized.

[0335] As described above, according to the embodiment of the present invention, a user can easily create, check, and modify a comic that reflects emotions based on natural language text.

[0336] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0337] Step 1:

[0338] Enter text and set characters from the user interface

[0339] Terminal

[0340] The user accesses the system's interface and inputs natural language text into a text area. They then specify specific character appearances and characteristics in a character configuration form. At this stage, they also input facial expressions and voice data for emotion recognition. The input text, character configuration, and emotion information are collected on the terminal.

[0341] Input: User-entered natural language text, character settings, and emotional information

[0342] Output: Input data collected on the terminal

[0343] Step 2:

[0344] Sending and receiving input data

[0345] Terminal

[0346] When the user completes the input and presses the send button, the device sends the collected text, character settings, and emotional information to the server in JSON format.

[0347] server

[0348] The server receives the data sent from the terminal and begins preparations for data processing.

[0349] Input: User input data sent from the device (JSON format)

[0350] Output: The input data received by the server

[0351] Step 3:

[0352] Dividing the text and checking the character settings

[0353] server

[0354] The server uses natural language processing technology to divide the received text into paragraphs. Each paragraph represents a different scene or event, making this an essential operation for generating effective manga. The server also checks the specified character settings and automatically selects multiple character patterns if the user has not specified them.

[0355] Input: JSON format user input data (natural language text, character settings)

[0356] Output: Separated paragraphs and confirmed character settings

[0357] Step 4:

[0358] Emotion recognition by emotion engine

[0359] server

[0360] The server runs an emotion engine, analyzes the received user facial and voice data, and recognizes emotions. For example, if the user's facial expression is "smiling," it is recognized as "joy," and if the user's voice is trembling, it is recognized as "tension." This emotional information is reflected in the character's expression when generating the manga.

[0361] Input: Received user facial expression and voice data

[0362] Output: Analyzed user emotion information

[0363] Step 5:

[0364] Manga panel generation using generative AI models

[0365] server

[0366] The server calls a generative AI model for each paragraph of the divided text. This model extracts key information from the text (characters, actions, background, etc.) and generates manga panels. In doing so, it sets the character's expressions and poses based on the results of the emotion engine. For example, in the paragraph "He confessed to her in the sunset," a sunset background and cat illustrations are generated to depict the confession scene.

[0367] Input: Divided paragraphs, sentiment information

[0368] Output: Generated comic panels

[0369] Step 6:

[0370] Panel layout and final manga generation

[0371] server

[0372] The generated panels are arranged in order based on a template and laid out as pages. The laid out pages are compiled into a manga book and the final manga data is generated. The generated manga data is converted into image data or PDF format.

[0373] Input: Multiple generated comic panels

[0374] Output: Final manga data (image data or PDF format)

[0375] Step 7:

[0376] Sending and displaying the final manga

[0377] server

[0378] The generated final manga data is sent to the terminal.

[0379] Terminal

[0380] The device receives the manga data sent from the server and displays it to the user, using an interactive viewer to allow easy page browsing.

[0381] Input: Final manga data

[0382] Output: Comic data in a format that can be viewed by users

[0383] Step 8:

[0384] Correction Request Processing

[0385] User

[0386] If the user wishes to make corrections to the generated comic, he or she inputs changes to specific frames or characters.

[0387] Terminal

[0388] The device sends a modification request in JSON format to the server.

[0389] server

[0390] The server accepts the modification request, regenerates the comic with the specified modifications reflected, and sends the regenerated comic to the terminal again.

[0391] Input: Correction Request

[0392] Output: New manga data with the corrections reflected

[0393] (Application example 2)

[0394] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0395] Systems already exist that automatically generate manga based on natural language text, but few systems can naturally express the emotions of the user in the panels. Furthermore, to improve the quality of the generated manga and user satisfaction, it is necessary to adjust the character expressions based on emotions. Furthermore, there is a demand for an interface that allows users to quickly view the generated manga and quickly request corrections. Current technology does not yet realize such advanced emotional reflection and interactive operation, so there is a need to solve these issues.

[0396] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0397] In this invention, the server includes means for inputting natural language sentences, means for dividing the natural language sentences into paragraphs, means for allowing the user to specify specific character settings, means for calling a generative model that generates comic panels for each paragraph, means for laying out multiple comic panels in an appropriate order, means for generating a final comic, means for returning the generated comic to the user, and means for analyzing the user's emotions and adjusting the expression of the comic panels based on the analysis results. This makes it possible to automatically generate comics based on the natural language sentences input by the user, and further to reflect the user's emotions in the expression of the panels.

[0398] A "natural language sentence" is a sentence that is input by a user and is composed of words that are used daily.

[0399] The "paragraph division method" is a method for automatically dividing an input natural language sentence into multiple paragraphs based on grammatical rules and punctuation marks.

[0400] "Character settings" are setting information about the character's appearance and personality specified by the user.

[0401] A "generative model" is an AI technology that extracts key information from natural language sentences and generates manga panels based on that information.

[0402] The "layout method" is a method for arranging the generated multiple comic panels in the appropriate order and formatting them as a page.

[0403] The "final manga" is the final product of the manga, with all panels properly laid out and put together.

[0404] The "returning means" is a method for sending the generated comic to the user.

[0405] "Means for analyzing emotions" refers to technology that recognizes the user's emotions and adjusts the character's expression based on the analysis results.

[0406] This invention is a system for automatically generating and providing comics that reflect emotions based on natural language sentences entered by a user. A specific embodiment of the system and its operation will be described below.

[0407] System Program

[0408] 1. Entering and dividing text

[0409] The terminal provides an interface for the user to input natural language sentences. Once the user has completed input, the sentence is sent to the server, which divides the sentence into paragraphs. This process uses natural language processing technology. For example, if the sentence "He confessed to her in the sunset" is received, the server divides the sentence into paragraphs.

[0410] 2. Applying character settings

[0411] If the user specifies a specific character setting, the server will obtain the character's appearance and characteristics based on that information. If no character setting is available, the server will present the user with multiple pre-prepared character patterns and apply the selected pattern. At this time, the device also provides an interface for recognizing the user's emotions, detecting them through facial expression recognition and voice analysis.

[0412] 3. Invoking the generative AI model

[0413] The server calls a generative AI model for each divided paragraph. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that information. For example, from the paragraph "He confessed to her in the sunset," the model extracts the character settings of "he" and "she" and the situations of "sunset" and "confession," and draws a picture based on those. The results of the emotion engine are used to adjust the presentation of the panels.

[0414] 4. Panel layout and final manga generation

[0415] The server arranges the generated frames in order based on the provided template and lays them out as pages. It then combines multiple comic frames into one and finally generates a comic. It then converts this comic into image data or PDF format and returns it to the user.

[0416] Hardware and software used

[0417] Hardware:

[0418] Smartphone (Android or iOS device)

[0419] software:

[0420] Flask (Python framework, backend)

[0421] Transformers (NLP library, sentiment analysis)

[0422] Remote server (running the manga generation AI model)

[0423] Specific examples

[0424] The user launches the application and enters the following text:

[0425] Example prompt statement:

[0426] He confessed to her in the sunset.

[0427] Given this sentence, the system will do the following:

[0428] Use natural language processing to divide the text into paragraphs.

[0429] The character setting specified by the user is applied, and if there is no setting, the user is allowed to select from a plurality of character patterns.

[0430] The emotion engine is used to analyze the user's emotions, and based on the results, a generative AI model is called to generate manga panels.

[0431] The generated panels are laid out in the appropriate order to generate the final cartoon, which is then returned to the user.

[0432] In this way, users can easily create original comics that reflect their own emotions based on input natural language sentences, and then review and share them.

[0433] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0434] Step 1:

[0435] The terminal provides an interface for the user to input a natural language sentence. The user inputs the natural language sentence and sends the sentence to the server. At this time, the input natural language sentence might be, for example, "He confessed to her in the sunset."

[0436] Step 2:

[0437] The server divides the received natural language sentence into paragraphs. It uses natural language processing technology for sentence analysis and automatically divides the sentence into multiple paragraphs based on grammatical rules and punctuation. For example, it divides the sentence into paragraphs like "He was in the sunset" and "He confessed to her."

[0438] Step 3:

[0439] The terminal provides an interface for the user to specify specific character settings. When the user inputs the character settings (appearance and personality), that information is sent to the server. If there is no character setting, the server presents multiple character patterns to the user and applies the selected pattern.

[0440] Step 4:

[0441] The server calls a generative AI model for each paragraph. This model extracts key information from the text (such as characters, actions, and background) and generates manga panels based on that information. For example, the model extracts the character settings of "him" and "her" and the situations such as "sunset" and "confession."

[0442] Step 5:

[0443] The server uses an emotion engine to analyze emotions based on the user's input text. Emotion analysis uses the user's emotional information sent from the device (e.g., the results of facial expression recognition and voice analysis). For example, if the emotion "tension" is recognized, the character's facial expression and pose are adjusted accordingly.

[0444] Step 6:

[0445] The server lays out the generated panels in an orderly manner, placing each panel in the appropriate order based on the provided template to form a page, and then combining multiple pages to generate the final manga.

[0446] Step 7:

[0447] The server converts the generated comic into image data or PDF format and sends it back to the terminal. The user can check the generated comic on the terminal. For example, if the user inputs the prompt sentence "He confessed to her in the sunset," a comic depicting that story will be displayed on the terminal.

[0448] Step 8:

[0449] The terminal provides an interface for accepting a request for modification of the generated cartoon from the user. When the user inputs the modification request and sends it to the server, the server accepts the modification request and returns the regenerated cartoon to the user.

[0450] In this way, by performing specific actions at each step, users can easily create original manga that reflect their emotions based on the natural language text they input, and then review and share them.

[0451] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0452] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0453] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0454] [Second embodiment]

[0455] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0456] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0457] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0458] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0459] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0460] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0461] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0462] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0463] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0464] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0465] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0466] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0467] This invention relates to a system that automatically generates manga based on natural language text entered by a user. The system divides the input text into paragraphs, applies specific character settings, and then uses a generative AI model to create manga panels for each paragraph, laying them out appropriately to generate the final manga. In addition, if the user does not specify character settings, the system also has the ability to select from multiple character patterns and to accept requests for revisions to the generated manga.

[0468] System Configuration

[0469] Server Program Processing

[0470] Receiving and dividing text

[0471] The server first receives the natural language text sent by the user. Next, it divides the text into paragraphs using natural language processing technology. For example, if the text is "He confessed to her in the sunset," it will divide it into paragraphs based on specific grammatical rules and punctuation.

[0472] Check and apply character settings

[0473] If the user specifies a specific character setting, the server acquires the character's appearance and characteristics based on that information. If no setting is specified, the server presents the user with multiple pre-prepared character patterns and applies the selected pattern.

[0474] Invoking a generative AI model

[0475] For each paragraph, a generative AI model is invoked. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that. For example, from the paragraph "He confessed to her in the sunset," the model extracts the character settings of "he" and "she" as well as the situations of "sunset" and "confession," and draws a picture based on that.

[0476] Panel layout and final manga generation

[0477] The generated frames are arranged in order based on the provided template and laid out as pages. Then, they are combined into a multi-page manga to generate the final manga. This is then converted into image data or PDF format and prepared for delivery to the user.

[0478] Terminal Programming

[0479] Providing a user interface

[0480] The terminal provides an interface for users to input natural language sentences. Here, a text area is provided, and a sentence input form is provided. There are also character setting selection boxes and a form for inputting specific character settings.

[0481] Submitting data and viewing the generated results

[0482] After the user enters text and character settings and presses the send button, the device sends the data in JSON format to the server. When the server returns the generated manga, the device displays it to the user. An interactive viewer is provided to make it easy to browse the page.

[0483] User operation method

[0484] Entering text and character settings

[0485] The user accesses the system's interface and inputs the text of the novel or story. If necessary, the user can also set up the characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako."

[0486] Manga generation and review

[0487] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[0488] Correction request

[0489] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends the data back to the server. The server accepts the modification request and provides the user with a regenerated cartoon.

[0490] As described above, according to the embodiment of the present invention, a user can easily create a comic from a natural language text, and can check and modify the created comic.

[0491] The processing flow will be explained below.

[0492] Specific processing steps of the program

[0493] Server Processing

[0494] Step 1: Receiving data

[0495] The server receives the natural language sentence and character setting information sent from the terminal.

[0496] Step 2: Segmentation

[0497] The server uses natural language processing techniques to divide the received natural language text into paragraphs, for example, based on grammar rules and punctuation.

[0498] Step 3: Check your character settings

[0499] The server checks whether or not a character setting exists. If a setting exists, the server acquires that information. If not, it selects one of several pre-prepared character patterns.

[0500] Step 4: Invoke the generative AI model

[0501] The server calls a generative AI model for each paragraph. The model extracts key information from the paragraph and generates a comic panel. For example, it draws a scene based on the paragraph, "He confessed his love to her in the sunset."

[0502] Step 5: Creating frames

[0503] The generative AI model generates panels based on each paragraph, including character appearances and backgrounds.

[0504] Step 6: Panel layout

[0505] The server arranges the generated frames in order and lays them out as pages. The frames are arranged based on the layout.

[0506] Step 7: Generate the final cartoon

[0507] After all the paragraph panels have been laid out, the server generates the final comic, converts it into image data or PDF format, and prepares it for delivery back to the user.

[0508] Terminal handling

[0509] Step 8: Displaying the User Interface

[0510] The terminal displays an interface for inputting natural language text, providing text areas and a character configuration form.

[0511] Step 9: Send data

[0512] The text and character setting information entered by the user is sent to the server, packaged in JSON format, and sent via an HTTP request.

[0513] User operations

[0514] Step 10: Enter text

[0515] The user enters a natural language sentence into the text area.

[0516] Step 11: Character Setup

[0517] The user configures the character as desired, entering the character's name, gender, and appearance.

[0518] Step 12: Send

[0519] The user presses the send button to send the entered text and character settings to the server.

[0520] Step 13: Check the manga

[0521] The manga returned from the server can be viewed on the user's device. All pages can be viewed in an interactive viewer.

[0522] Step 14: Request a correction

[0523] The user requests modifications to the generated manga, inputs the specific changes, and sends the request back to the server.

[0524] By following the above steps, a user can easily create, check, and modify a manga based on natural language text.

[0525] Example 1

[0526] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0527] Traditionally, manga production has been done manually, requiring a great deal of time and effort. It also requires specialized skills and experience, making it difficult for ordinary users to turn their own stories into manga. Furthermore, corrections and changes take time, making it inefficient.

[0528] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0529] In this invention, the server includes means for inputting natural language sentences, means for dividing the natural language sentences into paragraphs, means for allowing the user to specify specific character settings, means for calling a generative model that generates comic panels for each paragraph, means for inputting prompt sentences to the generative model and generating comic panels based on key information, means for laying out multiple comic panels in an appropriate order, means for generating the final comic, and means for returning the generated comic to the user. This enables users to easily generate and edit comics based on natural language sentences without having specialized knowledge or skills.

[0530] A "natural language sentence" is a text entered by a user and written in a normal human language.

[0531] The "means for dividing into paragraphs" is a function that uses natural language processing technology to divide an input natural language sentence into multiple paragraphs based on grammatical rules and punctuation marks.

[0532] "Character settings" are information about the appearance and characteristics of the characters in the manga, specified by the user.

[0533] A "generative model" is an AI model that extracts key information from natural language sentences and generates manga panels based on that information.

[0534] A "prompt sentence" is an instruction sentence for inputting key information extracted from a natural language sentence to a generative model.

[0535] A "manga panel" is an individual image or frame that depicts a scene or situation in a manga.

[0536] The "layout means" is a function that arranges the generated comic panels in the appropriate order and position.

[0537] The "final manga" refers to the entire manga work that is completed by laying out the multiple manga panels that have been generated.

[0538] "Means for returning" is a function for sending the final comic created to the user.

[0539] This invention relates to a system that automatically generates manga based on natural language text entered by a user. The system divides the input text into paragraphs, applies specific character settings, and then uses a generative AI model to create manga panels for each paragraph, laying them out appropriately to generate the final manga. In addition, if the user does not specify character settings, the system also has the ability to select from multiple character patterns and to accept requests for corrections to the generated manga.

[0540] Server Program Processing

[0541] First, the server receives the natural language text sent by the user. This text is received in JSON format. Next, the server uses a Python natural language processing library (e.g., NLTK or spaCy) to divide the text into paragraphs. For example, if the server receives the text "He confessed to her in the sunset," it divides it into paragraphs based on grammar rules and punctuation.

[0542] Next, if the user specifies a specific character configuration, the server retrieves that information from a database that stores the character's appearance and characteristics. If no configuration is specified, the server presents the user with a list of character configurations and applies the pattern selected by the user.

[0543] Next, for each paragraph, the server calls a generative AI model (for example, OpenAI's GPT-4). The model extracts key information from the paragraph (characters, action, background) and generates manga panels based on that information. A prompt is created and input to the model. For example, for the paragraph "He confessed to her in the sunset," the prompt "The scene should include a situation where 'He confesses to her in the sunset'" is input to the generative AI model.

[0544] The generated comic panels are arranged in order based on a prepared template. They are laid out as pages using HTML5 and CSS3, and combined into a single multi-page comic. The final comic is converted into image data or PDF format, and prepared for delivery to the user. The conversion is performed using the Python ReportLab library.

[0545] Terminal Programming

[0546] The terminal provides an interface for the user to input natural language text. For example, it uses HTML and JavaScript to provide text areas, character setting selection boxes, and forms for entering specific character settings. When the user enters text and character settings and presses the submit button, the terminal sends the data to the server in JSON format.

[0547] Once the generated comic is sent back from the server, the device displays it to the user, allowing them to easily browse the page using an interactive viewer (e.g., React or Vue.js).

[0548] User operation method

[0549] The user accesses the system's interface and inputs the text of the novel or story. If necessary, the user can also set up the characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako."

[0550] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[0551] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends the data back to the server. The server accepts the modification request and provides the user with a regenerated cartoon.

[0552] In this way, according to the embodiment of the present invention, the user can easily convert natural language text into a cartoon, and can check and correct the generated cartoon.

[0553] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0554] Specific explanation of processing steps

[0555] Step 1: Enter and send natural language text

[0556] User Actions

[0557] The user accesses the system's interface and inputs a natural language sentence. For example, they can input a sentence such as "He confessed his love to her in the sunset," and set up the characters as needed. They can also specify the appearance and personality of "Taro" and "Hanako." When the user has completed input and pressed the send button, the device sends this data to the server in JSON format.

[0558] input

[0559] Natural language text and character settings.

[0560] output

[0561] The data is sent to the server in JSON format.

[0562] Step 2: Receiving text and dividing it into paragraphs

[0563] Server Operation

[0564] The server receives the JSON data sent from the device. The received data is stored in memory. Then, using Python's natural language processing library (NLTK or spaCy), the server divides the sentence into paragraphs based on grammar rules and punctuation. For example, the sentence "He confessed to her in the sunset" is divided into paragraphs such as "He confessed to her in the sunset" and "He confessed to her."

[0565] input

[0566] JSON data of natural language text sent from the device.

[0567] output

[0568] A list of separated paragraphs.

[0569] Step 3: Check and apply character settings

[0570] Server Operation

[0571] The server checks whether the user has specified a specific character setting. If so, it retrieves that information from a database, which stores the character's appearance and characteristics. If no setting is specified, it presents the user with a list of multiple character patterns and applies the pattern selected by the user.

[0572] input

[0573] User-specified character settings or multiple character patterns.

[0574] output

[0575] Applied character settings.

[0576] Step 4: Generate manga panels using a generative AI model

[0577] Server Operation

[0578] For each paragraph, the server calls a generative AI model (for example, OpenAI's GPT-4). A prompt is created and input to the generative AI model. The model extracts key information (characters, action, background) from the paragraph and generates manga panels based on that information. For example, for the paragraph "He confessed to her in the sunset," the prompt "The scene should include the situation where 'He confesses to her in the sunset'" is input to the generative AI model.

[0579] input

[0580] A prompt sentence created for each paragraph.

[0581] output

[0582] Generated comic panels.

[0583] Step 5: Layout of the comic panels

[0584] Server Operation

[0585] The generated comic panels are arranged in order based on a prepared template. The page is laid out using HTML5 and CSS3. The order and position of the panels are determined by the template.

[0586] input

[0587] Generated comic panels.

[0588] output

[0589] Layout of the manga page.

[0590] Step 6: Generate and return the final comic

[0591] Server Operation

[0592] The process combines multiple pages of comics into one to generate the final comic. The generated comic is then converted into image data or PDF format and prepared for delivery to the user. Specifically, the process converts to PDF format using the Python ReportLab library.

[0593] input

[0594] Layout of the manga page.

[0595] output

[0596] Final manga data (image data or PDF format).

[0597] Step 7: View and modify the generated results

[0598] Device behavior

[0599] The device receives the final comic data sent back from the server. It displays the results to the user using an interactive viewer (such as React or Vue.js). If the user requests corrections, they enter the corrections and send them back to the server. The server processes the regenerated comic and provides it to the user.

[0600] input

[0601] The final comic data returned from the server and the user's modification request.

[0602] output

[0603] Displayed cartoon and modified cartoon data.

[0604] In this way, each processing step proceeds with specific data inputs and outputs, ultimately providing the comic to the user.

[0605] (Application example 1)

[0606] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0607] Conventional manga generation systems using natural language processing have limited ways for users to view the generated manga, making it difficult to provide an interactive experience. In addition, there is a lack of means to visually present product information in a virtual space, which limits the user experience.

[0608] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0609] In this invention, the server includes means for inputting a natural language sentence, means for dividing the natural language sentence into paragraphs, means for allowing a user to specify specific character settings, means for calling a generative model that generates comic panels for each paragraph, means for laying out multiple comic panels in an appropriate order, means for generating a final comic, means for returning the generated comic to the user, means for displaying the generated comic using a head-mounted display, and means for the user to input natural language in a virtual space, thereby enabling the user to visually understand product information through an interactive experience.

[0610] "Natural language sentences" are text data entered by a user using a general language.

[0611] A "paragraph" refers to a coherent part of a natural language sentence.

[0612] "Character settings" is information about the characters and attributes that appear in the manga specified by the user.

[0613] A "generative model" is an algorithm or program that automatically generates manga panels from natural language text.

[0614] A "manga panel" is a small screen or frame in a manga that depicts part of a story using pictures and text.

[0615] "Layout" refers to arranging multiple comic panels in the appropriate order.

[0616] A "head-mounted display" is a display device that provides visual information when worn by a user.

[0617] "Virtual space" is a virtual three-dimensional space generated by a computer.

[0618] "Product information" is detailed data about products and services that interest users.

[0619] This invention relates to a system that automatically generates manga based on natural language text entered by a user and displays the content in a virtual space. The system uses natural language processing technology to divide the text into paragraphs and apply specific character settings. Next, a generative AI model is used to create manga panels for each paragraph, and the panels are appropriately laid out to generate the final manga. The generated manga is then displayed to the user via a head-mounted display.

[0620] System Configuration

[0621] Server Program Processing

[0622] Receiving and dividing text

[0623] The server first receives the natural language sentence sent by the user. This sentence is divided into paragraphs using natural language processing techniques (such as BERT or GPT-3). For example, if the sentence received is "He confessed to her in the sunset," it is divided into paragraphs based on specific grammatical rules and punctuation.

[0624] Check and apply character settings

[0625] If the user specifies a specific character setting, the server acquires the character's appearance and characteristics based on that information. If no setting is specified, the server presents the user with multiple pre-prepared character patterns and applies the selected pattern.

[0626] Invoking a generative AI model

[0627] For each paragraph, a generative AI model (such as DALL-E or Stable Diffusion) is called. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that. For example, from the paragraph "He confessed to her in the sunset," the character settings of "he" and "she" and the situations of "sunset" and "confession" are extracted, and a picture is drawn based on that.

[0628] Panel layout and final manga generation

[0629] The generated frames are arranged in order based on the provided template and laid out as pages. Then, they are combined into a multi-page manga to generate the final manga. This is then converted into image data or PDF format and prepared for delivery to the user.

[0630] Display in virtual space

[0631] The final manga is displayed in a virtual space using a head-mounted display, allowing users to interactively view the generated manga in the virtual space.

[0632] Terminal Programming

[0633] Providing a user interface

[0634] The terminal provides an interface for users to input natural language sentences. Here, a text area is provided, and a sentence input form is provided. There are also character setting selection boxes and a form for inputting specific character settings.

[0635] Submitting data and viewing the generated results

[0636] After the user inputs text and character settings and presses the send button, the device sends the data in JSON format to the server. The generated manga is then sent back from the server, and the device displays it in a virtual space via a head-mounted display.

[0637] User operation method

[0638] Entering text and character settings

[0639] The user accesses the system's interface and inputs the text of the novel or story. If necessary, the user can also set up characters. For example, the user can specify the appearance and personality of characters "A" and "B."

[0640] Manga generation and review

[0641] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is then sent back to the device from the server, where the user can view it through the head-mounted display. For example, if the user enters the text "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[0642] Correction request

[0643] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends the data back to the server. The server accepts the modification request and provides the user with a regenerated cartoon.

[0644] Examples and prompts

[0645] Specific examples

[0646] The user inputs the following natural language in the virtual space.

[0647] Input: "What is the history of this item?"

[0648] Result: The history and background related to the product is displayed in a virtual space in comic form.

[0649] Prompt Sentence Examples

[0650] Write a detailed scene description for a manga panel where a character named A explains the history of a particular product.

[0651] As described above, according to the embodiment of the present invention, a user can easily turn natural language text into a cartoon, and can check and modify the generated cartoon in a virtual space.

[0652] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0653] Step 1:

[0654] The user accesses the terminal's user interface and inputs natural language text. This interface provides a text area, a text input form, a selection box for character settings, and a form for inputting specific character settings. Input: Natural language text and character settings. Output: Data (natural language text, character settings) from the terminal to the server.

[0655] Step 2:

[0656] The server receives the natural language text sent from the device and divides it into paragraphs using natural language processing technology (e.g., BERT or GPT-3). Input: The natural language text received by the server. Output: The paragraphs after division.

[0657] Step 3:

[0658] The server checks the character settings specified by the user. If no character settings are specified, it presents multiple character patterns to the user and applies the selected pattern. Input: User's character settings or unspecified information. Output: Character setting information.

[0659] Step 4:

[0660] For each paragraph, the server invokes a generative AI model (e.g., DALL-E or Stable Diffusion) to extract key information (characters, actions, background, etc.) from the natural language text and generate a manga panel. Input: paragraph, character settings. Output: generated manga panel.

[0661] Step 5:

[0662] The server arranges the generated comic panels in order based on the provided template and lays them out as a page. This layout information becomes the final comic. Input: Generated comic panels. Output: Laid-out comic page.

[0663] Step 6:

[0664] The server converts the final manga into image data or PDF format and sends it back to the terminal. Input: Layout of manga page. Output: Manga in image data or PDF format.

[0665] Step 7:

[0666] The terminal displays the returned manga to the user. This display is done in a virtual space through a head-mounted display. The user can view the manga interactively. Input: Manga in image data or PDF format. Output: Manga displayed on the head-mounted display.

[0667] Step 8:

[0668] When a user requests corrections to the generated manga, they input the changes to a specific frame or character and send the data back to the server from their device. Input: User's correction request. Output: Correction data sent to the server.

[0669] Step 9:

[0670] The server accepts the modification request and returns the regenerated comic to the terminal. Input: Modified data. Output: Modified comic data.

[0671] Step 10:

[0672] The device then displays the revised manga to the user via the head-mounted display. Input: Revised manga data. Output: Revised manga displayed on the head-mounted display.

[0673] Through the above processing steps, the user can generate a manga based on natural language text and interactively check and modify its content in a virtual space.

[0674] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0675] This invention relates to a system that automatically generates manga based on natural language text entered by a user, and by combining it with an emotion engine, it is possible to create manga that reflects the user's emotions. The system configuration and specific program processing are described below.

[0676] System Configuration

[0677] Server Program Processing

[0678] Receiving and dividing text

[0679] The server first receives the natural language text sent by the user. Next, it divides the text into paragraphs using natural language processing technology. For example, if the text is "He confessed to her in the sunset," it will divide it into paragraphs based on specific grammatical rules and punctuation.

[0680] Check and apply character settings

[0681] If the user specifies a specific character setting, the server acquires the character's appearance and characteristics based on that information. If no setting is specified, the server presents the user with multiple pre-prepared character patterns and applies the selected pattern. At this time, the server uses an emotion engine to confirm the user's emotion and adjusts the character's facial expression and pose based on that emotion.

[0682] Invoking a generative AI model

[0683] For each paragraph, a generative AI model is invoked. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that information. For example, from the paragraph "He confessed to her in the sunset," the character settings of "he" and "she" and the situations of "sunset" and "confession" are extracted, and a picture is drawn based on those. The results of the emotion engine are used to adjust the presentation of the panels.

[0684] Panel layout and final manga generation

[0685] The generated frames are arranged in order based on the provided template and laid out as pages. Then, they are combined into a multi-page manga to generate the final manga. This is then converted into image data or PDF format and prepared for delivery to the user.

[0686] Terminal Programming

[0687] Providing a user interface

[0688] The terminal provides an interface for users to input natural language sentences. Here, a text area is provided and a sentence input form is provided. There are also character setting selection boxes and a form for inputting specific character settings. Furthermore, an interface is provided for recognizing the user's emotions using an emotion engine. For example, the user's emotions can be detected by facial expression recognition or voice analysis.

[0689] Submitting data and viewing the generated results

[0690] After the user enters text, character settings, and emotion information and presses the send button, the device sends the data in JSON format to the server. When the server returns the generated manga, the device displays it to the user. An interactive viewer is provided to make it easy to browse the page.

[0691] User operation method

[0692] Entering text and character settings

[0693] The user accesses the system's interface and inputs the text of a novel or story. If necessary, the user can also set up characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako." Furthermore, the user can input or detect their own emotions through the interface.

[0694] Manga generation and review

[0695] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[0696] Correction request

[0697] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends them back to the server. The server accepts the modification requests and provides the user with a regenerated cartoon.

[0698] Harnessing emotions

[0699] The emotion engine optimizes the expression of each panel of the manga based on the user's emotions recognized by the engine. For example, if joy is recognized, the character's expression will be changed to a smile, or other appropriate effect will be applied.

[0700] For example, if a user inputs the sentence "He confessed his feelings to her in the sunset," and the emotion engine recognizes the user's emotion as "tension," the character in the generated manga frame will have a tense expression and pose. In this way, real-time manga generation that reflects the user's emotions is realized.

[0701] As described above, according to the embodiment of the present invention, a user can easily create, check, and modify a comic that reflects emotions based on natural language text.

[0702] The processing flow will be explained below.

[0703] Specific processing steps of the program

[0704] Server Processing

[0705] Step 1: Receiving data

[0706] The server receives the natural language sentences, character setting information, and emotion information sent from the terminal.

[0707] Step 2: Segmentation

[0708] The server uses natural language processing technology to divide the received natural language sentence into paragraphs. For example, it divides the sentence "He confessed to her in the sunset" into paragraphs based on punctuation marks.

[0709] Step 3: Check your character settings

[0710] The server checks whether or not a character setting exists. If a setting exists, the server acquires that information. If not, the server presents multiple pre-prepared character patterns.

[0711] Step 4: Extracting emotional information

[0712] The server analyzes the received emotional information and extracts a specific emotional state (e.g., joy, sadness, tension, etc.) based on the results.

[0713] Step 5: Invoke the generative AI model

[0714] The server calls a generative AI model for each paragraph. The model extracts key information from the paragraph and generates manga panels taking into account emotional information. For example, a scene is drawn based on the paragraph "He confessed his love to her in the sunset," and the character's facial expression and pose are adjusted based on the emotional information.

[0715] Step 6: Creating frames

[0716] The generative AI model generates manga panels for each paragraph based on the extracted key information and emotional information. For example, if the emotional information is "tension," the model will make the character's facial expression tense.

[0717] Step 7: Panel layout

[0718] The server arranges the generated frames in the appropriate order and lays them out as pages, arranging the frames according to a specific storyboard.

[0719] Step 8: Generate the final cartoon

[0720] After all the paragraph panels have been laid out, the server generates the final comic, converts it into image data or PDF format, and prepares it for delivery back to the user.

[0721] Terminal handling

[0722] Step 9: Displaying the User Interface

[0723] The terminal displays an interface for inputting natural language sentences, providing a text area and an input box for character settings, as well as a form for recognizing the user's emotions using an emotion engine.

[0724] Step 10: Send data

[0725] The text, character settings, and emotion information entered by the user are sent to the server, packaged in JSON format and sent via an HTTP request.

[0726] User operations

[0727] Step 11: Enter text

[0728] The user inputs a natural language sentence into the text area, for example, "He confessed to her in the sunset."

[0729] Step 12: Character Setup

[0730] The user configures the character as desired, entering the character's name, gender, and appearance.

[0731] Step 13: Input or Recognize Emotions

[0732] Users input their emotions through the emotion engine, which then automatically detects emotions through facial expression recognition and voice analysis, for example.

[0733] Step 14: Send

[0734] The user presses the send button to send the entered text, character settings, and emotion information to the server.

[0735] Step 15: Check the manga

[0736] Check the manga returned from the server on your device. You can easily browse all pages in the interactive viewer. For example, you can confirm that the generated manga scene depicts a character expressing the emotion of "tension."

[0737] Step 16: Request a correction

[0738] The user requests modifications to the generated cartoon. The user inputs the specific changes and sends them back to the server. The server accepts the modification request and provides the regenerated cartoon to the user.

[0739] These steps allow users to use a system that uses natural language text and emotional information to easily generate, review, and revise more emotionally rich comics.

[0740] Example 2

[0741] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0742] In today's world, systems that automatically generate manga based on users' natural language input are required to meet diverse entertainment needs. However, current technology for manga generation systems that incorporate character expressions that reflect the user's emotions is insufficient. Furthermore, there is a lack of flexibility in the ability to request corrections to the generated manga and in character settings. A system that can resolve these issues is needed.

[0743] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0744] In this invention, the server includes means for inputting a natural language sentence, means for dividing the natural language sentence into paragraphs, means for allowing the user to specify a specific character setting, means for acquiring the character's appearance and characteristics based on the character setting, means for confirming the user's emotions using an emotion engine and adjusting the character's facial expression and pose, means for calling a generative model for generating manga panels for each paragraph, means for laying out multiple manga panels in an appropriate order, means for generating the final manga, and means for returning the generated manga to the user. This enables real-time manga generation that reflects the user's emotions and also realizes a flexible system that can respond to requests for revisions.

[0745] A "natural language sentence" is a sentence written in a language that a user uses on a daily basis.

[0746] A "paragraph" is a portion of text that uses line breaks and spaces to separate meanings or content within a sentence.

[0747] "Character settings" are setting information that allows a user to specify the appearance, personality, and characteristics of a specific character.

[0748] A "generative model" is an algorithm or program that automatically generates manga panels from input natural language text.

[0749] The "emotion engine" is a system that recognizes emotions from the user's facial expressions and voice, and adjusts the character's expressions based on that information.

[0750] A "manga panel" is a combination of pictures and text that separates scenes in a manga.

[0751] A "template" is a predetermined framework or format for determining the page layout of a manga.

[0752] The "final manga" is a manga work that is compiled into a single story by appropriately laying out the multiple frames that have been generated.

[0753] A "request for correction" is a request made by a user to notify the server of any corrections or changes that the user wishes to make to a work.

[0754] This invention relates to a system that automatically generates manga based on natural language text entered by a user, and by combining it with an emotion engine, it is possible to create manga that reflects the user's emotions. The system configuration and specific program processing are described below.

[0755] Server Program Processing

[0756] The server first receives the natural language text sent by the user. It then uses natural language processing technology to divide the text into paragraphs and invokes a generative AI model to extract key information (such as characters, actions, and background) from each paragraph. If the user has specified a specific character setting, the server obtains the character's appearance and characteristics based on that information, uses an emotion engine to confirm the user's emotions, and adjusts the character's facial expressions and poses based on those emotions.

[0757] For example, when processing the paragraph "He confessed his love to her in the sunset," the generative AI model extracts the character settings of "him" and "her" and the situations of "sunset" and "confession," and draws a picture based on these. The results of the emotion engine are used to adjust the presentation of the panels. The generated panels are then arranged in order based on the provided template and laid out as pages. The resulting multi-page manga is then compiled into a single page, generating the final manga. This is then converted into image data or PDF format, ready to be sent back to the user.

[0758] Terminal Programming

[0759] The terminal provides an interface for users to input natural language text. A text area is provided here, providing a text input form. Selection boxes for character settings and a form for inputting specific character settings are also provided. An interface is also provided for recognizing the user's emotions using an emotion engine. For example, the user's emotions are detected using facial expression recognition or voice analysis. After the user enters the text, character settings, and emotional information and presses the send button, the terminal sends this data in JSON format to the server. When the generated manga is returned from the server, the terminal displays it to the user. An interactive viewer is provided to make it easy to browse the page.

[0760] User operation method

[0761] The user accesses the system's interface and inputs the text of a novel or story. If necessary, the user can also set up characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako." Furthermore, the user can input or detect their own emotions through the interface.

[0762] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[0763] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends them back to the server. The server accepts the modification requests and provides the user with a regenerated cartoon.

[0764] The emotion engine optimizes the expression of each panel of the manga based on the user's emotions recognized by the engine. For example, if joy is recognized, the character's expression will be changed to a smile, or other appropriate effect will be applied.

[0765] For example, if a user inputs the sentence "He confessed his feelings to her in the sunset," and the emotion engine recognizes the user's emotion as "tension," the character in the generated manga frame will have a tense expression and pose. In this way, real-time manga generation that reflects the user's emotions is realized.

[0766] As described above, according to the embodiment of the present invention, a user can easily create, check, and modify a comic that reflects emotions based on natural language text.

[0767] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0768] Step 1:

[0769] Enter text and set characters from the user interface

[0770] Terminal

[0771] The user accesses the system's interface and inputs natural language text into a text area. They then specify specific character appearances and characteristics in a character configuration form. At this stage, they also input facial expressions and voice data for emotion recognition. The input text, character configuration, and emotion information are collected on the terminal.

[0772] Input: User-entered natural language text, character settings, and emotional information

[0773] Output: Input data collected on the terminal

[0774] Step 2:

[0775] Sending and receiving input data

[0776] Terminal

[0777] When the user completes the input and presses the send button, the device sends the collected text, character settings, and emotional information to the server in JSON format.

[0778] server

[0779] The server receives the data sent from the terminal and begins preparations for data processing.

[0780] Input: User input data sent from the device (JSON format)

[0781] Output: The input data received by the server

[0782] Step 3:

[0783] Dividing the text and checking the character settings

[0784] server

[0785] The server uses natural language processing technology to divide the received text into paragraphs. Each paragraph represents a different scene or event, making this an essential operation for generating effective manga. The server also checks the specified character settings and automatically selects multiple character patterns if the user has not specified them.

[0786] Input: JSON format user input data (natural language text, character settings)

[0787] Output: Separated paragraphs and confirmed character settings

[0788] Step 4:

[0789] Emotion recognition by emotion engine

[0790] server

[0791] The server runs an emotion engine, analyzes the received user facial and voice data, and recognizes emotions. For example, if the user's facial expression is "smiling," it is recognized as "joy," and if the user's voice is trembling, it is recognized as "tension." This emotional information is reflected in the character's expression when generating the manga.

[0792] Input: Received user facial expression and voice data

[0793] Output: Analyzed user emotion information

[0794] Step 5:

[0795] Manga panel generation using generative AI models

[0796] server

[0797] The server calls a generative AI model for each paragraph of the divided text. This model extracts key information from the text (characters, actions, background, etc.) and generates manga panels. In doing so, it sets the character's expressions and poses based on the results of the emotion engine. For example, in the paragraph "He confessed to her in the sunset," a sunset background and cat illustrations are generated to depict the confession scene.

[0798] Input: Divided paragraphs, sentiment information

[0799] Output: Generated comic panels

[0800] Step 6:

[0801] Panel layout and final manga generation

[0802] server

[0803] The generated panels are arranged in order based on a template and laid out as pages. The laid out pages are compiled into a manga book and the final manga data is generated. The generated manga data is converted into image data or PDF format.

[0804] Input: Multiple generated comic panels

[0805] Output: Final manga data (image data or PDF format)

[0806] Step 7:

[0807] Sending and displaying the final manga

[0808] server

[0809] The generated final manga data is sent to the terminal.

[0810] Terminal

[0811] The device receives the manga data sent from the server and displays it to the user, using an interactive viewer to allow easy page browsing.

[0812] Input: Final manga data

[0813] Output: Comic data in a format that can be viewed by users

[0814] Step 8:

[0815] Correction Request Processing

[0816] User

[0817] If the user wishes to make corrections to the generated comic, he or she inputs changes to specific frames or characters.

[0818] Terminal

[0819] The device sends a modification request in JSON format to the server.

[0820] server

[0821] The server accepts the modification request, regenerates the comic with the specified modifications reflected, and sends the regenerated comic to the terminal again.

[0822] Input: Correction Request

[0823] Output: New manga data with the corrections reflected

[0824] (Application example 2)

[0825] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0826] Systems already exist that automatically generate manga based on natural language text, but few systems can naturally express the emotions of the user in the panels. Furthermore, to improve the quality of the generated manga and user satisfaction, it is necessary to adjust the character expressions based on emotions. Furthermore, there is a demand for an interface that allows users to quickly view the generated manga and quickly request corrections. Current technology does not yet realize such advanced emotional reflection and interactive operation, so there is a need to solve these issues.

[0827] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0828] In this invention, the server includes means for inputting natural language sentences, means for dividing the natural language sentences into paragraphs, means for allowing the user to specify specific character settings, means for calling a generative model that generates comic panels for each paragraph, means for laying out multiple comic panels in an appropriate order, means for generating a final comic, means for returning the generated comic to the user, and means for analyzing the user's emotions and adjusting the expression of the comic panels based on the analysis results. This makes it possible to automatically generate comics based on the natural language sentences input by the user, and further to reflect the user's emotions in the expression of the panels.

[0829] A "natural language sentence" is a sentence that is input by a user and is composed of words that are used daily.

[0830] The "paragraph division method" is a method for automatically dividing an input natural language sentence into multiple paragraphs based on grammatical rules and punctuation marks.

[0831] "Character settings" are setting information about the character's appearance and personality specified by the user.

[0832] A "generative model" is an AI technology that extracts key information from natural language sentences and generates manga panels based on that information.

[0833] The "layout method" is a method for arranging the generated multiple comic panels in the appropriate order and formatting them as a page.

[0834] The "final manga" is the final product of the manga, with all panels properly laid out and put together.

[0835] The "returning means" is a method for sending the generated comic to the user.

[0836] "Means for analyzing emotions" refers to technology that recognizes the user's emotions and adjusts the character's expression based on the analysis results.

[0837] This invention is a system for automatically generating and providing comics that reflect emotions based on natural language sentences entered by a user. A specific embodiment of the system and its operation will be described below.

[0838] System Program

[0839] 1. Entering and dividing text

[0840] The terminal provides an interface for the user to input natural language sentences. Once the user has completed input, the sentence is sent to the server, which divides the sentence into paragraphs. This process uses natural language processing technology. For example, if the sentence "He confessed to her in the sunset" is received, the server divides the sentence into paragraphs.

[0841] 2. Applying character settings

[0842] If the user specifies a specific character setting, the server will obtain the character's appearance and characteristics based on that information. If no character setting is available, the server will present the user with multiple pre-prepared character patterns and apply the selected pattern. At this time, the device also provides an interface for recognizing the user's emotions, detecting them through facial expression recognition and voice analysis.

[0843] 3. Invoking the generative AI model

[0844] The server calls a generative AI model for each divided paragraph. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that information. For example, from the paragraph "He confessed to her in the sunset," the model extracts the character settings of "he" and "she" and the situations of "sunset" and "confession," and draws a picture based on those. The results of the emotion engine are used to adjust the presentation of the panels.

[0845] 4. Panel layout and final manga generation

[0846] The server arranges the generated frames in order based on the provided template and lays them out as pages. It then combines multiple comic frames into one and finally generates a comic. It then converts this comic into image data or PDF format and returns it to the user.

[0847] Hardware and software used

[0848] Hardware:

[0849] Smartphone (Android or iOS device)

[0850] software:

[0851] Flask (Python framework, backend)

[0852] Transformers (NLP library, sentiment analysis)

[0853] Remote server (running the manga generation AI model)

[0854] Specific examples

[0855] The user launches the application and enters the following text:

[0856] Example prompt statement:

[0857] He confessed to her in the sunset.

[0858] Given this sentence, the system will do the following:

[0859] Use natural language processing to divide the text into paragraphs.

[0860] The character setting specified by the user is applied, and if there is no setting, the user is allowed to select from a plurality of character patterns.

[0861] The emotion engine is used to analyze the user's emotions, and based on the results, a generative AI model is called to generate manga panels.

[0862] The generated panels are laid out in the appropriate order to generate the final cartoon, which is then returned to the user.

[0863] In this way, users can easily create original comics that reflect their own emotions based on input natural language sentences, and then review and share them.

[0864] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0865] Step 1:

[0866] The terminal provides an interface for the user to input a natural language sentence. The user inputs the natural language sentence and sends the sentence to the server. At this time, the input natural language sentence might be, for example, "He confessed to her in the sunset."

[0867] Step 2:

[0868] The server divides the received natural language sentence into paragraphs. It uses natural language processing technology for sentence analysis and automatically divides the sentence into multiple paragraphs based on grammatical rules and punctuation. For example, it divides the sentence into paragraphs like "He was in the sunset" and "He confessed to her."

[0869] Step 3:

[0870] The terminal provides an interface for the user to specify specific character settings. When the user inputs the character settings (appearance and personality), that information is sent to the server. If there is no character setting, the server presents multiple character patterns to the user and applies the selected pattern.

[0871] Step 4:

[0872] The server calls a generative AI model for each paragraph. This model extracts key information from the text (such as characters, actions, and background) and generates manga panels based on that information. For example, the model extracts the character settings of "him" and "her" and the situations such as "sunset" and "confession."

[0873] Step 5:

[0874] The server uses an emotion engine to analyze emotions based on the user's input text. Emotion analysis uses the user's emotional information sent from the device (e.g., the results of facial expression recognition and voice analysis). For example, if the emotion "tension" is recognized, the character's facial expression and pose are adjusted accordingly.

[0875] Step 6:

[0876] The server lays out the generated panels in an orderly manner, placing each panel in the appropriate order based on the provided template to form a page, and then combining multiple pages to generate the final manga.

[0877] Step 7:

[0878] The server converts the generated comic into image data or PDF format and sends it back to the terminal. The user can check the generated comic on the terminal. For example, if the user inputs the prompt sentence "He confessed to her in the sunset," a comic depicting that story will be displayed on the terminal.

[0879] Step 8:

[0880] The terminal provides an interface for accepting a request for modification of the generated cartoon from the user. When the user inputs the modification request and sends it to the server, the server accepts the modification request and returns the regenerated cartoon to the user.

[0881] In this way, by performing specific actions at each step, users can easily create original manga that reflect their emotions based on the natural language text they input, and then review and share them.

[0882] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0883] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0884] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0885] [Third embodiment]

[0886] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0887] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0888] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0889] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0890] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0891] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0892] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0893] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0894] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0895] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0896] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0897] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0898] This invention relates to a system that automatically generates manga based on natural language text entered by a user. The system divides the input text into paragraphs, applies specific character settings, and then uses a generative AI model to create manga panels for each paragraph, laying them out appropriately to generate the final manga. In addition, if the user does not specify character settings, the system also has the ability to select from multiple character patterns and to accept requests for revisions to the generated manga.

[0899] System Configuration

[0900] Server Program Processing

[0901] Receiving and dividing text

[0902] The server first receives the natural language text sent by the user. Next, it divides the text into paragraphs using natural language processing technology. For example, if the text is "He confessed to her in the sunset," it will divide it into paragraphs based on specific grammatical rules and punctuation.

[0903] Check and apply character settings

[0904] If the user specifies a specific character setting, the server acquires the character's appearance and characteristics based on that information. If no setting is specified, the server presents the user with multiple pre-prepared character patterns and applies the selected pattern.

[0905] Invoking a generative AI model

[0906] For each paragraph, a generative AI model is invoked. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that. For example, from the paragraph "He confessed to her in the sunset," the model extracts the character settings of "he" and "she" as well as the situations of "sunset" and "confession," and draws a picture based on that.

[0907] Panel layout and final manga generation

[0908] The generated frames are arranged in order based on the provided template and laid out as pages. Then, they are combined into a multi-page manga to generate the final manga. This is then converted into image data or PDF format and prepared for delivery to the user.

[0909] Terminal Programming

[0910] Providing a user interface

[0911] The terminal provides an interface for users to input natural language sentences. Here, a text area is provided, and a sentence input form is provided. There are also character setting selection boxes and a form for inputting specific character settings.

[0912] Submitting data and viewing the generated results

[0913] After the user enters text and character settings and presses the send button, the device sends the data in JSON format to the server. When the server returns the generated manga, the device displays it to the user. An interactive viewer is provided to make it easy to browse the page.

[0914] User operation method

[0915] Entering text and character settings

[0916] The user accesses the system's interface and inputs the text of the novel or story. If necessary, the user can also set up the characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako."

[0917] Manga generation and review

[0918] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[0919] Correction request

[0920] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends the data back to the server. The server accepts the modification request and provides the user with a regenerated cartoon.

[0921] As described above, according to the embodiment of the present invention, a user can easily create a comic from a natural language text, and can check and modify the created comic.

[0922] The processing flow will be explained below.

[0923] Specific processing steps of the program

[0924] Server Processing

[0925] Step 1: Receiving data

[0926] The server receives the natural language sentence and character setting information sent from the terminal.

[0927] Step 2: Segmentation

[0928] The server uses natural language processing techniques to divide the received natural language text into paragraphs, for example, based on grammar rules and punctuation.

[0929] Step 3: Check your character settings

[0930] The server checks whether or not a character setting exists. If a setting exists, the server acquires that information. If not, it selects one of several pre-prepared character patterns.

[0931] Step 4: Invoke the generative AI model

[0932] The server calls a generative AI model for each paragraph. The model extracts key information from the paragraph and generates a comic panel. For example, it draws a scene based on the paragraph, "He confessed his love to her in the sunset."

[0933] Step 5: Creating frames

[0934] The generative AI model generates panels based on each paragraph, including character appearances and backgrounds.

[0935] Step 6: Panel layout

[0936] The server arranges the generated frames in order and lays them out as pages. The frames are arranged based on the layout.

[0937] Step 7: Generate the final cartoon

[0938] After all the paragraph panels have been laid out, the server generates the final comic, converts it into image data or PDF format, and prepares it for delivery back to the user.

[0939] Terminal handling

[0940] Step 8: Displaying the User Interface

[0941] The terminal displays an interface for inputting natural language text, providing text areas and a character configuration form.

[0942] Step 9: Send data

[0943] The text and character setting information entered by the user is sent to the server, packaged in JSON format, and sent via an HTTP request.

[0944] User operations

[0945] Step 10: Enter text

[0946] The user enters a natural language sentence into the text area.

[0947] Step 11: Character Setup

[0948] The user configures the character as desired, entering the character's name, gender, and appearance.

[0949] Step 12: Send

[0950] The user presses the send button to send the entered text and character settings to the server.

[0951] Step 13: Check the manga

[0952] The manga returned from the server can be viewed on the user's device. All pages can be viewed in an interactive viewer.

[0953] Step 14: Request a correction

[0954] The user requests modifications to the generated manga, inputs the specific changes, and sends the request back to the server.

[0955] By following the above steps, a user can easily create, check, and modify a manga based on natural language text.

[0956] Example 1

[0957] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0958] Traditionally, manga production has been done manually, requiring a great deal of time and effort. It also requires specialized skills and experience, making it difficult for ordinary users to turn their own stories into manga. Furthermore, corrections and changes take time, making it inefficient.

[0959] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0960] In this invention, the server includes means for inputting natural language sentences, means for dividing the natural language sentences into paragraphs, means for allowing the user to specify specific character settings, means for calling a generative model that generates comic panels for each paragraph, means for inputting prompt sentences to the generative model and generating comic panels based on key information, means for laying out multiple comic panels in an appropriate order, means for generating the final comic, and means for returning the generated comic to the user. This enables users to easily generate and edit comics based on natural language sentences without having specialized knowledge or skills.

[0961] A "natural language sentence" is a text entered by a user and written in a normal human language.

[0962] The "means for dividing into paragraphs" is a function that uses natural language processing technology to divide an input natural language sentence into multiple paragraphs based on grammatical rules and punctuation marks.

[0963] "Character settings" are information about the appearance and characteristics of the characters in the manga, specified by the user.

[0964] A "generative model" is an AI model that extracts key information from natural language sentences and generates manga panels based on that information.

[0965] A "prompt sentence" is an instruction sentence for inputting key information extracted from a natural language sentence to a generative model.

[0966] A "manga panel" is an individual image or frame that depicts a scene or situation in a manga.

[0967] The "layout means" is a function that arranges the generated comic panels in the appropriate order and position.

[0968] The "final manga" refers to the entire manga work that is completed by laying out the multiple manga panels that have been generated.

[0969] "Means for returning" is a function for sending the final comic created to the user.

[0970] This invention relates to a system that automatically generates manga based on natural language text entered by a user. The system divides the input text into paragraphs, applies specific character settings, and then uses a generative AI model to create manga panels for each paragraph, laying them out appropriately to generate the final manga. In addition, if the user does not specify character settings, the system also has the ability to select from multiple character patterns and to accept requests for corrections to the generated manga.

[0971] Server Program Processing

[0972] First, the server receives the natural language text sent by the user. This text is received in JSON format. Next, the server uses a Python natural language processing library (e.g., NLTK or spaCy) to divide the text into paragraphs. For example, if the server receives the text "He confessed to her in the sunset," it divides it into paragraphs based on grammar rules and punctuation.

[0973] Next, if the user specifies a specific character configuration, the server retrieves that information from a database that stores the character's appearance and characteristics. If no configuration is specified, the server presents the user with a list of character configurations and applies the pattern selected by the user.

[0974] Next, for each paragraph, the server calls a generative AI model (for example, OpenAI's GPT-4). The model extracts key information from the paragraph (characters, action, background) and generates manga panels based on that information. A prompt is created and input to the model. For example, for the paragraph "He confessed to her in the sunset," the prompt "The scene should include a situation where 'He confesses to her in the sunset'" is input to the generative AI model.

[0975] The generated comic panels are arranged in order based on a prepared template. They are laid out as pages using HTML5 and CSS3, and combined into a single multi-page comic. The final comic is converted into image data or PDF format, and prepared for delivery to the user. The conversion is performed using the Python ReportLab library.

[0976] Terminal Programming

[0977] The terminal provides an interface for the user to input natural language text. For example, it uses HTML and JavaScript to provide text areas, character setting selection boxes, and forms for entering specific character settings. When the user enters text and character settings and presses the submit button, the terminal sends the data to the server in JSON format.

[0978] Once the generated comic is sent back from the server, the device displays it to the user, allowing them to easily browse the page using an interactive viewer (e.g., React or Vue.js).

[0979] User operation method

[0980] The user accesses the system's interface and inputs the text of the novel or story. If necessary, the user can also set up the characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako."

[0981] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[0982] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends the data back to the server. The server accepts the modification request and provides the user with a regenerated cartoon.

[0983] In this way, according to the embodiment of the present invention, the user can easily convert natural language text into a cartoon, and can check and correct the generated cartoon.

[0984] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0985] Specific explanation of processing steps

[0986] Step 1: Enter and send natural language text

[0987] User Actions

[0988] The user accesses the system's interface and inputs a natural language sentence. For example, they can input a sentence such as "He confessed his love to her in the sunset," and set up the characters as needed. They can also specify the appearance and personality of "Taro" and "Hanako." When the user has completed input and pressed the send button, the device sends this data to the server in JSON format.

[0989] input

[0990] Natural language text and character settings.

[0991] output

[0992] The data is sent to the server in JSON format.

[0993] Step 2: Receiving text and dividing it into paragraphs

[0994] Server Operation

[0995] The server receives the JSON data sent from the device. The received data is stored in memory. Then, using Python's natural language processing library (NLTK or spaCy), the server divides the sentence into paragraphs based on grammar rules and punctuation. For example, the sentence "He confessed to her in the sunset" is divided into paragraphs such as "He confessed to her in the sunset" and "He confessed to her."

[0996] input

[0997] JSON data of natural language text sent from the device.

[0998] output

[0999] A list of separated paragraphs.

[1000] Step 3: Check and apply character settings

[1001] Server Operation

[1002] The server checks whether the user has specified a specific character setting. If so, it retrieves that information from a database, which stores the character's appearance and characteristics. If no setting is specified, it presents the user with a list of multiple character patterns and applies the pattern selected by the user.

[1003] input

[1004] User-specified character settings or multiple character patterns.

[1005] output

[1006] Applied character settings.

[1007] Step 4: Generate manga panels using a generative AI model

[1008] Server Operation

[1009] For each paragraph, the server calls a generative AI model (for example, OpenAI's GPT-4). A prompt is created and input to the generative AI model. The model extracts key information (characters, action, background) from the paragraph and generates manga panels based on that information. For example, for the paragraph "He confessed to her in the sunset," the prompt "The scene should include the situation where 'He confesses to her in the sunset'" is input to the generative AI model.

[1010] input

[1011] A prompt sentence created for each paragraph.

[1012] output

[1013] Generated comic panels.

[1014] Step 5: Layout of the comic panels

[1015] Server Operation

[1016] The generated comic panels are arranged in order based on a prepared template. The page is laid out using HTML5 and CSS3. The order and position of the panels are determined by the template.

[1017] input

[1018] Generated comic panels.

[1019] output

[1020] Layout of the manga page.

[1021] Step 6: Generate and return the final comic

[1022] Server Operation

[1023] The process combines multiple pages of comics into one to generate the final comic. The generated comic is then converted into image data or PDF format and prepared for delivery to the user. Specifically, the process converts to PDF format using the Python ReportLab library.

[1024] input

[1025] Layout of the manga page.

[1026] output

[1027] Final manga data (image data or PDF format).

[1028] Step 7: View and modify the generated results

[1029] Device behavior

[1030] The device receives the final comic data sent back from the server. It displays the results to the user using an interactive viewer (such as React or Vue.js). If the user requests corrections, they enter the corrections and send them back to the server. The server processes the regenerated comic and provides it to the user.

[1031] input

[1032] The final comic data returned from the server and the user's modification request.

[1033] output

[1034] Displayed cartoon and modified cartoon data.

[1035] In this way, each processing step proceeds with specific data inputs and outputs, ultimately providing the comic to the user.

[1036] (Application example 1)

[1037] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1038] Conventional manga generation systems using natural language processing have limited ways for users to view the generated manga, making it difficult to provide an interactive experience. In addition, there is a lack of means to visually present product information in a virtual space, which limits the user experience.

[1039] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1040] In this invention, the server includes means for inputting a natural language sentence, means for dividing the natural language sentence into paragraphs, means for allowing a user to specify specific character settings, means for calling a generative model that generates comic panels for each paragraph, means for laying out multiple comic panels in an appropriate order, means for generating a final comic, means for returning the generated comic to the user, means for displaying the generated comic using a head-mounted display, and means for the user to input natural language in a virtual space, thereby enabling the user to visually understand product information through an interactive experience.

[1041] "Natural language sentences" are text data entered by a user using a general language.

[1042] A "paragraph" refers to a coherent part of a natural language sentence.

[1043] "Character settings" is information about the characters and attributes that appear in the manga specified by the user.

[1044] A "generative model" is an algorithm or program that automatically generates manga panels from natural language text.

[1045] A "manga panel" is a small screen or frame in a manga that depicts part of a story using pictures and text.

[1046] "Layout" refers to arranging multiple comic panels in the appropriate order.

[1047] A "head-mounted display" is a display device that provides visual information when worn by a user.

[1048] "Virtual space" is a virtual three-dimensional space generated by a computer.

[1049] "Product information" is detailed data about products and services that interest users.

[1050] This invention relates to a system that automatically generates manga based on natural language text entered by a user and displays the content in a virtual space. The system uses natural language processing technology to divide the text into paragraphs and apply specific character settings. Next, a generative AI model is used to create manga panels for each paragraph, and the panels are appropriately laid out to generate the final manga. The generated manga is then displayed to the user via a head-mounted display.

[1051] System Configuration

[1052] Server Program Processing

[1053] Receiving and dividing text

[1054] The server first receives the natural language sentence sent by the user. This sentence is divided into paragraphs using natural language processing techniques (such as BERT or GPT-3). For example, if the sentence received is "He confessed to her in the sunset," it is divided into paragraphs based on specific grammatical rules and punctuation.

[1055] Check and apply character settings

[1056] If the user specifies a specific character setting, the server acquires the character's appearance and characteristics based on that information. If no setting is specified, the server presents the user with multiple pre-prepared character patterns and applies the selected pattern.

[1057] Invoking a generative AI model

[1058] For each paragraph, a generative AI model (such as DALL-E or Stable Diffusion) is called. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that. For example, from the paragraph "He confessed to her in the sunset," the character settings of "he" and "she" and the situations of "sunset" and "confession" are extracted, and a picture is drawn based on that.

[1059] Panel layout and final manga generation

[1060] The generated frames are arranged in order based on the provided template and laid out as pages. Then, they are combined into a multi-page manga to generate the final manga. This is then converted into image data or PDF format and prepared for delivery to the user.

[1061] Display in virtual space

[1062] The final manga is displayed in a virtual space using a head-mounted display, allowing users to interactively view the generated manga in the virtual space.

[1063] Terminal Programming

[1064] Providing a user interface

[1065] The terminal provides an interface for users to input natural language sentences. Here, a text area is provided, and a sentence input form is provided. There are also character setting selection boxes and a form for inputting specific character settings.

[1066] Submitting data and viewing the generated results

[1067] After the user inputs text and character settings and presses the send button, the device sends the data in JSON format to the server. The generated manga is then sent back from the server, and the device displays it in a virtual space via a head-mounted display.

[1068] User operation method

[1069] Entering text and character settings

[1070] The user accesses the system's interface and inputs the text of the novel or story. If necessary, the user can also set up characters. For example, the user can specify the appearance and personality of characters "A" and "B."

[1071] Manga generation and review

[1072] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is then sent back to the device from the server, where the user can view it through the head-mounted display. For example, if the user enters the text "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[1073] Correction request

[1074] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends the data back to the server. The server accepts the modification request and provides the user with a regenerated cartoon.

[1075] Examples and prompts

[1076] Specific examples

[1077] The user inputs the following natural language in the virtual space.

[1078] Input: "What is the history of this item?"

[1079] Result: The history and background related to the product is displayed in a virtual space in comic form.

[1080] Prompt Sentence Examples

[1081] Write a detailed scene description for a manga panel where a character named A explains the history of a particular product.

[1082] As described above, according to the embodiment of the present invention, a user can easily turn natural language text into a cartoon, and can check and modify the generated cartoon in a virtual space.

[1083] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1084] Step 1:

[1085] The user accesses the terminal's user interface and inputs natural language text. This interface provides a text area, a text input form, a selection box for character settings, and a form for inputting specific character settings. Input: Natural language text and character settings. Output: Data (natural language text, character settings) from the terminal to the server.

[1086] Step 2:

[1087] The server receives the natural language text sent from the device and divides it into paragraphs using natural language processing technology (e.g., BERT or GPT-3). Input: The natural language text received by the server. Output: The paragraphs after division.

[1088] Step 3:

[1089] The server checks the character settings specified by the user. If no character settings are specified, it presents multiple character patterns to the user and applies the selected pattern. Input: User's character settings or unspecified information. Output: Character setting information.

[1090] Step 4:

[1091] For each paragraph, the server invokes a generative AI model (e.g., DALL-E or Stable Diffusion) to extract key information (characters, actions, background, etc.) from the natural language text and generate a manga panel. Input: paragraph, character settings. Output: generated manga panel.

[1092] Step 5:

[1093] The server arranges the generated comic panels in order based on the provided template and lays them out as a page. This layout information becomes the final comic. Input: Generated comic panels. Output: Laid-out comic page.

[1094] Step 6:

[1095] The server converts the final manga into image data or PDF format and sends it back to the terminal. Input: Layout of manga page. Output: Manga in image data or PDF format.

[1096] Step 7:

[1097] The terminal displays the returned manga to the user. This display is done in a virtual space through a head-mounted display. The user can view the manga interactively. Input: Manga in image data or PDF format. Output: Manga displayed on the head-mounted display.

[1098] Step 8:

[1099] When a user requests corrections to the generated manga, they input the changes to a specific frame or character and send the data back to the server from their device. Input: User's correction request. Output: Correction data sent to the server.

[1100] Step 9:

[1101] The server accepts the modification request and returns the regenerated comic to the terminal. Input: Modified data. Output: Modified comic data.

[1102] Step 10:

[1103] The device then displays the revised manga to the user via the head-mounted display. Input: Revised manga data. Output: Revised manga displayed on the head-mounted display.

[1104] Through the above processing steps, the user can generate a manga based on natural language text and interactively check and modify its content in a virtual space.

[1105] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1106] This invention relates to a system that automatically generates manga based on natural language text entered by a user, and by combining it with an emotion engine, it is possible to create manga that reflects the user's emotions. The system configuration and specific program processing are described below.

[1107] System Configuration

[1108] Server Program Processing

[1109] Receiving and dividing text

[1110] The server first receives the natural language text sent by the user. Next, it divides the text into paragraphs using natural language processing technology. For example, if the text is "He confessed to her in the sunset," it will divide it into paragraphs based on specific grammatical rules and punctuation.

[1111] Check and apply character settings

[1112] If the user specifies a specific character setting, the server acquires the character's appearance and characteristics based on that information. If no setting is specified, the server presents the user with multiple pre-prepared character patterns and applies the selected pattern. At this time, the server uses an emotion engine to confirm the user's emotion and adjusts the character's facial expression and pose based on that emotion.

[1113] Invoking a generative AI model

[1114] For each paragraph, a generative AI model is invoked. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that information. For example, from the paragraph "He confessed to her in the sunset," the character settings of "he" and "she" and the situations of "sunset" and "confession" are extracted, and a picture is drawn based on those. The results of the emotion engine are used to adjust the presentation of the panels.

[1115] Panel layout and final manga generation

[1116] The generated frames are arranged in order based on the provided template and laid out as pages. Then, they are combined into a multi-page manga to generate the final manga. This is then converted into image data or PDF format and prepared for delivery to the user.

[1117] Terminal Programming

[1118] Providing a user interface

[1119] The terminal provides an interface for users to input natural language sentences. Here, a text area is provided and a sentence input form is provided. There are also character setting selection boxes and a form for inputting specific character settings. Furthermore, an interface is provided for recognizing the user's emotions using an emotion engine. For example, the user's emotions can be detected by facial expression recognition or voice analysis.

[1120] Submitting data and viewing the generated results

[1121] After the user enters text, character settings, and emotion information and presses the send button, the device sends the data in JSON format to the server. When the server returns the generated manga, the device displays it to the user. An interactive viewer is provided to make it easy to browse the page.

[1122] User operation method

[1123] Entering text and character settings

[1124] The user accesses the system's interface and inputs the text of a novel or story. If necessary, the user can also set up characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako." Furthermore, the user can input or detect their own emotions through the interface.

[1125] Manga generation and review

[1126] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[1127] Correction request

[1128] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends them back to the server. The server accepts the modification requests and provides the user with a regenerated cartoon.

[1129] Harnessing emotions

[1130] The emotion engine optimizes the expression of each panel of the manga based on the user's emotions recognized by the engine. For example, if joy is recognized, the character's expression will be changed to a smile, or other appropriate effect will be applied.

[1131] For example, if a user inputs the sentence "He confessed his feelings to her in the sunset," and the emotion engine recognizes the user's emotion as "tension," the character in the generated manga frame will have a tense expression and pose. In this way, real-time manga generation that reflects the user's emotions is realized.

[1132] As described above, according to the embodiment of the present invention, a user can easily create, check, and modify a comic that reflects emotions based on natural language text.

[1133] The processing flow will be explained below.

[1134] Specific processing steps of the program

[1135] Server Processing

[1136] Step 1: Receiving data

[1137] The server receives the natural language sentences, character setting information, and emotion information sent from the terminal.

[1138] Step 2: Segmentation

[1139] The server uses natural language processing technology to divide the received natural language sentence into paragraphs. For example, it divides the sentence "He confessed to her in the sunset" into paragraphs based on punctuation marks.

[1140] Step 3: Check your character settings

[1141] The server checks whether or not a character setting exists. If a setting exists, the server acquires that information. If not, the server presents multiple pre-prepared character patterns.

[1142] Step 4: Extracting emotional information

[1143] The server analyzes the received emotional information and extracts a specific emotional state (e.g., joy, sadness, tension, etc.) based on the results.

[1144] Step 5: Invoke the generative AI model

[1145] The server calls a generative AI model for each paragraph. The model extracts key information from the paragraph and generates manga panels taking into account emotional information. For example, a scene is drawn based on the paragraph "He confessed his love to her in the sunset," and the character's facial expression and pose are adjusted based on the emotional information.

[1146] Step 6: Creating frames

[1147] The generative AI model generates manga panels for each paragraph based on the extracted key information and emotional information. For example, if the emotional information is "tension," the model will make the character's facial expression tense.

[1148] Step 7: Panel layout

[1149] The server arranges the generated frames in the appropriate order and lays them out as pages, arranging the frames according to a specific storyboard.

[1150] Step 8: Generate the final cartoon

[1151] After all the paragraph panels have been laid out, the server generates the final comic, converts it into image data or PDF format, and prepares it for delivery back to the user.

[1152] Terminal handling

[1153] Step 9: Displaying the User Interface

[1154] The terminal displays an interface for inputting natural language sentences, providing a text area and an input box for character settings, as well as a form for recognizing the user's emotions using an emotion engine.

[1155] Step 10: Send data

[1156] The text, character settings, and emotion information entered by the user are sent to the server, packaged in JSON format and sent via an HTTP request.

[1157] User operations

[1158] Step 11: Enter text

[1159] The user inputs a natural language sentence into the text area, for example, "He confessed to her in the sunset."

[1160] Step 12: Character Setup

[1161] The user configures the character as desired, entering the character's name, gender, and appearance.

[1162] Step 13: Input or Recognize Emotions

[1163] Users input their emotions through the emotion engine, which then automatically detects emotions through facial expression recognition and voice analysis, for example.

[1164] Step 14: Send

[1165] The user presses the send button to send the entered text, character settings, and emotion information to the server.

[1166] Step 15: Check the manga

[1167] Check the manga returned from the server on your device. You can easily browse all pages in the interactive viewer. For example, you can confirm that the generated manga scene depicts a character expressing the emotion of "tension."

[1168] Step 16: Request a correction

[1169] The user requests modifications to the generated cartoon. The user inputs the specific changes and sends them back to the server. The server accepts the modification request and provides the regenerated cartoon to the user.

[1170] These steps allow users to use a system that uses natural language text and emotional information to easily generate, review, and revise more emotionally rich comics.

[1171] Example 2

[1172] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1173] In today's world, systems that automatically generate manga based on users' natural language input are required to meet diverse entertainment needs. However, current technology for manga generation systems that incorporate character expressions that reflect the user's emotions is insufficient. Furthermore, there is a lack of flexibility in the ability to request corrections to the generated manga and in character settings. A system that can resolve these issues is needed.

[1174] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1175] In this invention, the server includes means for inputting a natural language sentence, means for dividing the natural language sentence into paragraphs, means for allowing the user to specify a specific character setting, means for acquiring the character's appearance and characteristics based on the character setting, means for confirming the user's emotions using an emotion engine and adjusting the character's facial expression and pose, means for calling a generative model for generating manga panels for each paragraph, means for laying out multiple manga panels in an appropriate order, means for generating the final manga, and means for returning the generated manga to the user. This enables real-time manga generation that reflects the user's emotions and also realizes a flexible system that can respond to requests for revisions.

[1176] A "natural language sentence" is a sentence written in a language that a user uses on a daily basis.

[1177] A "paragraph" is a portion of text that uses line breaks and spaces to separate meanings or content within a sentence.

[1178] "Character settings" are setting information that allows a user to specify the appearance, personality, and characteristics of a specific character.

[1179] A "generative model" is an algorithm or program that automatically generates manga panels from input natural language text.

[1180] The "emotion engine" is a system that recognizes emotions from the user's facial expressions and voice, and adjusts the character's expressions based on that information.

[1181] A "manga panel" is a combination of pictures and text that separates scenes in a manga.

[1182] A "template" is a predetermined framework or format for determining the page layout of a manga.

[1183] The "final manga" is a manga work that is compiled into a single story by appropriately laying out the multiple frames that have been generated.

[1184] A "request for correction" is a request made by a user to notify the server of any corrections or changes that the user wishes to make to a work.

[1185] This invention relates to a system that automatically generates manga based on natural language text entered by a user, and by combining it with an emotion engine, it is possible to create manga that reflects the user's emotions. The system configuration and specific program processing are described below.

[1186] Server Program Processing

[1187] The server first receives the natural language text sent by the user. It then uses natural language processing technology to divide the text into paragraphs and invokes a generative AI model to extract key information (such as characters, actions, and background) from each paragraph. If the user has specified a specific character setting, the server obtains the character's appearance and characteristics based on that information, uses an emotion engine to confirm the user's emotions, and adjusts the character's facial expressions and poses based on those emotions.

[1188] For example, when processing the paragraph "He confessed his love to her in the sunset," the generative AI model extracts the character settings of "him" and "her" and the situations of "sunset" and "confession," and draws a picture based on these. The results of the emotion engine are used to adjust the presentation of the panels. The generated panels are then arranged in order based on the provided template and laid out as pages. The resulting multi-page manga is then compiled into a single page, generating the final manga. This is then converted into image data or PDF format, ready to be sent back to the user.

[1189] Terminal Programming

[1190] The terminal provides an interface for users to input natural language text. A text area is provided here, providing a text input form. Selection boxes for character settings and a form for inputting specific character settings are also provided. An interface is also provided for recognizing the user's emotions using an emotion engine. For example, the user's emotions are detected using facial expression recognition or voice analysis. After the user enters the text, character settings, and emotional information and presses the send button, the terminal sends this data in JSON format to the server. When the generated manga is returned from the server, the terminal displays it to the user. An interactive viewer is provided to make it easy to browse the page.

[1191] User operation method

[1192] The user accesses the system's interface and inputs the text of a novel or story. If necessary, the user can also set up characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako." Furthermore, the user can input or detect their own emotions through the interface.

[1193] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[1194] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends them back to the server. The server accepts the modification requests and provides the user with a regenerated cartoon.

[1195] The emotion engine optimizes the expression of each panel of the manga based on the user's emotions recognized by the engine. For example, if joy is recognized, the character's expression will be changed to a smile, or other appropriate effect will be applied.

[1196] For example, if a user inputs the sentence "He confessed his feelings to her in the sunset," and the emotion engine recognizes the user's emotion as "tension," the character in the generated manga frame will have a tense expression and pose. In this way, real-time manga generation that reflects the user's emotions is realized.

[1197] As described above, according to the embodiment of the present invention, a user can easily create, check, and modify a comic that reflects emotions based on natural language text.

[1198] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1199] Step 1:

[1200] Enter text and set characters from the user interface

[1201] Terminal

[1202] The user accesses the system's interface and inputs natural language text into a text area. They then specify specific character appearances and characteristics in a character configuration form. At this stage, they also input facial expressions and voice data for emotion recognition. The input text, character configuration, and emotion information are collected on the terminal.

[1203] Input: User-entered natural language text, character settings, and emotional information

[1204] Output: Input data collected on the terminal

[1205] Step 2:

[1206] Sending and receiving input data

[1207] Terminal

[1208] When the user completes the input and presses the send button, the device sends the collected text, character settings, and emotional information to the server in JSON format.

[1209] server

[1210] The server receives the data sent from the terminal and begins preparations for data processing.

[1211] Input: User input data sent from the device (JSON format)

[1212] Output: The input data received by the server

[1213] Step 3:

[1214] Dividing the text and checking the character settings

[1215] server

[1216] The server uses natural language processing technology to divide the received text into paragraphs. Each paragraph represents a different scene or event, making this an essential operation for generating effective manga. The server also checks the specified character settings and automatically selects multiple character patterns if the user has not specified them.

[1217] Input: JSON format user input data (natural language text, character settings)

[1218] Output: Separated paragraphs and confirmed character settings

[1219] Step 4:

[1220] Emotion recognition by emotion engine

[1221] server

[1222] The server runs an emotion engine, analyzes the received user facial and voice data, and recognizes emotions. For example, if the user's facial expression is "smiling," it is recognized as "joy," and if the user's voice is trembling, it is recognized as "tension." This emotional information is reflected in the character's expression when generating the manga.

[1223] Input: Received user facial expression and voice data

[1224] Output: Analyzed user emotion information

[1225] Step 5:

[1226] Manga panel generation using generative AI models

[1227] server

[1228] The server calls a generative AI model for each paragraph of the divided text. This model extracts key information from the text (characters, actions, background, etc.) and generates manga panels. In doing so, it sets the character's expressions and poses based on the results of the emotion engine. For example, in the paragraph "He confessed to her in the sunset," a sunset background and cat illustrations are generated to depict the confession scene.

[1229] Input: Divided paragraphs, sentiment information

[1230] Output: Generated comic panels

[1231] Step 6:

[1232] Panel layout and final manga generation

[1233] server

[1234] The generated panels are arranged in order based on a template and laid out as pages. The laid out pages are compiled into a manga book and the final manga data is generated. The generated manga data is converted into image data or PDF format.

[1235] Input: Multiple generated comic panels

[1236] Output: Final manga data (image data or PDF format)

[1237] Step 7:

[1238] Sending and displaying the final manga

[1239] server

[1240] The generated final manga data is sent to the terminal.

[1241] Terminal

[1242] The device receives the manga data sent from the server and displays it to the user, using an interactive viewer to allow easy page browsing.

[1243] Input: Final manga data

[1244] Output: Comic data in a format that can be viewed by users

[1245] Step 8:

[1246] Correction Request Processing

[1247] User

[1248] If the user wishes to make corrections to the generated comic, he or she inputs changes to specific frames or characters.

[1249] Terminal

[1250] The device sends a modification request in JSON format to the server.

[1251] server

[1252] The server accepts the modification request, regenerates the comic with the specified modifications reflected, and sends the regenerated comic to the terminal again.

[1253] Input: Correction Request

[1254] Output: New manga data with the corrections reflected

[1255] (Application example 2)

[1256] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1257] Systems already exist that automatically generate manga based on natural language text, but few systems can naturally express the emotions of the user in the panels. Furthermore, to improve the quality of the generated manga and user satisfaction, it is necessary to adjust the character expressions based on emotions. Furthermore, there is a demand for an interface that allows users to quickly view the generated manga and quickly request corrections. Current technology does not yet realize such advanced emotional reflection and interactive operation, so there is a need to solve these issues.

[1258] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1259] In this invention, the server includes means for inputting natural language sentences, means for dividing the natural language sentences into paragraphs, means for allowing the user to specify specific character settings, means for calling a generative model that generates comic panels for each paragraph, means for laying out multiple comic panels in an appropriate order, means for generating a final comic, means for returning the generated comic to the user, and means for analyzing the user's emotions and adjusting the expression of the comic panels based on the analysis results. This makes it possible to automatically generate comics based on the natural language sentences input by the user, and further to reflect the user's emotions in the expression of the panels.

[1260] A "natural language sentence" is a sentence that is input by a user and is composed of words that are used daily.

[1261] The "paragraph division method" is a method for automatically dividing an input natural language sentence into multiple paragraphs based on grammatical rules and punctuation marks.

[1262] "Character settings" are setting information about the character's appearance and personality specified by the user.

[1263] A "generative model" is an AI technology that extracts key information from natural language sentences and generates manga panels based on that information.

[1264] The "layout method" is a method for arranging the generated multiple comic panels in the appropriate order and formatting them as a page.

[1265] The "final manga" is the final product of the manga, with all panels properly laid out and put together.

[1266] The "returning means" is a method for sending the generated comic to the user.

[1267] "Means for analyzing emotions" refers to technology that recognizes the user's emotions and adjusts the character's expression based on the analysis results.

[1268] This invention is a system for automatically generating and providing comics that reflect emotions based on natural language sentences entered by a user. A specific embodiment of the system and its operation will be described below.

[1269] System Program

[1270] 1. Entering and dividing text

[1271] The terminal provides an interface for the user to input natural language sentences. Once the user has completed input, the sentence is sent to the server, which divides the sentence into paragraphs. This process uses natural language processing technology. For example, if the sentence "He confessed to her in the sunset" is received, the server divides the sentence into paragraphs.

[1272] 2. Applying character settings

[1273] If the user specifies a specific character setting, the server will obtain the character's appearance and characteristics based on that information. If no character setting is available, the server will present the user with multiple pre-prepared character patterns and apply the selected pattern. At this time, the device also provides an interface for recognizing the user's emotions, detecting them through facial expression recognition and voice analysis.

[1274] 3. Invoking the generative AI model

[1275] The server calls a generative AI model for each divided paragraph. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that information. For example, from the paragraph "He confessed to her in the sunset," the model extracts the character settings of "he" and "she" and the situations of "sunset" and "confession," and draws a picture based on those. The results of the emotion engine are used to adjust the presentation of the panels.

[1276] 4. Panel layout and final manga generation

[1277] The server arranges the generated frames in order based on the provided template and lays them out as pages. It then combines multiple comic frames into one and finally generates a comic. It then converts this comic into image data or PDF format and returns it to the user.

[1278] Hardware and software used

[1279] Hardware:

[1280] Smartphone (Android or iOS device)

[1281] software:

[1282] Flask (Python framework, backend)

[1283] Transformers (NLP library, sentiment analysis)

[1284] Remote server (running the manga generation AI model)

[1285] Specific examples

[1286] The user launches the application and enters the following text:

[1287] Example prompt statement:

[1288] He confessed to her in the sunset.

[1289] Given this sentence, the system will do the following:

[1290] Use natural language processing to divide the text into paragraphs.

[1291] The character setting specified by the user is applied, and if there is no setting, the user is allowed to select from a plurality of character patterns.

[1292] The emotion engine is used to analyze the user's emotions, and based on the results, a generative AI model is called to generate manga panels.

[1293] The generated panels are laid out in the appropriate order to generate the final cartoon, which is then returned to the user.

[1294] In this way, users can easily create original comics that reflect their own emotions based on input natural language sentences, and then review and share them.

[1295] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1296] Step 1:

[1297] The terminal provides an interface for the user to input a natural language sentence. The user inputs the natural language sentence and sends the sentence to the server. At this time, the input natural language sentence might be, for example, "He confessed to her in the sunset."

[1298] Step 2:

[1299] The server divides the received natural language sentence into paragraphs. It uses natural language processing technology for sentence analysis and automatically divides the sentence into multiple paragraphs based on grammatical rules and punctuation. For example, it divides the sentence into paragraphs like "He was in the sunset" and "He confessed to her."

[1300] Step 3:

[1301] The terminal provides an interface for the user to specify specific character settings. When the user inputs the character settings (appearance and personality), that information is sent to the server. If there is no character setting, the server presents multiple character patterns to the user and applies the selected pattern.

[1302] Step 4:

[1303] The server calls a generative AI model for each paragraph. This model extracts key information from the text (such as characters, actions, and background) and generates manga panels based on that information. For example, the model extracts the character settings of "him" and "her" and the situations such as "sunset" and "confession."

[1304] Step 5:

[1305] The server uses an emotion engine to analyze emotions based on the user's input text. Emotion analysis uses the user's emotional information sent from the device (e.g., the results of facial expression recognition and voice analysis). For example, if the emotion "tension" is recognized, the character's facial expression and pose are adjusted accordingly.

[1306] Step 6:

[1307] The server lays out the generated panels in an orderly manner, placing each panel in the appropriate order based on the provided template to form a page, and then combining multiple pages to generate the final manga.

[1308] Step 7:

[1309] The server converts the generated comic into image data or PDF format and sends it back to the terminal. The user can check the generated comic on the terminal. For example, if the user inputs the prompt sentence "He confessed to her in the sunset," a comic depicting that story will be displayed on the terminal.

[1310] Step 8:

[1311] The terminal provides an interface for accepting a request for modification of the generated cartoon from the user. When the user inputs the modification request and sends it to the server, the server accepts the modification request and returns the regenerated cartoon to the user.

[1312] In this way, by performing specific actions at each step, users can easily create original manga that reflect their emotions based on the natural language text they input, and then review and share them.

[1313] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1314] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1315] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1316] [Fourth embodiment]

[1317] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1318] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1319] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1320] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1321] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1322] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1323] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1324] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1325] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1326] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1327] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1328] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1329] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1330] This invention relates to a system that automatically generates manga based on natural language text entered by a user. The system divides the input text into paragraphs, applies specific character settings, and then uses a generative AI model to create manga panels for each paragraph, laying them out appropriately to generate the final manga. In addition, if the user does not specify character settings, the system also has the ability to select from multiple character patterns and to accept requests for revisions to the generated manga.

[1331] System Configuration

[1332] Server Program Processing

[1333] Receiving and dividing text

[1334] The server first receives the natural language text sent by the user. Next, it divides the text into paragraphs using natural language processing technology. For example, if the text is "He confessed to her in the sunset," it will divide it into paragraphs based on specific grammatical rules and punctuation.

[1335] Check and apply character settings

[1336] If the user specifies a specific character setting, the server acquires the character's appearance and characteristics based on that information. If no setting is specified, the server presents the user with multiple pre-prepared character patterns and applies the selected pattern.

[1337] Invoking a generative AI model

[1338] For each paragraph, a generative AI model is invoked. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that. For example, from the paragraph "He confessed to her in the sunset," the model extracts the character settings of "he" and "she" as well as the situations of "sunset" and "confession," and draws a picture based on that.

[1339] Panel layout and final manga generation

[1340] The generated frames are arranged in order based on the provided template and laid out as pages. Then, they are combined into a multi-page manga to generate the final manga. This is then converted into image data or PDF format and prepared for delivery to the user.

[1341] Terminal Programming

[1342] Providing a user interface

[1343] The terminal provides an interface for users to input natural language sentences. Here, a text area is provided, and a sentence input form is provided. There are also character setting selection boxes and a form for inputting specific character settings.

[1344] Submitting data and viewing the generated results

[1345] After the user enters text and character settings and presses the send button, the device sends the data in JSON format to the server. When the server returns the generated manga, the device displays it to the user. An interactive viewer is provided to make it easy to browse the page.

[1346] User operation method

[1347] Entering text and character settings

[1348] The user accesses the system's interface and inputs the text of the novel or story. If necessary, the user can also set up the characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako."

[1349] Manga generation and review

[1350] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[1351] Correction request

[1352] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends the data back to the server. The server accepts the modification request and provides the user with a regenerated cartoon.

[1353] As described above, according to the embodiment of the present invention, a user can easily create a comic from a natural language text, and can check and modify the created comic.

[1354] The processing flow will be explained below.

[1355] Specific processing steps of the program

[1356] Server Processing

[1357] Step 1: Receiving data

[1358] The server receives the natural language sentence and character setting information sent from the terminal.

[1359] Step 2: Segmentation

[1360] The server uses natural language processing techniques to divide the received natural language text into paragraphs, for example, based on grammar rules and punctuation.

[1361] Step 3: Check your character settings

[1362] The server checks whether or not a character setting exists. If a setting exists, the server acquires that information. If not, it selects one of several pre-prepared character patterns.

[1363] Step 4: Invoke the generative AI model

[1364] The server calls a generative AI model for each paragraph. The model extracts key information from the paragraph and generates a comic panel. For example, it draws a scene based on the paragraph, "He confessed his love to her in the sunset."

[1365] Step 5: Creating frames

[1366] The generative AI model generates panels based on each paragraph, including character appearances and backgrounds.

[1367] Step 6: Panel layout

[1368] The server arranges the generated frames in order and lays them out as pages. The frames are arranged based on the layout.

[1369] Step 7: Generate the final cartoon

[1370] After all the paragraph panels have been laid out, the server generates the final comic, converts it into image data or PDF format, and prepares it for delivery back to the user.

[1371] Terminal handling

[1372] Step 8: Displaying the User Interface

[1373] The terminal displays an interface for inputting natural language text, providing text areas and a character configuration form.

[1374] Step 9: Send data

[1375] The text and character setting information entered by the user is sent to the server, packaged in JSON format, and sent via an HTTP request.

[1376] User operations

[1377] Step 10: Enter text

[1378] The user enters a natural language sentence into the text area.

[1379] Step 11: Character Setup

[1380] The user configures the character as desired, entering the character's name, gender, and appearance.

[1381] Step 12: Send

[1382] The user presses the send button to send the entered text and character settings to the server.

[1383] Step 13: Check the manga

[1384] The manga returned from the server can be viewed on the user's device. All pages can be viewed in an interactive viewer.

[1385] Step 14: Request a correction

[1386] The user requests modifications to the generated manga, inputs the specific changes, and sends the request back to the server.

[1387] By following the above steps, a user can easily create, check, and modify a manga based on natural language text.

[1388] Example 1

[1389] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1390] Traditionally, manga production has been done manually, requiring a great deal of time and effort. It also requires specialized skills and experience, making it difficult for ordinary users to turn their own stories into manga. Furthermore, corrections and changes take time, making it inefficient.

[1391] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1392] In this invention, the server includes means for inputting natural language sentences, means for dividing the natural language sentences into paragraphs, means for allowing the user to specify specific character settings, means for calling a generative model that generates comic panels for each paragraph, means for inputting prompt sentences to the generative model and generating comic panels based on key information, means for laying out multiple comic panels in an appropriate order, means for generating the final comic, and means for returning the generated comic to the user. This enables users to easily generate and edit comics based on natural language sentences without having specialized knowledge or skills.

[1393] A "natural language sentence" is a text entered by a user and written in a normal human language.

[1394] The "means for dividing into paragraphs" is a function that uses natural language processing technology to divide an input natural language sentence into multiple paragraphs based on grammatical rules and punctuation marks.

[1395] "Character settings" are information about the appearance and characteristics of the characters in the manga, specified by the user.

[1396] A "generative model" is an AI model that extracts key information from natural language sentences and generates manga panels based on that information.

[1397] A "prompt sentence" is an instruction sentence for inputting key information extracted from a natural language sentence to a generative model.

[1398] A "manga panel" is an individual image or frame that depicts a scene or situation in a manga.

[1399] The "layout means" is a function that arranges the generated comic panels in the appropriate order and position.

[1400] The "final manga" refers to the entire manga work that is completed by laying out the multiple manga panels that have been generated.

[1401] "Means for returning" is a function for sending the final comic created to the user.

[1402] This invention relates to a system that automatically generates manga based on natural language text entered by a user. The system divides the input text into paragraphs, applies specific character settings, and then uses a generative AI model to create manga panels for each paragraph, laying them out appropriately to generate the final manga. In addition, if the user does not specify character settings, the system also has the ability to select from multiple character patterns and to accept requests for corrections to the generated manga.

[1403] Server Program Processing

[1404] First, the server receives the natural language text sent by the user. This text is received in JSON format. Next, the server uses a Python natural language processing library (e.g., NLTK or spaCy) to divide the text into paragraphs. For example, if the server receives the text "He confessed to her in the sunset," it divides it into paragraphs based on grammar rules and punctuation.

[1405] Next, if the user specifies a specific character configuration, the server retrieves that information from a database that stores the character's appearance and characteristics. If no configuration is specified, the server presents the user with a list of character configurations and applies the pattern selected by the user.

[1406] Next, for each paragraph, the server calls a generative AI model (for example, OpenAI's GPT-4). The model extracts key information from the paragraph (characters, action, background) and generates manga panels based on that information. A prompt is created and input to the model. For example, for the paragraph "He confessed to her in the sunset," the prompt "The scene should include a situation where 'He confesses to her in the sunset'" is input to the generative AI model.

[1407] The generated comic panels are arranged in order based on a prepared template. They are laid out as pages using HTML5 and CSS3, and combined into a single multi-page comic. The final comic is converted into image data or PDF format, and prepared for delivery to the user. The conversion is performed using the Python ReportLab library.

[1408] Terminal Programming

[1409] The terminal provides an interface for the user to input natural language text. For example, it uses HTML and JavaScript to provide text areas, character setting selection boxes, and forms for entering specific character settings. When the user enters text and character settings and presses the submit button, the terminal sends the data to the server in JSON format.

[1410] Once the generated comic is sent back from the server, the device displays it to the user, allowing them to easily browse the page using an interactive viewer (e.g., React or Vue.js).

[1411] User operation method

[1412] The user accesses the system's interface and inputs the text of the novel or story. If necessary, the user can also set up the characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako."

[1413] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[1414] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends the data back to the server. The server accepts the modification request and provides the user with a regenerated cartoon.

[1415] In this way, according to the embodiment of the present invention, the user can easily convert natural language text into a cartoon, and can check and correct the generated cartoon.

[1416] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1417] Specific explanation of processing steps

[1418] Step 1: Enter and send natural language text

[1419] User Actions

[1420] The user accesses the system's interface and inputs a natural language sentence. For example, they can input a sentence such as "He confessed his love to her in the sunset," and set up the characters as needed. They can also specify the appearance and personality of "Taro" and "Hanako." When the user has completed input and pressed the send button, the device sends this data to the server in JSON format.

[1421] input

[1422] Natural language text and character settings.

[1423] output

[1424] The data is sent to the server in JSON format.

[1425] Step 2: Receiving text and dividing it into paragraphs

[1426] Server Operation

[1427] The server receives the JSON data sent from the device. The received data is stored in memory. Then, using Python's natural language processing library (NLTK or spaCy), the server divides the sentence into paragraphs based on grammar rules and punctuation. For example, the sentence "He confessed to her in the sunset" is divided into paragraphs such as "He confessed to her in the sunset" and "He confessed to her."

[1428] input

[1429] JSON data of natural language text sent from the device.

[1430] output

[1431] A list of separated paragraphs.

[1432] Step 3: Check and apply character settings

[1433] Server Operation

[1434] The server checks whether the user has specified a specific character setting. If so, it retrieves that information from a database, which stores the character's appearance and characteristics. If no setting is specified, it presents the user with a list of multiple character patterns and applies the pattern selected by the user.

[1435] input

[1436] User-specified character settings or multiple character patterns.

[1437] output

[1438] Applied character settings.

[1439] Step 4: Generate manga panels using a generative AI model

[1440] Server Operation

[1441] For each paragraph, the server calls a generative AI model (for example, OpenAI's GPT-4). A prompt is created and input to the generative AI model. The model extracts key information (characters, action, background) from the paragraph and generates manga panels based on that information. For example, for the paragraph "He confessed to her in the sunset," the prompt "The scene should include the situation where 'He confesses to her in the sunset'" is input to the generative AI model.

[1442] input

[1443] A prompt sentence created for each paragraph.

[1444] output

[1445] Generated comic panels.

[1446] Step 5: Layout of the comic panels

[1447] Server Operation

[1448] The generated comic panels are arranged in order based on a prepared template. The page is laid out using HTML5 and CSS3. The order and position of the panels are determined by the template.

[1449] input

[1450] Generated comic panels.

[1451] output

[1452] Layout of the manga page.

[1453] Step 6: Generate and return the final comic

[1454] Server Operation

[1455] The process combines multiple pages of comics into one to generate the final comic. The generated comic is then converted into image data or PDF format and prepared for delivery to the user. Specifically, the process converts to PDF format using the Python ReportLab library.

[1456] input

[1457] Layout of the manga page.

[1458] output

[1459] Final manga data (image data or PDF format).

[1460] Step 7: View and modify the generated results

[1461] Device behavior

[1462] The device receives the final comic data sent back from the server. It displays the results to the user using an interactive viewer (such as React or Vue.js). If the user requests corrections, they enter the corrections and send them back to the server. The server processes the regenerated comic and provides it to the user.

[1463] input

[1464] The final comic data returned from the server and the user's modification request.

[1465] output

[1466] Displayed cartoon and modified cartoon data.

[1467] In this way, each processing step proceeds with specific data inputs and outputs, ultimately providing the comic to the user.

[1468] (Application example 1)

[1469] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1470] Conventional manga generation systems using natural language processing have limited ways for users to view the generated manga, making it difficult to provide an interactive experience. In addition, there is a lack of means to visually present product information in a virtual space, which limits the user experience.

[1471] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1472] In this invention, the server includes means for inputting a natural language sentence, means for dividing the natural language sentence into paragraphs, means for allowing a user to specify specific character settings, means for calling a generative model that generates comic panels for each paragraph, means for laying out multiple comic panels in an appropriate order, means for generating a final comic, means for returning the generated comic to the user, means for displaying the generated comic using a head-mounted display, and means for the user to input natural language in a virtual space, thereby enabling the user to visually understand product information through an interactive experience.

[1473] "Natural language sentences" are text data entered by a user using a general language.

[1474] A "paragraph" refers to a coherent part of a natural language sentence.

[1475] "Character settings" is information about the characters and attributes that appear in the manga specified by the user.

[1476] A "generative model" is an algorithm or program that automatically generates manga panels from natural language text.

[1477] A "manga panel" is a small screen or frame in a manga that depicts part of a story using pictures and text.

[1478] "Layout" refers to arranging multiple comic panels in the appropriate order.

[1479] A "head-mounted display" is a display device that provides visual information when worn by a user.

[1480] "Virtual space" is a virtual three-dimensional space generated by a computer.

[1481] "Product information" is detailed data about products and services that interest users.

[1482] This invention relates to a system that automatically generates manga based on natural language text entered by a user and displays the content in a virtual space. The system uses natural language processing technology to divide the text into paragraphs and apply specific character settings. Next, a generative AI model is used to create manga panels for each paragraph, and the panels are appropriately laid out to generate the final manga. The generated manga is then displayed to the user via a head-mounted display.

[1483] System Configuration

[1484] Server Program Processing

[1485] Receiving and dividing text

[1486] The server first receives the natural language sentence sent by the user. This sentence is divided into paragraphs using natural language processing techniques (such as BERT or GPT-3). For example, if the sentence received is "He confessed to her in the sunset," it is divided into paragraphs based on specific grammatical rules and punctuation.

[1487] Check and apply character settings

[1488] If the user specifies a specific character setting, the server acquires the character's appearance and characteristics based on that information. If no setting is specified, the server presents the user with multiple pre-prepared character patterns and applies the selected pattern.

[1489] Invoking a generative AI model

[1490] For each paragraph, a generative AI model (such as DALL-E or Stable Diffusion) is called. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that. For example, from the paragraph "He confessed to her in the sunset," the character settings of "he" and "she" and the situations of "sunset" and "confession" are extracted, and a picture is drawn based on that.

[1491] Panel layout and final manga generation

[1492] The generated frames are arranged in order based on the provided template and laid out as pages. Then, they are combined into a multi-page manga to generate the final manga. This is then converted into image data or PDF format and prepared for delivery to the user.

[1493] Display in virtual space

[1494] The final manga is displayed in a virtual space using a head-mounted display, allowing users to interactively view the generated manga in the virtual space.

[1495] Terminal Programming

[1496] Providing a user interface

[1497] The terminal provides an interface for users to input natural language sentences. Here, a text area is provided, and a sentence input form is provided. There are also character setting selection boxes and a form for inputting specific character settings.

[1498] Submitting data and viewing the generated results

[1499] After the user inputs text and character settings and presses the send button, the device sends the data in JSON format to the server. The generated manga is then sent back from the server, and the device displays it in a virtual space via a head-mounted display.

[1500] User operation method

[1501] Entering text and character settings

[1502] The user accesses the system's interface and inputs the text of the novel or story. If necessary, the user can also set up characters. For example, the user can specify the appearance and personality of characters "A" and "B."

[1503] Manga generation and review

[1504] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is then sent back to the device from the server, where the user can view it through the head-mounted display. For example, if the user enters the text "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[1505] Correction request

[1506] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends the data back to the server. The server accepts the modification request and provides the user with a regenerated cartoon.

[1507] Examples and prompts

[1508] Specific examples

[1509] The user inputs the following natural language in the virtual space.

[1510] Input: "What is the history of this item?"

[1511] Result: The history and background related to the product is displayed in a virtual space in comic form.

[1512] Prompt Sentence Examples

[1513] Write a detailed scene description for a manga panel where a character named A explains the history of a particular product.

[1514] As described above, according to the embodiment of the present invention, a user can easily turn natural language text into a cartoon, and can check and modify the generated cartoon in a virtual space.

[1515] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1516] Step 1:

[1517] The user accesses the terminal's user interface and inputs natural language text. This interface provides a text area, a text input form, a selection box for character settings, and a form for inputting specific character settings. Input: Natural language text and character settings. Output: Data (natural language text, character settings) from the terminal to the server.

[1518] Step 2:

[1519] The server receives the natural language text sent from the device and divides it into paragraphs using natural language processing technology (e.g., BERT or GPT-3). Input: The natural language text received by the server. Output: The paragraphs after division.

[1520] Step 3:

[1521] The server checks the character settings specified by the user. If no character settings are specified, it presents multiple character patterns to the user and applies the selected pattern. Input: User's character settings or unspecified information. Output: Character setting information.

[1522] Step 4:

[1523] For each paragraph, the server invokes a generative AI model (e.g., DALL-E or Stable Diffusion) to extract key information (characters, actions, background, etc.) from the natural language text and generate a manga panel. Input: paragraph, character settings. Output: generated manga panel.

[1524] Step 5:

[1525] The server arranges the generated comic panels in order based on the provided template and lays them out as a page. This layout information becomes the final comic. Input: Generated comic panels. Output: Laid-out comic page.

[1526] Step 6:

[1527] The server converts the final manga into image data or PDF format and sends it back to the terminal. Input: Layout of manga page. Output: Manga in image data or PDF format.

[1528] Step 7:

[1529] The terminal displays the returned manga to the user. This display is done in a virtual space through a head-mounted display. The user can view the manga interactively. Input: Manga in image data or PDF format. Output: Manga displayed on the head-mounted display.

[1530] Step 8:

[1531] When a user requests corrections to the generated manga, they input the changes to a specific frame or character and send the data back to the server from their device. Input: User's correction request. Output: Correction data sent to the server.

[1532] Step 9:

[1533] The server accepts the modification request and returns the regenerated comic to the terminal. Input: Modified data. Output: Modified comic data.

[1534] Step 10:

[1535] The device then displays the revised manga to the user via the head-mounted display. Input: Revised manga data. Output: Revised manga displayed on the head-mounted display.

[1536] Through the above processing steps, the user can generate a manga based on natural language text and interactively check and modify its content in a virtual space.

[1537] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1538] This invention relates to a system that automatically generates manga based on natural language text entered by a user, and by combining it with an emotion engine, it is possible to create manga that reflects the user's emotions. The system configuration and specific program processing are described below.

[1539] System Configuration

[1540] Server Program Processing

[1541] Receiving and dividing text

[1542] The server first receives the natural language text sent by the user. Next, it divides the text into paragraphs using natural language processing technology. For example, if the text is "He confessed to her in the sunset," it will divide it into paragraphs based on specific grammatical rules and punctuation.

[1543] Check and apply character settings

[1544] If the user specifies a specific character setting, the server acquires the character's appearance and characteristics based on that information. If no setting is specified, the server presents the user with multiple pre-prepared character patterns and applies the selected pattern. At this time, the server uses an emotion engine to confirm the user's emotion and adjusts the character's facial expression and pose based on that emotion.

[1545] Invoking a generative AI model

[1546] For each paragraph, a generative AI model is invoked. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that information. For example, from the paragraph "He confessed to her in the sunset," the character settings of "he" and "she" and the situations of "sunset" and "confession" are extracted, and a picture is drawn based on those. The results of the emotion engine are used to adjust the presentation of the panels.

[1547] Panel layout and final manga generation

[1548] The generated frames are arranged in order based on the provided template and laid out as pages. Then, they are combined into a multi-page manga to generate the final manga. This is then converted into image data or PDF format and prepared for delivery to the user.

[1549] Terminal Programming

[1550] Providing a user interface

[1551] The terminal provides an interface for users to input natural language sentences. Here, a text area is provided and a sentence input form is provided. There are also character setting selection boxes and a form for inputting specific character settings. Furthermore, an interface is provided for recognizing the user's emotions using an emotion engine. For example, the user's emotions can be detected by facial expression recognition or voice analysis.

[1552] Submitting data and viewing the generated results

[1553] After the user enters text, character settings, and emotion information and presses the send button, the device sends the data in JSON format to the server. When the server returns the generated manga, the device displays it to the user. An interactive viewer is provided to make it easy to browse the page.

[1554] User operation method

[1555] Entering text and character settings

[1556] The user accesses the system's interface and inputs the text of a novel or story. If necessary, the user can also set up characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako." Furthermore, the user can input or detect their own emotions through the interface.

[1557] Manga generation and review

[1558] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[1559] Correction request

[1560] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends them back to the server. The server accepts the modification requests and provides the user with a regenerated cartoon.

[1561] Harnessing emotions

[1562] The emotion engine optimizes the expression of each panel of the manga based on the user's emotions recognized by the engine. For example, if joy is recognized, the character's expression will be changed to a smile, or other appropriate effect will be applied.

[1563] For example, if a user inputs the sentence "He confessed his feelings to her in the sunset," and the emotion engine recognizes the user's emotion as "tension," the character in the generated manga frame will have a tense expression and pose. In this way, real-time manga generation that reflects the user's emotions is realized.

[1564] As described above, according to the embodiment of the present invention, a user can easily create, check, and modify a comic that reflects emotions based on natural language text.

[1565] The processing flow will be explained below.

[1566] Specific processing steps of the program

[1567] Server Processing

[1568] Step 1: Receiving data

[1569] The server receives the natural language sentences, character setting information, and emotion information sent from the terminal.

[1570] Step 2: Segmentation

[1571] The server uses natural language processing technology to divide the received natural language sentence into paragraphs. For example, it divides the sentence "He confessed to her in the sunset" into paragraphs based on punctuation marks.

[1572] Step 3: Check your character settings

[1573] The server checks whether or not a character setting exists. If a setting exists, the server acquires that information. If not, the server presents multiple pre-prepared character patterns.

[1574] Step 4: Extracting emotional information

[1575] The server analyzes the received emotional information and extracts a specific emotional state (e.g., joy, sadness, tension, etc.) based on the results.

[1576] Step 5: Invoke the generative AI model

[1577] The server calls a generative AI model for each paragraph. The model extracts key information from the paragraph and generates manga panels taking into account emotional information. For example, a scene is drawn based on the paragraph "He confessed his love to her in the sunset," and the character's facial expression and pose are adjusted based on the emotional information.

[1578] Step 6: Creating frames

[1579] The generative AI model generates manga panels for each paragraph based on the extracted key information and emotional information. For example, if the emotional information is "tension," the model will make the character's facial expression tense.

[1580] Step 7: Panel layout

[1581] The server arranges the generated frames in the appropriate order and lays them out as pages, arranging the frames according to a specific storyboard.

[1582] Step 8: Generate the final cartoon

[1583] After all the paragraph panels have been laid out, the server generates the final comic, converts it into image data or PDF format, and prepares it for delivery back to the user.

[1584] Terminal handling

[1585] Step 9: Displaying the User Interface

[1586] The terminal displays an interface for inputting natural language sentences, providing a text area and an input box for character settings, as well as a form for recognizing the user's emotions using an emotion engine.

[1587] Step 10: Send data

[1588] The text, character settings, and emotion information entered by the user are sent to the server, packaged in JSON format and sent via an HTTP request.

[1589] User operations

[1590] Step 11: Enter text

[1591] The user inputs a natural language sentence into the text area, for example, "He confessed to her in the sunset."

[1592] Step 12: Character Setup

[1593] The user configures the character as desired, entering the character's name, gender, and appearance.

[1594] Step 13: Input or Recognize Emotions

[1595] Users input their emotions through the emotion engine, which then automatically detects emotions through facial expression recognition and voice analysis, for example.

[1596] Step 14: Send

[1597] The user presses the send button to send the entered text, character settings, and emotion information to the server.

[1598] Step 15: Check the manga

[1599] Check the manga returned from the server on your device. You can easily browse all pages in the interactive viewer. For example, you can confirm that the generated manga scene depicts a character expressing the emotion of "tension."

[1600] Step 16: Request a correction

[1601] The user requests modifications to the generated cartoon. The user inputs the specific changes and sends them back to the server. The server accepts the modification request and provides the regenerated cartoon to the user.

[1602] These steps allow users to use a system that uses natural language text and emotional information to easily generate, review, and revise more emotionally rich comics.

[1603] Example 2

[1604] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1605] In today's world, systems that automatically generate manga based on users' natural language input are required to meet diverse entertainment needs. However, current technology for manga generation systems that incorporate character expressions that reflect the user's emotions is insufficient. Furthermore, there is a lack of flexibility in the ability to request corrections to the generated manga and in character settings. A system that can resolve these issues is needed.

[1606] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1607] In this invention, the server includes means for inputting a natural language sentence, means for dividing the natural language sentence into paragraphs, means for allowing the user to specify a specific character setting, means for acquiring the character's appearance and characteristics based on the character setting, means for confirming the user's emotions using an emotion engine and adjusting the character's facial expression and pose, means for calling a generative model for generating manga panels for each paragraph, means for laying out multiple manga panels in an appropriate order, means for generating the final manga, and means for returning the generated manga to the user. This enables real-time manga generation that reflects the user's emotions and also realizes a flexible system that can respond to requests for revisions.

[1608] A "natural language sentence" is a sentence written in a language that a user uses on a daily basis.

[1609] A "paragraph" is a portion of text that uses line breaks and spaces to separate meanings or content within a sentence.

[1610] "Character settings" are setting information that allows a user to specify the appearance, personality, and characteristics of a specific character.

[1611] A "generative model" is an algorithm or program that automatically generates manga panels from input natural language text.

[1612] The "emotion engine" is a system that recognizes emotions from the user's facial expressions and voice, and adjusts the character's expressions based on that information.

[1613] A "manga panel" is a combination of pictures and text that separates scenes in a manga.

[1614] A "template" is a predetermined framework or format for determining the page layout of a manga.

[1615] The "final manga" is a manga work that is compiled into a single story by appropriately laying out the multiple frames that have been generated.

[1616] A "request for correction" is a request made by a user to notify the server of any corrections or changes that the user wishes to make to a work.

[1617] This invention relates to a system that automatically generates manga based on natural language text entered by a user, and by combining it with an emotion engine, it is possible to create manga that reflects the user's emotions. The system configuration and specific program processing are described below.

[1618] Server Program Processing

[1619] The server first receives the natural language text sent by the user. It then uses natural language processing technology to divide the text into paragraphs and invokes a generative AI model to extract key information (such as characters, actions, and background) from each paragraph. If the user has specified a specific character setting, the server obtains the character's appearance and characteristics based on that information, uses an emotion engine to confirm the user's emotions, and adjusts the character's facial expressions and poses based on those emotions.

[1620] For example, when processing the paragraph "He confessed his love to her in the sunset," the generative AI model extracts the character settings of "him" and "her" and the situations of "sunset" and "confession," and draws a picture based on these. The results of the emotion engine are used to adjust the presentation of the panels. The generated panels are then arranged in order based on the provided template and laid out as pages. The resulting multi-page manga is then compiled into a single page, generating the final manga. This is then converted into image data or PDF format, ready to be sent back to the user.

[1621] Terminal Programming

[1622] The terminal provides an interface for users to input natural language text. A text area is provided here, providing a text input form. Selection boxes for character settings and a form for inputting specific character settings are also provided. An interface is also provided for recognizing the user's emotions using an emotion engine. For example, the user's emotions are detected using facial expression recognition or voice analysis. After the user enters the text, character settings, and emotional information and presses the send button, the terminal sends this data in JSON format to the server. When the generated manga is returned from the server, the terminal displays it to the user. An interactive viewer is provided to make it easy to browse the page.

[1623] User operation method

[1624] The user accesses the system's interface and inputs the text of a novel or story. If necessary, the user can also set up characters. For example, the user can specify the appearance and personality of the characters "Taro" and "Hanako." Furthermore, the user can input or detect their own emotions through the interface.

[1625] Once the text and settings are complete, the user presses the send button to send the data to the server. The generated manga is sent back to the device from the server, where the user can check it. For example, if the text is entered as "He confessed his love to her in the sunset," the user can see that the story is depicted in the generated manga scene.

[1626] If a user requests modifications to the generated cartoon, the user inputs changes to specific frames or characters and sends them back to the server. The server accepts the modification requests and provides the user with a regenerated cartoon.

[1627] The emotion engine optimizes the expression of each panel of the manga based on the user's emotions recognized by the engine. For example, if joy is recognized, the character's expression will be changed to a smile, or other appropriate effect will be applied.

[1628] For example, if a user inputs the sentence "He confessed his feelings to her in the sunset," and the emotion engine recognizes the user's emotion as "tension," the character in the generated manga frame will have a tense expression and pose. In this way, real-time manga generation that reflects the user's emotions is realized.

[1629] As described above, according to the embodiment of the present invention, a user can easily create, check, and modify a comic that reflects emotions based on natural language text.

[1630] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1631] Step 1:

[1632] Enter text and set characters from the user interface

[1633] Terminal

[1634] The user accesses the system's interface and inputs natural language text into a text area. They then specify specific character appearances and characteristics in a character configuration form. At this stage, they also input facial expressions and voice data for emotion recognition. The input text, character configuration, and emotion information are collected on the terminal.

[1635] Input: User-entered natural language text, character settings, and emotional information

[1636] Output: Input data collected on the terminal

[1637] Step 2:

[1638] Sending and receiving input data

[1639] Terminal

[1640] When the user completes the input and presses the send button, the device sends the collected text, character settings, and emotional information to the server in JSON format.

[1641] server

[1642] The server receives the data sent from the terminal and begins preparations for data processing.

[1643] Input: User input data sent from the device (JSON format)

[1644] Output: The input data received by the server

[1645] Step 3:

[1646] Dividing the text and checking the character settings

[1647] server

[1648] The server uses natural language processing technology to divide the received text into paragraphs. Each paragraph represents a different scene or event, making this an essential operation for generating effective manga. The server also checks the specified character settings and automatically selects multiple character patterns if the user has not specified them.

[1649] Input: JSON format user input data (natural language text, character settings)

[1650] Output: Separated paragraphs and confirmed character settings

[1651] Step 4:

[1652] Emotion recognition by emotion engine

[1653] server

[1654] The server runs an emotion engine, analyzes the received user facial and voice data, and recognizes emotions. For example, if the user's facial expression is "smiling," it is recognized as "joy," and if the user's voice is trembling, it is recognized as "tension." This emotional information is reflected in the character's expression when generating the manga.

[1655] Input: Received user facial expression and voice data

[1656] Output: Analyzed user emotion information

[1657] Step 5:

[1658] Manga panel generation using generative AI models

[1659] server

[1660] The server calls a generative AI model for each paragraph of the divided text. This model extracts key information from the text (characters, actions, background, etc.) and generates manga panels. In doing so, it sets the character's expressions and poses based on the results of the emotion engine. For example, in the paragraph "He confessed to her in the sunset," a sunset background and cat illustrations are generated to depict the confession scene.

[1661] Input: Divided paragraphs, sentiment information

[1662] Output: Generated comic panels

[1663] Step 6:

[1664] Panel layout and final manga generation

[1665] server

[1666] The generated panels are arranged in order based on a template and laid out as pages. The laid out pages are compiled into a manga book and the final manga data is generated. The generated manga data is converted into image data or PDF format.

[1667] Input: Multiple generated comic panels

[1668] Output: Final manga data (image data or PDF format)

[1669] Step 7:

[1670] Sending and displaying the final manga

[1671] server

[1672] The generated final manga data is sent to the terminal.

[1673] Terminal

[1674] The device receives the manga data sent from the server and displays it to the user, using an interactive viewer to allow easy page browsing.

[1675] Input: Final manga data

[1676] Output: Comic data in a format that can be viewed by users

[1677] Step 8:

[1678] Correction Request Processing

[1679] User

[1680] If the user wishes to make corrections to the generated comic, he or she inputs changes to specific frames or characters.

[1681] Terminal

[1682] The device sends a modification request in JSON format to the server.

[1683] server

[1684] The server accepts the modification request, regenerates the comic with the specified modifications reflected, and sends the regenerated comic to the terminal again.

[1685] Input: Correction Request

[1686] Output: New manga data with the corrections reflected

[1687] (Application example 2)

[1688] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1689] Systems already exist that automatically generate manga based on natural language text, but few systems can naturally express the emotions of the user in the panels. Furthermore, to improve the quality of the generated manga and user satisfaction, it is necessary to adjust the character expressions based on emotions. Furthermore, there is a demand for an interface that allows users to quickly view the generated manga and quickly request corrections. Current technology does not yet realize such advanced emotional reflection and interactive operation, so there is a need to solve these issues.

[1690] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1691] In this invention, the server includes means for inputting natural language sentences, means for dividing the natural language sentences into paragraphs, means for allowing the user to specify specific character settings, means for calling a generative model that generates comic panels for each paragraph, means for laying out multiple comic panels in an appropriate order, means for generating a final comic, means for returning the generated comic to the user, and means for analyzing the user's emotions and adjusting the expression of the comic panels based on the analysis results. This makes it possible to automatically generate comics based on the natural language sentences input by the user, and further to reflect the user's emotions in the expression of the panels.

[1692] A "natural language sentence" is a sentence that is input by a user and is composed of words that are used daily.

[1693] The "paragraph division method" is a method for automatically dividing an input natural language sentence into multiple paragraphs based on grammatical rules and punctuation marks.

[1694] "Character settings" are setting information about the character's appearance and personality specified by the user.

[1695] A "generative model" is an AI technology that extracts key information from natural language sentences and generates manga panels based on that information.

[1696] The "layout method" is a method for arranging the generated multiple comic panels in the appropriate order and formatting them as a page.

[1697] The "final manga" is the final product of the manga, with all panels properly laid out and put together.

[1698] The "returning means" is a method for sending the generated comic to the user.

[1699] "Means for analyzing emotions" refers to technology that recognizes the user's emotions and adjusts the character's expression based on the analysis results.

[1700] This invention is a system for automatically generating and providing comics that reflect emotions based on natural language sentences entered by a user. A specific embodiment of the system and its operation will be described below.

[1701] System Program

[1702] 1. Entering and dividing text

[1703] The terminal provides an interface for the user to input natural language sentences. Once the user has completed input, the sentence is sent to the server, which divides the sentence into paragraphs. This process uses natural language processing technology. For example, if the sentence "He confessed to her in the sunset" is received, the server divides the sentence into paragraphs.

[1704] 2. Applying character settings

[1705] If the user specifies a specific character setting, the server will obtain the character's appearance and characteristics based on that information. If no character setting is available, the server will present the user with multiple pre-prepared character patterns and apply the selected pattern. At this time, the device also provides an interface for recognizing the user's emotions, detecting them through facial expression recognition and voice analysis.

[1706] 3. Invoking the generative AI model

[1707] The server calls a generative AI model for each divided paragraph. This model extracts key information (characters, actions, background, etc.) from the natural language text and generates manga panels based on that information. For example, from the paragraph "He confessed to her in the sunset," the model extracts the character settings of "he" and "she" and the situations of "sunset" and "confession," and draws a picture based on those. The results of the emotion engine are used to adjust the presentation of the panels.

[1708] 4. Panel layout and final manga generation

[1709] The server arranges the generated frames in order based on the provided template and lays them out as pages. It then combines multiple comic frames into one and finally generates a comic. It then converts this comic into image data or PDF format and returns it to the user.

[1710] Hardware and software used

[1711] Hardware:

[1712] Smartphone (Android or iOS device)

[1713] software:

[1714] Flask (Python framework, backend)

[1715] Transformers (NLP library, sentiment analysis)

[1716] Remote server (running the manga generation AI model)

[1717] Specific examples

[1718] The user launches the application and enters the following text:

[1719] Example prompt statement:

[1720] He confessed to her in the sunset.

[1721] Given this sentence, the system will do the following:

[1722] Use natural language processing to divide the text into paragraphs.

[1723] The character setting specified by the user is applied, and if there is no setting, the user is allowed to select from a plurality of character patterns.

[1724] The emotion engine is used to analyze the user's emotions, and based on the results, a generative AI model is called to generate manga panels.

[1725] The generated panels are laid out in the appropriate order to generate the final cartoon, which is then returned to the user.

[1726] In this way, users can easily create original comics that reflect their own emotions based on input natural language sentences, and then review and share them.

[1727] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1728] Step 1:

[1729] The terminal provides an interface for the user to input a natural language sentence. The user inputs the natural language sentence and sends the sentence to the server. At this time, the input natural language sentence might be, for example, "He confessed to her in the sunset."

[1730] Step 2:

[1731] The server divides the received natural language sentence into paragraphs. It uses natural language processing technology for sentence analysis and automatically divides the sentence into multiple paragraphs based on grammatical rules and punctuation. For example, it divides the sentence into paragraphs like "He was in the sunset" and "He confessed to her."

[1732] Step 3:

[1733] The terminal provides an interface for the user to specify specific character settings. When the user inputs the character settings (appearance and personality), that information is sent to the server. If there is no character setting, the server presents multiple character patterns to the user and applies the selected pattern.

[1734] Step 4:

[1735] The server calls a generative AI model for each paragraph. This model extracts key information from the text (such as characters, actions, and background) and generates manga panels based on that information. For example, the model extracts the character settings of "him" and "her" and the situations such as "sunset" and "confession."

[1736] Step 5:

[1737] The server uses an emotion engine to analyze emotions based on the user's input text. Emotion analysis uses the user's emotional information sent from the device (e.g., the results of facial expression recognition and voice analysis). For example, if the emotion "tension" is recognized, the character's facial expression and pose are adjusted accordingly.

[1738] Step 6:

[1739] The server lays out the generated panels in an orderly manner, placing each panel in the appropriate order based on the provided template to form a page, and then combining multiple pages to generate the final manga.

[1740] Step 7:

[1741] The server converts the generated comic into image data or PDF format and sends it back to the terminal. The user can check the generated comic on the terminal. For example, if the user inputs the prompt sentence "He confessed to her in the sunset," a comic depicting that story will be displayed on the terminal.

[1742] Step 8:

[1743] The terminal provides an interface for accepting a request for modification of the generated cartoon from the user. When the user inputs the modification request and sends it to the server, the server accepts the modification request and returns the regenerated cartoon to the user.

[1744] In this way, by performing specific actions at each step, users can easily create original manga that reflect their emotions based on the natural language text they input, and then review and share them.

[1745] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1746] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1747] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1748] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1749] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1750] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1751] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1752] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1753] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1754] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1755] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1756] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1757] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1758] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1759] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1760] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1761] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1762] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1763] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1764] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1765] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1766] The following is further disclosed regarding the above embodiment.

[1767] (Claim 1)

[1768] a means for inputting natural language text;

[1769] means for dividing the natural language text into paragraphs;

[1770] a means by which a user can specify a particular character setting;

[1771] A means for calling a generative model that generates comic panels for each paragraph;

[1772] A means of laying out multiple comic panels in the appropriate order;

[1773] a means for generating the final cartoon;

[1774] means for returning the generated cartoon to the user;

[1775] A system including:

[1776] (Claim 2)

[1777] Further provided is a means for selecting multiple character patterns when there is no character setting.

[1778] 10. The system of claim 1.

[1779] (Claim 3)

[1780] The apparatus further includes a means for accepting a request for correction from a user regarding the generated cartoon.

[1781] 10. The system of claim 1.

[1782] "Example 1"

[1783] (Claim 1)

[1784] a means for inputting natural language text;

[1785] means for dividing the natural language text into paragraphs;

[1786] a means by which a user can specify a particular character setting;

[1787] A means for calling a generative model that generates comic panels for each paragraph;

[1788] A means to input a prompt sentence into the generative model and generate a comic frame based on key information;

[1789] A means of laying out multiple comic panels in the appropriate order;

[1790] a means for generating the final cartoon;

[1791] means for returning the generated cartoon to the user;

[1792] A system including:

[1793] (Claim 2)

[1794] Further provided is a means for selecting multiple character patterns when there is no character setting.

[1795] 10. The system of claim 1.

[1796] (Claim 3)

[1797] The apparatus further includes a means for accepting a request for correction from a user regarding the generated cartoon.

[1798] 10. The system of claim 1.

[1799] "Application Example 1"

[1800] (Claim 1)

[1801] a means for inputting natural language text;

[1802] means for dividing the natural language text into paragraphs;

[1803] a means by which a user can specify a particular character setting;

[1804] A means for calling a generative model that generates comic panels for each paragraph;

[1805] A means of laying out multiple comic panels in the appropriate order;

[1806] a means for generating the final cartoon;

[1807] means for returning the generated cartoon to the user;

[1808] a means for displaying the generated manga using a head-mounted display;

[1809] A means for a user to input natural language in a virtual space;

[1810] A system including:

[1811] (Claim 2)

[1812] Further provided is a means for selecting multiple character patterns when there is no character setting.

[1813] 10. The system of claim 1.

[1814] (Claim 3)

[1815] The apparatus further includes a means for accepting a request for correction from a user regarding the generated cartoon.

[1816] 10. The system of claim 1.

[1817] (Claim 4)

[1818] The apparatus further includes a means for providing product information based on the comic.

[1819] 10. The system of claim 1.

[1820] "Example 2: Combining Emotion Engines"

[1821] (Claim 1)

[1822] a means for inputting natural language text;

[1823] means for dividing the natural language text into paragraphs;

[1824] a means by which a user can specify a particular character setting;

[1825] A means for acquiring the appearance and characteristics of the character based on the character setting;

[1826] A means to check the user's emotions using an emotion engine and adjust the character's facial expressions and poses;

[1827] A means for calling a generative model that generates comic panels for each paragraph;

[1828] A means of laying out multiple comic panels in the appropriate order;

[1829] a means for generating the final cartoon;

[1830] means for returning the generated cartoon to the user;

[1831] A system including:

[1832] (Claim 2)

[1833] 10. The system according to claim 1, further comprising means for selecting a plurality of character patterns when no character settings are available.

[1834] (Claim 3)

[1835] 2. The system according to claim 1, further comprising means for accepting a request from a user to modify the generated cartoon.

[1836] "Application example 2 when combining emotion engines"

[1837] (Claim 1)

[1838] a means for inputting natural language text;

[1839] means for dividing the natural language text into paragraphs;

[1840] a means by which a user can specify a particular character setting;

[1841] A means for calling a generative model that generates comic panels for each paragraph;

[1842] A means of laying out multiple comic panels in the appropriate order;

[1843] a means for generating the final cartoon;

[1844] means for returning the generated cartoon to the user;

[1845] means for analyzing the user's emotions and adjusting the expression of the comic panels based on the analysis results;

[1846] A system including:

[1847] (Claim 2)

[1848] Further provided is a means for selecting multiple character patterns when there is no character setting.

[1849] 10. The system of claim 1.

[1850] (Claim 3)

[1851] The apparatus further includes a means for accepting a request for correction from a user regarding the generated cartoon.

[1852] 10. The system of claim 1. [Explanation of symbols]

[1853] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for inputting natural language text; means for dividing the natural language text into paragraphs; a means by which a user can specify a particular character setting; A means for calling a generative model that generates comic panels for each paragraph; A means of laying out multiple comic panels in the appropriate order; a means for generating the final cartoon; means for returning the generated cartoon to the user; A system including:

2. Further provided is a means for selecting multiple character patterns when there is no character setting. The system of claim 1 .

3. The apparatus further includes a means for accepting a request for correction from a user regarding the generated cartoon. The system of claim 1 .

Images