System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting text data and image data by a user using an electronic device; story generation means for generating a story based on the text data; image generation means for generating a comic image based on the story generated by the story generation means; means for transmitting the comic image generated by the image generation means to the electronic device; and means for displaying the transmitted comic image on the electronic device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's world, with the widespread use of electronic devices such as smartphones and tablets, many people can easily save their memories as digital data, but there is a problem in that it is difficult to enjoy these digital data in a memorable way. Furthermore, saving memories as manga requires specialized knowledge and skills, making it difficult for average users to easily do so. Therefore, there is a demand for a way for users to easily save and enjoy their memories in the form of manga. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following means: A means for a user to input text data and image data using an electronic device is provided. Next, a story generation means is provided for generating a story based on the text data. Furthermore, an image generation means is included for generating a comic image based on the story generated by the story generation means. By providing a means for transmitting the comic image generated by the image generation means to the electronic device and finally including a means for displaying the transmitted comic image on the electronic device, a system is realized that allows users to easily save and display memories in the form of a comic, even without specialized knowledge.

[0006] A "user" is a person who operates an electronic device to input text data and image data.

[0007] "Electronic devices" refer to devices that can input and display digital data, such as smartphones, tablets, and personal computers.

[0008] "Text data" is character information entered by the user, and mainly includes memorable episodes.

[0009] "Image data" refers to image information such as photographs and illustrations uploaded by users.

[0010] A "narrative generation means" is an algorithm or program for automatically generating a story, i.e., a narrative text, based on text data.

[0011] The "image generation means" is an algorithm or program for automatically generating comic-style images based on the story generated by the story generation means.

[0012] The "transmitting means" is a communication means for transmitting the generated comic image to an electronic device.

[0013] The "display means" is a screen display function for visually displaying the comic image received in the electronic device to the user. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention relates to a system that automatically generates and displays a manga based on memorable episodes and multiple photos entered by a user.

[0036] System Overview

[0037] This system works by linking electronic devices (smartphones, tablets, PCs, etc.) with a server. Users input text data (memorable episodes) and image data (photos) through the electronic device, and the server processes this data to generate a manga, which is then sent back to the electronic device for display.

[0038] Detailed embodiment

[0039] User operation

[0040] First, users launch the app and enter a memorable episode in the text input field. They also select relevant photos from their smartphone or tablet's camera roll and upload them to the app. When the user presses the "Generate Manga" button, the entered data is sent to the server.

[0041] Terminal handling

[0042] The device sends the text data and image data entered by the user to the server. The transmission is via internet communication, and the data is encoded in JSON format or similar. This process converts the data into a format that can be received by the server.

[0043] Server-side story and image generation

[0044] The server receives the text data and image data sent from the device. The server inputs the text data into a narrative generation algorithm (e.g., a natural language processing model) to generate an episode-based story. The generated story is then input into an image generation algorithm (e.g., a generative AI model) to generate manga-style images.

[0045] Send and Display

[0046] The server sends the generated manga image to the electronic device. The image data is transferred from the server to the device via internet communication. The device displays the received manga image to the user. The user can then view the manga on the app screen and enjoy their memories.

[0047] Specific examples

[0048] Enter your story and photos

[0049] For example, if a user inputs the following episode:

[0050] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[0051] Also, upload the following photos:

[0052] 1. Family photos

[0053] 2. Ocean photos

[0054] 3. Fireworks photos

[0055] System processing flow

[0056] 1. The device sends the above data to the server.

[0057] 2. Based on the episodes received by the server, the narrative generation algorithm generates a story containing the following scenes:

[0058] Family playing at the beach

[0059] A scene of riding a wave

[0060] Fireworks scene at night

[0061] 3. Based on the story generated by the server, an image generation algorithm draws each scene of the manga.

[0062] 4. The server sends the generated manga image to the device.

[0063] 5. The device displays the cartoon image to the user.

[0064] In this way, we have created a system that allows users without specialized knowledge or skills to easily save and enjoy their own memories as manga.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] Users launch the app, enter a memorable story into the text input field, and select related photos from their smartphone or tablet's camera roll and upload them to the app.

[0068] Step 2:

[0069] The user presses the "Generate Manga" button. This action starts the process in which the device sends the input data (text data and image data) to the server.

[0070] Step 3:

[0071] The terminal encodes the text data and image data entered by the user into JSON format and sends a POST request to the specified server URL.

[0072] Step 4:

[0073] The server receives the data sent from the terminal, decodes the JSON data, and extracts the text data and image data.

[0074] Step 5:

[0075] The server inputs the extracted text data into a narrative generation algorithm to generate a detailed story based on the episodes, for example, by using a natural language processing model to automatically create each scene of the story.

[0076] Step 6:

[0077] The server feeds the generated story into an image generation algorithm, which combines the provided image data to generate manga-style pages, choosing the right images for each scene, inserting text, and dividing the panels.

[0078] Step 7:

[0079] The server then encodes the generated manga image into JSON format again and sends it back to the device. Communication takes place over the Internet.

[0080] Step 8:

[0081] The device decodes the JSON data returned from the server and extracts the manga image.

[0082] Step 9:

[0083] The device displays the received comic image on the user interface, and the user can view the generated comic through the app and visually enjoy their memories.

[0084] The above steps realize a system that allows users to easily save and enjoy their memories in the form of comics.

[0085] Example 1

[0086] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0087] Currently, there are limited ways to save and share memories and experiences as visual content, and many users find the process technically challenging. In particular, generating visually rich content like manga from text and photos requires advanced skills and a lot of time, making it difficult for average users. For this reason, there is a demand for a system that can easily and automatically generate and display manga from episodes and photos.

[0088] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0089] In this invention, the server includes means for a user to input text data and image data using an electronic device, means for transmitting the text data and image data to the server via Internet communication, means for executing a story generation algorithm on the server based on the text data to generate a story, means for executing an image generation algorithm on the server based on the generated story to generate a comic image, means for transmitting the generated comic image to the electronic device, and means for displaying the transmitted comic image on the electronic device, thereby enabling users to easily enjoy their memories and experiences visually in comic form.

[0090] "Electronic device" means an electronic device used by a user to input and display data, such as a smartphone, tablet, or personal computer.

[0091] "Text data" refers to character information that a user inputs into an electronic device, and is primarily text data that serves as material for memorable episodes and stories.

[0092] "Image data" refers to photographs and graphics files uploaded by users from electronic devices, and is used as material for creating manga.

[0093] "Internet communication" refers to a means of communication for sending and receiving data over a network, and refers to the technology used to exchange information between a terminal and a server.

[0094] "Server" refers to the central computing device that receives and processes data submitted by users and generates the final output.

[0095] A "narrative generation algorithm" is an algorithm that takes text data as input and generates a series of stories based on it, and works using natural language processing techniques.

[0096] An "image generation algorithm" is an algorithm for creating visual comic images based on a generated story, and refers to the technology of drawing pictures using a generative AI model.

[0097] "Manga images" refer to a series of images that visually represent each scene of a story, created using a narrative generation algorithm and an image generation algorithm.

[0098] "Transmitting means" refers to a means including techniques and processes for transferring the generated data to a terminal via the Internet.

[0099] "Displaying means" refers to the technology and method for displaying the received cartoon image on the screen of the user's electronic device so that the user can visually confirm it.

[0100] MODE FOR CARRYING OUT THE INVENTION

[0101] This invention is a system that automatically generates and displays manga based on memorable episodes and multiple photos entered by the user. This system operates by linking electronic devices (smartphones, tablets, PCs, etc.) with a server.

[0102] Hardware and software used

[0103] Electronic devices: smartphones, tablets, computers

[0104] Server: Cloud server or dedicated server

[0105] Communication method: Internet communication

[0106] Narrative generation algorithms: Natural language processing models (e.g., GPT-4 (registered trademark))

[0107] Image generation algorithm: Generative AI model (e.g., DALL-E)

[0108] System Overview

[0109] User operations

[0110] First, users launch the app on their electronic device. They enter a memorable episode into the app's text input field, then select and upload related photos from their camera roll. By pressing the "Generate Manga" button, the episode and photo data are sent to the server.

[0111] Sending and Receiving Data

[0112] The device encodes the text and image data entered by the user into JSON format and sends it over the Internet to the server, which receives the data and decodes it into the appropriate format.

[0113] Narrative generation and story creation

[0114] The server then inputs the decoded text data into a narrative generation algorithm (natural language processing model) to generate an episodic story that includes scenes and events that match the content of the text entered by the user.

[0115] Image generation and cartoon creation

[0116] The generated story is then fed into an image generation algorithm (a generative AI model), which generates comic-style images based on each scene in the story, resulting in a visual representation that matches the episode.

[0117] Sending and displaying the generated comic

[0118] The server then sends the generated manga image back to the device via the Internet. The device then decodes the received manga image and displays it in a dedicated app, allowing the user to visually enjoy the generated manga.

[0119] Specific examples

[0120] For example, if a user inputs the following episode:

[0121] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[0122] The next photos you upload will include:

[0123] 1. Family photos

[0124] 2. Ocean photos

[0125] 3. Fireworks photos

[0126] Prompt Sentence Examples

[0127] "Based on the story entered by the user: 'Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and created beautiful memories.' Please create a story that includes scenes of playing at the beach with the family, riding the waves, and watching fireworks at night, and describe each scene in comic form."

[0128] This system allows users without specialized knowledge or skills to easily save and enjoy their memories as manga.

[0129] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0130] Step 1:

[0131] The user launches the app and enters a memorable episode into the text input field. Next, they select and upload a related photo from their camera roll. When they press the "Generate Manga" button, the text and image data are entered.

[0132] Input: Memories (text data), photos (image data)

[0133] Output: Text data and image data are prepared within the app.

[0134] Specific actions: Open the app, enter an episode, select a photo, press a button

[0135] Step 2:

[0136] The device encodes the text and image data entered by the user into JSON format, which is then sent to the server via internet communication.

[0137] Input: Text and image data entered by the user

[0138] Output: JSON encoded data sent to the server

[0139] Specific operations: Convert text data to JSON format, convert image data to JSON format, and send the data via the Internet

[0140] Step 3:

[0141] The server receives and decodes the JSON formatted data sent from the device. The decoded text data and image data are used in the respective processing steps.

[0142] Input: JSON format text data and image data

[0143] Output: Decoded text and image data

[0144] Specific operations: Receive JSON data, decode the data, and prepare text and image data.

[0145] Step 4:

[0146] The server inputs the decoded text data into a narrative generation algorithm (natural language processing model), which generates a story based on the input episodes.

[0147] Input: Decoded text data

[0148] Output: Generated stories

[0149] Specific operations: Input text data into the algorithm, generate a story, and temporarily save the generated story

[0150] Step 5:

[0151] The server inputs the generated story into an image generation algorithm (a generative AI model), which generates manga-style images based on each scene in the story.

[0152] Input: Generated stories

[0153] Output: Generated cartoon image

[0154] Specific operation: Input the story into the algorithm, generate a manga scene, and save the generated manga image.

[0155] Step 6:

[0156] The server then sends the generated manga image to the device via internet communication, converting it into an appropriate format before transferring it.

[0157] Input: Generated cartoon image

[0158] Output: Cartoon image sent to the device

[0159] Specific operation: Converting comic image formats and sending data

[0160] Step 7:

[0161] The device decodes the manga images received from the server and displays them in a dedicated app, allowing users to visually enjoy the generated manga.

[0162] Input: Cartoon image received from the server

[0163] Output: Cartoon image displayed in Appro

[0164] Specific operations: Decoding image data, displaying it in the app, and viewing it by the user

[0165] (Application example 1)

[0166] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0167] Conventional online shopping site review display methods separate text and images, making it difficult for users to visually understand the content of reviews. Furthermore, many users are reluctant to write detailed reviews, resulting in a tendency for the number of reviews to be small. This creates a problem of a lack of information for other users considering a purchase. This invention aims to display user reviews in comic format, making it easier for users to visually understand reviews and increasing the number of reviews submitted.

[0168] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0169] In this invention, the server includes means for a user to input text data and image data using an electronic device, story generation means for generating a story based on the text data, image generation means for generating a comic image based on the story generated by the story generation means, means for transmitting the comic image generated by the image generation means to the electronic device, means for displaying the transmitted comic image on the electronic device, and means for displaying product reviews in comic format, thereby making it possible to display the content of the review input by the user in a visually easy-to-understand manner.

[0170] "Electronic devices" refers to all devices for inputting and displaying data, such as smartphones, tablets, and personal computers.

[0171] "Text data" refers to character string information entered by the user, including descriptive content such as memorable anecdotes and product reviews.

[0172] "Image data" refers to photos and image files uploaded by users.

[0173] A "story" is a narrative that depicts a series of events generated based on text data.

[0174] "Narrative generation means" refers to algorithms and methods for generating stories based on text data.

[0175] "Manga images" refer to images that visually represent stories or episodes.

[0176] The "image generation means" refers to a means for generating images in comic format based on the story generated by the story generation means.

[0177] The "transmission means" refers to a means for transmitting the cartoon image generated by the image generation means to an electronic device.

[0178] "Display means" refers to the means by which the electronic device displays the transmitted cartoon image to the user.

[0179] "Product reviews" refer to text data that describes the user's impressions and evaluations of products they have purchased.

[0180] "Means for displaying in cartoon format" refers to means for visually displaying the generated cartoon image on a user's electronic device.

[0181] This invention relates to a system that automatically generates and displays manga based on memorable episodes, product reviews, and multiple photos entered by the user. The system consists of a front-end (smartphone app) and a back-end (server).

[0182] User operation

[0183] First, users launch the smartphone app and enter their reviews or stories in the text input field. They also select relevant photos from their smartphone or tablet's camera roll and upload them to the app. When the user presses the "Send" button, the entered text and image data are sent to the server.

[0184] Terminal handling

[0185] The device sends the text data and image data entered by the user to the server. The transmission is via internet communication, and the data is encoded in JSON format or similar. This process converts the data into a format that can be received by the server.

[0186] Server-side processing

[0187] The server receives the text data and image data sent from the device. The received text data is first input into a narrative generation algorithm, which generates a story based on episodes and reviews. The generated story is then input into an image generation algorithm, which generates images in the form of a manga. The technologies used in this process include a natural language processing model and an image caption generation model (generative AI model).

[0188] Sending generated data

[0189] The server sends the generated manga image to the device. The image data is transferred from the server to the device via internet communication. The device displays the received manga image to the user, who can then enjoy reading the manga on the app screen.

[0190] Specific examples

[0191] For example, a user might write a review like this: "This handheld vacuum cleaner is very useful. The first time I used it, it quickly cleaned my whole house." They might also upload photos of the product in use. Based on this data, the system can generate a story and cartoon-style images like the following: a scene of using the vacuum cleaner to clean the house, and a scene of the cleaned room.

[0192] Hardware and software used

[0193] The hardware used includes electronic devices such as smartphones, tablets, and PCs. The software uses React Native on the front end and Node.js and Express on the back end. The AI ​​models used include natural language processing models (e.g., GPT-3 (registered trademark)) and image caption generation models.

[0194] Example prompts to be input to the generative AI model

[0195] "Generate a unique story based on the following reviews:

[0196] "This handheld vacuum cleaner is so useful. The first time I used it, my whole house was clean in no time."

[0197] This system allows users to easily enjoy their own memories and product reviews visually in comic format.

[0198] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0199] Step 1:

[0200] A user launches a smartphone app and enters a review or anecdote in the text input field. They also select related photos and upload them to the app. This generates text data and image data. The inputs are the text data of the user's review or anecdote and the selected image data. The output is a set of text data and image data.

[0201] Step 2:

[0202] The device sends the text data and image data entered by the user to the server. Internet communication is used for transmission, and the data is encoded in JSON format or similar. Specifically, the text data and image data are combined into a single JSON object and sent to the server via an HTTP POST request. The input includes the text data and image data entered by the user. The output is the encoded data sent to the server.

[0203] Step 3:

[0204] The server receives text data and image data sent from the terminal. Specifically, it receives an HTTP request, decodes the JSON format data, and extracts the text data and image data. The input is the encoded data sent from the terminal. The output is the decoded text data and image data.

[0205] Step 4:

[0206] The server inputs the received text data into a narrative generation algorithm to generate a story based on the reviews and episodes. For example, it uses a natural language processing model (generative AI model) to generate a series of stories based on the content of the reviews. The input is the user's text data. The output is the generated story.

[0207] Step 5:

[0208] The server inputs the generated story into an image generation algorithm to generate cartoon-style images. For example, it uses an image caption generation model to generate cartoon images corresponding to each scene in the story. The inputs are the generated story and image data. The output is a cartoon-style image.

[0209] Step 6:

[0210] The server sends the generated manga image to the terminal. The transmission uses internet communication, and the image data is transferred from the server to the terminal. Specifically, the generated manga image is returned to the terminal as an HTTP response. The input is the generated manga image. The output is the manga image sent to the terminal.

[0211] Step 7:

[0212] The device displays the received manga image to the user. Specifically, the manga image is displayed on the app screen so that the user can easily view it. The input is the manga image sent from the server. The output is a manga-format image that the user can view.

[0213] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0214] The present invention is a system that generates and displays manga based on text data and image data entered by a user, and further recognizes the user's emotions and generates manga in accordance with those emotions.

[0215] System Overview

[0216] This system works by linking electronic devices (smartphones, tablets, PCs, etc.) with a server. Users input text data (memorable episodes) and image data (photos) through their electronic devices, and the server processes this data to generate a manga, which is then sent back to the electronic device for display. Furthermore, an emotion engine is used to recognize the user's emotions, and the story and manga images are adjusted accordingly.

[0217] Detailed embodiment

[0218] User operation

[0219] First, users launch the app and enter a memorable episode in the text input field. They also select relevant photos from their smartphone or tablet's camera roll and upload them to the app. When the user presses the "Generate Manga" button, the entered data is sent to the server.

[0220] Terminal handling

[0221] The device sends the text data and image data entered by the user to the server. The transmission is via internet communication, and the data is encoded in JSON format or similar. This process converts the data into a format that can be received by the server.

[0222] Server-side processing

[0223] The server receives the data sent from the device, decodes the JSON data, and extracts text and image data. It then uses an emotion engine to recognize the user's emotions from the text and image data. This emotion recognition information is used for subsequent processing.

[0224] Narrative generation and emotion regulation

[0225] The server inputs the recognized emotion information into a narrative generation algorithm to generate a detailed story based on episodes. For example, if the user expresses joy, it generates a story that emphasizes positive episodes. Each scene of the story is automatically created using a natural language processing model.

[0226] Image generation and emotion regulation

[0227] The server then uses an image generation algorithm to create a manga-style image based on the generated story and image data. The server then adjusts the image tone and color based on emotion recognition information. For example, if the emotion indicates sadness, the color tone can be darkened and a function to track facial expressions can be added.

[0228] Send and Display

[0229] The server sends the generated manga image to the electronic device. The image data is transferred from the server to the device via internet communication. The device displays the received manga image on the user interface. The user can view the manga on the app screen and enjoy their memories in a way that suits their emotions.

[0230] Specific examples

[0231] Enter your story and photos

[0232] For example, if a user inputs the following episode:

[0233] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[0234] Also, upload the following photos:

[0235] 1. Family photos

[0236] 2. Ocean photos

[0237] 3. Fireworks photos

[0238] Specific examples of emotion recognition

[0239] When a user writes an episode, the emotion engine recognizes the emotion "fun." This information is used by the story generation algorithm to represent positive scenes.

[0240] System processing flow

[0241] 1. The device sends the above data to the server.

[0242] 2. Based on the episodes received by the server, the emotion engine recognizes the user's emotions and generates a positive story.

[0243] 3. Based on the story generated by the server, an image generation algorithm generates cartoon images with a positive tone.

[0244] 4. The server sends the generated manga image to the device.

[0245] 5. The device displays the cartoon image to the user.

[0246] In this way, a system is realized that allows users to easily create and display comics of memories that reflect the user's emotions, allowing the user to enjoy memories in a more emotional way.

[0247] The processing flow will be explained below.

[0248] Step 1:

[0249] Users launch the app, enter a memorable story into the text input field, and select related photos from their smartphone or tablet's camera roll and upload them to the app.

[0250] Step 2:

[0251] The user presses the "Generate Manga" button. This action starts the process in which the device sends the input data (text data and image data) to the server.

[0252] Step 3:

[0253] The terminal encodes the text data and image data entered by the user into JSON format and sends a POST request to the specified server URL.

[0254] Step 4:

[0255] The server receives the data sent from the terminal, decodes the JSON data, and extracts the text data and image data.

[0256] Step 5:

[0257] The server inputs the extracted text and image data into an emotion engine to recognize the user's emotions. The emotion engine uses natural language processing and image analysis to extract emotional information from the episodes and photos entered by the user.

[0258] Step 6:

[0259] The server then adjusts the text data fed into the story generation algorithm based on the recognized emotion information. For example, if the user expresses the emotion "fun," the story content will be emphasized with a positive tone.

[0260] Step 7:

[0261] The server inputs the adjusted text data into a narrative generation algorithm to generate a detailed story based on the episodes, and automatically creates each scene of the story using a natural language processing model.

[0262] Step 8:

[0263] The server uses an image generation algorithm to generate a cartoon-style image based on the generated story and the recognized emotional information. The image tone and color are adjusted based on the emotional information. For example, if the emotion indicates "fun," bright colors and smiling faces are emphasized.

[0264] Step 9:

[0265] The server encodes the generated manga image in JSON format and sends it back to the device. The image data is transferred via the Internet.

[0266] Step 10:

[0267] The device decodes the JSON data returned from the server and extracts the manga image.

[0268] Step 11:

[0269] The device displays the received manga image on the user interface, which is displayed on the app screen, allowing the user to view the generated manga.

[0270] Step 12:

[0271] Users can browse the manga on the app screen and enjoy memorable episodes and the manga based on them.

[0272] The above steps realize a system that allows users to easily save and enjoy memories that match their emotions in the form of comics.

[0273] Example 2

[0274] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0275] Conventional systems simply convert text data entered by users into a story, making it difficult to generate manga that takes into account the user's emotions and image data. This limited the user experience and prevented the creation of more emotionally rich stories. Furthermore, adjustments to the tone and color of the generated manga images could not be automated, making it difficult to generate manga that reflects the user's emotions.

[0276] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0277] In this invention, the server includes means for receiving text data and image data and recognizing a user's emotions using an emotion engine, story generation means for generating a story based on the result of the emotion recognition, image generation means for generating a comic image based on the story and image data generated by the story generation means, and means for adjusting the tone and color of the comic image according to the result of the emotion recognition. This makes it possible to generate a comic according to the user's emotions, providing a more emotional user experience.

[0278] A "user" is a person who uses the system and inputs text data and image data using an electronic device.

[0279] "Electronic device" is a general term for digital devices such as smartphones, tablets, and personal computers, which provide a means for users to input text data and image data.

[0280] "Text data" refers to text information entered by the user, including descriptions of memorable episodes and the like.

[0281] "Image data" refers to data including photos and image files uploaded by users.

[0282] A "server" refers to a computer system for receiving and processing data sent by users.

[0283] "Emotion engine" refers to an algorithm or software for recognizing a user's emotions from text data and image data.

[0284] "Narrative generation means" refers to an algorithm or software for generating a story based on text data and emotion recognition results.

[0285] "Image generation means" refers to an algorithm or software for generating comic-style images based on the generated story and image data.

[0286] "Adjustment means" refers to an algorithm or software that provides the functionality to adjust the tone and color of the generated cartoon image according to the user's emotion recognition results.

[0287] "Transmission means" refers to an algorithm or software that provides the functionality for transmitting the generated cartoon image to an electronic device.

[0288] The present invention is a system that generates and displays manga based on text data and image data entered by a user. Furthermore, the system recognizes the user's emotions and generates manga in accordance with those emotions. Specific embodiments of this system are described below.

[0289] System Overview

[0290] This system works by linking electronic devices (smartphones, tablets, PCs, etc.) with a server. Users input text data (memorable episodes) and image data (photos) through their electronic devices, and the server processes this data to generate a manga, which is then sent back to the electronic device for display. Furthermore, an emotion engine is used to recognize the user's emotions, and the story and manga images are adjusted accordingly.

[0291] Hardware and software used

[0292] Hardware: smartphones, tablets, PCs, servers

[0293] Software: Dedicated app, emotion engine, natural language processing model, image generation algorithm, internet communication (HTTP)

[0294] Processing flow and specific operations

[0295] User operations

[0296] First, the user launches the dedicated app and writes a memorable episode in the text input field. For example, they might write, "Last summer, I went to the beach with my family. We had a great time riding the waves. At night, we watched fireworks and created beautiful memories." They then select relevant photos from their smartphone or tablet's camera roll and upload them to the app. For example, they could use family photos, photos of the beach, or photos of fireworks. When the user presses the "Generate Manga" button, this data is sent to the server.

[0297] Sending and Receiving Data

[0298] The device encodes the text data and image data entered by the user in JSON format and sends it to the server via Internet communication (HTTP protocol). The server decodes the received JSON data and extracts the text data and image data.

[0299] Emotion Recognition Processing

[0300] The server inputs the decoded text data into an emotion engine to recognize the user's emotion. For example, it extracts the emotion "fun" from the text. It also inputs image data into the emotion engine to analyze emotions from the images.

[0301] Story Generation

[0302] The server inputs the emotion recognition results into a story generation algorithm. If the emotion recognition identifies the emotion as "fun," the story generation algorithm generates a story that emphasizes positive episodes. Using a natural language processing model, detailed story scenes are automatically created based on the user's episodes.

[0303] Image generation and emotion regulation

[0304] The server inputs the generated story and image data uploaded by the user into an image generation algorithm, using a generative AI model. The image tone and color are adjusted based on the emotion recognition results. For example, a happy emotion is represented by a bright, colorful tone, while a sad emotion is represented by a dark tone. A comic-style image corresponding to each scene in the story is generated.

[0305] Sending and displaying the generated cartoon image

[0306] The server sends the generated manga image to the device. This is again done via Internet communication (HTTP protocol). The device decodes the received manga image and displays it to the user. The user then views the generated manga through the app. The app's user interface provides functions such as turning pages and zooming in and out of the manga.

[0307] Prompt Sentence Examples

[0308] Prompt: "When the user inputs an emotional story, generate a cartoon that reflects that particular emotion. For example, use bright and colorful cartoons for happy stories, and darker cartoons for sad stories."

[0309] This allows the system to easily create and display manga of memories that reflect the user's emotions, allowing the user to enjoy their memories in a more emotional way.

[0310] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0311] Step 1:

[0312] Users launch the app on their smartphone or tablet, enter their memorable experiences in the text input field, and select related photos from their camera roll and upload them to the app.

[0313] Input: Memorable episodes (text data) and related photos (image data)

[0314] Output: Text and image data that can be viewed within the app

[0315] Step 2:

[0316] The terminal encodes the text and image data entered by the user in JSON format.

[0317] Input: Text data and image data

[0318] Output: A data object in JSON format

[0319] Step 3:

[0320] The device sends the encoded JSON data to the server over the Internet using the HTTP protocol.

[0321] Input: A JSON formatted data object

[0322] Output: Data sent to the server

[0323] Step 4:

[0324] The server receives and decodes the JSON data sent from the device. This process extracts text data and image data. At this time, the data integrity is also checked.

[0325] Input: JSON data submitted

[0326] Output: Extracted text and image data

[0327] Step 5:

[0328] The server inputs the decoded text data into an emotion engine to recognize the user's emotion. Processing such as extracting the emotion "fun" from the text is performed. Image data is also input into the emotion engine for emotion analysis.

[0329] Input: Text data and image data

[0330] Output: User's emotional information (e.g., happy, sad)

[0331] Step 6:

[0332] The server inputs the emotion recognition results into a story generation algorithm to generate a story. If the emotion recognition identifies the emotion as "fun," the story generation algorithm generates a story that emphasizes positive episodes.

[0333] Input: Emotion information and text data

[0334] Output: Generated stories

[0335] Step 7:

[0336] The server inputs the generated story and user-uploaded image data into an image generation algorithm, which adjusts the tone and color of the image based on emotion recognition results and generates manga-style images corresponding to each scene in the story.

[0337] Input: Story, image data, emotional information

[0338] Output: Generated cartoon image

[0339] Step 8:

[0340] The server sends the generated manga image to the terminal using the Internet (HTTP protocol).

[0341] Input: Generated cartoon image

[0342] Output: Data sent to the terminal

[0343] Step 9:

[0344] The device decodes the received manga images and displays them on the app's user interface. The user can then browse the generated manga. The app also provides page-turning and zooming functions.

[0345] Input: Submitted cartoon image

[0346] Output: Comic image displayed on the app screen

[0347] In this way, the present invention allows users to easily create and display cartoons of their memories that reflect their own emotions.

[0348] (Application example 2)

[0349] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0350] Conventional manga generation systems were able to generate static manga based on text and image data entered by users, but the generated manga could not be expressed in a way that adapted to the user's emotions. This made it difficult to provide a more personalized experience that matched the user's emotions. Furthermore, it was not possible to generate a story that properly reflected the main points and emotional nuances of the episodes entered by the user. As a result, it was difficult to provide a more emotionally rich and personalized manga experience for users.

[0351] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0352] In this invention, the server includes means for a user to input text data and image data using an electronic device, emotion recognition means for recognizing the user's emotion from the text data and image data, story generation means for generating a story based on the emotion recognized by the text data and emotion recognition means, image generation means for generating a comic image based on the story generated by the story generation means, means for transmitting the comic image generated by the image generation means to the electronic device, and means for displaying the transmitted comic image on the electronic device, thereby enabling the generation and display of a more personalized comic tailored to the emotion felt by the user.

[0353] "Electronic devices" refer to devices such as smartphones, tablets, and personal computers that users use to input text data and image data.

[0354] "Text data" refers to sentence data that describes the contents of an episode or story and is input by a user using an electronic device.

[0355] "Image data" refers to photographs and image files that users upload using electronic devices.

[0356] The "emotion recognition means" is an algorithm or system that has the function of analyzing text data and image data to recognize the user's emotions.

[0357] A "narrative generator" is an algorithm or system that generates a detailed story based on text data and recognized emotions.

[0358] The "image generation means" is an algorithm or system that creates comic-style images based on the story generated by the story generation means.

[0359] The "transmission means" is a communication means for transferring the generated comic image data from the server to the electronic device.

[0360] The "display means" is a function for displaying the comic image received by the electronic device on the user interface.

[0361] "Memorable episodes" are text data relating to events and memories that the user has experienced in the past.

[0362] "Uploaded photos" are image data sent by users to the system via their electronic devices.

[0363] "Adjusting the tone and color tone based on emotion recognition" means adjusting the color and brightness of the generated cartoon image in accordance with the user's emotion information recognized by the emotion recognition means.

[0364] The purpose of this system is to generate and display a manga that corresponds to the user's emotions based on text data and image data entered by the user. A specific embodiment of the system is as follows.

[0365] User operation

[0366] First, the user launches a dedicated application on an electronic device such as a smartphone or tablet. The user writes the episode in the text input field on the application, and then selects and uploads related photos from the device's camera roll. When the user presses the "Generate Manga" button, the entered text and image data are sent from the device to the server.

[0367] Terminal handling

[0368] The device encodes the text and image data entered by the user in a format such as JSON and sends it to a server over the Internet, using a secure and efficient communication protocol for data transfer.

[0369] Server-side processing

[0370] The server receives the data sent from the device, decodes the JSON data, and separates and extracts the text data and image data. Next, it uses an emotion recognition engine to recognize the user's emotions from the text data and image data. This emotion recognition information plays an important role in subsequent processing.

[0371] Specifically, the server does the following:

[0372] 1. Emotion recognition means: An emotion recognition engine (e.g., the EmotionRecognition library) is used to recognize specific emotions (e.g., "happy," "sad," etc.) from the user's text data.

[0373] 2. Narrative generation method: Using natural language processing algorithms, a detailed story is generated from text data according to the recognized emotions.

[0374] 3. Image generation method: Based on the generated story and uploaded image data, an image generation algorithm (e.g., ComicGenerator library) is used to generate a comic-style image, adjusting the tone and color based on the emotional information.

[0375] Send and Display

[0376] The generated manga image is sent from the server to the device, and the device displays the received manga image on the user interface. The user can view the manga on the app screen and enjoy the input episode in a way that matches their emotions.

[0377] Specific examples

[0378] For example, if a user enters an episode like this:

[0379] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[0380] Also, upload the following photos:

[0381] 1. Family photos

[0382] 2. Ocean photos

[0383] 3. Fireworks photos

[0384] Based on this episode, the system recognizes the emotion of "fun" and generates a story and cartoon image with a positive tone. Specific examples of prompts used during system processing are as follows:

[0385] Prompt Sentence Examples

[0386] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[0387] In this way, a personalized comic experience can be provided according to the user's emotions.

[0388] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0389] Step 1:

[0390] The user inputs text and image data using an electronic device. The user launches a dedicated application, writes the episode in the text input field, and selects and uploads related photos. This input data is converted into JSON format and prepared for data transmission.

[0391] Input: Text data (episodes), image data (photos)

[0392] Output: JSON formatted data packet

[0393] Step 2:

[0394] The device sends a JSON-formatted data packet to the server, which then transmits the encoded data over the internet using a secure and efficient communications protocol.

[0395] Input: JSON formatted data packet

[0396] Output: Data packets sent

[0397] Step 3:

[0398] The server receives the data packets and decodes the JSON data to extract text and image data, which is then structured and prepared for processing by the emotion recognition engine.

[0399] Input: Data packets sent

[0400] Output: Text data, image data

[0401] Step 4:

[0402] The server uses an emotion recognition means to recognize the user's emotion from the text data and image data. An emotion recognition engine (e.g., EmotionRecognition library) is used to extract specific emotions (e.g., "happy" or "sad") from the text data.

[0403] Input: Text data, image data

[0404] Output: Emotion information (emotion classification)

[0405] Step 5:

[0406] The server uses a narrative generation means to generate a detailed story based on the text data and the recognized emotion information, and uses a natural language processing algorithm to generate an emotion-adapted story based on the episodes.

[0407] Input: Text data, emotion information

[0408] Output: Detailed story

[0409] Step 6:

[0410] The server uses the image generation means to generate a comic-style image based on the story generated by the story generation means and the uploaded image data, and adjusts the tone and color based on the emotional information (e.g., bright tones for happy emotions, dark tones for sad emotions).

[0411] Input: detailed story, image data, emotional information

[0412] Output: Cartoon image

[0413] Step 7:

[0414] The server encodes the generated comic image in JSON format and sends it to the device. The receiving device receives and decodes the data packet to obtain the comic image.

[0415] Input: Cartoon image

[0416] Output: JSON formatted manga image data packet

[0417] Step 8:

[0418] The terminal displays the received comic image on the user interface, and the user can view the comic generated according to their emotion through the application.

[0419] Input: JSON formatted manga image data packet

[0420] Output: Displayed cartoon image

[0421] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0422] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0423] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0424] [Second embodiment]

[0425] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0426] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0427] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0428] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0429] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0430] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0431] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0432] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0433] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0434] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0435] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0436] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0437] The present invention relates to a system that automatically generates and displays a manga based on memorable episodes and multiple photos entered by a user.

[0438] System Overview

[0439] This system works by linking electronic devices (smartphones, tablets, PCs, etc.) with a server. Users input text data (memorable episodes) and image data (photos) through the electronic device, and the server processes this data to generate a manga, which is then sent back to the electronic device for display.

[0440] Detailed embodiment

[0441] User operation

[0442] First, users launch the app and enter a memorable episode in the text input field. They also select relevant photos from their smartphone or tablet's camera roll and upload them to the app. When the user presses the "Generate Manga" button, the entered data is sent to the server.

[0443] Terminal handling

[0444] The device sends the text data and image data entered by the user to the server. The transmission is via internet communication, and the data is encoded in JSON format or similar. This process converts the data into a format that can be received by the server.

[0445] Server-side story and image generation

[0446] The server receives the text data and image data sent from the device. The server inputs the text data into a narrative generation algorithm (e.g., a natural language processing model) to generate an episode-based story. The generated story is then input into an image generation algorithm (e.g., a generative AI model) to generate manga-style images.

[0447] Send and Display

[0448] The server sends the generated manga image to the electronic device. The image data is transferred from the server to the device via internet communication. The device displays the received manga image to the user. The user can then view the manga on the app screen and enjoy their memories.

[0449] Specific examples

[0450] Enter your story and photos

[0451] For example, if a user inputs the following episode:

[0452] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[0453] Also, upload the following photos:

[0454] 1. Family photos

[0455] 2. Ocean photos

[0456] 3. Fireworks photos

[0457] System processing flow

[0458] 1. The device sends the above data to the server.

[0459] 2. Based on the episodes received by the server, the narrative generation algorithm generates a story containing the following scenes:

[0460] Family playing at the beach

[0461] A scene of riding a wave

[0462] Fireworks scene at night

[0463] 3. Based on the story generated by the server, an image generation algorithm draws each scene of the manga.

[0464] 4. The server sends the generated manga image to the device.

[0465] 5. The device displays the cartoon image to the user.

[0466] In this way, we have created a system that allows users without specialized knowledge or skills to easily save and enjoy their own memories as manga.

[0467] The processing flow will be explained below.

[0468] Step 1:

[0469] Users launch the app, enter a memorable story into the text input field, and select related photos from their smartphone or tablet's camera roll and upload them to the app.

[0470] Step 2:

[0471] The user presses the "Generate Manga" button. This action starts the process in which the device sends the input data (text data and image data) to the server.

[0472] Step 3:

[0473] The terminal encodes the text data and image data entered by the user into JSON format and sends a POST request to the specified server URL.

[0474] Step 4:

[0475] The server receives the data sent from the terminal, decodes the JSON data, and extracts the text data and image data.

[0476] Step 5:

[0477] The server inputs the extracted text data into a narrative generation algorithm to generate a detailed story based on the episodes, for example, by using a natural language processing model to automatically create each scene of the story.

[0478] Step 6:

[0479] The server feeds the generated story into an image generation algorithm, which combines the provided image data to generate manga-style pages, choosing the right images for each scene, inserting text, and dividing the panels.

[0480] Step 7:

[0481] The server then encodes the generated manga image into JSON format again and sends it back to the device. Communication takes place over the Internet.

[0482] Step 8:

[0483] The device decodes the JSON data returned from the server and extracts the manga image.

[0484] Step 9:

[0485] The device displays the received comic image on the user interface, and the user can view the generated comic through the app and visually enjoy their memories.

[0486] The above steps realize a system that allows users to easily save and enjoy their memories in the form of comics.

[0487] Example 1

[0488] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0489] Currently, there are limited ways to save and share memories and experiences as visual content, and many users find the process technically challenging. In particular, generating visually rich content like manga from text and photos requires advanced skills and a lot of time, making it difficult for average users. For this reason, there is a demand for a system that can easily and automatically generate and display manga from episodes and photos.

[0490] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0491] In this invention, the server includes means for a user to input text data and image data using an electronic device, means for transmitting the text data and image data to the server via Internet communication, means for executing a story generation algorithm on the server based on the text data to generate a story, means for executing an image generation algorithm on the server based on the generated story to generate a comic image, means for transmitting the generated comic image to the electronic device, and means for displaying the transmitted comic image on the electronic device, thereby enabling users to easily enjoy their memories and experiences visually in comic form.

[0492] "Electronic device" means an electronic device used by a user to input and display data, such as a smartphone, tablet, or personal computer.

[0493] "Text data" refers to character information that a user inputs into an electronic device, and is primarily text data that serves as material for memorable episodes and stories.

[0494] "Image data" refers to photographs and graphics files uploaded by users from electronic devices, and is used as material for creating manga.

[0495] "Internet communication" refers to a means of communication for sending and receiving data over a network, and refers to the technology used to exchange information between a terminal and a server.

[0496] "Server" refers to the central computing device that receives and processes data submitted by users and generates the final output.

[0497] A "narrative generation algorithm" is an algorithm that takes text data as input and generates a series of stories based on it, and works using natural language processing techniques.

[0498] An "image generation algorithm" is an algorithm for creating visual comic images based on a generated story, and refers to the technology of drawing pictures using a generative AI model.

[0499] "Manga images" refer to a series of images that visually represent each scene of a story, created using a narrative generation algorithm and an image generation algorithm.

[0500] "Transmitting means" refers to a means including techniques and processes for transferring the generated data to a terminal via the Internet.

[0501] "Displaying means" refers to the technology and method for displaying the received cartoon image on the screen of the user's electronic device so that the user can visually confirm it.

[0502] MODE FOR CARRYING OUT THE INVENTION

[0503] This invention is a system that automatically generates and displays manga based on memorable episodes and multiple photos entered by the user. This system operates by linking electronic devices (smartphones, tablets, PCs, etc.) with a server.

[0504] Hardware and software used

[0505] Electronic devices: smartphones, tablets, computers

[0506] Server: Cloud server or dedicated server

[0507] Communication method: Internet communication

[0508] Narrative generation algorithms: Natural language processing models (e.g., GPT-4)

[0509] Image generation algorithm: Generative AI model (e.g., DALL-E)

[0510] System Overview

[0511] User operations

[0512] First, users launch the app on their electronic device. They enter a memorable episode into the app's text input field, then select and upload related photos from their camera roll. By pressing the "Generate Manga" button, the episode and photo data are sent to the server.

[0513] Sending and Receiving Data

[0514] The device encodes the text and image data entered by the user into JSON format and sends it over the Internet to the server, which receives the data and decodes it into the appropriate format.

[0515] Narrative generation and story creation

[0516] The server then inputs the decoded text data into a narrative generation algorithm (natural language processing model) to generate an episodic story that includes scenes and events that match the content of the text entered by the user.

[0517] Image generation and cartoon creation

[0518] The generated story is then fed into an image generation algorithm (a generative AI model), which generates comic-style images based on each scene in the story, resulting in a visual representation that matches the episode.

[0519] Sending and displaying the generated comic

[0520] The server then sends the generated manga image back to the device via the Internet. The device then decodes the received manga image and displays it in a dedicated app, allowing the user to visually enjoy the generated manga.

[0521] Specific examples

[0522] For example, if a user inputs the following episode:

[0523] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[0524] The next photos you upload will include:

[0525] 1. Family photos

[0526] 2. Ocean photos

[0527] 3. Fireworks photos

[0528] Prompt Sentence Examples

[0529] "Based on the story entered by the user: 'Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and created beautiful memories.' Please create a story that includes scenes of playing at the beach with the family, riding the waves, and watching fireworks at night, and describe each scene in comic form."

[0530] This system allows users without specialized knowledge or skills to easily save and enjoy their memories as manga.

[0531] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0532] Step 1:

[0533] The user launches the app and enters a memorable episode into the text input field. Next, they select and upload a related photo from their camera roll. When they press the "Generate Manga" button, the text and image data are entered.

[0534] Input: Memories (text data), photos (image data)

[0535] Output: Text data and image data are prepared within the app.

[0536] Specific actions: Open the app, enter an episode, select a photo, press a button

[0537] Step 2:

[0538] The device encodes the text and image data entered by the user into JSON format, which is then sent to the server via internet communication.

[0539] Input: Text and image data entered by the user

[0540] Output: JSON encoded data sent to the server

[0541] Specific operations: Convert text data to JSON format, convert image data to JSON format, and send the data via the Internet

[0542] Step 3:

[0543] The server receives and decodes the JSON formatted data sent from the device. The decoded text data and image data are used in the respective processing steps.

[0544] Input: JSON format text data and image data

[0545] Output: Decoded text and image data

[0546] Specific operations: Receive JSON data, decode the data, and prepare text and image data.

[0547] Step 4:

[0548] The server inputs the decoded text data into a narrative generation algorithm (natural language processing model), which generates a story based on the input episodes.

[0549] Input: Decoded text data

[0550] Output: Generated stories

[0551] Specific operations: Input text data into the algorithm, generate a story, and temporarily save the generated story

[0552] Step 5:

[0553] The server inputs the generated story into an image generation algorithm (a generative AI model), which generates manga-style images based on each scene in the story.

[0554] Input: Generated stories

[0555] Output: Generated cartoon image

[0556] Specific operation: Input the story into the algorithm, generate a manga scene, and save the generated manga image.

[0557] Step 6:

[0558] The server then sends the generated manga image to the device via internet communication, converting it into an appropriate format before transferring it.

[0559] Input: Generated cartoon image

[0560] Output: Cartoon image sent to the device

[0561] Specific operation: Converting comic image formats and sending data

[0562] Step 7:

[0563] The device decodes the manga images received from the server and displays them in a dedicated app, allowing users to visually enjoy the generated manga.

[0564] Input: Cartoon image received from the server

[0565] Output: Cartoon image displayed in Appro

[0566] Specific operations: Decoding image data, displaying it in the app, and viewing it by the user

[0567] (Application example 1)

[0568] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0569] Conventional online shopping site review display methods separate text and images, making it difficult for users to visually understand the content of reviews. Furthermore, many users are reluctant to write detailed reviews, resulting in a tendency for the number of reviews to be small. This creates a problem of a lack of information for other users considering a purchase. This invention aims to display user reviews in comic format, making it easier for users to visually understand reviews and increasing the number of reviews submitted.

[0570] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0571] In this invention, the server includes means for a user to input text data and image data using an electronic device, story generation means for generating a story based on the text data, image generation means for generating a comic image based on the story generated by the story generation means, means for transmitting the comic image generated by the image generation means to the electronic device, means for displaying the transmitted comic image on the electronic device, and means for displaying product reviews in comic format, thereby making it possible to display the content of the review input by the user in a visually easy-to-understand manner.

[0572] "Electronic devices" refers to all devices for inputting and displaying data, such as smartphones, tablets, and personal computers.

[0573] "Text data" refers to character string information entered by the user, including descriptive content such as memorable anecdotes and product reviews.

[0574] "Image data" refers to photos and image files uploaded by users.

[0575] A "story" is a narrative that depicts a series of events generated based on text data.

[0576] "Narrative generation means" refers to algorithms and methods for generating stories based on text data.

[0577] "Manga images" refer to images that visually represent stories or episodes.

[0578] The "image generation means" refers to a means for generating images in comic format based on the story generated by the story generation means.

[0579] The "transmission means" refers to a means for transmitting the cartoon image generated by the image generation means to an electronic device.

[0580] "Display means" refers to the means by which the electronic device displays the transmitted cartoon image to the user.

[0581] "Product reviews" refer to text data that describes the user's impressions and evaluations of products they have purchased.

[0582] "Means for displaying in cartoon format" refers to means for visually displaying the generated cartoon image on a user's electronic device.

[0583] This invention relates to a system that automatically generates and displays manga based on memorable episodes, product reviews, and multiple photos entered by the user. The system consists of a front-end (smartphone app) and a back-end (server).

[0584] User operation

[0585] First, users launch the smartphone app and enter their reviews or stories in the text input field. They also select relevant photos from their smartphone or tablet's camera roll and upload them to the app. When the user presses the "Send" button, the entered text and image data are sent to the server.

[0586] Terminal handling

[0587] The device sends the text data and image data entered by the user to the server. The transmission is via internet communication, and the data is encoded in JSON format or similar. This process converts the data into a format that can be received by the server.

[0588] Server-side processing

[0589] The server receives the text data and image data sent from the device. The received text data is first input into a narrative generation algorithm, which generates a story based on episodes and reviews. The generated story is then input into an image generation algorithm, which generates images in the form of a manga. The technologies used in this process include a natural language processing model and an image caption generation model (generative AI model).

[0590] Sending generated data

[0591] The server sends the generated manga image to the device. The image data is transferred from the server to the device via internet communication. The device displays the received manga image to the user, who can then enjoy reading the manga on the app screen.

[0592] Specific examples

[0593] For example, a user might write a review like this: "This handheld vacuum cleaner is very useful. The first time I used it, it quickly cleaned my whole house." They might also upload photos of the product in use. Based on this data, the system can generate a story and cartoon-style images like the following: a scene of using the vacuum cleaner to clean the house, and a scene of the cleaned room.

[0594] Hardware and software used

[0595] The hardware used includes electronic devices such as smartphones, tablets, and PCs. The software uses React Native on the front end and Node.js and Express on the back end. The AI ​​models used include natural language processing models (e.g., GPT-3) and image caption generation models.

[0596] Example prompts to be input to the generative AI model

[0597] "Generate a unique story based on the following reviews:

[0598] "This handheld vacuum cleaner is so useful. The first time I used it, my whole house was clean in no time."

[0599] This system allows users to easily enjoy their own memories and product reviews visually in comic format.

[0600] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0601] Step 1:

[0602] A user launches a smartphone app and enters a review or anecdote in the text input field. They also select related photos and upload them to the app. This generates text data and image data. The inputs are the text data of the user's review or anecdote and the selected image data. The output is a set of text data and image data.

[0603] Step 2:

[0604] The device sends the text data and image data entered by the user to the server. Internet communication is used for transmission, and the data is encoded in JSON format or similar. Specifically, the text data and image data are combined into a single JSON object and sent to the server via an HTTP POST request. The input includes the text data and image data entered by the user. The output is the encoded data sent to the server.

[0605] Step 3:

[0606] The server receives text data and image data sent from the terminal. Specifically, it receives an HTTP request, decodes the JSON format data, and extracts the text data and image data. The input is the encoded data sent from the terminal. The output is the decoded text data and image data.

[0607] Step 4:

[0608] The server inputs the received text data into a narrative generation algorithm to generate a story based on the reviews and episodes. For example, it uses a natural language processing model (generative AI model) to generate a series of stories based on the content of the reviews. The input is the user's text data. The output is the generated story.

[0609] Step 5:

[0610] The server inputs the generated story into an image generation algorithm to generate cartoon-style images. For example, it uses an image caption generation model to generate cartoon images corresponding to each scene in the story. The inputs are the generated story and image data. The output is a cartoon-style image.

[0611] Step 6:

[0612] The server sends the generated manga image to the terminal. The transmission uses internet communication, and the image data is transferred from the server to the terminal. Specifically, the generated manga image is returned to the terminal as an HTTP response. The input is the generated manga image. The output is the manga image sent to the terminal.

[0613] Step 7:

[0614] The device displays the received manga image to the user. Specifically, the manga image is displayed on the app screen so that the user can easily view it. The input is the manga image sent from the server. The output is a manga-format image that the user can view.

[0615] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0616] The present invention is a system that generates and displays manga based on text data and image data entered by a user, and further recognizes the user's emotions and generates manga in accordance with those emotions.

[0617] System Overview

[0618] This system works by linking electronic devices (smartphones, tablets, PCs, etc.) with a server. Users input text data (memorable episodes) and image data (photos) through their electronic devices, and the server processes this data to generate a manga, which is then sent back to the electronic device for display. Furthermore, an emotion engine is used to recognize the user's emotions, and the story and manga images are adjusted accordingly.

[0619] Detailed embodiment

[0620] User operation

[0621] First, users launch the app and enter a memorable episode in the text input field. They also select relevant photos from their smartphone or tablet's camera roll and upload them to the app. When the user presses the "Generate Manga" button, the entered data is sent to the server.

[0622] Terminal handling

[0623] The device sends the text data and image data entered by the user to the server. The transmission is via internet communication, and the data is encoded in JSON format or similar. This process converts the data into a format that can be received by the server.

[0624] Server-side processing

[0625] The server receives the data sent from the device, decodes the JSON data, and extracts text and image data. It then uses an emotion engine to recognize the user's emotions from the text and image data. This emotion recognition information is used for subsequent processing.

[0626] Narrative generation and emotion regulation

[0627] The server inputs the recognized emotion information into a narrative generation algorithm to generate a detailed story based on episodes. For example, if the user expresses joy, it generates a story that emphasizes positive episodes. Each scene of the story is automatically created using a natural language processing model.

[0628] Image generation and emotion regulation

[0629] The server then uses an image generation algorithm to create a manga-style image based on the generated story and image data. The server then adjusts the image tone and color based on emotion recognition information. For example, if the emotion indicates sadness, the color tone can be darkened and a function to track facial expressions can be added.

[0630] Send and Display

[0631] The server sends the generated manga image to the electronic device. The image data is transferred from the server to the device via internet communication. The device displays the received manga image on the user interface. The user can view the manga on the app screen and enjoy their memories in a way that suits their emotions.

[0632] Specific examples

[0633] Enter your story and photos

[0634] For example, if a user inputs the following episode:

[0635] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[0636] Also, upload the following photos:

[0637] 1. Family photos

[0638] 2. Ocean photos

[0639] 3. Fireworks photos

[0640] Specific examples of emotion recognition

[0641] When a user writes an episode, the emotion engine recognizes the emotion "fun." This information is used by the story generation algorithm to represent positive scenes.

[0642] System processing flow

[0643] 1. The device sends the above data to the server.

[0644] 2. Based on the episodes received by the server, the emotion engine recognizes the user's emotions and generates a positive story.

[0645] 3. Based on the story generated by the server, an image generation algorithm generates cartoon images with a positive tone.

[0646] 4. The server sends the generated manga image to the device.

[0647] 5. The device displays the cartoon image to the user.

[0648] In this way, a system is realized that allows users to easily create and display comics of memories that reflect the user's emotions, allowing the user to enjoy memories in a more emotional way.

[0649] The processing flow will be explained below.

[0650] Step 1:

[0651] Users launch the app, enter a memorable story into the text input field, and select related photos from their smartphone or tablet's camera roll and upload them to the app.

[0652] Step 2:

[0653] The user presses the "Generate Manga" button. This action starts the process in which the device sends the input data (text data and image data) to the server.

[0654] Step 3:

[0655] The terminal encodes the text data and image data entered by the user into JSON format and sends a POST request to the specified server URL.

[0656] Step 4:

[0657] The server receives the data sent from the terminal, decodes the JSON data, and extracts the text data and image data.

[0658] Step 5:

[0659] The server inputs the extracted text and image data into an emotion engine to recognize the user's emotions. The emotion engine uses natural language processing and image analysis to extract emotional information from the episodes and photos entered by the user.

[0660] Step 6:

[0661] The server then adjusts the text data fed into the story generation algorithm based on the recognized emotion information. For example, if the user expresses the emotion "fun," the story content will be emphasized with a positive tone.

[0662] Step 7:

[0663] The server inputs the adjusted text data into a narrative generation algorithm to generate a detailed story based on the episodes, and automatically creates each scene of the story using a natural language processing model.

[0664] Step 8:

[0665] The server uses an image generation algorithm to generate a cartoon-style image based on the generated story and the recognized emotional information. The image tone and color are adjusted based on the emotional information. For example, if the emotion indicates "fun," bright colors and smiling faces are emphasized.

[0666] Step 9:

[0667] The server encodes the generated manga image in JSON format and sends it back to the device. The image data is transferred via the Internet.

[0668] Step 10:

[0669] The device decodes the JSON data returned from the server and extracts the manga image.

[0670] Step 11:

[0671] The device displays the received manga image on the user interface, which is displayed on the app screen, allowing the user to view the generated manga.

[0672] Step 12:

[0673] Users can browse the manga on the app screen and enjoy memorable episodes and the manga based on them.

[0674] The above steps realize a system that allows users to easily save and enjoy memories that match their emotions in the form of comics.

[0675] Example 2

[0676] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0677] Conventional systems simply convert text data entered by users into a story, making it difficult to generate manga that takes into account the user's emotions and image data. This limited the user experience and prevented the creation of more emotionally rich stories. Furthermore, adjustments to the tone and color of the generated manga images could not be automated, making it difficult to generate manga that reflects the user's emotions.

[0678] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0679] In this invention, the server includes means for receiving text data and image data and recognizing a user's emotions using an emotion engine, story generation means for generating a story based on the result of the emotion recognition, image generation means for generating a comic image based on the story and image data generated by the story generation means, and means for adjusting the tone and color of the comic image according to the result of the emotion recognition. This makes it possible to generate a comic according to the user's emotions, providing a more emotional user experience.

[0680] A "user" is a person who uses the system and inputs text data and image data using an electronic device.

[0681] "Electronic device" is a general term for digital devices such as smartphones, tablets, and personal computers, which provide a means for users to input text data and image data.

[0682] "Text data" refers to text information entered by the user, including descriptions of memorable episodes and the like.

[0683] "Image data" refers to data including photos and image files uploaded by users.

[0684] A "server" refers to a computer system for receiving and processing data sent by users.

[0685] "Emotion engine" refers to an algorithm or software for recognizing a user's emotions from text data and image data.

[0686] "Narrative generation means" refers to an algorithm or software for generating a story based on text data and emotion recognition results.

[0687] "Image generation means" refers to an algorithm or software for generating comic-style images based on the generated story and image data.

[0688] "Adjustment means" refers to an algorithm or software that provides the functionality to adjust the tone and color of the generated cartoon image according to the user's emotion recognition results.

[0689] "Transmission means" refers to an algorithm or software that provides the functionality for transmitting the generated cartoon image to an electronic device.

[0690] The present invention is a system that generates and displays manga based on text data and image data entered by a user. Furthermore, the system recognizes the user's emotions and generates manga in accordance with those emotions. Specific embodiments of this system are described below.

[0691] System Overview

[0692] This system works by linking electronic devices (smartphones, tablets, PCs, etc.) with a server. Users input text data (memorable episodes) and image data (photos) through their electronic devices, and the server processes this data to generate a manga, which is then sent back to the electronic device for display. Furthermore, an emotion engine is used to recognize the user's emotions, and the story and manga images are adjusted accordingly.

[0693] Hardware and software used

[0694] Hardware: smartphones, tablets, PCs, servers

[0695] Software: Dedicated app, emotion engine, natural language processing model, image generation algorithm, internet communication (HTTP)

[0696] Processing flow and specific operations

[0697] User operations

[0698] First, the user launches the dedicated app and writes a memorable episode in the text input field. For example, they might write, "Last summer, I went to the beach with my family. We had a great time riding the waves. At night, we watched fireworks and created beautiful memories." They then select relevant photos from their smartphone or tablet's camera roll and upload them to the app. For example, they could use family photos, photos of the beach, or photos of fireworks. When the user presses the "Generate Manga" button, this data is sent to the server.

[0699] Sending and Receiving Data

[0700] The device encodes the text data and image data entered by the user in JSON format and sends it to the server via Internet communication (HTTP protocol). The server decodes the received JSON data and extracts the text data and image data.

[0701] Emotion Recognition Processing

[0702] The server inputs the decoded text data into an emotion engine to recognize the user's emotion. For example, it extracts the emotion "fun" from the text. It also inputs image data into the emotion engine to analyze emotions from the images.

[0703] Story Generation

[0704] The server inputs the emotion recognition results into a story generation algorithm. If the emotion recognition identifies the emotion as "fun," the story generation algorithm generates a story that emphasizes positive episodes. Using a natural language processing model, detailed story scenes are automatically created based on the user's episodes.

[0705] Image generation and emotion regulation

[0706] The server inputs the generated story and image data uploaded by the user into an image generation algorithm, using a generative AI model. The image tone and color are adjusted based on the emotion recognition results. For example, a happy emotion is represented by a bright, colorful tone, while a sad emotion is represented by a dark tone. A comic-style image corresponding to each scene in the story is generated.

[0707] Sending and displaying the generated cartoon image

[0708] The server sends the generated manga image to the device. This is again done via Internet communication (HTTP protocol). The device decodes the received manga image and displays it to the user. The user then views the generated manga through the app. The app's user interface provides functions such as turning pages and zooming in and out of the manga.

[0709] Prompt Sentence Examples

[0710] Prompt: "When the user inputs an emotional story, generate a cartoon that reflects that particular emotion. For example, use bright and colorful cartoons for happy stories, and darker cartoons for sad stories."

[0711] This allows the system to easily create and display manga of memories that reflect the user's emotions, allowing the user to enjoy their memories in a more emotional way.

[0712] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0713] Step 1:

[0714] Users launch the app on their smartphone or tablet, enter their memorable experiences in the text input field, and select related photos from their camera roll and upload them to the app.

[0715] Input: Memorable episodes (text data) and related photos (image data)

[0716] Output: Text and image data that can be viewed within the app

[0717] Step 2:

[0718] The terminal encodes the text and image data entered by the user in JSON format.

[0719] Input: Text data and image data

[0720] Output: A data object in JSON format

[0721] Step 3:

[0722] The device sends the encoded JSON data to the server over the Internet using the HTTP protocol.

[0723] Input: A JSON formatted data object

[0724] Output: Data sent to the server

[0725] Step 4:

[0726] The server receives and decodes the JSON data sent from the device. This process extracts text data and image data. At this time, the data integrity is also checked.

[0727] Input: JSON data submitted

[0728] Output: Extracted text and image data

[0729] Step 5:

[0730] The server inputs the decoded text data into an emotion engine to recognize the user's emotion. Processing such as extracting the emotion "fun" from the text is performed. Image data is also input into the emotion engine for emotion analysis.

[0731] Input: Text data and image data

[0732] Output: User's emotional information (e.g., happy, sad)

[0733] Step 6:

[0734] The server inputs the emotion recognition results into a story generation algorithm to generate a story. If the emotion recognition identifies the emotion as "fun," the story generation algorithm generates a story that emphasizes positive episodes.

[0735] Input: Emotion information and text data

[0736] Output: Generated stories

[0737] Step 7:

[0738] The server inputs the generated story and user-uploaded image data into an image generation algorithm, which adjusts the tone and color of the image based on emotion recognition results and generates manga-style images corresponding to each scene in the story.

[0739] Input: Story, image data, emotional information

[0740] Output: Generated cartoon image

[0741] Step 8:

[0742] The server sends the generated manga image to the terminal using the Internet (HTTP protocol).

[0743] Input: Generated cartoon image

[0744] Output: Data sent to the terminal

[0745] Step 9:

[0746] The device decodes the received manga images and displays them on the app's user interface. The user can then browse the generated manga. The app also provides page-turning and zooming functions.

[0747] Input: Submitted cartoon image

[0748] Output: Comic image displayed on the app screen

[0749] In this way, the present invention allows users to easily create and display cartoons of their memories that reflect their own emotions.

[0750] (Application example 2)

[0751] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0752] Conventional manga generation systems were able to generate static manga based on text and image data entered by users, but the generated manga could not be expressed in a way that adapted to the user's emotions. This made it difficult to provide a more personalized experience that matched the user's emotions. Furthermore, it was not possible to generate a story that properly reflected the main points and emotional nuances of the episodes entered by the user. As a result, it was difficult to provide a more emotionally rich and personalized manga experience for users.

[0753] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0754] In this invention, the server includes means for a user to input text data and image data using an electronic device, emotion recognition means for recognizing the user's emotion from the text data and image data, story generation means for generating a story based on the emotion recognized by the text data and emotion recognition means, image generation means for generating a comic image based on the story generated by the story generation means, means for transmitting the comic image generated by the image generation means to the electronic device, and means for displaying the transmitted comic image on the electronic device, thereby enabling the generation and display of a more personalized comic tailored to the emotion felt by the user.

[0755] "Electronic devices" refer to devices such as smartphones, tablets, and personal computers that users use to input text data and image data.

[0756] "Text data" refers to sentence data that describes the contents of an episode or story and is input by a user using an electronic device.

[0757] "Image data" refers to photographs and image files that users upload using electronic devices.

[0758] The "emotion recognition means" is an algorithm or system that has the function of analyzing text data and image data to recognize the user's emotions.

[0759] A "narrative generator" is an algorithm or system that generates a detailed story based on text data and recognized emotions.

[0760] The "image generation means" is an algorithm or system that creates comic-style images based on the story generated by the story generation means.

[0761] The "transmission means" is a communication means for transferring the generated comic image data from the server to the electronic device.

[0762] The "display means" is a function for displaying the comic image received by the electronic device on the user interface.

[0763] "Memorable episodes" are text data relating to events and memories that the user has experienced in the past.

[0764] "Uploaded photos" are image data sent by users to the system via their electronic devices.

[0765] "Adjusting the tone and color tone based on emotion recognition" means adjusting the color and brightness of the generated cartoon image in accordance with the user's emotion information recognized by the emotion recognition means.

[0766] The purpose of this system is to generate and display a manga that corresponds to the user's emotions based on text data and image data entered by the user. A specific embodiment of the system is as follows.

[0767] User operation

[0768] First, the user launches a dedicated application on an electronic device such as a smartphone or tablet. The user writes the episode in the text input field on the application, and then selects and uploads related photos from the device's camera roll. When the user presses the "Generate Manga" button, the entered text and image data are sent from the device to the server.

[0769] Terminal handling

[0770] The device encodes the text and image data entered by the user in a format such as JSON and sends it to a server over the Internet, using a secure and efficient communication protocol for data transfer.

[0771] Server-side processing

[0772] The server receives the data sent from the device, decodes the JSON data, and separates and extracts the text data and image data. Next, it uses an emotion recognition engine to recognize the user's emotions from the text data and image data. This emotion recognition information plays an important role in subsequent processing.

[0773] Specifically, the server does the following:

[0774] 1. Emotion recognition means: An emotion recognition engine (e.g., the EmotionRecognition library) is used to recognize specific emotions (e.g., "happy," "sad," etc.) from the user's text data.

[0775] 2. Narrative generation method: Using natural language processing algorithms, a detailed story is generated from text data according to the recognized emotions.

[0776] 3. Image generation method: Based on the generated story and uploaded image data, an image generation algorithm (e.g., ComicGenerator library) is used to generate a comic-style image, adjusting the tone and color based on the emotional information.

[0777] Send and Display

[0778] The generated manga image is sent from the server to the device, and the device displays the received manga image on the user interface. The user can view the manga on the app screen and enjoy the input episode in a way that matches their emotions.

[0779] Specific examples

[0780] For example, if a user enters an episode like this:

[0781] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[0782] Also, upload the following photos:

[0783] 1. Family photos

[0784] 2. Ocean photos

[0785] 3. Fireworks photos

[0786] Based on this episode, the system recognizes the emotion of "fun" and generates a story and cartoon image with a positive tone. Specific examples of prompts used during system processing are as follows:

[0787] Prompt Sentence Examples

[0788] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[0789] In this way, a personalized comic experience can be provided according to the user's emotions.

[0790] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0791] Step 1:

[0792] The user inputs text and image data using an electronic device. The user launches a dedicated application, writes the episode in the text input field, and selects and uploads related photos. This input data is converted into JSON format and prepared for data transmission.

[0793] Input: Text data (episodes), image data (photos)

[0794] Output: JSON formatted data packet

[0795] Step 2:

[0796] The device sends a JSON-formatted data packet to the server, which then transmits the encoded data over the internet using a secure and efficient communications protocol.

[0797] Input: JSON formatted data packet

[0798] Output: Data packets sent

[0799] Step 3:

[0800] The server receives the data packets and decodes the JSON data to extract text and image data, which is then structured and prepared for processing by the emotion recognition engine.

[0801] Input: Data packets sent

[0802] Output: Text data, image data

[0803] Step 4:

[0804] The server uses an emotion recognition means to recognize the user's emotion from the text data and image data. An emotion recognition engine (e.g., EmotionRecognition library) is used to extract specific emotions (e.g., "happy" or "sad") from the text data.

[0805] Input: Text data, image data

[0806] Output: Emotion information (emotion classification)

[0807] Step 5:

[0808] The server uses a narrative generation means to generate a detailed story based on the text data and the recognized emotion information, and uses a natural language processing algorithm to generate an emotion-adapted story based on the episodes.

[0809] Input: Text data, emotion information

[0810] Output: Detailed story

[0811] Step 6:

[0812] The server uses the image generation means to generate a comic-style image based on the story generated by the story generation means and the uploaded image data, and adjusts the tone and color based on the emotional information (e.g., bright tones for happy emotions, dark tones for sad emotions).

[0813] Input: detailed story, image data, emotional information

[0814] Output: Cartoon image

[0815] Step 7:

[0816] The server encodes the generated comic image in JSON format and sends it to the device. The receiving device receives and decodes the data packet to obtain the comic image.

[0817] Input: Cartoon image

[0818] Output: JSON formatted manga image data packet

[0819] Step 8:

[0820] The terminal displays the received comic image on the user interface, and the user can view the comic generated according to their emotion through the application.

[0821] Input: JSON formatted manga image data packet

[0822] Output: Displayed cartoon image

[0823] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0824] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0825] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0826] [Third embodiment]

[0827] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0828] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0829] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0830] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0831] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0832] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0833] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0834] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0835] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0836] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0837] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0838] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0839] The present invention relates to a system that automatically generates and displays a manga based on memorable episodes and multiple photos entered by a user.

[0840] System Overview

[0841] This system works by linking electronic devices (smartphones, tablets, PCs, etc.) with a server. Users input text data (memorable episodes) and image data (photos) through the electronic device, and the server processes this data to generate a manga, which is then sent back to the electronic device for display.

[0842] Detailed embodiment

[0843] User operation

[0844] First, users launch the app and enter a memorable episode in the text input field. They also select relevant photos from their smartphone or tablet's camera roll and upload them to the app. When the user presses the "Generate Manga" button, the entered data is sent to the server.

[0845] Terminal handling

[0846] The device sends the text data and image data entered by the user to the server. The transmission is via internet communication, and the data is encoded in JSON format or similar. This process converts the data into a format that can be received by the server.

[0847] Server-side story and image generation

[0848] The server receives the text data and image data sent from the device. The server inputs the text data into a narrative generation algorithm (e.g., a natural language processing model) to generate an episode-based story. The generated story is then input into an image generation algorithm (e.g., a generative AI model) to generate manga-style images.

[0849] Send and Display

[0850] The server sends the generated manga image to the electronic device. The image data is transferred from the server to the device via internet communication. The device displays the received manga image to the user. The user can then view the manga on the app screen and enjoy their memories.

[0851] Specific examples

[0852] Enter your story and photos

[0853] For example, if a user inputs the following episode:

[0854] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[0855] Also, upload the following photos:

[0856] 1. Family photos

[0857] 2. Ocean photos

[0858] 3. Fireworks photos

[0859] System processing flow

[0860] 1. The device sends the above data to the server.

[0861] 2. Based on the episodes received by the server, the narrative generation algorithm generates a story containing the following scenes:

[0862] Family playing at the beach

[0863] A scene of riding a wave

[0864] Fireworks scene at night

[0865] 3. Based on the story generated by the server, an image generation algorithm draws each scene of the manga.

[0866] 4. The server sends the generated manga image to the device.

[0867] 5. The device displays the cartoon image to the user.

[0868] In this way, we have created a system that allows users without specialized knowledge or skills to easily save and enjoy their own memories as manga.

[0869] The processing flow will be explained below.

[0870] Step 1:

[0871] Users launch the app, enter a memorable story into the text input field, and select related photos from their smartphone or tablet's camera roll and upload them to the app.

[0872] Step 2:

[0873] The user presses the "Generate Manga" button. This action starts the process in which the device sends the input data (text data and image data) to the server.

[0874] Step 3:

[0875] The terminal encodes the text data and image data entered by the user into JSON format and sends a POST request to the specified server URL.

[0876] Step 4:

[0877] The server receives the data sent from the terminal, decodes the JSON data, and extracts the text data and image data.

[0878] Step 5:

[0879] The server inputs the extracted text data into a narrative generation algorithm to generate a detailed story based on the episodes, for example, by using a natural language processing model to automatically create each scene of the story.

[0880] Step 6:

[0881] The server feeds the generated story into an image generation algorithm, which combines the provided image data to generate manga-style pages, choosing the right images for each scene, inserting text, and dividing the panels.

[0882] Step 7:

[0883] The server then encodes the generated manga image into JSON format again and sends it back to the device. Communication takes place over the Internet.

[0884] Step 8:

[0885] The device decodes the JSON data returned from the server and extracts the manga image.

[0886] Step 9:

[0887] The device displays the received comic image on the user interface, and the user can view the generated comic through the app and visually enjoy their memories.

[0888] The above steps realize a system that allows users to easily save and enjoy their memories in the form of comics.

[0889] Example 1

[0890] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0891] Currently, there are limited ways to save and share memories and experiences as visual content, and many users find the process technically challenging. In particular, generating visually rich content like manga from text and photos requires advanced skills and a lot of time, making it difficult for average users. For this reason, there is a demand for a system that can easily and automatically generate and display manga from episodes and photos.

[0892] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0893] In this invention, the server includes means for a user to input text data and image data using an electronic device, means for transmitting the text data and image data to the server via Internet communication, means for executing a story generation algorithm on the server based on the text data to generate a story, means for executing an image generation algorithm on the server based on the generated story to generate a comic image, means for transmitting the generated comic image to the electronic device, and means for displaying the transmitted comic image on the electronic device, thereby enabling users to easily enjoy their memories and experiences visually in comic form.

[0894] "Electronic device" means an electronic device used by a user to input and display data, such as a smartphone, tablet, or personal computer.

[0895] "Text data" refers to character information that a user inputs into an electronic device, and is primarily text data that serves as material for memorable episodes and stories.

[0896] "Image data" refers to photographs and graphics files uploaded by users from electronic devices, and is used as material for creating manga.

[0897] "Internet communication" refers to a means of communication for sending and receiving data over a network, and refers to the technology used to exchange information between a terminal and a server.

[0898] "Server" refers to the central computing device that receives and processes data submitted by users and generates the final output.

[0899] A "narrative generation algorithm" is an algorithm that takes text data as input and generates a series of stories based on it, and works using natural language processing techniques.

[0900] An "image generation algorithm" is an algorithm for creating visual comic images based on a generated story, and refers to the technology of drawing pictures using a generative AI model.

[0901] "Manga images" refer to a series of images that visually represent each scene of a story, created using a narrative generation algorithm and an image generation algorithm.

[0902] "Transmitting means" refers to a means including techniques and processes for transferring the generated data to a terminal via the Internet.

[0903] "Displaying means" refers to the technology and method for displaying the received cartoon image on the screen of the user's electronic device so that the user can visually confirm it.

[0904] MODE FOR CARRYING OUT THE INVENTION

[0905] This invention is a system that automatically generates and displays manga based on memorable episodes and multiple photos entered by the user. This system operates by linking electronic devices (smartphones, tablets, PCs, etc.) with a server.

[0906] Hardware and software used

[0907] Electronic devices: smartphones, tablets, computers

[0908] Server: Cloud server or dedicated server

[0909] Communication method: Internet communication

[0910] Narrative generation algorithms: Natural language processing models (e.g., GPT-4)

[0911] Image generation algorithm: Generative AI model (e.g., DALL-E)

[0912] System Overview

[0913] User operations

[0914] First, users launch the app on their electronic device. They enter a memorable episode into the app's text input field, then select and upload related photos from their camera roll. By pressing the "Generate Manga" button, the episode and photo data are sent to the server.

[0915] Sending and Receiving Data

[0916] The device encodes the text and image data entered by the user into JSON format and sends it over the Internet to the server, which receives the data and decodes it into the appropriate format.

[0917] Narrative generation and story creation

[0918] The server then inputs the decoded text data into a narrative generation algorithm (natural language processing model) to generate an episodic story that includes scenes and events that match the content of the text entered by the user.

[0919] Image generation and cartoon creation

[0920] The generated story is then fed into an image generation algorithm (a generative AI model), which generates comic-style images based on each scene in the story, resulting in a visual representation that matches the episode.

[0921] Sending and displaying the generated comic

[0922] The server then sends the generated manga image back to the device via the Internet. The device then decodes the received manga image and displays it in a dedicated app, allowing the user to visually enjoy the generated manga.

[0923] Specific examples

[0924] For example, if a user inputs the following episode:

[0925] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[0926] The next photos you upload will include:

[0927] 1. Family photos

[0928] 2. Ocean photos

[0929] 3. Fireworks photos

[0930] Prompt Sentence Examples

[0931] "Based on the story entered by the user: 'Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and created beautiful memories.' Please create a story that includes scenes of playing at the beach with the family, riding the waves, and watching fireworks at night, and describe each scene in comic form."

[0932] This system allows users without specialized knowledge or skills to easily save and enjoy their memories as manga.

[0933] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0934] Step 1:

[0935] The user launches the app and enters a memorable episode into the text input field. Next, they select and upload a related photo from their camera roll. When they press the "Generate Manga" button, the text and image data are entered.

[0936] Input: Memories (text data), photos (image data)

[0937] Output: Text data and image data are prepared within the app.

[0938] Specific actions: Open the app, enter an episode, select a photo, press a button

[0939] Step 2:

[0940] The device encodes the text and image data entered by the user into JSON format, which is then sent to the server via internet communication.

[0941] Input: Text and image data entered by the user

[0942] Output: JSON encoded data sent to the server

[0943] Specific operations: Convert text data to JSON format, convert image data to JSON format, and send the data via the Internet

[0944] Step 3:

[0945] The server receives and decodes the JSON formatted data sent from the device. The decoded text data and image data are used in the respective processing steps.

[0946] Input: JSON format text data and image data

[0947] Output: Decoded text and image data

[0948] Specific operations: Receive JSON data, decode the data, and prepare text and image data.

[0949] Step 4:

[0950] The server inputs the decoded text data into a narrative generation algorithm (natural language processing model), which generates a story based on the input episodes.

[0951] Input: Decoded text data

[0952] Output: Generated stories

[0953] Specific operations: Input text data into the algorithm, generate a story, and temporarily save the generated story

[0954] Step 5:

[0955] The server inputs the generated story into an image generation algorithm (a generative AI model), which generates manga-style images based on each scene in the story.

[0956] Input: Generated stories

[0957] Output: Generated cartoon image

[0958] Specific operation: Input the story into the algorithm, generate a manga scene, and save the generated manga image.

[0959] Step 6:

[0960] The server then sends the generated manga image to the device via internet communication, converting it into an appropriate format before transferring it.

[0961] Input: Generated cartoon image

[0962] Output: Cartoon image sent to the device

[0963] Specific operation: Converting comic image formats and sending data

[0964] Step 7:

[0965] The device decodes the manga images received from the server and displays them in a dedicated app, allowing users to visually enjoy the generated manga.

[0966] Input: Cartoon image received from the server

[0967] Output: Cartoon image displayed in Appro

[0968] Specific operations: Decoding image data, displaying it in the app, and viewing it by the user

[0969] (Application example 1)

[0970] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0971] Conventional online shopping site review display methods separate text and images, making it difficult for users to visually understand the content of reviews. Furthermore, many users are reluctant to write detailed reviews, resulting in a tendency for the number of reviews to be small. This creates a problem of a lack of information for other users considering a purchase. This invention aims to display user reviews in comic format, making it easier for users to visually understand reviews and increasing the number of reviews submitted.

[0972] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0973] In this invention, the server includes means for a user to input text data and image data using an electronic device, story generation means for generating a story based on the text data, image generation means for generating a comic image based on the story generated by the story generation means, means for transmitting the comic image generated by the image generation means to the electronic device, means for displaying the transmitted comic image on the electronic device, and means for displaying product reviews in comic format, thereby making it possible to display the content of the review input by the user in a visually easy-to-understand manner.

[0974] "Electronic devices" refers to all devices for inputting and displaying data, such as smartphones, tablets, and personal computers.

[0975] "Text data" refers to character string information entered by the user, including descriptive content such as memorable anecdotes and product reviews.

[0976] "Image data" refers to photos and image files uploaded by users.

[0977] A "story" is a narrative that depicts a series of events generated based on text data.

[0978] "Narrative generation means" refers to algorithms and methods for generating stories based on text data.

[0979] "Manga images" refer to images that visually represent stories or episodes.

[0980] The "image generation means" refers to a means for generating images in comic format based on the story generated by the story generation means.

[0981] The "transmission means" refers to a means for transmitting the cartoon image generated by the image generation means to an electronic device.

[0982] "Display means" refers to the means by which the electronic device displays the transmitted cartoon image to the user.

[0983] "Product reviews" refer to text data that describes the user's impressions and evaluations of products they have purchased.

[0984] "Means for displaying in cartoon format" refers to means for visually displaying the generated cartoon image on a user's electronic device.

[0985] This invention relates to a system that automatically generates and displays manga based on memorable episodes, product reviews, and multiple photos entered by the user. The system consists of a front-end (smartphone app) and a back-end (server).

[0986] User operation

[0987] First, users launch the smartphone app and enter their reviews or stories in the text input field. They also select relevant photos from their smartphone or tablet's camera roll and upload them to the app. When the user presses the "Send" button, the entered text and image data are sent to the server.

[0988] Terminal handling

[0989] The device sends the text data and image data entered by the user to the server. The transmission is via internet communication, and the data is encoded in JSON format or similar. This process converts the data into a format that can be received by the server.

[0990] Server-side processing

[0991] The server receives the text data and image data sent from the device. The received text data is first input into a narrative generation algorithm, which generates a story based on episodes and reviews. The generated story is then input into an image generation algorithm, which generates images in the form of a manga. The technologies used in this process include a natural language processing model and an image caption generation model (generative AI model).

[0992] Sending generated data

[0993] The server sends the generated manga image to the device. The image data is transferred from the server to the device via internet communication. The device displays the received manga image to the user, who can then enjoy reading the manga on the app screen.

[0994] Specific examples

[0995] For example, a user might write a review like this: "This handheld vacuum cleaner is very useful. The first time I used it, it quickly cleaned my whole house." They might also upload photos of the product in use. Based on this data, the system can generate a story and cartoon-style images like the following: a scene of using the vacuum cleaner to clean the house, and a scene of the cleaned room.

[0996] Hardware and software used

[0997] The hardware used includes electronic devices such as smartphones, tablets, and PCs. The software uses React Native on the front end and Node.js and Express on the back end. The AI ​​models used include natural language processing models (e.g., GPT-3) and image caption generation models.

[0998] Example prompts to be input to the generative AI model

[0999] "Generate a unique story based on the following reviews:

[1000] "This handheld vacuum cleaner is so useful. The first time I used it, my whole house was clean in no time."

[1001] This system allows users to easily enjoy their own memories and product reviews visually in comic format.

[1002] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1003] Step 1:

[1004] A user launches a smartphone app and enters a review or anecdote in the text input field. They also select related photos and upload them to the app. This generates text data and image data. The inputs are the text data of the user's review or anecdote and the selected image data. The output is a set of text data and image data.

[1005] Step 2:

[1006] The device sends the text data and image data entered by the user to the server. Internet communication is used for transmission, and the data is encoded in JSON format or similar. Specifically, the text data and image data are combined into a single JSON object and sent to the server via an HTTP POST request. The input includes the text data and image data entered by the user. The output is the encoded data sent to the server.

[1007] Step 3:

[1008] The server receives text data and image data sent from the terminal. Specifically, it receives an HTTP request, decodes the JSON format data, and extracts the text data and image data. The input is the encoded data sent from the terminal. The output is the decoded text data and image data.

[1009] Step 4:

[1010] The server inputs the received text data into a narrative generation algorithm to generate a story based on the reviews and episodes. For example, it uses a natural language processing model (generative AI model) to generate a series of stories based on the content of the reviews. The input is the user's text data. The output is the generated story.

[1011] Step 5:

[1012] The server inputs the generated story into an image generation algorithm to generate cartoon-style images. For example, it uses an image caption generation model to generate cartoon images corresponding to each scene in the story. The inputs are the generated story and image data. The output is a cartoon-style image.

[1013] Step 6:

[1014] The server sends the generated manga image to the terminal. The transmission uses internet communication, and the image data is transferred from the server to the terminal. Specifically, the generated manga image is returned to the terminal as an HTTP response. The input is the generated manga image. The output is the manga image sent to the terminal.

[1015] Step 7:

[1016] The device displays the received manga image to the user. Specifically, the manga image is displayed on the app screen so that the user can easily view it. The input is the manga image sent from the server. The output is a manga-format image that the user can view.

[1017] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1018] The present invention is a system that generates and displays manga based on text data and image data entered by a user, and further recognizes the user's emotions and generates manga in accordance with those emotions.

[1019] System Overview

[1020] This system works by linking electronic devices (smartphones, tablets, PCs, etc.) with a server. Users input text data (memorable episodes) and image data (photos) through their electronic devices, and the server processes this data to generate a manga, which is then sent back to the electronic device for display. Furthermore, an emotion engine is used to recognize the user's emotions, and the story and manga images are adjusted accordingly.

[1021] Detailed embodiment

[1022] User operation

[1023] First, users launch the app and enter a memorable episode in the text input field. They also select relevant photos from their smartphone or tablet's camera roll and upload them to the app. When the user presses the "Generate Manga" button, the entered data is sent to the server.

[1024] Terminal handling

[1025] The device sends the text data and image data entered by the user to the server. The transmission is via internet communication, and the data is encoded in JSON format or similar. This process converts the data into a format that can be received by the server.

[1026] Server-side processing

[1027] The server receives the data sent from the device, decodes the JSON data, and extracts text and image data. It then uses an emotion engine to recognize the user's emotions from the text and image data. This emotion recognition information is used for subsequent processing.

[1028] Narrative generation and emotion regulation

[1029] The server inputs the recognized emotion information into a narrative generation algorithm to generate a detailed story based on episodes. For example, if the user expresses joy, it generates a story that emphasizes positive episodes. Each scene of the story is automatically created using a natural language processing model.

[1030] Image generation and emotion regulation

[1031] The server then uses an image generation algorithm to create a manga-style image based on the generated story and image data. The server then adjusts the image tone and color based on emotion recognition information. For example, if the emotion indicates sadness, the color tone can be darkened and a function to track facial expressions can be added.

[1032] Send and Display

[1033] The server sends the generated manga image to the electronic device. The image data is transferred from the server to the device via internet communication. The device displays the received manga image on the user interface. The user can view the manga on the app screen and enjoy their memories in a way that suits their emotions.

[1034] Specific examples

[1035] Enter your story and photos

[1036] For example, if a user inputs the following episode:

[1037] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[1038] Also, upload the following photos:

[1039] 1. Family photos

[1040] 2. Ocean photos

[1041] 3. Fireworks photos

[1042] Specific examples of emotion recognition

[1043] When a user writes an episode, the emotion engine recognizes the emotion "fun." This information is used by the story generation algorithm to represent positive scenes.

[1044] System processing flow

[1045] 1. The device sends the above data to the server.

[1046] 2. Based on the episodes received by the server, the emotion engine recognizes the user's emotions and generates a positive story.

[1047] 3. Based on the story generated by the server, an image generation algorithm generates cartoon images with a positive tone.

[1048] 4. The server sends the generated manga image to the device.

[1049] 5. The device displays the cartoon image to the user.

[1050] In this way, a system is realized that allows users to easily create and display comics of memories that reflect the user's emotions, allowing the user to enjoy memories in a more emotional way.

[1051] The processing flow will be explained below.

[1052] Step 1:

[1053] Users launch the app, enter a memorable story into the text input field, and select related photos from their smartphone or tablet's camera roll and upload them to the app.

[1054] Step 2:

[1055] The user presses the "Generate Manga" button. This action starts the process in which the device sends the input data (text data and image data) to the server.

[1056] Step 3:

[1057] The terminal encodes the text data and image data entered by the user into JSON format and sends a POST request to the specified server URL.

[1058] Step 4:

[1059] The server receives the data sent from the terminal, decodes the JSON data, and extracts the text data and image data.

[1060] Step 5:

[1061] The server inputs the extracted text and image data into an emotion engine to recognize the user's emotions. The emotion engine uses natural language processing and image analysis to extract emotional information from the episodes and photos entered by the user.

[1062] Step 6:

[1063] The server then adjusts the text data fed into the story generation algorithm based on the recognized emotion information. For example, if the user expresses the emotion "fun," the story content will be emphasized with a positive tone.

[1064] Step 7:

[1065] The server inputs the adjusted text data into a narrative generation algorithm to generate a detailed story based on the episodes, and automatically creates each scene of the story using a natural language processing model.

[1066] Step 8:

[1067] The server uses an image generation algorithm to generate a cartoon-style image based on the generated story and the recognized emotional information. The image tone and color are adjusted based on the emotional information. For example, if the emotion indicates "fun," bright colors and smiling faces are emphasized.

[1068] Step 9:

[1069] The server encodes the generated manga image in JSON format and sends it back to the device. The image data is transferred via the Internet.

[1070] Step 10:

[1071] The device decodes the JSON data returned from the server and extracts the manga image.

[1072] Step 11:

[1073] The device displays the received manga image on the user interface, which is displayed on the app screen, allowing the user to view the generated manga.

[1074] Step 12:

[1075] Users can browse the manga on the app screen and enjoy memorable episodes and the manga based on them.

[1076] The above steps realize a system that allows users to easily save and enjoy memories that match their emotions in the form of comics.

[1077] Example 2

[1078] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1079] Conventional systems simply convert text data entered by users into a story, making it difficult to generate manga that takes into account the user's emotions and image data. This limited the user experience and prevented the creation of more emotionally rich stories. Furthermore, adjustments to the tone and color of the generated manga images could not be automated, making it difficult to generate manga that reflects the user's emotions.

[1080] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1081] In this invention, the server includes means for receiving text data and image data and recognizing a user's emotions using an emotion engine, story generation means for generating a story based on the result of the emotion recognition, image generation means for generating a comic image based on the story and image data generated by the story generation means, and means for adjusting the tone and color of the comic image according to the result of the emotion recognition. This makes it possible to generate a comic according to the user's emotions, providing a more emotional user experience.

[1082] A "user" is a person who uses the system and inputs text data and image data using an electronic device.

[1083] "Electronic device" is a general term for digital devices such as smartphones, tablets, and personal computers, which provide a means for users to input text data and image data.

[1084] "Text data" refers to text information entered by the user, including descriptions of memorable episodes and the like.

[1085] "Image data" refers to data including photos and image files uploaded by users.

[1086] A "server" refers to a computer system for receiving and processing data sent by users.

[1087] "Emotion engine" refers to an algorithm or software for recognizing a user's emotions from text data and image data.

[1088] "Narrative generation means" refers to an algorithm or software for generating a story based on text data and emotion recognition results.

[1089] "Image generation means" refers to an algorithm or software for generating comic-style images based on the generated story and image data.

[1090] "Adjustment means" refers to an algorithm or software that provides the functionality to adjust the tone and color of the generated cartoon image according to the user's emotion recognition results.

[1091] "Transmission means" refers to an algorithm or software that provides the functionality for transmitting the generated cartoon image to an electronic device.

[1092] The present invention is a system that generates and displays manga based on text data and image data entered by a user. Furthermore, the system recognizes the user's emotions and generates manga in accordance with those emotions. Specific embodiments of this system are described below.

[1093] System Overview

[1094] This system works by linking electronic devices (smartphones, tablets, PCs, etc.) with a server. Users input text data (memorable episodes) and image data (photos) through their electronic devices, and the server processes this data to generate a manga, which is then sent back to the electronic device for display. Furthermore, an emotion engine is used to recognize the user's emotions, and the story and manga images are adjusted accordingly.

[1095] Hardware and software used

[1096] Hardware: smartphones, tablets, PCs, servers

[1097] Software: Dedicated app, emotion engine, natural language processing model, image generation algorithm, internet communication (HTTP)

[1098] Processing flow and specific operations

[1099] User operations

[1100] First, the user launches the dedicated app and writes a memorable episode in the text input field. For example, they might write, "Last summer, I went to the beach with my family. We had a great time riding the waves. At night, we watched fireworks and created beautiful memories." They then select relevant photos from their smartphone or tablet's camera roll and upload them to the app. For example, they could use family photos, photos of the beach, or photos of fireworks. When the user presses the "Generate Manga" button, this data is sent to the server.

[1101] Sending and Receiving Data

[1102] The device encodes the text data and image data entered by the user in JSON format and sends it to the server via Internet communication (HTTP protocol). The server decodes the received JSON data and extracts the text data and image data.

[1103] Emotion Recognition Processing

[1104] The server inputs the decoded text data into an emotion engine to recognize the user's emotion. For example, it extracts the emotion "fun" from the text. It also inputs image data into the emotion engine to analyze emotions from the images.

[1105] Story Generation

[1106] The server inputs the emotion recognition results into a story generation algorithm. If the emotion recognition identifies the emotion as "fun," the story generation algorithm generates a story that emphasizes positive episodes. Using a natural language processing model, detailed story scenes are automatically created based on the user's episodes.

[1107] Image generation and emotion regulation

[1108] The server inputs the generated story and image data uploaded by the user into an image generation algorithm, using a generative AI model. The image tone and color are adjusted based on the emotion recognition results. For example, a happy emotion is represented by a bright, colorful tone, while a sad emotion is represented by a dark tone. A comic-style image corresponding to each scene in the story is generated.

[1109] Sending and displaying the generated cartoon image

[1110] The server sends the generated manga image to the device. This is again done via Internet communication (HTTP protocol). The device decodes the received manga image and displays it to the user. The user then views the generated manga through the app. The app's user interface provides functions such as turning pages and zooming in and out of the manga.

[1111] Prompt Sentence Examples

[1112] Prompt: "When the user inputs an emotional story, generate a cartoon that reflects that particular emotion. For example, use bright and colorful cartoons for happy stories, and darker cartoons for sad stories."

[1113] This allows the system to easily create and display manga of memories that reflect the user's emotions, allowing the user to enjoy their memories in a more emotional way.

[1114] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1115] Step 1:

[1116] Users launch the app on their smartphone or tablet, enter their memorable experiences in the text input field, and select related photos from their camera roll and upload them to the app.

[1117] Input: Memorable episodes (text data) and related photos (image data)

[1118] Output: Text and image data that can be viewed within the app

[1119] Step 2:

[1120] The terminal encodes the text and image data entered by the user in JSON format.

[1121] Input: Text data and image data

[1122] Output: A data object in JSON format

[1123] Step 3:

[1124] The device sends the encoded JSON data to the server over the Internet using the HTTP protocol.

[1125] Input: A JSON formatted data object

[1126] Output: Data sent to the server

[1127] Step 4:

[1128] The server receives and decodes the JSON data sent from the device. This process extracts text data and image data. At this time, the data integrity is also checked.

[1129] Input: JSON data submitted

[1130] Output: Extracted text and image data

[1131] Step 5:

[1132] The server inputs the decoded text data into an emotion engine to recognize the user's emotion. Processing such as extracting the emotion "fun" from the text is performed. Image data is also input into the emotion engine for emotion analysis.

[1133] Input: Text data and image data

[1134] Output: User's emotional information (e.g., happy, sad)

[1135] Step 6:

[1136] The server inputs the emotion recognition results into a story generation algorithm to generate a story. If the emotion recognition identifies the emotion as "fun," the story generation algorithm generates a story that emphasizes positive episodes.

[1137] Input: Emotion information and text data

[1138] Output: Generated stories

[1139] Step 7:

[1140] The server inputs the generated story and user-uploaded image data into an image generation algorithm, which adjusts the tone and color of the image based on emotion recognition results and generates manga-style images corresponding to each scene in the story.

[1141] Input: Story, image data, emotional information

[1142] Output: Generated cartoon image

[1143] Step 8:

[1144] The server sends the generated manga image to the terminal using the Internet (HTTP protocol).

[1145] Input: Generated cartoon image

[1146] Output: Data sent to the terminal

[1147] Step 9:

[1148] The device decodes the received manga images and displays them on the app's user interface. The user can then browse the generated manga. The app also provides page-turning and zooming functions.

[1149] Input: Submitted cartoon image

[1150] Output: Comic image displayed on the app screen

[1151] In this way, the present invention allows users to easily create and display cartoons of their memories that reflect their own emotions.

[1152] (Application example 2)

[1153] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1154] Conventional manga generation systems were able to generate static manga based on text and image data entered by users, but the generated manga could not be expressed in a way that adapted to the user's emotions. This made it difficult to provide a more personalized experience that matched the user's emotions. Furthermore, it was not possible to generate a story that properly reflected the main points and emotional nuances of the episodes entered by the user. As a result, it was difficult to provide a more emotionally rich and personalized manga experience for users.

[1155] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1156] In this invention, the server includes means for a user to input text data and image data using an electronic device, emotion recognition means for recognizing the user's emotion from the text data and image data, story generation means for generating a story based on the emotion recognized by the text data and emotion recognition means, image generation means for generating a comic image based on the story generated by the story generation means, means for transmitting the comic image generated by the image generation means to the electronic device, and means for displaying the transmitted comic image on the electronic device, thereby enabling the generation and display of a more personalized comic tailored to the emotion felt by the user.

[1157] "Electronic devices" refer to devices such as smartphones, tablets, and personal computers that users use to input text data and image data.

[1158] "Text data" refers to sentence data that describes the contents of an episode or story and is input by a user using an electronic device.

[1159] "Image data" refers to photographs and image files that users upload using electronic devices.

[1160] The "emotion recognition means" is an algorithm or system that has the function of analyzing text data and image data to recognize the user's emotions.

[1161] A "narrative generator" is an algorithm or system that generates a detailed story based on text data and recognized emotions.

[1162] The "image generation means" is an algorithm or system that creates comic-style images based on the story generated by the story generation means.

[1163] The "transmission means" is a communication means for transferring the generated comic image data from the server to the electronic device.

[1164] The "display means" is a function for displaying the comic image received by the electronic device on the user interface.

[1165] "Memorable episodes" are text data relating to events and memories that the user has experienced in the past.

[1166] "Uploaded photos" are image data sent by users to the system via their electronic devices.

[1167] "Adjusting the tone and color tone based on emotion recognition" means adjusting the color and brightness of the generated cartoon image in accordance with the user's emotion information recognized by the emotion recognition means.

[1168] The purpose of this system is to generate and display a manga that corresponds to the user's emotions based on text data and image data entered by the user. A specific embodiment of the system is as follows.

[1169] User operation

[1170] First, the user launches a dedicated application on an electronic device such as a smartphone or tablet. The user writes the episode in the text input field on the application, and then selects and uploads related photos from the device's camera roll. When the user presses the "Generate Manga" button, the entered text and image data are sent from the device to the server.

[1171] Terminal handling

[1172] The device encodes the text and image data entered by the user in a format such as JSON and sends it to a server over the Internet, using a secure and efficient communication protocol for data transfer.

[1173] Server-side processing

[1174] The server receives the data sent from the device, decodes the JSON data, and separates and extracts the text data and image data. Next, it uses an emotion recognition engine to recognize the user's emotions from the text data and image data. This emotion recognition information plays an important role in subsequent processing.

[1175] Specifically, the server does the following:

[1176] 1. Emotion recognition means: An emotion recognition engine (e.g., the EmotionRecognition library) is used to recognize specific emotions (e.g., "happy," "sad," etc.) from the user's text data.

[1177] 2. Narrative generation method: Using natural language processing algorithms, a detailed story is generated from text data according to the recognized emotions.

[1178] 3. Image generation method: Based on the generated story and uploaded image data, an image generation algorithm (e.g., ComicGenerator library) is used to generate a comic-style image, adjusting the tone and color based on the emotional information.

[1179] Send and Display

[1180] The generated manga image is sent from the server to the device, and the device displays the received manga image on the user interface. The user can view the manga on the app screen and enjoy the input episode in a way that matches their emotions.

[1181] Specific examples

[1182] For example, if a user enters an episode like this:

[1183] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[1184] Also, upload the following photos:

[1185] 1. Family photos

[1186] 2. Ocean photos

[1187] 3. Fireworks photos

[1188] Based on this episode, the system recognizes the emotion of "fun" and generates a story and cartoon image with a positive tone. Specific examples of prompts used during system processing are as follows:

[1189] Prompt Sentence Examples

[1190] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[1191] In this way, a personalized comic experience can be provided according to the user's emotions.

[1192] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1193] Step 1:

[1194] The user inputs text and image data using an electronic device. The user launches a dedicated application, writes the episode in the text input field, and selects and uploads related photos. This input data is converted into JSON format and prepared for data transmission.

[1195] Input: Text data (episodes), image data (photos)

[1196] Output: JSON formatted data packet

[1197] Step 2:

[1198] The device sends a JSON-formatted data packet to the server, which then transmits the encoded data over the internet using a secure and efficient communications protocol.

[1199] Input: JSON formatted data packet

[1200] Output: Data packets sent

[1201] Step 3:

[1202] The server receives the data packets and decodes the JSON data to extract text and image data, which is then structured and prepared for processing by the emotion recognition engine.

[1203] Input: Data packets sent

[1204] Output: Text data, image data

[1205] Step 4:

[1206] The server uses an emotion recognition means to recognize the user's emotion from the text data and image data. An emotion recognition engine (e.g., EmotionRecognition library) is used to extract specific emotions (e.g., "happy" or "sad") from the text data.

[1207] Input: Text data, image data

[1208] Output: Emotion information (emotion classification)

[1209] Step 5:

[1210] The server uses a narrative generation means to generate a detailed story based on the text data and the recognized emotion information, and uses a natural language processing algorithm to generate an emotion-adapted story based on the episodes.

[1211] Input: Text data, emotion information

[1212] Output: Detailed story

[1213] Step 6:

[1214] The server uses the image generation means to generate a comic-style image based on the story generated by the story generation means and the uploaded image data, and adjusts the tone and color based on the emotional information (e.g., bright tones for happy emotions, dark tones for sad emotions).

[1215] Input: detailed story, image data, emotional information

[1216] Output: Cartoon image

[1217] Step 7:

[1218] The server encodes the generated comic image in JSON format and sends it to the device. The receiving device receives and decodes the data packet to obtain the comic image.

[1219] Input: Cartoon image

[1220] Output: JSON formatted manga image data packet

[1221] Step 8:

[1222] The terminal displays the received comic image on the user interface, and the user can view the comic generated according to their emotion through the application.

[1223] Input: JSON formatted manga image data packet

[1224] Output: Displayed cartoon image

[1225] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1226] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1227] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1228] [Fourth embodiment]

[1229] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1230] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1231] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1232] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1233] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1234] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1235] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1236] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1237] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1238] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1239] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1240] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1241] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1242] The present invention relates to a system that automatically generates and displays a manga based on memorable episodes and multiple photos entered by a user.

[1243] System Overview

[1244] This system works by linking electronic devices (smartphones, tablets, PCs, etc.) with a server. Users input text data (memorable episodes) and image data (photos) through the electronic device, and the server processes this data to generate a manga, which is then sent back to the electronic device for display.

[1245] Detailed embodiment

[1246] User operation

[1247] First, users launch the app and enter a memorable episode in the text input field. They also select relevant photos from their smartphone or tablet's camera roll and upload them to the app. When the user presses the "Generate Manga" button, the entered data is sent to the server.

[1248] Terminal handling

[1249] The device sends the text data and image data entered by the user to the server. The transmission is via internet communication, and the data is encoded in JSON format or similar. This process converts the data into a format that can be received by the server.

[1250] Server-side story and image generation

[1251] The server receives the text data and image data sent from the device. The server inputs the text data into a narrative generation algorithm (e.g., a natural language processing model) to generate an episode-based story. The generated story is then input into an image generation algorithm (e.g., a generative AI model) to generate manga-style images.

[1252] Send and Display

[1253] The server sends the generated manga image to the electronic device. The image data is transferred from the server to the device via internet communication. The device displays the received manga image to the user. The user can then view the manga on the app screen and enjoy their memories.

[1254] Specific examples

[1255] Enter your story and photos

[1256] For example, if a user inputs the following episode:

[1257] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[1258] Also, upload the following photos:

[1259] 1. Family photos

[1260] 2. Ocean photos

[1261] 3. Fireworks photos

[1262] System processing flow

[1263] 1. The device sends the above data to the server.

[1264] 2. Based on the episodes received by the server, the narrative generation algorithm generates a story containing the following scenes:

[1265] Family playing at the beach

[1266] A scene of riding a wave

[1267] Fireworks scene at night

[1268] 3. Based on the story generated by the server, an image generation algorithm draws each scene of the manga.

[1269] 4. The server sends the generated manga image to the device.

[1270] 5. The device displays the cartoon image to the user.

[1271] In this way, we have created a system that allows users without specialized knowledge or skills to easily save and enjoy their own memories as manga.

[1272] The processing flow will be explained below.

[1273] Step 1:

[1274] Users launch the app, enter a memorable story into the text input field, and select related photos from their smartphone or tablet's camera roll and upload them to the app.

[1275] Step 2:

[1276] The user presses the "Generate Manga" button. This action starts the process in which the device sends the input data (text data and image data) to the server.

[1277] Step 3:

[1278] The terminal encodes the text data and image data entered by the user into JSON format and sends a POST request to the specified server URL.

[1279] Step 4:

[1280] The server receives the data sent from the terminal, decodes the JSON data, and extracts the text data and image data.

[1281] Step 5:

[1282] The server inputs the extracted text data into a narrative generation algorithm to generate a detailed story based on the episodes, for example, by using a natural language processing model to automatically create each scene of the story.

[1283] Step 6:

[1284] The server feeds the generated story into an image generation algorithm, which combines the provided image data to generate manga-style pages, choosing the right images for each scene, inserting text, and dividing the panels.

[1285] Step 7:

[1286] The server then encodes the generated manga image into JSON format again and sends it back to the device. Communication takes place over the Internet.

[1287] Step 8:

[1288] The device decodes the JSON data returned from the server and extracts the manga image.

[1289] Step 9:

[1290] The device displays the received comic image on the user interface, and the user can view the generated comic through the app and visually enjoy their memories.

[1291] The above steps realize a system that allows users to easily save and enjoy their memories in the form of comics.

[1292] Example 1

[1293] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1294] Currently, there are limited ways to save and share memories and experiences as visual content, and many users find the process technically challenging. In particular, generating visually rich content like manga from text and photos requires advanced skills and a lot of time, making it difficult for average users. For this reason, there is a demand for a system that can easily and automatically generate and display manga from episodes and photos.

[1295] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1296] In this invention, the server includes means for a user to input text data and image data using an electronic device, means for transmitting the text data and image data to the server via Internet communication, means for executing a story generation algorithm on the server based on the text data to generate a story, means for executing an image generation algorithm on the server based on the generated story to generate a comic image, means for transmitting the generated comic image to the electronic device, and means for displaying the transmitted comic image on the electronic device, thereby enabling users to easily enjoy their memories and experiences visually in comic form.

[1297] "Electronic device" means an electronic device used by a user to input and display data, such as a smartphone, tablet, or personal computer.

[1298] "Text data" refers to character information that a user inputs into an electronic device, and is primarily text data that serves as material for memorable episodes and stories.

[1299] "Image data" refers to photographs and graphics files uploaded by users from electronic devices, and is used as material for creating manga.

[1300] "Internet communication" refers to a means of communication for sending and receiving data over a network, and refers to the technology used to exchange information between a terminal and a server.

[1301] "Server" refers to the central computing device that receives and processes data submitted by users and generates the final output.

[1302] A "narrative generation algorithm" is an algorithm that takes text data as input and generates a series of stories based on it, and works using natural language processing techniques.

[1303] An "image generation algorithm" is an algorithm for creating visual comic images based on a generated story, and refers to the technology of drawing pictures using a generative AI model.

[1304] "Manga images" refer to a series of images that visually represent each scene of a story, created using a narrative generation algorithm and an image generation algorithm.

[1305] "Transmitting means" refers to a means including techniques and processes for transferring the generated data to a terminal via the Internet.

[1306] "Displaying means" refers to the technology and method for displaying the received cartoon image on the screen of the user's electronic device so that the user can visually confirm it.

[1307] MODE FOR CARRYING OUT THE INVENTION

[1308] This invention is a system that automatically generates and displays manga based on memorable episodes and multiple photos entered by the user. This system operates by linking electronic devices (smartphones, tablets, PCs, etc.) with a server.

[1309] Hardware and software used

[1310] Electronic devices: smartphones, tablets, computers

[1311] Server: Cloud server or dedicated server

[1312] Communication method: Internet communication

[1313] Narrative generation algorithms: Natural language processing models (e.g., GPT-4)

[1314] Image generation algorithm: Generative AI model (e.g., DALL-E)

[1315] System Overview

[1316] User operations

[1317] First, users launch the app on their electronic device. They enter a memorable episode into the app's text input field, then select and upload related photos from their camera roll. By pressing the "Generate Manga" button, the episode and photo data are sent to the server.

[1318] Sending and Receiving Data

[1319] The device encodes the text and image data entered by the user into JSON format and sends it over the Internet to the server, which receives the data and decodes it into the appropriate format.

[1320] Narrative generation and story creation

[1321] The server then inputs the decoded text data into a narrative generation algorithm (natural language processing model) to generate an episodic story that includes scenes and events that match the content of the text entered by the user.

[1322] Image generation and cartoon creation

[1323] The generated story is then fed into an image generation algorithm (a generative AI model), which generates comic-style images based on each scene in the story, resulting in a visual representation that matches the episode.

[1324] Sending and displaying the generated comic

[1325] The server then sends the generated manga image back to the device via the Internet. The device then decodes the received manga image and displays it in a dedicated app, allowing the user to visually enjoy the generated manga.

[1326] Specific examples

[1327] For example, if a user inputs the following episode:

[1328] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[1329] The next photos you upload will include:

[1330] 1. Family photos

[1331] 2. Ocean photos

[1332] 3. Fireworks photos

[1333] Prompt Sentence Examples

[1334] "Based on the story entered by the user: 'Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and created beautiful memories.' Please create a story that includes scenes of playing at the beach with the family, riding the waves, and watching fireworks at night, and describe each scene in comic form."

[1335] This system allows users without specialized knowledge or skills to easily save and enjoy their memories as manga.

[1336] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1337] Step 1:

[1338] The user launches the app and enters a memorable episode into the text input field. Next, they select and upload a related photo from their camera roll. When they press the "Generate Manga" button, the text and image data are entered.

[1339] Input: Memories (text data), photos (image data)

[1340] Output: Text data and image data are prepared within the app.

[1341] Specific actions: Open the app, enter an episode, select a photo, press a button

[1342] Step 2:

[1343] The device encodes the text and image data entered by the user into JSON format, which is then sent to the server via internet communication.

[1344] Input: Text and image data entered by the user

[1345] Output: JSON encoded data sent to the server

[1346] Specific operations: Convert text data to JSON format, convert image data to JSON format, and send the data via the Internet

[1347] Step 3:

[1348] The server receives and decodes the JSON formatted data sent from the device. The decoded text data and image data are used in the respective processing steps.

[1349] Input: JSON format text data and image data

[1350] Output: Decoded text and image data

[1351] Specific operations: Receive JSON data, decode the data, and prepare text and image data.

[1352] Step 4:

[1353] The server inputs the decoded text data into a narrative generation algorithm (natural language processing model), which generates a story based on the input episodes.

[1354] Input: Decoded text data

[1355] Output: Generated stories

[1356] Specific operations: Input text data into the algorithm, generate a story, and temporarily save the generated story

[1357] Step 5:

[1358] The server inputs the generated story into an image generation algorithm (a generative AI model), which generates manga-style images based on each scene in the story.

[1359] Input: Generated stories

[1360] Output: Generated cartoon image

[1361] Specific operation: Input the story into the algorithm, generate a manga scene, and save the generated manga image.

[1362] Step 6:

[1363] The server then sends the generated manga image to the device via internet communication, converting it into an appropriate format before transferring it.

[1364] Input: Generated cartoon image

[1365] Output: Cartoon image sent to the device

[1366] Specific operation: Converting comic image formats and sending data

[1367] Step 7:

[1368] The device decodes the manga images received from the server and displays them in a dedicated app, allowing users to visually enjoy the generated manga.

[1369] Input: Cartoon image received from the server

[1370] Output: Cartoon image displayed in Appro

[1371] Specific operations: Decoding image data, displaying it in the app, and viewing it by the user

[1372] (Application example 1)

[1373] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1374] Conventional online shopping site review display methods separate text and images, making it difficult for users to visually understand the content of reviews. Furthermore, many users are reluctant to write detailed reviews, resulting in a tendency for the number of reviews to be small. This creates a problem of a lack of information for other users considering a purchase. This invention aims to display user reviews in comic format, making it easier for users to visually understand reviews and increasing the number of reviews submitted.

[1375] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1376] In this invention, the server includes means for a user to input text data and image data using an electronic device, story generation means for generating a story based on the text data, image generation means for generating a comic image based on the story generated by the story generation means, means for transmitting the comic image generated by the image generation means to the electronic device, means for displaying the transmitted comic image on the electronic device, and means for displaying product reviews in comic format, thereby making it possible to display the content of the review input by the user in a visually easy-to-understand manner.

[1377] "Electronic devices" refers to all devices for inputting and displaying data, such as smartphones, tablets, and personal computers.

[1378] "Text data" refers to character string information entered by the user, including descriptive content such as memorable anecdotes and product reviews.

[1379] "Image data" refers to photos and image files uploaded by users.

[1380] A "story" is a narrative that depicts a series of events generated based on text data.

[1381] "Narrative generation means" refers to algorithms and methods for generating stories based on text data.

[1382] "Manga images" refer to images that visually represent stories or episodes.

[1383] The "image generation means" refers to a means for generating images in comic format based on the story generated by the story generation means.

[1384] The "transmission means" refers to a means for transmitting the cartoon image generated by the image generation means to an electronic device.

[1385] "Display means" refers to the means by which the electronic device displays the transmitted cartoon image to the user.

[1386] "Product reviews" refer to text data that describes the user's impressions and evaluations of products they have purchased.

[1387] "Means for displaying in cartoon format" refers to means for visually displaying the generated cartoon image on a user's electronic device.

[1388] This invention relates to a system that automatically generates and displays manga based on memorable episodes, product reviews, and multiple photos entered by the user. The system consists of a front-end (smartphone app) and a back-end (server).

[1389] User operation

[1390] First, users launch the smartphone app and enter their reviews or stories in the text input field. They also select relevant photos from their smartphone or tablet's camera roll and upload them to the app. When the user presses the "Send" button, the entered text and image data are sent to the server.

[1391] Terminal handling

[1392] The device sends the text data and image data entered by the user to the server. The transmission is via internet communication, and the data is encoded in JSON format or similar. This process converts the data into a format that can be received by the server.

[1393] Server-side processing

[1394] The server receives the text data and image data sent from the device. The received text data is first input into a narrative generation algorithm, which generates a story based on episodes and reviews. The generated story is then input into an image generation algorithm, which generates images in the form of a manga. The technologies used in this process include a natural language processing model and an image caption generation model (generative AI model).

[1395] Sending generated data

[1396] The server sends the generated manga image to the device. The image data is transferred from the server to the device via internet communication. The device displays the received manga image to the user, who can then enjoy reading the manga on the app screen.

[1397] Specific examples

[1398] For example, a user might write a review like this: "This handheld vacuum cleaner is very useful. The first time I used it, it quickly cleaned my whole house." They might also upload photos of the product in use. Based on this data, the system can generate a story and cartoon-style images like the following: a scene of using the vacuum cleaner to clean the house, and a scene of the cleaned room.

[1399] Hardware and software used

[1400] The hardware used includes electronic devices such as smartphones, tablets, and PCs. The software uses React Native on the front end and Node.js and Express on the back end. The AI ​​models used include natural language processing models (e.g., GPT-3) and image caption generation models.

[1401] Example prompts to be input to the generative AI model

[1402] "Generate a unique story based on the following reviews:

[1403] "This handheld vacuum cleaner is so useful. The first time I used it, my whole house was clean in no time."

[1404] This system allows users to easily enjoy their own memories and product reviews visually in comic format.

[1405] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1406] Step 1:

[1407] A user launches a smartphone app and enters a review or anecdote in the text input field. They also select related photos and upload them to the app. This generates text data and image data. The inputs are the text data of the user's review or anecdote and the selected image data. The output is a set of text data and image data.

[1408] Step 2:

[1409] The device sends the text data and image data entered by the user to the server. Internet communication is used for transmission, and the data is encoded in JSON format or similar. Specifically, the text data and image data are combined into a single JSON object and sent to the server via an HTTP POST request. The input includes the text data and image data entered by the user. The output is the encoded data sent to the server.

[1410] Step 3:

[1411] The server receives text data and image data sent from the terminal. Specifically, it receives an HTTP request, decodes the JSON format data, and extracts the text data and image data. The input is the encoded data sent from the terminal. The output is the decoded text data and image data.

[1412] Step 4:

[1413] The server inputs the received text data into a narrative generation algorithm to generate a story based on the reviews and episodes. For example, it uses a natural language processing model (generative AI model) to generate a series of stories based on the content of the reviews. The input is the user's text data. The output is the generated story.

[1414] Step 5:

[1415] The server inputs the generated story into an image generation algorithm to generate cartoon-style images. For example, it uses an image caption generation model to generate cartoon images corresponding to each scene in the story. The inputs are the generated story and image data. The output is a cartoon-style image.

[1416] Step 6:

[1417] The server sends the generated manga image to the terminal. The transmission uses internet communication, and the image data is transferred from the server to the terminal. Specifically, the generated manga image is returned to the terminal as an HTTP response. The input is the generated manga image. The output is the manga image sent to the terminal.

[1418] Step 7:

[1419] The device displays the received manga image to the user. Specifically, the manga image is displayed on the app screen so that the user can easily view it. The input is the manga image sent from the server. The output is a manga-format image that the user can view.

[1420] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1421] The present invention is a system that generates and displays manga based on text data and image data entered by a user, and further recognizes the user's emotions and generates manga in accordance with those emotions.

[1422] System Overview

[1423] This system works by linking electronic devices (smartphones, tablets, PCs, etc.) with a server. Users input text data (memorable episodes) and image data (photos) through their electronic devices, and the server processes this data to generate a manga, which is then sent back to the electronic device for display. Furthermore, an emotion engine is used to recognize the user's emotions, and the story and manga images are adjusted accordingly.

[1424] Detailed embodiment

[1425] User operation

[1426] First, users launch the app and enter a memorable episode in the text input field. They also select relevant photos from their smartphone or tablet's camera roll and upload them to the app. When the user presses the "Generate Manga" button, the entered data is sent to the server.

[1427] Terminal handling

[1428] The device sends the text data and image data entered by the user to the server. The transmission is via internet communication, and the data is encoded in JSON format or similar. This process converts the data into a format that can be received by the server.

[1429] Server-side processing

[1430] The server receives the data sent from the device, decodes the JSON data, and extracts text and image data. It then uses an emotion engine to recognize the user's emotions from the text and image data. This emotion recognition information is used for subsequent processing.

[1431] Narrative generation and emotion regulation

[1432] The server inputs the recognized emotion information into a narrative generation algorithm to generate a detailed story based on episodes. For example, if the user expresses joy, it generates a story that emphasizes positive episodes. Each scene of the story is automatically created using a natural language processing model.

[1433] Image generation and emotion regulation

[1434] The server then uses an image generation algorithm to create a manga-style image based on the generated story and image data. The server then adjusts the image tone and color based on emotion recognition information. For example, if the emotion indicates sadness, the color tone can be darkened and a function to track facial expressions can be added.

[1435] Send and Display

[1436] The server sends the generated manga image to the electronic device. The image data is transferred from the server to the device via internet communication. The device displays the received manga image on the user interface. The user can view the manga on the app screen and enjoy their memories in a way that suits their emotions.

[1437] Specific examples

[1438] Enter your story and photos

[1439] For example, if a user inputs the following episode:

[1440] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[1441] Also, upload the following photos:

[1442] 1. Family photos

[1443] 2. Ocean photos

[1444] 3. Fireworks photos

[1445] Specific examples of emotion recognition

[1446] When a user writes an episode, the emotion engine recognizes the emotion "fun." This information is used by the story generation algorithm to represent positive scenes.

[1447] System processing flow

[1448] 1. The device sends the above data to the server.

[1449] 2. Based on the episodes received by the server, the emotion engine recognizes the user's emotions and generates a positive story.

[1450] 3. Based on the story generated by the server, an image generation algorithm generates cartoon images with a positive tone.

[1451] 4. The server sends the generated manga image to the device.

[1452] 5. The device displays the cartoon image to the user.

[1453] In this way, a system is realized that allows users to easily create and display comics of memories that reflect the user's emotions, allowing the user to enjoy memories in a more emotional way.

[1454] The processing flow will be explained below.

[1455] Step 1:

[1456] Users launch the app, enter a memorable story into the text input field, and select related photos from their smartphone or tablet's camera roll and upload them to the app.

[1457] Step 2:

[1458] The user presses the "Generate Manga" button. This action starts the process in which the device sends the input data (text data and image data) to the server.

[1459] Step 3:

[1460] The terminal encodes the text data and image data entered by the user into JSON format and sends a POST request to the specified server URL.

[1461] Step 4:

[1462] The server receives the data sent from the terminal, decodes the JSON data, and extracts the text data and image data.

[1463] Step 5:

[1464] The server inputs the extracted text and image data into an emotion engine to recognize the user's emotions. The emotion engine uses natural language processing and image analysis to extract emotional information from the episodes and photos entered by the user.

[1465] Step 6:

[1466] The server then adjusts the text data fed into the story generation algorithm based on the recognized emotion information. For example, if the user expresses the emotion "fun," the story content will be emphasized with a positive tone.

[1467] Step 7:

[1468] The server inputs the adjusted text data into a narrative generation algorithm to generate a detailed story based on the episodes, and automatically creates each scene of the story using a natural language processing model.

[1469] Step 8:

[1470] The server uses an image generation algorithm to generate a cartoon-style image based on the generated story and the recognized emotional information. The image tone and color are adjusted based on the emotional information. For example, if the emotion indicates "fun," bright colors and smiling faces are emphasized.

[1471] Step 9:

[1472] The server encodes the generated manga image in JSON format and sends it back to the device. The image data is transferred via the Internet.

[1473] Step 10:

[1474] The device decodes the JSON data returned from the server and extracts the manga image.

[1475] Step 11:

[1476] The device displays the received manga image on the user interface, which is displayed on the app screen, allowing the user to view the generated manga.

[1477] Step 12:

[1478] Users can browse the manga on the app screen and enjoy memorable episodes and the manga based on them.

[1479] The above steps realize a system that allows users to easily save and enjoy memories that match their emotions in the form of comics.

[1480] Example 2

[1481] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1482] Conventional systems simply convert text data entered by users into a story, making it difficult to generate manga that takes into account the user's emotions and image data. This limited the user experience and prevented the creation of more emotionally rich stories. Furthermore, adjustments to the tone and color of the generated manga images could not be automated, making it difficult to generate manga that reflects the user's emotions.

[1483] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1484] In this invention, the server includes means for receiving text data and image data and recognizing a user's emotions using an emotion engine, story generation means for generating a story based on the result of the emotion recognition, image generation means for generating a comic image based on the story and image data generated by the story generation means, and means for adjusting the tone and color of the comic image according to the result of the emotion recognition. This makes it possible to generate a comic according to the user's emotions, providing a more emotional user experience.

[1485] A "user" is a person who uses the system and inputs text data and image data using an electronic device.

[1486] "Electronic device" is a general term for digital devices such as smartphones, tablets, and personal computers, which provide a means for users to input text data and image data.

[1487] "Text data" refers to text information entered by the user, including descriptions of memorable episodes and the like.

[1488] "Image data" refers to data including photos and image files uploaded by users.

[1489] A "server" refers to a computer system for receiving and processing data sent by users.

[1490] "Emotion engine" refers to an algorithm or software for recognizing a user's emotions from text data and image data.

[1491] "Narrative generation means" refers to an algorithm or software for generating a story based on text data and emotion recognition results.

[1492] "Image generation means" refers to an algorithm or software for generating comic-style images based on the generated story and image data.

[1493] "Adjustment means" refers to an algorithm or software that provides the functionality to adjust the tone and color of the generated cartoon image according to the user's emotion recognition results.

[1494] "Transmission means" refers to an algorithm or software that provides the functionality for transmitting the generated cartoon image to an electronic device.

[1495] The present invention is a system that generates and displays manga based on text data and image data entered by a user. Furthermore, the system recognizes the user's emotions and generates manga in accordance with those emotions. Specific embodiments of this system are described below.

[1496] System Overview

[1497] This system works by linking electronic devices (smartphones, tablets, PCs, etc.) with a server. Users input text data (memorable episodes) and image data (photos) through their electronic devices, and the server processes this data to generate a manga, which is then sent back to the electronic device for display. Furthermore, an emotion engine is used to recognize the user's emotions, and the story and manga images are adjusted accordingly.

[1498] Hardware and software used

[1499] Hardware: smartphones, tablets, PCs, servers

[1500] Software: Dedicated app, emotion engine, natural language processing model, image generation algorithm, internet communication (HTTP)

[1501] Processing flow and specific operations

[1502] User operations

[1503] First, the user launches the dedicated app and writes a memorable episode in the text input field. For example, they might write, "Last summer, I went to the beach with my family. We had a great time riding the waves. At night, we watched fireworks and created beautiful memories." They then select relevant photos from their smartphone or tablet's camera roll and upload them to the app. For example, they could use family photos, photos of the beach, or photos of fireworks. When the user presses the "Generate Manga" button, this data is sent to the server.

[1504] Sending and Receiving Data

[1505] The device encodes the text data and image data entered by the user in JSON format and sends it to the server via Internet communication (HTTP protocol). The server decodes the received JSON data and extracts the text data and image data.

[1506] Emotion Recognition Processing

[1507] The server inputs the decoded text data into an emotion engine to recognize the user's emotion. For example, it extracts the emotion "fun" from the text. It also inputs image data into the emotion engine to analyze emotions from the images.

[1508] Story Generation

[1509] The server inputs the emotion recognition results into a story generation algorithm. If the emotion recognition identifies the emotion as "fun," the story generation algorithm generates a story that emphasizes positive episodes. Using a natural language processing model, detailed story scenes are automatically created based on the user's episodes.

[1510] Image generation and emotion regulation

[1511] The server inputs the generated story and image data uploaded by the user into an image generation algorithm, using a generative AI model. The image tone and color are adjusted based on the emotion recognition results. For example, a happy emotion is represented by a bright, colorful tone, while a sad emotion is represented by a dark tone. A comic-style image corresponding to each scene in the story is generated.

[1512] Sending and displaying the generated cartoon image

[1513] The server sends the generated manga image to the device. This is again done via Internet communication (HTTP protocol). The device decodes the received manga image and displays it to the user. The user then views the generated manga through the app. The app's user interface provides functions such as turning pages and zooming in and out of the manga.

[1514] Prompt Sentence Examples

[1515] Prompt: "When the user inputs an emotional story, generate a cartoon that reflects that particular emotion. For example, use bright and colorful cartoons for happy stories, and darker cartoons for sad stories."

[1516] This allows the system to easily create and display manga of memories that reflect the user's emotions, allowing the user to enjoy their memories in a more emotional way.

[1517] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1518] Step 1:

[1519] Users launch the app on their smartphone or tablet, enter their memorable experiences in the text input field, and select related photos from their camera roll and upload them to the app.

[1520] Input: Memorable episodes (text data) and related photos (image data)

[1521] Output: Text and image data that can be viewed within the app

[1522] Step 2:

[1523] The terminal encodes the text and image data entered by the user in JSON format.

[1524] Input: Text data and image data

[1525] Output: A data object in JSON format

[1526] Step 3:

[1527] The device sends the encoded JSON data to the server over the Internet using the HTTP protocol.

[1528] Input: A JSON formatted data object

[1529] Output: Data sent to the server

[1530] Step 4:

[1531] The server receives and decodes the JSON data sent from the device. This process extracts text data and image data. At this time, the data integrity is also checked.

[1532] Input: JSON data submitted

[1533] Output: Extracted text and image data

[1534] Step 5:

[1535] The server inputs the decoded text data into an emotion engine to recognize the user's emotion. Processing such as extracting the emotion "fun" from the text is performed. Image data is also input into the emotion engine for emotion analysis.

[1536] Input: Text data and image data

[1537] Output: User's emotional information (e.g., happy, sad)

[1538] Step 6:

[1539] The server inputs the emotion recognition results into a story generation algorithm to generate a story. If the emotion recognition identifies the emotion as "fun," the story generation algorithm generates a story that emphasizes positive episodes.

[1540] Input: Emotion information and text data

[1541] Output: Generated stories

[1542] Step 7:

[1543] The server inputs the generated story and user-uploaded image data into an image generation algorithm, which adjusts the tone and color of the image based on emotion recognition results and generates manga-style images corresponding to each scene in the story.

[1544] Input: Story, image data, emotional information

[1545] Output: Generated cartoon image

[1546] Step 8:

[1547] The server sends the generated manga image to the terminal using the Internet (HTTP protocol).

[1548] Input: Generated cartoon image

[1549] Output: Data sent to the terminal

[1550] Step 9:

[1551] The device decodes the received manga images and displays them on the app's user interface. The user can then browse the generated manga. The app also provides page-turning and zooming functions.

[1552] Input: Submitted cartoon image

[1553] Output: Comic image displayed on the app screen

[1554] In this way, the present invention allows users to easily create and display cartoons of their memories that reflect their own emotions.

[1555] (Application example 2)

[1556] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1557] Conventional manga generation systems were able to generate static manga based on text and image data entered by users, but the generated manga could not be expressed in a way that adapted to the user's emotions. This made it difficult to provide a more personalized experience that matched the user's emotions. Furthermore, it was not possible to generate a story that properly reflected the main points and emotional nuances of the episodes entered by the user. As a result, it was difficult to provide a more emotionally rich and personalized manga experience for users.

[1558] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1559] In this invention, the server includes means for a user to input text data and image data using an electronic device, emotion recognition means for recognizing the user's emotion from the text data and image data, story generation means for generating a story based on the emotion recognized by the text data and emotion recognition means, image generation means for generating a comic image based on the story generated by the story generation means, means for transmitting the comic image generated by the image generation means to the electronic device, and means for displaying the transmitted comic image on the electronic device, thereby enabling the generation and display of a more personalized comic tailored to the emotion felt by the user.

[1560] "Electronic devices" refer to devices such as smartphones, tablets, and personal computers that users use to input text data and image data.

[1561] "Text data" refers to sentence data that describes the contents of an episode or story and is input by a user using an electronic device.

[1562] "Image data" refers to photographs and image files that users upload using electronic devices.

[1563] The "emotion recognition means" is an algorithm or system that has the function of analyzing text data and image data to recognize the user's emotions.

[1564] A "narrative generator" is an algorithm or system that generates a detailed story based on text data and recognized emotions.

[1565] The "image generation means" is an algorithm or system that creates comic-style images based on the story generated by the story generation means.

[1566] The "transmission means" is a communication means for transferring the generated comic image data from the server to the electronic device.

[1567] The "display means" is a function for displaying the comic image received by the electronic device on the user interface.

[1568] "Memorable episodes" are text data relating to events and memories that the user has experienced in the past.

[1569] "Uploaded photos" are image data sent by users to the system via their electronic devices.

[1570] "Adjusting the tone and color tone based on emotion recognition" means adjusting the color and brightness of the generated cartoon image in accordance with the user's emotion information recognized by the emotion recognition means.

[1571] The purpose of this system is to generate and display a manga that corresponds to the user's emotions based on text data and image data entered by the user. A specific embodiment of the system is as follows.

[1572] User operation

[1573] First, the user launches a dedicated application on an electronic device such as a smartphone or tablet. The user writes the episode in the text input field on the application, and then selects and uploads related photos from the device's camera roll. When the user presses the "Generate Manga" button, the entered text and image data are sent from the device to the server.

[1574] Terminal handling

[1575] The device encodes the text and image data entered by the user in a format such as JSON and sends it to a server over the Internet, using a secure and efficient communication protocol for data transfer.

[1576] Server-side processing

[1577] The server receives the data sent from the device, decodes the JSON data, and separates and extracts the text data and image data. Next, it uses an emotion recognition engine to recognize the user's emotions from the text data and image data. This emotion recognition information plays an important role in subsequent processing.

[1578] Specifically, the server does the following:

[1579] 1. Emotion recognition means: An emotion recognition engine (e.g., the EmotionRecognition library) is used to recognize specific emotions (e.g., "happy," "sad," etc.) from the user's text data.

[1580] 2. Narrative generation method: Using natural language processing algorithms, a detailed story is generated from text data according to the recognized emotions.

[1581] 3. Image generation method: Based on the generated story and uploaded image data, an image generation algorithm (e.g., ComicGenerator library) is used to generate a comic-style image, adjusting the tone and color based on the emotional information.

[1582] Send and Display

[1583] The generated manga image is sent from the server to the device, and the device displays the received manga image on the user interface. The user can view the manga on the app screen and enjoy the input episode in a way that matches their emotions.

[1584] Specific examples

[1585] For example, if a user enters an episode like this:

[1586] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[1587] Also, upload the following photos:

[1588] 1. Family photos

[1589] 2. Ocean photos

[1590] 3. Fireworks photos

[1591] Based on this episode, the system recognizes the emotion of "fun" and generates a story and cartoon image with a positive tone. Specific examples of prompts used during system processing are as follows:

[1592] Prompt Sentence Examples

[1593] "Last summer, I went to the beach with my family. We had a great time riding the waves. We watched fireworks at night and made some beautiful memories."

[1594] In this way, a personalized comic experience can be provided according to the user's emotions.

[1595] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1596] Step 1:

[1597] The user inputs text and image data using an electronic device. The user launches a dedicated application, writes the episode in the text input field, and selects and uploads related photos. This input data is converted into JSON format and prepared for data transmission.

[1598] Input: Text data (episodes), image data (photos)

[1599] Output: JSON formatted data packet

[1600] Step 2:

[1601] The device sends a JSON-formatted data packet to the server, which then transmits the encoded data over the internet using a secure and efficient communications protocol.

[1602] Input: JSON formatted data packet

[1603] Output: Data packets sent

[1604] Step 3:

[1605] The server receives the data packets and decodes the JSON data to extract text and image data, which is then structured and prepared for processing by the emotion recognition engine.

[1606] Input: Data packets sent

[1607] Output: Text data, image data

[1608] Step 4:

[1609] The server uses an emotion recognition means to recognize the user's emotion from the text data and image data. An emotion recognition engine (e.g., EmotionRecognition library) is used to extract specific emotions (e.g., "happy" or "sad") from the text data.

[1610] Input: Text data, image data

[1611] Output: Emotion information (emotion classification)

[1612] Step 5:

[1613] The server uses a narrative generation means to generate a detailed story based on the text data and the recognized emotion information, and uses a natural language processing algorithm to generate an emotion-adapted story based on the episodes.

[1614] Input: Text data, emotion information

[1615] Output: Detailed story

[1616] Step 6:

[1617] The server uses the image generation means to generate a comic-style image based on the story generated by the story generation means and the uploaded image data, and adjusts the tone and color based on the emotional information (e.g., bright tones for happy emotions, dark tones for sad emotions).

[1618] Input: detailed story, image data, emotional information

[1619] Output: Cartoon image

[1620] Step 7:

[1621] The server encodes the generated comic image in JSON format and sends it to the device. The receiving device receives and decodes the data packet to obtain the comic image.

[1622] Input: Cartoon image

[1623] Output: JSON formatted manga image data packet

[1624] Step 8:

[1625] The terminal displays the received comic image on the user interface, and the user can view the comic generated according to their emotion through the application.

[1626] Input: JSON formatted manga image data packet

[1627] Output: Displayed cartoon image

[1628] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1629] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1630] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1631] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1632] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1633] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1634] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1635] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1636] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1637] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1638] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1639] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1640] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1641] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1642] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1643] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1644] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1645] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1646] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1647] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1648] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1649] The following is further disclosed regarding the above embodiment.

[1650] (Claim 1)

[1651] A means for a user to input text data and image data using an electronic device;

[1652] a story generation means for generating a story based on the text data;

[1653] an image generation means for generating a comic image based on the story generated by the story generation means;

[1654] means for transmitting the cartoon image generated by the image generation means to the electronic device;

[1655] a means for displaying the transmitted comic image in the electronic device;

[1656] A system including:

[1657] (Claim 2)

[1658] The system according to claim 1, wherein the text data is a memorable episode.

[1659] (Claim 3)

[1660] 2. The system of claim 1, wherein the image data is a photograph uploaded by a user.

[1661] "Example 1"

[1662] (Claim 1)

[1663] A means for a user to input text data and image data using an electronic device;

[1664] means for transmitting the text data and image data to a server via internet communication;

[1665] means for executing a story generation algorithm based on the text data in the server to generate a story;

[1666] means for executing an image generation algorithm based on the generated story to generate a cartoon image;

[1667] means for transmitting the generated cartoon image to the electronic device;

[1668] a means for displaying the transmitted comic image in the electronic device;

[1669] A system including:

[1670] (Claim 2)

[1671] The system according to claim 1, wherein the text data is a memorable episode.

[1672] (Claim 3)

[1673] 2. The system of claim 1, wherein the image data is a photograph uploaded by a user.

[1674] "Application Example 1"

[1675] Novel feature extraction

[1676] 1. Displaying product reviews in comic format

[1677] 2. Generate manga from review text and image data

[1678] 3. Generate narratives and captions using generative AI models

[1679] (Claim 1)

[1680] A means for a user to input text data and image data using an electronic device;

[1681] a story generation means for generating a story based on the text data;

[1682] an image generation means for generating a comic image based on the story generated by the story generation means;

[1683] means for transmitting the cartoon image generated by the image generation means to the electronic device;

[1684] a means for displaying the transmitted comic image in the electronic device;

[1685] A means to display product reviews in comic form,

[1686] A system including:

[1687] (Claim 2)

[1688] The system according to claim 1, wherein the text data is a memorable episode or a product review.

[1689] (Claim 3)

[1690] 2. The system of claim 1, wherein the image data is a photograph uploaded by a user.

[1691] "Example 2: Combining Emotion Engines"

[1692] (Claim 1)

[1693] A means for a user to input text data and image data using an electronic device;

[1694] means for transmitting the text data and image data to a server;

[1695] a server receiving the text data and image data and recognizing the user's emotion using an emotion engine;

[1696] a story generation means for generating a story based on the result of the emotion recognition;

[1697] an image generation means for generating a comic image based on the story and image data generated by the story generation means;

[1698] means for adjusting the tone and color of the cartoon image in response to the result of the emotion recognition;

[1699] means for transmitting the cartoon image generated by the image generation means to the electronic device;

[1700] a means for displaying the transmitted comic image in the electronic device;

[1701] A system including:

[1702] (Claim 2)

[1703] The system according to claim 1, wherein the text data is a memorable episode.

[1704] (Claim 3)

[1705] 2. The system of claim 1, wherein the image data is a photograph uploaded by a user.

[1706] "Application example 2 when combining emotion engines"

[1707] (Claim 1)

[1708] A means for a user to input text data and image data using an electronic device;

[1709] emotion recognition means for recognizing a user's emotion from the text data and image data;

[1710] a story generation means for generating a story based on the text data and the emotion recognized by the emotion recognition means;

[1711] an image generation means for generating a comic image based on the story generated by the story generation means;

[1712] means for transmitting the cartoon image generated by the image generation means to the electronic device;

[1713] a means for displaying the transmitted comic image in the electronic device;

[1714] A system including:

[1715] (Claim 2)

[1716] 2. The system according to claim 1, wherein the text data is a memorable episode, and the emotion recognition means recognizes the emotion felt by the user based on the memorable episode.

[1717] (Claim 3)

[1718] 2. The system of claim 1, wherein the image data is a photograph uploaded by a user, and the image generating means adjusts the tone and color of the cartoon image based on emotion recognition. [Explanation of symbols]

[1719] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for a user to input text data and image data using an electronic device; a story generation means for generating a story based on the text data; an image generation means for generating a comic image based on the story generated by the story generation means; means for transmitting the cartoon image generated by the image generation means to the electronic device; a means for displaying the transmitted comic image in the electronic device; A system including:

2. The system according to claim 1 , wherein the text data is a memorable episode.

3. 2. The system of claim 1, wherein the image data is a photograph uploaded by a user.

Images