System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting a title and theme of a story from a user device; means for generating an initial plot with a generative AI model; means for displaying the plot received from the generative AI model on the user device; means for passing the story development input by the user to the generative AI model; means for displaying a new development proposed by the generative AI model on the user device; means for saving a final story edited by the user; and means for processing a bookbinding request based on the saved story.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Due to the widespread use of digital devices and busy activities, today's children have less time for free, creative play and storytelling. Many children also find it difficult to ask questions or express their opinions. Furthermore, when learning and enjoyment are separated, learning can feel obligatory and it becomes difficult to develop interest. Given this background, there is a need for a way for children to learn while being creative and having fun, and to improve their communication skills and imagination. The purpose of this invention is to solve these problems by providing a system that allows children to create original story picture books by interacting with a generative AI model, providing them with an opportunity to learn while having fun and gaining a sense of accomplishment and satisfaction. [Means for solving the problem]

[0005] The present invention is characterized by a system including: a means for inputting a story title and theme from a user terminal; a means for generating an initial plot using a generative AI model; a means for displaying the plot received from the generative AI model on the user terminal; a means for passing the story development input by the user to the generative AI model; a means for displaying new developments proposed by the generative AI model on the user terminal; a means for saving the final story edited by the user; and a means for processing a binding request based on the saved story. The system also includes a text-to-speech function and multilingual support, which promotes listening skills and multilingual comprehension. This system allows children to develop their own ideas through interaction with the generative AI model and create original story picture books. Having the stories they create bound and delivered to their homes allows them to feel a sense of accomplishment and satisfaction.

[0006] A "user terminal" is a device used by a user to input or view data, and includes PCs, smartphones, tablets, etc.

[0007] A "title" is a name that succinctly expresses the content or theme of a story.

[0008] A "theme" refers to the main theme or central idea of ​​a story, and is the element that determines the direction of the story.

[0009] A "generative AI model" is an algorithm that uses artificial intelligence technology to generate storylines and suggestions based on user input.

[0010] An "initial plot" is a rough storyline that provides the first part or outline of a story, and is the basic structure that guides the progression of the story.

[0011] "Story development" refers to describing the specific content of the story, such as the progression of the story, the actions of characters, and the occurrence of events.

[0012] "Storing" refers to recording the data edited by the user in a database or other storage so that it can be accessed or used later.

[0013] A "bookbinding request" is a request from a user to have their completed story produced and delivered as a physical book.

[0014] The "text-to-speech function" is a function in which an electronic device converts text data into speech and reads it out loud to the user.

[0015] "Multilingual support" refers to the ability of a system to understand and display multiple languages ​​and to provide services to users in the language of their choice. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This paper describes a system that allows users to create original story picture books while interacting with a generative AI model. Specifically, it describes the configuration of the system, which includes a user terminal, a server, and a generative AI model, and the processing of the program.

[0038] System Overview

[0039] The system of the present invention receives input from the user's device, sends it to the generative AI model via the server, and then provides the output from the AI ​​back to the user. Through this process, users (especially children) are given the experience of creating their own story picture book. The system configuration and the functions of each element are described in detail below.

[0040] User terminal

[0041] The user device (e.g., PC, smartphone, tablet) provides an interface for the child to input the title and theme of the story.

[0042] 1. Input interface:

[0043] Provide a form for children to enter the title and theme of their story.

[0044] 2. Display interface:

[0045] Displays plot suggestions from AI and story developments entered by users.

[0046] 3. Editing features:

[0047] It provides an editor where users can edit, modify and save stories.

[0048] 4. Binding Request:

[0049] Provides an interface for submitting completed stories as binding requests.

[0050] server

[0051] The server is responsible for sending input data from users to the generative AI model and returning output data from the generative AI model to the user's device. It also stores data and manages binding requests. Its main functions are as follows:

[0052] 1. User authentication:

[0053] Verify your login information and authenticate.

[0054] 2. Data processing and analysis:

[0055] Receives input data from the user and passes it to the generative AI model.

[0056] 3. Data Retention:

[0057] Store user-created stories in a database.

[0058] 4. Generative AI model collaboration:

[0059] Send data to the AI ​​model and receive the results.

[0060] 5. Binding Request Management:

[0061] Receives binding requests and coordinates with binding services.

[0062] Generative AI Models

[0063] The generative AI model generates plot suggestions and storylines based on user input data.

[0064] 1. Initial plot generation:

[0065] Generates an initial plot based on the title and theme entered by the user.

[0066] 2. Storyline proposal:

[0067] Based on the story development entered by the user, new ideas and developments are suggested.

[0068] Specific examples

[0069] Imagine a child is creating a picture book with the theme of "adventure story."

[0070] Operation on the user device

[0071] The user (child) launches the application on the user's device and enters their login information. If login is successful, the home screen is displayed and the child clicks the "Create a new story" button. Next, they enter the story title "Adventure with a Dragon" and the theme "Adventure" in the form that appears, and press the submit button.

[0072] Processing on the server

[0073] The server stores the title and theme received from the user in a database and sends a request to the generative AI model to generate an initial plot. When the generative AI model receives the initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure..."), it sends the content to the user's device.

[0074] Viewing and editing on user devices

[0075] The user device displays the received initial plot, and the child can think up a new storyline. For example, they can input and submit a story in which a boy named Taro appears, becomes friends with a dragon, and goes on a treasure hunt together.

[0076] Collaboration between the server and the generative AI model

[0077] The server sends the new storyline input by the child to the generative AI model again and receives the next suggestion. When the server receives a new storyline from the AI ​​model (e.g., "Taro and Dragon find a treasure chest in a cave deep in the mountains, and inside is an old map..."), it sends it to the user's device.

[0078] Submit a binding request

[0079] After the story is completed, the user device displays a button for the child to send the completed story as a bookbinding request. When the child clicks the button, the request is sent to the server, which then works with the bookbinding service to arrange for the picture book to be produced and delivered.

[0080] This is the process by which children can create their own original story picture books using the system of the present invention, which not only allows children to exercise their creativity and learn while having fun, but also gives them a sense of accomplishment by having their own unique picture book.

[0081] The processing flow will be explained below.

[0082] Step 1:

[0083] The user starts the service application on the user device and the login screen is displayed. The child enters the user name and password and clicks the login button.

[0084] Step 2:

[0085] The server receives the login information and performs authentication. If authentication is successful, the server starts the user session and returns the home screen data to the device.

[0086] Step 3:

[0087] The device will display a home screen and offer a "Create a new story" button that the child can click to display a form where they can enter a title and theme.

[0088] Step 4:

[0089] The user enters a story title (e.g., "Adventure with a Dragon") and a theme (e.g., "Adventure") and clicks the submit button.

[0090] Step 5:

[0091] The server receives the input title and theme, stores them in a database, and then sends a request to the generative AI model to generate the initial plot.

[0092] Step 6:

[0093] The generative AI model generates an initial plot based on the title and theme, and sends the generated plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure...") back to the server.

[0094] Step 7:

[0095] The server sends the initial plot received from the generative AI model to the terminal.

[0096] Step 8:

[0097] The device displays the initial plot. The user (child) reads the plot and thinks of a new storyline to input (e.g., "Taro becomes friends with the dragon and they go on a treasure hunt together."). When they're done, they click the send button.

[0098] Step 9:

[0099] The server receives the new storyline input from the user and sends it back to the generative AI model, requesting new proposals.

[0100] Step 10:

[0101] The generative AI model generates the next proposal based on the new storyline and sends the result back to the server (e.g., "Tarro and Dragon find a treasure chest in a cave deep in the mountains, inside which is an old map").

[0102] Step 11:

[0103] The server sends the next proposal received from the generative AI model to the device.

[0104] Step 12:

[0105] The device will display a new suggestion, and the user can view the suggestion and edit the story. Once they've finished editing, they can click the save button again. When the final story is complete, they'll be given the option to request binding.

[0106] Step 13:

[0107] The user checks the completed story and clicks the binding request button.

[0108] Step 14:

[0109] The server receives the bookbinding request, stores it in a database, and sends the request to the bookbinding service. After receiving confirmation from the bookbinding service, the server arranges for the production and delivery of the picture book.

[0110] Step 15:

[0111] Users receive a bound picture book delivered to their home and enjoy their own original story.

[0112] Example 1

[0113] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0114] Conventional story creation systems require users to do a lot of manual work when creating a story, making the creation process complicated and time-consuming. They also lacked adequate support to fully bring out users' creativity, especially functions that allow children to enjoy creative activities. Furthermore, the process of binding completed stories into books was complicated, making it difficult for users to easily obtain the finished product.

[0115] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0116] In this invention, the server includes a means for transmitting data input from a user terminal to the server, a means for the server to authenticate the user and store the data in a database, and a means for the server to transmit a request to the generative AI model and receive the generated plot. This allows the user to intuitively input data for story creation, utilize the generative AI model to propose and edit the story development, and finally easily request bookbinding.

[0117] A "user terminal" is a device operated by a user, and examples include PCs, smartphones, tablets, etc.

[0118] "Input means" refers to an interface for a user to input data, and includes a keyboard, a touch screen, a microphone, etc.

[0119] A "generative AI model" is a system that uses artificial intelligence to generate data, specifically a model that generates a plot or storyline based on a prompt.

[0120] "Initial plot generator" refers to a process for generating an initial plot for a story based on a title and theme entered by a user.

[0121] "Display means" refers to an interface for visually displaying data on a user terminal, including a screen or monitor.

[0122] "Storage" refers to a method for long-term storage of data, including databases and file systems.

[0123] "Bookbinding request processing means" refers to a system that processes requests from users to bind their completed stories into actual books.

[0124] A "server" refers to a computer or a series of computer systems that connects to multiple user terminals via a network and processes and stores data.

[0125] "Authentication measures" refers to the process of verifying a user's identity and controlling access to a system, including verifying user IDs and passwords.

[0126] A "database" refers to a system for efficiently storing, retrieving, and managing structured data.

[0127] "Request sending means" refers to a method for sending a user's request to a generative AI model or other system component.

[0128] The present invention relates to a system in which a user creates an original story picture book while interacting with a generative AI model. The system includes a user terminal, a server, and a generative AI model, each of which plays a specific role.

[0129] User terminal

[0130] The user device provides an interface for users (especially children) to input the title and theme of the story. It also has an editor function that displays the plot and story development generated by the generative AI model and allows users to edit, modify, and save the story. User devices can be PCs, smartphones, tablets, or other devices.

[0131] As a concrete example, a user (child) starts an application on the device and enters login information. After successful login, the user clicks the "Create a new story" button, enters the title "Adventure with a Dragon" and the theme "Adventure" in the form, and presses the submit button.

[0132] server

[0133] The server receives input data from user devices and sends it to the generative AI model. It also handles user authentication, data storage, plot generation requests, and bookbinding requests. The server uses standard server hardware and database software (e.g., MySQL (registered trademark), PostgreSQL).

[0134] The server receives the title and theme entered by the user, stores them in a database, and sends a request to the generative AI model to generate an initial plot. For example, the server sends the following prompt to the generative AI model:

[0135] Title: Adventure with Dragons

[0136] Theme: Adventure

[0137] Prompt: One day, a brave boy meets a dragon who guards a hidden treasure...

[0138] Generative AI Models

[0139] The generative AI model generates a plot and storyline based on the input title and theme. For example, OpenAI's GPT-3 model is used. The generative AI model generates an initial plot based on prompts received from the server and sends the plot back to the server.

[0140] A generative AI model might generate a plot like this:

[0141] One day, a brave boy meets a dragon who is guarding a secret treasure. The boy and the dragon decide to join forces to search for the treasure...

[0142] A concrete example of the program's operation flow

[0143] When a user inputs the story title and theme, the user's device sends the data in JSON format to the server. The server stores the data in a database and sends a prompt to the generative AI model to generate an initial plot. The server receives the plot returned by the generative AI model and sends it to the user's device for display.

[0144] When a user begins to input a new storyline, that data is also sent via the server to the generative AI model, which then generates a new storyline. By repeating this process, the user and the generative AI model work together to create a unique story picture book. Once completed, the user submits a binding request, and the server connects the request to the binding service to produce and deliver the picture book.

[0145] Specific technical features

[0146] This system uses a generative AI model to automatically generate plots and story developments, and is equipped with the ability to efficiently process, store, and display data entered by users. The server acts as a central link between the user's device and the generative AI model, ensuring a smooth story creation process. It is also designed to automate the processing of binding requests, allowing users to easily obtain physical picture books.

[0147] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0148] Step 1:

[0149] The user uses the user terminal to input the title and theme of the story. Specifically, the user starts the application, enters the title "Adventure with a Dragon" and the theme "Adventure" in the story creation form, and presses the submit button. This operation generates the input data (title and theme) from the user terminal.

[0150] Step 2:

[0151] The user's device sends the entered title and theme data to the server. The entered data is converted to JSON format and sent as an HTTP POST request to the server's API endpoint. Authentication information such as the user ID must also be included in this process.

[0152] Step 3:

[0153] The server parses the received JSON data and authenticates the user. After successful authentication, it saves the entered title and theme in the database. The server then connects to the database and adds the data as a new entry to the corresponding table.

[0154] Step 4:

[0155] The server sends a request to the generative AI model to generate an initial plot based on the title and theme stored in the database, generates a prompt text, and sends a request including the prompt text to the API endpoint of the generative AI model.

[0156] Step 5:

[0157] The generative AI model generates an initial plot based on the prompt received from the server. For example, it might generate an initial plot such as, "One day, a brave boy meets a dragon. The dragon is guarding a secret treasure..." and send it back to the server.

[0158] Step 6:

[0159] The server receives the initial plot returned from the generative AI model, stores it in a database, and sends an HTTP response containing the initial plot to the user device.

[0160] Step 7:

[0161] The user terminal displays the initial plot received from the server on the user interface. The user checks the plot and inputs a new story development. For example, the user inputs a development such as "A boy named Taro appears, becomes friends with a dragon, and goes on a treasure hunt together."

[0162] Step 8:

[0163] The user device sends the newly entered storyline data to the server, which again converts the input data into JSON format and sends it to the server as an HTTP POST request.

[0164] Step 9:

[0165] The server receives the new storyline data and resends it to the generative AI model. It then generates a new prompt and sends a request to the generative AI model based on that.

[0166] Step 10:

[0167] The generative AI model generates new developments based on the new storylines sent in. For example, it generates a new plot like, "Tarro and Dragon find a treasure chest in a cave deep in the mountains, and inside is an old map..."

[0168] Step 11:

[0169] The server receives the newly generated deployment from the generative AI model, stores it in a database, and sends it to the user's device as an HTTP response.

[0170] Step 12:

[0171] The user's device displays the new developments received from the server on the user interface. The user continues to edit and revise the story until it is complete.

[0172] Step 13:

[0173] The user confirms the completion of the final story and sends a binding request, which is then sent to the server from the user terminal.

[0174] Step 14:

[0175] The server stores the received bookbinding request in a database and sends the request to the bookbinding service, which then processes the book for production and delivery.

[0176] Step 15:

[0177] The server receives a completion notification from the bookbinding service and notifies the user of the completion of binding and delivery information. The user receives the notification and receives the completed picture book.

[0178] (Application example 1)

[0179] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0180] Existing story generation systems lack a means for users to easily share the content they create with other users when creating original story picture books. Another issue is the lack of convenience for use as a content distribution service for parents and children to enjoy together.

[0181] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0182] In this invention, the server includes means for inputting a story title and theme from a user terminal, means for generating an initial plot using a generative AI model, means for displaying the plot received from the generative AI model on the user terminal, means for passing the story development input by the user to the generative AI model, means for displaying a new development proposed by the generative AI model on the user terminal, means for saving the final story edited by the user, means for processing a binding request based on the saved story, and means for delivering the generated plot or story development as a content delivery service, thereby enabling users to easily share the generated story with other users or use it as a content delivery service for parent-child enjoyment.

[0183] A "user terminal" is a terminal that a user uses to input the title and theme of a story and to display the generated plot and story development.

[0184] A "generative AI model" is an artificial intelligence model that automatically generates an initial plot and story development based on the title and theme entered by the user.

[0185] An "initial plot" is the basic structure of a story that the generative AI model initially generates based on the story title and theme provided by the user.

[0186] "Story development" is an item in which the user specifically describes the progress of the story or adds new scenes.

[0187] A "bookbinding request" is a request by a user to produce a completed story as a physical book.

[0188] A "content distribution service" is a service that distributes generated plots and storylines in a form that can be used by other users.

[0189] This invention relates to a system that allows users to create original story picture books while interacting with a generative AI model. This system includes a user terminal, a server, and a generative AI model, and describes the program processing.

[0190] System Overview

[0191] The system of the present invention receives input from the user's device, sends it to the generative AI model via the server, and provides the output from the AI ​​back to the user. Through this process, users (especially children) can experience creating their own story picture book.

[0192] User terminal

[0193] The user device (e.g., smartphone, tablet) provides an interface for inputting the story title and theme. It also has the function of displaying the generated plot and story development. The user device has the following functions:

[0194] 1. Input interface: Provides a form for users to enter the title and theme of the story.

[0195] 2. Display interface: Displays plot suggestions from the generative AI model and story developments entered by the user.

[0196] 3. Editing function: Provide an editor that allows users to edit, modify, and save stories.

[0197] 4. Binding Request: Provides an interface for submitting completed stories as binding requests.

[0198] server

[0199] The server is responsible for sending input data from the user to the generative AI model and returning output data from the generative AI model to the user's device. The hardware used is a cloud server (e.g., AWS (registered trademark), Google (registered trademark) Cloud), and the software used is Flask (a lightweight web server framework built in Python). The server has the following functions:

[0200] 1. User authentication: Verify login information and perform authentication.

[0201] 2. Data processing and analysis: Receives input data from the user and passes it to the generative AI model.

[0202] 3. Data storage: Store the stories created by users in a database.

[0203] 4. Generative AI model integration: Send data to the generative AI model and receive the results.

[0204] 5. Binding Request Management: Receive binding requests and coordinate with the binding service.

[0205] 6. Content Distribution: Distributing the generated plot or storyline as a content distribution service.

[0206] Generative AI Models

[0207] Generative AI models (e.g., OpenAI GPT) generate plot suggestions and storylines based on user input data. Generative AI models have the following capabilities:

[0208] 1. Initial plot generation: Generate an initial plot based on the title and theme entered by the user.

[0209] 2. Story development suggestions: Based on the story development input by the user, new ideas and developments are suggested.

[0210] Specific examples

[0211] Let's say a child wants to create a picture book with the theme of "Adventure Story." The user (child) starts the application on the user's device and enters their login information. After successful login, the home screen is displayed and the child clicks the "Create a new story" button. Next, in the form that appears, the child enters the story title "Adventure with a Dragon" and the theme "Adventure," and presses the submit button.

[0212] The server stores the title and theme received from the user in a database and sends a request to the generative AI model to generate an initial plot. When the generative AI model receives the initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure..."), it sends the content to the user's device.

[0213] The user device displays the received initial plot, and the child can think up a new storyline. For example, they can input and submit a story in which a boy named Taro appears, becomes friends with a dragon, and goes on a treasure hunt together.

[0214] The server sends the new storyline input by the child to the generative AI model again and receives the next suggestion. When the server receives a new storyline from the AI ​​model (e.g., "Taro and Dragon find a treasure chest in a cave deep in the mountains, and inside is an old map..."), it sends it to the user's device.

[0215] After the story is completed, the user device displays a button for the child to send the completed story as a bookbinding request. When the child clicks the button, the request is sent to the server, which then works with the bookbinding service to arrange for the picture book to be produced and delivered.

[0216] An example of a prompt is:

[0217] "Title: The Brave Girl and the Magical Forest

[0218] Theme: Adventure

[0219] Generate the plot."

[0220] The format is as follows.

[0221] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0222] Step 1:

[0223] The user device launches the application and enters login information. The user device sends the entered login information to the server, which then authenticates the user. If authentication is successful, the server returns a successful authentication response to the user device. The user device then displays the home screen.

[0224] Step 2:

[0225] The user clicks the "Create a new story" button, which displays a form for entering the story title and theme. The user enters the title and theme in the form and presses the submit button. The user's device sends the input data to the server. The server stores the received title and theme in a database and sends a request to the generative AI model to generate an initial plot.

[0226] Step 3:

[0227] The generative AI model generates an initial plot based on the title and theme received from the user. The generative AI model returns the generated initial plot to the server. After receiving the plot, the server sends its contents to the user's device. The user's device displays the initial plot.

[0228] Step 4:

[0229] The user thinks of a new storyline and inputs the content they want to add. The user's device sends the user's input to the server. The server then sends the input data back to the generative AI model and requests a storyline proposal.

[0230] Step 5:

[0231] The generative AI model proposes a new storyline. The generative AI model generates a storyline and returns the result to the server. The server receives the result and sends the next storyline to the user's device. The user's device displays the generated storyline.

[0232] Step 6:

[0233] The user edits and saves the final story. The user terminal sends the edited story to the server, which stores it in the database. When saving is complete, the server sends a confirmation message to the user terminal.

[0234] Step 7:

[0235] The user clicks a button to request binding. The user terminal sends the request to the server, which receives the request. The server then works with the binding service to arrange for the production and delivery of the picture book. The server also processes the generated plot or story development to distribute to the content distribution service.

[0236] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0237] This invention combines an emotion engine with a system that allows users to create original story picture books while interacting with a generative AI model. Specifically, we explain the configuration of the system, which includes a user terminal, a server, a generative AI model, and an emotion engine, and how each element works together.

[0238] System Overview

[0239] The system of the present invention receives input from a user terminal, sends it to a generative AI model and an emotion engine via a server, and provides the output from the AI ​​and emotion engine back to the user terminal. This allows users (especially children) to experience creating their own story picture books. The following is a detailed description of the system's components and their functions.

[0240] User terminal

[0241] The user terminal is a device that provides an interface for creating stories. Its main functions are as follows:

[0242] 1. Input interface:

[0243] Provide a form for children to enter the title and theme of their story.

[0244] 2. Display interface:

[0245] It displays plot suggestions from generative AI models and storylines input by users.

[0246] 3. Editing features:

[0247] It provides an editor where users can edit, modify and save stories.

[0248] 4. Binding Request:

[0249] Provides an interface for submitting completed stories as binding requests.

[0250] 5. Emotion recognition:

[0251] It has the ability to analyze the user's facial expressions and voice and send the data to the emotion engine.

[0252] server

[0253] The server is responsible for sending input data from users to the generative AI model and emotion engine, and returning output data to the user device. Its main functions are as follows:

[0254] 1. User authentication:

[0255] Verify the user's login information and perform authentication.

[0256] 2. Data processing and analysis:

[0257] User input data is passed to the emotion engine and generative AI model.

[0258] 3. Data Retention:

[0259] Store user-created stories in a database.

[0260] 4. Collaboration with generative AI models and emotion engines:

[0261] The title, theme, and user emotion data are sent to the emotion engine and generative AI model, and the results are returned to the user's device.

[0262] 5. Binding Request Management:

[0263] Receives binding requests and coordinates with binding services.

[0264] Generative AI Models

[0265] The generative AI model generates plot suggestions and story developments based on user input data and emotional data sent from the emotion engine.

[0266] 1. Initial plot generation:

[0267] Generates an initial plot based on the title and theme entered by the user.

[0268] 2. Storyline proposal:

[0269] Based on the story development entered by the user, new ideas and developments are suggested.

[0270] Emotion Engine

[0271] The emotion engine analyzes emotions from the user's facial expressions and voice and passes that data to a generative AI model.

[0272] 1. Sentiment analysis:

[0273] Performs real-time emotion analysis of the user's facial expressions and voice.

[0274] 2. Emotion data generation:

[0275] Emotional data is generated based on the analysis results and passed to the generative AI model.

[0276] Specific examples

[0277] Imagine a child is creating a picture book with the theme of "adventure story."

[0278] Operation on the user device

[0279] The user (child) starts the application on the user's device and enters their login information. After successful login, the home screen is displayed and the child clicks the "Create a new story" button. In the form that is then displayed, the child enters the story title "Adventure with a Dragon" and the theme "Adventure" and presses the submit button.

[0280] Processing on the server

[0281] The server stores the title and theme received from the user in a database and sends the data to the generative AI model and emotion engine. The emotion engine analyzes the user's facial expressions and voice data and passes the generated emotion data to the generative AI model.

[0282] How generative AI models work

[0283] The generative AI model generates an initial plot based on the title, theme, and emotional data, and sends the generated initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure...") back to the server.

[0284] Collaboration between the server and the emotion engine

[0285] The server sends the initial plot received from the generative AI model to the terminal and makes user-level adjustments, taking into account feedback from the emotion engine.

[0286] Viewing and editing on user devices

[0287] The user's device displays the initial plot, and the child thinks up and inputs a new storyline (e.g., "Taro becomes friends with the dragon and they go on a treasure hunt together"). At this time, the emotion engine analyzes the user's emotions from facial expressions and voice and provides feedback in real time. Once input is complete, the child clicks the send button.

[0288] Reconnecting the server and the generative AI model

[0289] The server resends new developments input from the user or emotion engine to the generative AI model, receives new proposals, and the generative AI model generates the next proposal and sends the result back to the server.

[0290] Submit a binding request

[0291] After the story is completed, the user device displays a button to request the completed story to be bound. When the child clicks the button, the request is sent to the server, which then works with the binding service to arrange for the book to be produced and delivered.

[0292] This is the process by which children create their own original story picture books using the system of this invention. By combining it with an emotion engine, dynamic story development and feedback are provided according to the user's emotions, creating a richer, more interactive experience.

[0293] The processing flow will be explained below.

[0294] Step 1:

[0295] The user starts the service application on the user device and the login screen is displayed. The child enters the user name and password and clicks the login button.

[0296] Step 2:

[0297] The server receives the login information and performs user authentication. If authentication is successful, the server starts a user session and returns the home screen data to the device.

[0298] Step 3:

[0299] The device will display a home screen and offer a "Create a new story" button that the child can click to display a form where they can enter a title and theme.

[0300] Step 4:

[0301] The user enters a story title (e.g., "Adventure with a Dragon") and a theme (e.g., "Adventure") and clicks the submit button.

[0302] Step 5:

[0303] The server stores the input title and theme in a database and sends a request to generate an initial plot to the generative AI model. At the same time, the server sends the user's facial expressions and voice data from the user's device to the emotion engine.

[0304] Step 6:

[0305] The generative AI model generates an initial plot based on the title and theme and sends the results back to the server. The emotion engine also analyzes the user's facial expressions and voice in real time to generate emotion data.

[0306] Step 7:

[0307] The server integrates the initial plot received from the generative AI model with the emotion data from the emotion engine and sends it to the terminal.

[0308] Step 8:

[0309] The device displays an initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure..."), which the user reads. The user then inputs a new storyline (e.g., "Taro becomes friends with the dragon and they go on a treasure hunt together") and submits it. At the same time, the emotion engine continues to analyze the user's emotions from their facial expressions and voice.

[0310] Step 9:

[0311] The server receives the newly input storyline and emotion data from the user and sends it to the generative AI model, which then proposes the next storyline.

[0312] Step 10:

[0313] The generative AI model generates the next proposal based on the new storyline and sends the result back to the server (e.g., "Tarro and Dragon find a treasure chest in a cave deep in the mountains, inside which is an old map").

[0314] Step 11:

[0315] The server receives the next suggestion from the generative AI model and sends it to the user device and the emotion engine, which continues to monitor the user's reactions and analyze the emotion data.

[0316] Step 12:

[0317] The device displays new suggestions, and the user can view them and edit the story. Once edits are complete, the user clicks the save button. During this time, the emotion engine continues to analyze emotions.

[0318] Step 13:

[0319] The server receives the final edited story and emotion data from the user and stores it in a database.

[0320] Step 14:

[0321] The user checks the completed story and clicks the binding request button.

[0322] Step 15:

[0323] The server receives the bookbinding request, stores it in a database, and sends the request to the bookbinding service. After receiving confirmation from the bookbinding service, the server arranges for the production and delivery of the picture book.

[0324] Step 16:

[0325] Users receive a bound picture book delivered to their home and enjoy their own original story.

[0326] These are the specific processing steps for creating a story picture book using the system of the present invention. By combining it with an emotion engine, it is possible to provide an interactive story based on the user's emotions, providing a richer experience.

[0327] Example 2

[0328] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0329] In today's digital society, it is important to provide children with the opportunity to express their creativity and create original story picture books. However, existing systems are unable to reflect users' emotions in real time, limiting the interactive story creation experience. Furthermore, the process of binding the created stories is time-consuming, making it difficult for users to use.

[0330] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0331] In this invention, the server includes a means including an emotion engine for analyzing user emotions, a means for transmitting data to the generative AI model, and a means for returning the results to the user terminal. This allows users to receive real-time emotional feedback as they create stories, enabling a richer, more interactive story creation experience. Furthermore, requests for binding the generated stories are efficiently processed, improving user convenience.

[0332] A "user terminal" is a device that allows a user to input and edit the title and theme of a story.

[0333] A "generative AI model" is an artificial intelligence that generates an initial plot based on the title and theme entered by the user and suggests new story developments.

[0334] An "initial plot" is a story outline initially created by a generative AI model based on the title and theme entered by the user.

[0335] The "emotion engine" is a system element that analyzes the user's facial expressions and voice, generates emotional data, and passes it to the generative AI model.

[0336] A "plot" is a basic component that shows the development and progression of a story.

[0337] "Emotion data" is data that indicates an emotional state generated by an emotion engine from the user's facial expressions and voice.

[0338] A "binding request" is a request that a user submits to have a completed story bound into a physical book.

[0339] "Saving" means recording the final story edited by the user in a storage device such as a database.

[0340] "Real-time feedback" means that the emotion engine analyzes the user's emotional state in real time and immediately reflects the results to the user.

[0341] This invention combines an emotion engine with a system that allows users to create original story picture books while interacting with a generative AI model. Specifically, we explain the configuration of the system, which includes a user terminal, a server, a generative AI model, and an emotion engine, and how each element works together.

[0342] System Overview

[0343] The system of the present invention receives input from a user terminal, sends it to a generative AI model and emotion engine via a server, and provides the output from the AI ​​and emotion engine back to the user terminal, thereby providing users (especially children) with the experience of creating their own story picture books.

[0344] User terminal

[0345] The user terminal is a device that provides an interface for creating stories. Its main functions are as follows:

[0346] 1. Input interface:

[0347] Provide a form for the child to enter the title and theme of the story. For example, enter the title "Adventure with a Dragon" and the theme "Adventure."

[0348] 2. Display interface:

[0349] It displays plot suggestions from generative AI models and storylines input by users.

[0350] 3. Editing features:

[0351] It provides an editor where users can edit, modify and save stories. Users can add their own ideas or modify existing stories.

[0352] 4. Binding Request:

[0353] It provides a button to submit the completed story as a binding request. Clicking the button sends the request to the server.

[0354] 5. Emotion recognition:

[0355] It has the ability to analyze the user's facial expressions and voice and send that data to the emotion engine.

[0356] server

[0357] The server is responsible for sending input data from users to the generative AI model and emotion engine, and returning output data to the user device. Its main functions are as follows:

[0358] 1. User authentication:

[0359] Verify and authenticate the user's login information. For example, the user enters login information and the server verifies it.

[0360] 2. Data processing and analysis:

[0361] User input data is passed to the emotion engine and generative AI model.

[0362] 3. Data Retention:

[0363] Store user-created stories in a database.

[0364] 4. Collaboration with generative AI models and emotion engines:

[0365] The title, theme, and user emotion data are sent to the emotion engine and generative AI model, and the results are returned to the user's device.

[0366] 5. Binding Request Management:

[0367] Receives binding requests and coordinates with binding services.

[0368] Generative AI Models

[0369] The generative AI model generates plot suggestions and story developments based on user input data and emotional data sent from the emotion engine.

[0370] 1. Initial plot generation:

[0371] Generates an initial plot based on the title and theme entered by the user. For example, create a plot based on "Adventures with Dragons."

[0372] 2. Storyline proposal:

[0373] Based on the story development entered by the user, new ideas and developments are suggested.

[0374] Emotion Engine

[0375] The emotion engine analyzes emotions from the user's facial expressions and voice and passes that data to a generative AI model.

[0376] 1. Sentiment analysis:

[0377] Performs real-time emotion analysis of the user's facial expressions and voice.

[0378] 2. Emotion data generation:

[0379] Emotional data is generated based on the analysis results and passed to the generative AI model.

[0380] Specific examples

[0381] Operation on the user device

[0382] The user (child) launches the application on their device and enters their login information. After successfully logging in, the home screen appears and the child clicks the "Create a new story" button. In the form that appears next, the child enters "Adventure with a Dragon" and "Adventure" and presses the submit button.

[0383] Processing on the server

[0384] The server stores the title and theme received from the user in a database and sends the data to the generative AI model and emotion engine. The emotion engine analyzes the user's facial expressions and voice data and passes the generated emotion data to the generative AI model.

[0385] How generative AI models work

[0386] The generative AI model generates an initial plot based on the title, theme, and emotional data, for example, "One day, a brave boy meets a dragon who is guarding a secret treasure," and sends it back to the server.

[0387] Collaboration between the server and the emotion engine

[0388] The server sends the initial plot received from the generative AI model to the user's device and makes adjustments taking into account feedback from the emotion engine.

[0389] Viewing and editing on user devices

[0390] The user's device displays the initial plot, and the user can think of and input a new story development. For example, they can input "Taro becomes friends with a dragon and they go on a treasure hunt together." At this time, the emotion engine analyzes the user's emotions from facial expressions and voice and provides feedback in real time. Once input is complete, the user clicks the send button.

[0391] Reconnecting the server and the generative AI model

[0392] The server resubmits the new deployment data to the generative AI model, which then generates the next proposal and sends the result back to the server.

[0393] Submit a binding request

[0394] After the story is completed, the user's device displays a button for sending the completed story as a bookbinding request. When the user clicks the button, the request is sent to the server, which then works with the bookbinding service to produce the picture book and arrange for its delivery.

[0395] Examples of prompt statements

[0396] As an example of a prompt for a generative AI model, the following could be considered as input to generate the initial plot of a story themed around "adventures with dragons":

[0397] "The user has entered the title 'Adventure with the Dragon' and the theme 'Adventure'. Based on this, please generate an initial plot for the story, starting with the scene where the boy meets the dragon."

[0398] This concludes the detailed explanation of how to use the system of the present invention. This system allows users to interactively create original stories while receiving emotional feedback, and ultimately to complete the process of bookbinding.

[0399] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0400] Step 1:

[0401] Log in

[0402] The user accesses the login screen and enters their login information. The server receives the login information (input) sent by the user and authenticates it by comparing it with information in the database (data calculation). If authentication is successful, the home screen is displayed on the user's device (output).

[0403] Step 2:

[0404] Start creating your story

[0405] The user clicks the "Create a new story" button on the home screen. The user's device detects the click event (input) and displays a new story creation form (output).

[0406] Step 3:

[0407] Enter the title and theme

[0408] The user enters the story title and theme on a new screen. The user's device receives the title "Adventure with a Dragon" and theme "Adventure" entered by the user through an input form (input). The information entered into the form is temporarily stored in the device's memory.

[0409] Step 4:

[0410] Sending input data

[0411] The user presses the send button to send the data to the server. The user's device sends the title and theme data (input) to the server and displays a notification of successful transmission to the user (output).

[0412] Step 5:

[0413] User Authentication

[0414] The server checks the user's login information and performs authentication. If authentication is successful, the server saves the user data in the database (data calculation).

[0415] Step 6:

[0416] Data storage

[0417] The server stores the received title and theme in a database (output). The server records this data in a database in an appropriate format for further processing.

[0418] Step 7:

[0419] Sending data to a generative AI model

[0420] The server sends the title and theme data to the generative AI model. It sends the title "Adventure with Dragons" and the theme "Adventure" (input) to the generative AI model via API, and receives the generated initial plot (output).

[0421] Step 8:

[0422] Sending data to the emotion engine

[0423] The server sends the user's emotional data to the emotion engine. It also sends facial expression and voice data (input) acquired from the user's device to the emotion engine and receives the analysis results (output).

[0424] Step 9:

[0425] Receiving plots from generative AI models

[0426] The server receives the initial plot from the generative AI model. The initial plot data (output) is "One day, a brave boy meets a dragon. The dragon guards a secret treasure..." and is used for the next process.

[0427] Step 10:

[0428] Receiving feedback from the emotion engine

[0429] The server receives feedback from the emotion engine and understands the user's emotional state. Based on the feedback data (output), it modifies the output results of the generative AI model (data calculation).

[0430] Step 11:

[0431] Displaying the initial plot

[0432] The user terminal displays the initial plot to the user, displays the received initial plot data (input) on the screen, and asks the user for feedback (output).

[0433] Step 12:

[0434] User edits

[0435] The user thinks up and inputs a new story development based on the initial plot. The user's input data, "Taro becomes friends with the dragon and goes on a treasure hunt together" (input), is acquired by the user's device and temporarily saved as edited data.

[0436] Step 13:

[0437] Send edited data

[0438] The user resends the edited data to the server. The user terminal sends the edited new story development data (input) to the server and displays a notification that transmission is complete (output).

[0439] Step 14:

[0440] Resubmit to generative AI model

[0441] The server resubmits new story development data to the generative AI model. It sends new story development data (input) via API and receives new proposals from the generative AI model (output).

[0442] Step 15:

[0443] Receiving a resubmission

[0444] The server receives new proposals from the generative AI model. It receives new story development proposal data (output) from the generative AI model and uses it for further processing.

[0445] Step 16:

[0446] View the final story

[0447] The user terminal displays the final story development to the user. The generated final story data (input) is displayed on the screen and the user is prompted to confirm it (output).

[0448] Step 17:

[0449] Submit a binding request

[0450] The user submits the completed story as a binding request. The user terminal transmits the binding request data (input) to the server and displays a notification of completion of transmission (output).

[0451] Step 18:

[0452] Collaboration with bookbinding services

[0453] The server receives the binding request and works with the binding service to produce and arrange for delivery of the picture book. It sends the binding request data (input) to the binding service and records the binding progress in a database (output).

[0454] (Application example 2)

[0455] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0456] Conventional story creation systems lacked the ability to provide feedback based on the user's emotions or adjust the storyline, which prevented them from fully enriching the user's experience. Furthermore, they lacked the interactive elements that children needed to have fun creating stories, which led to a lack of interest.

[0457] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting the title and theme of the story from a user terminal, means for generating an initial plot using a generative AI model, means for displaying the plot received from the generative AI model on the user terminal, means for passing the story development input by the user to the generative AI model, means for displaying a new development proposed by the generative AI model on the user terminal, means for saving the final story edited by the user, means for processing a binding request based on the saved story, and means for adjusting the story feedback displayed on the user terminal based on the emotion, including an emotion engine that analyzes the user's facial expressions and voice to generate emotion data. This allows for dynamic story development and feedback in accordance with the user's emotions, enabling a richer and more interactive experience.

[0458] A "user terminal" is a device that provides an interface for creating a story and receives input from a user.

[0459] A "generative AI model" is an artificial intelligence that generates an initial plot or new story development based on user input data.

[0460] The "initial plot" is the initial story development that is generated based on the story title and theme entered by the user.

[0461] A "plot" indicates the story's outline and development pattern, and is generated by a generative AI model.

[0462] Theme refers to the overall theme or central idea of ​​the story.

[0463] The "emotion engine" is an engine that analyzes emotions from the user's facial expressions and voice, and adjusts the story development through the generated emotional data.

[0464] A "bookbinding request" is a procedure in which a user requests that a completed story be produced as a physical picture book.

[0465] "Emotion data" refers to emotional information analyzed from the user's facial expressions and voice, and is reflected in the story development.

[0466] "Interface" refers to the screen display and input methods on a user terminal that provide the means for a user to interact with a system.

[0467] "Feedback" refers to the information or suggestions that a system returns in response to a user's input or behavior.

[0468] To implement this invention, a user terminal on which a dedicated application is installed, a server located on the cloud, a generative AI model, and an emotion engine are used. Details of these are described below.

[0469] Overall system configuration

[0470] The system includes the following major components:

[0471] 1. User Device:

[0472] Input interface: Accepts title and theme input from the user.

[0473] Display interface: Shows the plot from the generative AI model and the final story.

[0474] Editing capabilities: Users can create, edit, and save stories.

[0475] Emotion Recognition: Uses a camera and microphone to capture the user's facial expressions and voice to generate emotional data.

[0476] Bookbinding Requests: Processing requests to have completed stories produced as physical picture books.

[0477] 2. Server:

[0478] User authentication: Verify the user's login information and perform authentication.

[0479] Data processing and analysis: Sending user input data to the emotion engine and generative AI model.

[0480] Data storage: Store user-created stories in a database.

[0481] Collaboration with generative AI models and emotion engines: Processes title, theme, and emotion data and returns the results to the user's device.

[0482] Binding Request Management: Send binding requests to binding services.

[0483] 3. Generative AI Models (e.g., GPT-4(R)):

[0484] Initial plot generation: Generates the basic plot of your story based on the title and theme.

[0485] Story development proposal: Based on new development input from the user, the next development proposal is made.

[0486] 4. Emotion engine (e.g. Affectiva SDK):

[0487] Sentiment analysis: Analyzes the user's facial expressions and voice to generate emotional data.

[0488] Emotional feedback: Adjust the storyline based on the generated emotional data.

[0489] Software and Hardware Configuration

[0490] User device: Smartphone application (e.g., using React Native)

[0491] Server: Node.js, Express framework, database (e.g. MongoDB)

[0492] Emotion recognition: Expo Camera, Expo Audio

[0493] Generative AI model: OpenAI GPT-4 API

[0494] Data processing flow

[0495] 1. Getting user input:

[0496] The user enters the story title and theme on their device and sends it to the server.

[0497] 2. Emotion capture and analysis:

[0498] It uses the camera and microphone on the user's device to capture facial expressions and voice.

[0499] The captured data is sent to the emotion engine via the server, where emotions are analyzed.

[0500] 3. Collaboration with generative AI models:

[0501] The server sends user input and emotion data to the generative AI model.

[0502] The generative AI model generates an initial plot and sends it to the user's device via the server.

[0503] 4. Providing User Feedback:

[0504] As users edit the storyline, the emotion engine provides real-time emotional feedback that is sent to the generative AI model.

[0505] The generative AI model generates new suggestions, which are displayed on the user's device.

[0506] 5. Processing binding requests:

[0507] Users submit completed stories as binding requests.

[0508] The server works with the bookbinding service to create the physical picture book and arrange for delivery.

[0509] Examples and prompts

[0510] Title: "The Adventures of Brave Taro"

[0511] Theme: Adventure

[0512] Scene: "Tarro meets a dragon and becomes friends with him."

[0513] Example prompts to input to a generative AI model:

[0514] Title: The Adventures of the Brave Taro

[0515] Theme: Adventure

[0516] Feedback emotion: Fun

[0517] Story Current Scene: Taro meets a dragon and they go on a treasure hunt together to become friends.

[0518] Please suggest the following expansion:

[0519] In this way, a system can be implemented that provides users with a fun, emotionally driven story-creation experience.

[0520] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0521] Step 1:

[0522] The user inputs the title and theme of the story. The user's device receives these inputs and sends them to the server.

[0523] Input: User title and theme

[0524] Output: Title and theme sent to server

[0525] Specific operation: When a user enters characters into an input field on the user terminal and presses the send button, this data is sent to the server.

[0526] Step 2:

[0527] The server sends the title and theme to the generative AI model, which generates the initial plot and sends the results back to the server.

[0528] Input: Title and Theme

[0529] Output: Initial plot

[0530] What it does: The server sends the title and theme to the generative AI model and receives the generated initial plot.

[0531] Step 3:

[0532] The server sends the initial plot to the user terminal, which displays it.

[0533] Input: Initial plot

[0534] Output: Initial plot displayed on the user's terminal

[0535] Specific operation: The initial plot received from the server is displayed on the display screen of the user's terminal.

[0536] Step 4:

[0537] The user inputs the continuation of the story. The user's device receives this input and sends it to the server.

[0538] Input: User storyline

[0539] Output: Sending the storyline to the server

[0540] Specific behavior: When the user enters a storyline into the input field and presses the submit button, it is sent to the server.

[0541] Step 5:

[0542] The user device captures the user's facial expressions and voice in real time and sends them to the emotion engine for analysis.

[0543] Input: User's facial expressions and voice

[0544] Output: Emotion data

[0545] Specific operation: Uses the camera and microphone on the user's device to capture facial expression and voice data and sends it to the emotion engine.

[0546] Step 6:

[0547] The emotion engine analyzes the user's facial expressions and voice to generate emotion data, which is then sent to the server.

[0548] Input: facial expression data and voice data

[0549] Output: Emotion data

[0550] Specific operation: The emotion engine analyzes the received data, determines the user's emotions, generates emotion data, and sends it to the server.

[0551] Step 7:

[0552] The server sends the emotional data and the user's storyline to a generative AI model, which then generates a proposal for the next storyline.

[0553] Input: Emotion data and storyline

[0554] Output: Next storyline proposal

[0555] Specific operation: The server sends the story development and emotional data from the user to the generative AI model, which then generates a proposal for the next development and returns it to the server.

[0556] Step 8:

[0557] The server transmits the next story development proposal to the user terminal, which displays it.

[0558] Input: Next storyline suggestion

[0559] Output: Next expansion suggestion displayed on user terminal

[0560] Specific operation: The proposal received from the server is sent to the user terminal, and is displayed on the display screen of the user terminal.

[0561] Step 9:

[0562] The user completes the story and submits a binding request, which the user terminal sends to the server.

[0563] Input: Final story

[0564] Output: Binding request

[0565] Specific behavior: The user checks the final story and presses the binding request button, which sends the request to the server.

[0566] Step 10:

[0567] The server receives the binding request, works with the binding service to produce the picture book, and arranges for delivery.

[0568] Input: Binding request

[0569] Output: Physical storybook

[0570] Specific operation: After the server receives the binding request, it requests the binding service to produce and deliver the picture book.

[0571] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0572] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0573] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0574] [Second embodiment]

[0575] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0576] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0577] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0578] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0579] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0580] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0581] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0582] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0583] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0584] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0585] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0586] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0587] This paper describes a system that allows users to create original story picture books while interacting with a generative AI model. Specifically, it describes the configuration of the system, which includes a user terminal, a server, and a generative AI model, and the processing of the program.

[0588] System Overview

[0589] The system of the present invention receives input from the user's device, sends it to the generative AI model via the server, and then provides the output from the AI ​​back to the user. Through this process, users (especially children) are given the experience of creating their own story picture book. The system configuration and the functions of each element are described in detail below.

[0590] User terminal

[0591] The user device (e.g., PC, smartphone, tablet) provides an interface for the child to input the title and theme of the story.

[0592] 1. Input interface:

[0593] Provide a form for children to enter the title and theme of their story.

[0594] 2. Display interface:

[0595] Displays plot suggestions from AI and story developments entered by users.

[0596] 3. Editing features:

[0597] It provides an editor where users can edit, modify and save stories.

[0598] 4. Binding Request:

[0599] Provides an interface for submitting completed stories as binding requests.

[0600] server

[0601] The server is responsible for sending input data from users to the generative AI model and returning output data from the generative AI model to the user's device. It also stores data and manages binding requests. Its main functions are as follows:

[0602] 1. User authentication:

[0603] Verify your login information and authenticate.

[0604] 2. Data processing and analysis:

[0605] Receives input data from the user and passes it to the generative AI model.

[0606] 3. Data Retention:

[0607] Store user-created stories in a database.

[0608] 4. Generative AI model collaboration:

[0609] Send data to the AI ​​model and receive the results.

[0610] 5. Binding Request Management:

[0611] Receives binding requests and coordinates with binding services.

[0612] Generative AI Models

[0613] The generative AI model generates plot suggestions and storylines based on user input data.

[0614] 1. Initial plot generation:

[0615] Generates an initial plot based on the title and theme entered by the user.

[0616] 2. Storyline proposal:

[0617] Based on the story development entered by the user, new ideas and developments are suggested.

[0618] Specific examples

[0619] Imagine a child is creating a picture book with the theme of "adventure story."

[0620] Operation on the user device

[0621] The user (child) launches the application on the user's device and enters their login information. If login is successful, the home screen is displayed and the child clicks the "Create a new story" button. Next, they enter the story title "Adventure with a Dragon" and the theme "Adventure" in the form that appears, and press the submit button.

[0622] Processing on the server

[0623] The server stores the title and theme received from the user in a database and sends a request to the generative AI model to generate an initial plot. When the generative AI model receives the initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure..."), it sends the content to the user's device.

[0624] Viewing and editing on user devices

[0625] The user device displays the received initial plot, and the child can think up a new storyline. For example, they can input and submit a story in which a boy named Taro appears, becomes friends with a dragon, and goes on a treasure hunt together.

[0626] Collaboration between the server and the generative AI model

[0627] The server sends the new storyline input by the child to the generative AI model again and receives the next suggestion. When the server receives a new storyline from the AI ​​model (e.g., "Taro and Dragon find a treasure chest in a cave deep in the mountains, and inside is an old map..."), it sends it to the user's device.

[0628] Submit a binding request

[0629] After the story is completed, the user device displays a button for the child to send the completed story as a bookbinding request. When the child clicks the button, the request is sent to the server, which then works with the bookbinding service to arrange for the picture book to be produced and delivered.

[0630] This is the process by which children can create their own original story picture books using the system of the present invention, which not only allows children to exercise their creativity and learn while having fun, but also gives them a sense of accomplishment by having their own unique picture book.

[0631] The processing flow will be explained below.

[0632] Step 1:

[0633] The user starts the service application on the user device and the login screen is displayed. The child enters the user name and password and clicks the login button.

[0634] Step 2:

[0635] The server receives the login information and performs authentication. If authentication is successful, the server starts the user session and returns the home screen data to the device.

[0636] Step 3:

[0637] The device will display a home screen and offer a "Create a new story" button that the child can click to display a form where they can enter a title and theme.

[0638] Step 4:

[0639] The user enters a story title (e.g., "Adventure with a Dragon") and a theme (e.g., "Adventure") and clicks the submit button.

[0640] Step 5:

[0641] The server receives the input title and theme, stores them in a database, and then sends a request to the generative AI model to generate the initial plot.

[0642] Step 6:

[0643] The generative AI model generates an initial plot based on the title and theme, and sends the generated plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure...") back to the server.

[0644] Step 7:

[0645] The server sends the initial plot received from the generative AI model to the terminal.

[0646] Step 8:

[0647] The device displays the initial plot. The user (child) reads the plot and thinks of a new storyline to input (e.g., "Taro becomes friends with the dragon and they go on a treasure hunt together."). When they're done, they click the send button.

[0648] Step 9:

[0649] The server receives the new storyline input from the user and sends it back to the generative AI model, requesting new proposals.

[0650] Step 10:

[0651] The generative AI model generates the next proposal based on the new storyline and sends the result back to the server (e.g., "Tarro and Dragon find a treasure chest in a cave deep in the mountains, inside which is an old map").

[0652] Step 11:

[0653] The server sends the next proposal received from the generative AI model to the device.

[0654] Step 12:

[0655] The device will display a new suggestion, and the user can view the suggestion and edit the story. Once they've finished editing, they can click the save button again. When the final story is complete, they'll be given the option to request binding.

[0656] Step 13:

[0657] The user checks the completed story and clicks the binding request button.

[0658] Step 14:

[0659] The server receives the bookbinding request, stores it in a database, and sends the request to the bookbinding service. After receiving confirmation from the bookbinding service, the server arranges for the production and delivery of the picture book.

[0660] Step 15:

[0661] Users receive a bound picture book delivered to their home and enjoy their own original story.

[0662] Example 1

[0663] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0664] Conventional story creation systems require users to do a lot of manual work when creating a story, making the creation process complicated and time-consuming. They also lacked adequate support to fully bring out users' creativity, especially functions that allow children to enjoy creative activities. Furthermore, the process of binding completed stories into books was complicated, making it difficult for users to easily obtain the finished product.

[0665] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0666] In this invention, the server includes a means for transmitting data input from a user terminal to the server, a means for the server to authenticate the user and store the data in a database, and a means for the server to transmit a request to the generative AI model and receive the generated plot. This allows the user to intuitively input data for story creation, utilize the generative AI model to propose and edit the story development, and finally easily request bookbinding.

[0667] A "user terminal" is a device operated by a user, and examples include PCs, smartphones, tablets, etc.

[0668] "Input means" refers to an interface for a user to input data, and includes a keyboard, a touch screen, a microphone, etc.

[0669] A "generative AI model" is a system that uses artificial intelligence to generate data, specifically a model that generates a plot or storyline based on a prompt.

[0670] "Initial plot generator" refers to a process for generating an initial plot for a story based on a title and theme entered by a user.

[0671] "Display means" refers to an interface for visually displaying data on a user terminal, including a screen or monitor.

[0672] "Storage" refers to a method for long-term storage of data, including databases and file systems.

[0673] "Bookbinding request processing means" refers to a system that processes requests from users to bind their completed stories into actual books.

[0674] A "server" refers to a computer or a series of computer systems that connects to multiple user terminals via a network and processes and stores data.

[0675] "Authentication measures" refers to the process of verifying a user's identity and controlling access to a system, including verifying user IDs and passwords.

[0676] A "database" refers to a system for efficiently storing, retrieving, and managing structured data.

[0677] "Request sending means" refers to a method for sending a user's request to a generative AI model or other system component.

[0678] The present invention relates to a system in which a user creates an original story picture book while interacting with a generative AI model. The system includes a user terminal, a server, and a generative AI model, each of which plays a specific role.

[0679] User terminal

[0680] The user device provides an interface for users (especially children) to input the title and theme of the story. It also has an editor function that displays the plot and story development generated by the generative AI model and allows users to edit, modify, and save the story. User devices can be PCs, smartphones, tablets, or other devices.

[0681] As a concrete example, a user (child) starts an application on the device and enters login information. After successful login, the user clicks the "Create a new story" button, enters the title "Adventure with a Dragon" and the theme "Adventure" in the form, and presses the submit button.

[0682] server

[0683] The server receives input data from user devices and sends it to the generative AI model. It also authenticates users, stores data, sends plot generation requests, and manages binding requests. The server uses standard server hardware and database software (e.g., MySQL, PostgreSQL).

[0684] The server receives the title and theme entered by the user, stores them in a database, and sends a request to the generative AI model to generate an initial plot. For example, the server sends the following prompt to the generative AI model:

[0685] Title: Adventure with Dragons

[0686] Theme: Adventure

[0687] Prompt: One day, a brave boy meets a dragon who guards a hidden treasure...

[0688] Generative AI Models

[0689] The generative AI model generates a plot and storyline based on the input title and theme. For example, OpenAI's GPT-3 model is used. The generative AI model generates an initial plot based on prompts received from the server and sends the plot back to the server.

[0690] A generative AI model might generate a plot like this:

[0691] One day, a brave boy meets a dragon who is guarding a secret treasure. The boy and the dragon decide to join forces to search for the treasure...

[0692] A concrete example of the program's operation flow

[0693] When a user inputs the story title and theme, the user's device sends the data in JSON format to the server. The server stores the data in a database and sends a prompt to the generative AI model to generate an initial plot. The server receives the plot returned by the generative AI model and sends it to the user's device for display.

[0694] When a user begins to input a new storyline, that data is also sent via the server to the generative AI model, which then generates a new storyline. By repeating this process, the user and the generative AI model work together to create a unique story picture book. Once completed, the user submits a binding request, and the server connects the request to the binding service to produce and deliver the picture book.

[0695] Specific technical features

[0696] This system uses a generative AI model to automatically generate plots and story developments, and is equipped with the ability to efficiently process, store, and display data entered by users. The server acts as a central link between the user's device and the generative AI model, ensuring a smooth story creation process. It is also designed to automate the processing of binding requests, allowing users to easily obtain physical picture books.

[0697] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0698] Step 1:

[0699] The user uses the user terminal to input the title and theme of the story. Specifically, the user starts the application, enters the title "Adventure with a Dragon" and the theme "Adventure" in the story creation form, and presses the submit button. This operation generates the input data (title and theme) from the user terminal.

[0700] Step 2:

[0701] The user's device sends the entered title and theme data to the server. The entered data is converted to JSON format and sent as an HTTP POST request to the server's API endpoint. Authentication information such as the user ID must also be included in this process.

[0702] Step 3:

[0703] The server parses the received JSON data and authenticates the user. After successful authentication, it saves the entered title and theme in the database. The server then connects to the database and adds the data as a new entry to the corresponding table.

[0704] Step 4:

[0705] The server sends a request to the generative AI model to generate an initial plot based on the title and theme stored in the database, generates a prompt text, and sends a request including the prompt text to the API endpoint of the generative AI model.

[0706] Step 5:

[0707] The generative AI model generates an initial plot based on the prompt received from the server. For example, it might generate an initial plot such as, "One day, a brave boy meets a dragon. The dragon is guarding a secret treasure..." and send it back to the server.

[0708] Step 6:

[0709] The server receives the initial plot returned from the generative AI model, stores it in a database, and sends an HTTP response containing the initial plot to the user device.

[0710] Step 7:

[0711] The user terminal displays the initial plot received from the server on the user interface. The user checks the plot and inputs a new story development. For example, the user inputs a development such as "A boy named Taro appears, becomes friends with a dragon, and goes on a treasure hunt together."

[0712] Step 8:

[0713] The user device sends the newly entered storyline data to the server, which again converts the input data into JSON format and sends it to the server as an HTTP POST request.

[0714] Step 9:

[0715] The server receives the new storyline data and resends it to the generative AI model. It then generates a new prompt and sends a request to the generative AI model based on that.

[0716] Step 10:

[0717] The generative AI model generates new developments based on the new storylines sent in. For example, it generates a new plot like, "Tarro and Dragon find a treasure chest in a cave deep in the mountains, and inside is an old map..."

[0718] Step 11:

[0719] The server receives the newly generated deployment from the generative AI model, stores it in a database, and sends it to the user's device as an HTTP response.

[0720] Step 12:

[0721] The user's device displays the new developments received from the server on the user interface. The user continues to edit and revise the story until it is complete.

[0722] Step 13:

[0723] The user confirms the completion of the final story and sends a binding request, which is then sent to the server from the user terminal.

[0724] Step 14:

[0725] The server stores the received bookbinding request in a database and sends the request to the bookbinding service, which then processes the book for production and delivery.

[0726] Step 15:

[0727] The server receives a completion notification from the bookbinding service and notifies the user of the completion of binding and delivery information. The user receives the notification and receives the completed picture book.

[0728] (Application example 1)

[0729] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0730] Existing story generation systems lack a means for users to easily share the content they create with other users when creating original story picture books. Another issue is the lack of convenience for use as a content distribution service for parents and children to enjoy together.

[0731] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0732] In this invention, the server includes means for inputting a story title and theme from a user terminal, means for generating an initial plot using a generative AI model, means for displaying the plot received from the generative AI model on the user terminal, means for passing the story development input by the user to the generative AI model, means for displaying a new development proposed by the generative AI model on the user terminal, means for saving the final story edited by the user, means for processing a binding request based on the saved story, and means for delivering the generated plot or story development as a content delivery service, thereby enabling users to easily share the generated story with other users or use it as a content delivery service for parent-child enjoyment.

[0733] A "user terminal" is a terminal that a user uses to input the title and theme of a story and to display the generated plot and story development.

[0734] A "generative AI model" is an artificial intelligence model that automatically generates an initial plot and story development based on the title and theme entered by the user.

[0735] An "initial plot" is the basic structure of a story that the generative AI model initially generates based on the story title and theme provided by the user.

[0736] "Story development" is an item in which the user specifically describes the progress of the story or adds new scenes.

[0737] A "bookbinding request" is a request by a user to produce a completed story as a physical book.

[0738] A "content distribution service" is a service that distributes generated plots and storylines in a form that can be used by other users.

[0739] This invention relates to a system that allows users to create original story picture books while interacting with a generative AI model. This system includes a user terminal, a server, and a generative AI model, and describes the program processing.

[0740] System Overview

[0741] The system of the present invention receives input from the user's device, sends it to the generative AI model via the server, and provides the output from the AI ​​back to the user. Through this process, users (especially children) can experience creating their own story picture book.

[0742] User terminal

[0743] The user device (e.g., smartphone, tablet) provides an interface for inputting the story title and theme. It also has the function of displaying the generated plot and story development. The user device has the following functions:

[0744] 1. Input interface: Provides a form for users to enter the title and theme of the story.

[0745] 2. Display interface: Displays plot suggestions from the generative AI model and story developments entered by the user.

[0746] 3. Editing function: Provide an editor that allows users to edit, modify, and save stories.

[0747] 4. Binding Request: Provides an interface for submitting completed stories as binding requests.

[0748] server

[0749] The server is responsible for sending input data from the user to the generative AI model and returning output data from the generative AI model to the user's device. The hardware used is a cloud server (e.g., AWS, Google Cloud), and the software used is Flask (a lightweight web server framework built in Python). The server has the following functions:

[0750] 1. User authentication: Verify login information and perform authentication.

[0751] 2. Data processing and analysis: Receives input data from the user and passes it to the generative AI model.

[0752] 3. Data storage: Store the stories created by users in a database.

[0753] 4. Generative AI model integration: Send data to the generative AI model and receive the results.

[0754] 5. Binding Request Management: Receive binding requests and coordinate with the binding service.

[0755] 6. Content Distribution: Distributing the generated plot or storyline as a content distribution service.

[0756] Generative AI Models

[0757] Generative AI models (e.g., OpenAI GPT) generate plot suggestions and storylines based on user input data. Generative AI models have the following capabilities:

[0758] 1. Initial plot generation: Generate an initial plot based on the title and theme entered by the user.

[0759] 2. Story development suggestions: Based on the story development input by the user, new ideas and developments are suggested.

[0760] Specific examples

[0761] Let's say a child wants to create a picture book with the theme of "Adventure Story." The user (child) starts the application on the user's device and enters their login information. After successful login, the home screen is displayed and the child clicks the "Create a new story" button. Next, in the form that appears, the child enters the story title "Adventure with a Dragon" and the theme "Adventure," and presses the submit button.

[0762] The server stores the title and theme received from the user in a database and sends a request to the generative AI model to generate an initial plot. When the generative AI model receives the initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure..."), it sends the content to the user's device.

[0763] The user device displays the received initial plot, and the child can think up a new storyline. For example, they can input and submit a story in which a boy named Taro appears, becomes friends with a dragon, and goes on a treasure hunt together.

[0764] The server sends the new storyline input by the child to the generative AI model again and receives the next suggestion. When the server receives a new storyline from the AI ​​model (e.g., "Taro and Dragon find a treasure chest in a cave deep in the mountains, and inside is an old map..."), it sends it to the user's device.

[0765] After the story is completed, the user device displays a button for the child to send the completed story as a bookbinding request. When the child clicks the button, the request is sent to the server, which then works with the bookbinding service to arrange for the picture book to be produced and delivered.

[0766] An example of a prompt is:

[0767] "Title: The Brave Girl and the Magical Forest

[0768] Theme: Adventure

[0769] Generate the plot."

[0770] The format is as follows.

[0771] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0772] Step 1:

[0773] The user device launches the application and enters login information. The user device sends the entered login information to the server, which then authenticates the user. If authentication is successful, the server returns a successful authentication response to the user device. The user device then displays the home screen.

[0774] Step 2:

[0775] The user clicks the "Create a new story" button, which displays a form for entering the story title and theme. The user enters the title and theme in the form and presses the submit button. The user's device sends the input data to the server. The server stores the received title and theme in a database and sends a request to the generative AI model to generate an initial plot.

[0776] Step 3:

[0777] The generative AI model generates an initial plot based on the title and theme received from the user. The generative AI model returns the generated initial plot to the server. After receiving the plot, the server sends its contents to the user's device. The user's device displays the initial plot.

[0778] Step 4:

[0779] The user thinks of a new storyline and inputs the content they want to add. The user's device sends the user's input to the server. The server then sends the input data back to the generative AI model and requests a storyline proposal.

[0780] Step 5:

[0781] The generative AI model proposes a new storyline. The generative AI model generates a storyline and returns the result to the server. The server receives the result and sends the next storyline to the user's device. The user's device displays the generated storyline.

[0782] Step 6:

[0783] The user edits and saves the final story. The user terminal sends the edited story to the server, which stores it in the database. When saving is complete, the server sends a confirmation message to the user terminal.

[0784] Step 7:

[0785] The user clicks a button to request binding. The user terminal sends the request to the server, which receives the request. The server then works with the binding service to arrange for the production and delivery of the picture book. The server also processes the generated plot or story development to distribute to the content distribution service.

[0786] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0787] This invention combines an emotion engine with a system that allows users to create original story picture books while interacting with a generative AI model. Specifically, we explain the configuration of the system, which includes a user terminal, a server, a generative AI model, and an emotion engine, and how each element works together.

[0788] System Overview

[0789] The system of the present invention receives input from a user terminal, sends it to a generative AI model and an emotion engine via a server, and provides the output from the AI ​​and emotion engine back to the user terminal. This allows users (especially children) to experience creating their own story picture books. The following is a detailed description of the system's components and their functions.

[0790] User terminal

[0791] The user terminal is a device that provides an interface for creating stories. Its main functions are as follows:

[0792] 1. Input interface:

[0793] Provide a form for children to enter the title and theme of their story.

[0794] 2. Display interface:

[0795] It displays plot suggestions from generative AI models and storylines input by users.

[0796] 3. Editing features:

[0797] It provides an editor where users can edit, modify and save stories.

[0798] 4. Binding Request:

[0799] Provides an interface for submitting completed stories as binding requests.

[0800] 5. Emotion recognition:

[0801] It has the ability to analyze the user's facial expressions and voice and send the data to the emotion engine.

[0802] server

[0803] The server is responsible for sending input data from users to the generative AI model and emotion engine, and returning output data to the user device. Its main functions are as follows:

[0804] 1. User authentication:

[0805] Verify the user's login information and perform authentication.

[0806] 2. Data processing and analysis:

[0807] User input data is passed to the emotion engine and generative AI model.

[0808] 3. Data Retention:

[0809] Store user-created stories in a database.

[0810] 4. Collaboration with generative AI models and emotion engines:

[0811] The title, theme, and user emotion data are sent to the emotion engine and generative AI model, and the results are returned to the user's device.

[0812] 5. Binding Request Management:

[0813] Receives binding requests and coordinates with binding services.

[0814] Generative AI Models

[0815] The generative AI model generates plot suggestions and story developments based on user input data and emotional data sent from the emotion engine.

[0816] 1. Initial plot generation:

[0817] Generates an initial plot based on the title and theme entered by the user.

[0818] 2. Storyline proposal:

[0819] Based on the story development entered by the user, new ideas and developments are suggested.

[0820] Emotion Engine

[0821] The emotion engine analyzes emotions from the user's facial expressions and voice and passes that data to a generative AI model.

[0822] 1. Sentiment analysis:

[0823] Performs real-time emotion analysis of the user's facial expressions and voice.

[0824] 2. Emotion data generation:

[0825] Emotional data is generated based on the analysis results and passed to the generative AI model.

[0826] Specific examples

[0827] Imagine a child is creating a picture book with the theme of "adventure story."

[0828] Operation on the user device

[0829] The user (child) starts the application on the user's device and enters their login information. After successful login, the home screen is displayed and the child clicks the "Create a new story" button. In the form that is then displayed, the child enters the story title "Adventure with a Dragon" and the theme "Adventure" and presses the submit button.

[0830] Processing on the server

[0831] The server stores the title and theme received from the user in a database and sends the data to the generative AI model and emotion engine. The emotion engine analyzes the user's facial expressions and voice data and passes the generated emotion data to the generative AI model.

[0832] How generative AI models work

[0833] The generative AI model generates an initial plot based on the title, theme, and emotional data, and sends the generated initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure...") back to the server.

[0834] Collaboration between the server and the emotion engine

[0835] The server sends the initial plot received from the generative AI model to the terminal and makes user-level adjustments, taking into account feedback from the emotion engine.

[0836] Viewing and editing on user devices

[0837] The user's device displays the initial plot, and the child thinks up and inputs a new storyline (e.g., "Taro becomes friends with the dragon and they go on a treasure hunt together"). At this time, the emotion engine analyzes the user's emotions from facial expressions and voice and provides feedback in real time. Once input is complete, the child clicks the send button.

[0838] Reconnecting the server and the generative AI model

[0839] The server resends new developments input from the user or emotion engine to the generative AI model, receives new proposals, and the generative AI model generates the next proposal and sends the result back to the server.

[0840] Submit a binding request

[0841] After the story is completed, the user device displays a button to request the completed story to be bound. When the child clicks the button, the request is sent to the server, which then works with the binding service to arrange for the book to be produced and delivered.

[0842] This is the process by which children create their own original story picture books using the system of this invention. By combining it with an emotion engine, dynamic story development and feedback are provided according to the user's emotions, creating a richer, more interactive experience.

[0843] The processing flow will be explained below.

[0844] Step 1:

[0845] The user starts the service application on the user device and the login screen is displayed. The child enters the user name and password and clicks the login button.

[0846] Step 2:

[0847] The server receives the login information and performs user authentication. If authentication is successful, the server starts a user session and returns the home screen data to the device.

[0848] Step 3:

[0849] The device will display a home screen and offer a "Create a new story" button that the child can click to display a form where they can enter a title and theme.

[0850] Step 4:

[0851] The user enters a story title (e.g., "Adventure with a Dragon") and a theme (e.g., "Adventure") and clicks the submit button.

[0852] Step 5:

[0853] The server stores the input title and theme in a database and sends a request to generate an initial plot to the generative AI model. At the same time, the server sends the user's facial expressions and voice data from the user's device to the emotion engine.

[0854] Step 6:

[0855] The generative AI model generates an initial plot based on the title and theme and sends the results back to the server. The emotion engine also analyzes the user's facial expressions and voice in real time to generate emotion data.

[0856] Step 7:

[0857] The server integrates the initial plot received from the generative AI model with the emotion data from the emotion engine and sends it to the terminal.

[0858] Step 8:

[0859] The device displays an initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure..."), which the user reads. The user then inputs a new storyline (e.g., "Taro becomes friends with the dragon and they go on a treasure hunt together") and submits it. At the same time, the emotion engine continues to analyze the user's emotions from their facial expressions and voice.

[0860] Step 9:

[0861] The server receives the newly input storyline and emotion data from the user and sends it to the generative AI model, which then proposes the next storyline.

[0862] Step 10:

[0863] The generative AI model generates the next proposal based on the new storyline and sends the result back to the server (e.g., "Tarro and Dragon find a treasure chest in a cave deep in the mountains, inside which is an old map").

[0864] Step 11:

[0865] The server receives the next suggestion from the generative AI model and sends it to the user device and the emotion engine, which continues to monitor the user's reactions and analyze the emotion data.

[0866] Step 12:

[0867] The device displays new suggestions, and the user can view them and edit the story. Once edits are complete, the user clicks the save button. During this time, the emotion engine continues to analyze emotions.

[0868] Step 13:

[0869] The server receives the final edited story and emotion data from the user and stores it in a database.

[0870] Step 14:

[0871] The user checks the completed story and clicks the binding request button.

[0872] Step 15:

[0873] The server receives the bookbinding request, stores it in a database, and sends the request to the bookbinding service. After receiving confirmation from the bookbinding service, the server arranges for the production and delivery of the picture book.

[0874] Step 16:

[0875] Users receive a bound picture book delivered to their home and enjoy their own original story.

[0876] These are the specific processing steps for creating a story picture book using the system of the present invention. By combining it with an emotion engine, it is possible to provide an interactive story based on the user's emotions, providing a richer experience.

[0877] Example 2

[0878] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0879] In today's digital society, it is important to provide children with the opportunity to express their creativity and create original story picture books. However, existing systems are unable to reflect users' emotions in real time, limiting the interactive story creation experience. Furthermore, the process of binding the created stories is time-consuming, making it difficult for users to use.

[0880] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0881] In this invention, the server includes a means including an emotion engine for analyzing user emotions, a means for transmitting data to the generative AI model, and a means for returning the results to the user terminal. This allows users to receive real-time emotional feedback as they create stories, enabling a richer, more interactive story creation experience. Furthermore, requests for binding the generated stories are efficiently processed, improving user convenience.

[0882] A "user terminal" is a device that allows a user to input and edit the title and theme of a story.

[0883] A "generative AI model" is an artificial intelligence that generates an initial plot based on the title and theme entered by the user and suggests new story developments.

[0884] An "initial plot" is a story outline initially created by a generative AI model based on the title and theme entered by the user.

[0885] The "emotion engine" is a system element that analyzes the user's facial expressions and voice, generates emotional data, and passes it to the generative AI model.

[0886] A "plot" is a basic component that shows the development and progression of a story.

[0887] "Emotion data" is data that indicates an emotional state generated by an emotion engine from the user's facial expressions and voice.

[0888] A "binding request" is a request that a user submits to have a completed story bound into a physical book.

[0889] "Saving" means recording the final story edited by the user in a storage device such as a database.

[0890] "Real-time feedback" means that the emotion engine analyzes the user's emotional state in real time and immediately reflects the results to the user.

[0891] This invention combines an emotion engine with a system that allows users to create original story picture books while interacting with a generative AI model. Specifically, we explain the configuration of the system, which includes a user terminal, a server, a generative AI model, and an emotion engine, and how each element works together.

[0892] System Overview

[0893] The system of the present invention receives input from a user terminal, sends it to a generative AI model and emotion engine via a server, and provides the output from the AI ​​and emotion engine back to the user terminal, thereby providing users (especially children) with the experience of creating their own story picture books.

[0894] User terminal

[0895] The user terminal is a device that provides an interface for creating stories. Its main functions are as follows:

[0896] 1. Input interface:

[0897] Provide a form for the child to enter the title and theme of the story. For example, enter the title "Adventure with a Dragon" and the theme "Adventure."

[0898] 2. Display interface:

[0899] It displays plot suggestions from generative AI models and storylines input by users.

[0900] 3. Editing features:

[0901] It provides an editor where users can edit, modify and save stories. Users can add their own ideas or modify existing stories.

[0902] 4. Binding Request:

[0903] It provides a button to submit the completed story as a binding request. Clicking the button sends the request to the server.

[0904] 5. Emotion recognition:

[0905] It has the ability to analyze the user's facial expressions and voice and send that data to the emotion engine.

[0906] server

[0907] The server is responsible for sending input data from users to the generative AI model and emotion engine, and returning output data to the user device. Its main functions are as follows:

[0908] 1. User authentication:

[0909] Verify and authenticate the user's login information. For example, the user enters login information and the server verifies it.

[0910] 2. Data processing and analysis:

[0911] User input data is passed to the emotion engine and generative AI model.

[0912] 3. Data Retention:

[0913] Store user-created stories in a database.

[0914] 4. Collaboration with generative AI models and emotion engines:

[0915] The title, theme, and user emotion data are sent to the emotion engine and generative AI model, and the results are returned to the user's device.

[0916] 5. Binding Request Management:

[0917] Receives binding requests and coordinates with binding services.

[0918] Generative AI Models

[0919] The generative AI model generates plot suggestions and story developments based on user input data and emotional data sent from the emotion engine.

[0920] 1. Initial plot generation:

[0921] Generates an initial plot based on the title and theme entered by the user. For example, create a plot based on "Adventures with Dragons."

[0922] 2. Storyline proposal:

[0923] Based on the story development entered by the user, new ideas and developments are suggested.

[0924] Emotion Engine

[0925] The emotion engine analyzes emotions from the user's facial expressions and voice and passes that data to a generative AI model.

[0926] 1. Sentiment analysis:

[0927] Performs real-time emotion analysis of the user's facial expressions and voice.

[0928] 2. Emotion data generation:

[0929] Emotional data is generated based on the analysis results and passed to the generative AI model.

[0930] Specific examples

[0931] Operation on the user device

[0932] The user (child) launches the application on their device and enters their login information. After successfully logging in, the home screen appears and the child clicks the "Create a new story" button. In the form that appears next, the child enters "Adventure with a Dragon" and "Adventure" and presses the submit button.

[0933] Processing on the server

[0934] The server stores the title and theme received from the user in a database and sends the data to the generative AI model and emotion engine. The emotion engine analyzes the user's facial expressions and voice data and passes the generated emotion data to the generative AI model.

[0935] How generative AI models work

[0936] The generative AI model generates an initial plot based on the title, theme, and emotional data, for example, "One day, a brave boy meets a dragon who is guarding a secret treasure," and sends it back to the server.

[0937] Collaboration between the server and the emotion engine

[0938] The server sends the initial plot received from the generative AI model to the user's device and makes adjustments taking into account feedback from the emotion engine.

[0939] Viewing and editing on user devices

[0940] The user's device displays the initial plot, and the user can think of and input a new story development. For example, they can input "Taro becomes friends with a dragon and they go on a treasure hunt together." At this time, the emotion engine analyzes the user's emotions from facial expressions and voice and provides feedback in real time. Once input is complete, the user clicks the send button.

[0941] Reconnecting the server and the generative AI model

[0942] The server resubmits the new deployment data to the generative AI model, which then generates the next proposal and sends the result back to the server.

[0943] Submit a binding request

[0944] After the story is completed, the user's device displays a button for sending the completed story as a bookbinding request. When the user clicks the button, the request is sent to the server, which then works with the bookbinding service to produce the picture book and arrange for its delivery.

[0945] Examples of prompt statements

[0946] As an example of a prompt for a generative AI model, the following could be considered as input to generate the initial plot of a story themed around "adventures with dragons":

[0947] "The user has entered the title 'Adventure with the Dragon' and the theme 'Adventure'. Based on this, please generate an initial plot for the story, starting with the scene where the boy meets the dragon."

[0948] This concludes the detailed explanation of how to use the system of the present invention. This system allows users to interactively create original stories while receiving emotional feedback, and ultimately to complete the process of bookbinding.

[0949] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0950] Step 1:

[0951] Log in

[0952] The user accesses the login screen and enters their login information. The server receives the login information (input) sent by the user and authenticates it by comparing it with information in the database (data calculation). If authentication is successful, the home screen is displayed on the user's device (output).

[0953] Step 2:

[0954] Start creating your story

[0955] The user clicks the "Create a new story" button on the home screen. The user's device detects the click event (input) and displays a new story creation form (output).

[0956] Step 3:

[0957] Enter the title and theme

[0958] The user enters the story title and theme on a new screen. The user's device receives the title "Adventure with a Dragon" and theme "Adventure" entered by the user through an input form (input). The information entered into the form is temporarily stored in the device's memory.

[0959] Step 4:

[0960] Sending input data

[0961] The user presses the send button to send the data to the server. The user's device sends the title and theme data (input) to the server and displays a notification of successful transmission to the user (output).

[0962] Step 5:

[0963] User Authentication

[0964] The server checks the user's login information and performs authentication. If authentication is successful, the server saves the user data in the database (data calculation).

[0965] Step 6:

[0966] Data storage

[0967] The server stores the received title and theme in a database (output). The server records this data in a database in an appropriate format for further processing.

[0968] Step 7:

[0969] Sending data to a generative AI model

[0970] The server sends the title and theme data to the generative AI model. It sends the title "Adventure with Dragons" and the theme "Adventure" (input) to the generative AI model via API, and receives the generated initial plot (output).

[0971] Step 8:

[0972] Sending data to the emotion engine

[0973] The server sends the user's emotional data to the emotion engine. It also sends facial expression and voice data (input) acquired from the user's device to the emotion engine and receives the analysis results (output).

[0974] Step 9:

[0975] Receiving plots from generative AI models

[0976] The server receives the initial plot from the generative AI model. The initial plot data (output) is "One day, a brave boy meets a dragon. The dragon guards a secret treasure..." and is used for the next process.

[0977] Step 10:

[0978] Receiving feedback from the emotion engine

[0979] The server receives feedback from the emotion engine and understands the user's emotional state. Based on the feedback data (output), it modifies the output results of the generative AI model (data calculation).

[0980] Step 11:

[0981] Displaying the initial plot

[0982] The user terminal displays the initial plot to the user, displays the received initial plot data (input) on the screen, and asks the user for feedback (output).

[0983] Step 12:

[0984] User edits

[0985] The user thinks up and inputs a new story development based on the initial plot. The user's input data, "Taro becomes friends with the dragon and goes on a treasure hunt together" (input), is acquired by the user's device and temporarily saved as edited data.

[0986] Step 13:

[0987] Send edited data

[0988] The user resends the edited data to the server. The user terminal sends the edited new story development data (input) to the server and displays a notification that transmission is complete (output).

[0989] Step 14:

[0990] Resubmit to generative AI model

[0991] The server resubmits new story development data to the generative AI model. It sends new story development data (input) via API and receives new proposals from the generative AI model (output).

[0992] Step 15:

[0993] Receiving a resubmission

[0994] The server receives new proposals from the generative AI model. It receives new story development proposal data (output) from the generative AI model and uses it for further processing.

[0995] Step 16:

[0996] View the final story

[0997] The user terminal displays the final story development to the user. The generated final story data (input) is displayed on the screen and the user is prompted to confirm it (output).

[0998] Step 17:

[0999] Submit a binding request

[1000] The user submits the completed story as a binding request. The user terminal transmits the binding request data (input) to the server and displays a notification of completion of transmission (output).

[1001] Step 18:

[1002] Collaboration with bookbinding services

[1003] The server receives the binding request and works with the binding service to produce and arrange for delivery of the picture book. It sends the binding request data (input) to the binding service and records the binding progress in a database (output).

[1004] (Application example 2)

[1005] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1006] Conventional story creation systems lacked the ability to provide feedback based on the user's emotions or adjust the storyline, which prevented them from fully enriching the user's experience. Furthermore, they lacked the interactive elements that children needed to have fun creating stories, which led to a lack of interest.

[1007] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting the title and theme of the story from a user terminal, means for generating an initial plot using a generative AI model, means for displaying the plot received from the generative AI model on the user terminal, means for passing the story development input by the user to the generative AI model, means for displaying a new development proposed by the generative AI model on the user terminal, means for saving the final story edited by the user, means for processing a binding request based on the saved story, and means for adjusting the story feedback displayed on the user terminal based on the emotion, including an emotion engine that analyzes the user's facial expressions and voice to generate emotion data. This allows for dynamic story development and feedback in accordance with the user's emotions, enabling a richer and more interactive experience.

[1008] A "user terminal" is a device that provides an interface for creating a story and receives input from a user.

[1009] A "generative AI model" is an artificial intelligence that generates an initial plot or new story development based on user input data.

[1010] The "initial plot" is the initial story development that is generated based on the story title and theme entered by the user.

[1011] A "plot" indicates the story's outline and development pattern, and is generated by a generative AI model.

[1012] Theme refers to the overall theme or central idea of ​​the story.

[1013] The "emotion engine" is an engine that analyzes emotions from the user's facial expressions and voice, and adjusts the story development through the generated emotional data.

[1014] A "bookbinding request" is a procedure in which a user requests that a completed story be produced as a physical picture book.

[1015] "Emotion data" refers to emotional information analyzed from the user's facial expressions and voice, and is reflected in the story development.

[1016] "Interface" refers to the screen display and input methods on a user terminal that provide the means for a user to interact with a system.

[1017] "Feedback" refers to the information or suggestions that a system returns in response to a user's input or behavior.

[1018] To implement this invention, a user terminal on which a dedicated application is installed, a server located on the cloud, a generative AI model, and an emotion engine are used. Details of these are described below.

[1019] Overall system configuration

[1020] The system includes the following major components:

[1021] 1. User Device:

[1022] Input interface: Accepts title and theme input from the user.

[1023] Display interface: Shows the plot from the generative AI model and the final story.

[1024] Editing capabilities: Users can create, edit, and save stories.

[1025] Emotion Recognition: Uses a camera and microphone to capture the user's facial expressions and voice to generate emotional data.

[1026] Bookbinding Requests: Processing requests to have completed stories produced as physical picture books.

[1027] 2. Server:

[1028] User authentication: Verify the user's login information and perform authentication.

[1029] Data processing and analysis: Sending user input data to the emotion engine and generative AI model.

[1030] Data storage: Store user-created stories in a database.

[1031] Collaboration with generative AI models and emotion engines: Processes title, theme, and emotion data and returns the results to the user's device.

[1032] Binding Request Management: Send binding requests to binding services.

[1033] 3. Generative AI models (e.g., GPT-4):

[1034] Initial plot generation: Generates the basic plot of your story based on the title and theme.

[1035] Story development proposal: Based on new development input from the user, the next development proposal is made.

[1036] 4. Emotion engine (e.g. Affectiva SDK):

[1037] Sentiment analysis: Analyzes the user's facial expressions and voice to generate emotional data.

[1038] Emotional feedback: Adjust the storyline based on the generated emotional data.

[1039] Software and Hardware Configuration

[1040] User device: Smartphone application (e.g., using React Native)

[1041] Server: Node.js, Express framework, database (e.g. MongoDB)

[1042] Emotion recognition: Expo Camera, Expo Audio

[1043] Generative AI model: OpenAI GPT-4 API

[1044] Data processing flow

[1045] 1. Getting user input:

[1046] The user enters the story title and theme on their device and sends it to the server.

[1047] 2. Emotion capture and analysis:

[1048] It uses the camera and microphone on the user's device to capture facial expressions and voice.

[1049] The captured data is sent to the emotion engine via the server, where emotions are analyzed.

[1050] 3. Collaboration with generative AI models:

[1051] The server sends user input and emotion data to the generative AI model.

[1052] The generative AI model generates an initial plot and sends it to the user's device via the server.

[1053] 4. Providing User Feedback:

[1054] As users edit the storyline, the emotion engine provides real-time emotional feedback that is sent to the generative AI model.

[1055] The generative AI model generates new suggestions, which are displayed on the user's device.

[1056] 5. Processing binding requests:

[1057] Users submit completed stories as binding requests.

[1058] The server works with the bookbinding service to create the physical picture book and arrange for delivery.

[1059] Examples and prompts

[1060] Title: "The Adventures of Brave Taro"

[1061] Theme: Adventure

[1062] Scene: "Tarro meets a dragon and becomes friends with him."

[1063] Example prompts to input to a generative AI model:

[1064] Title: The Adventures of the Brave Taro

[1065] Theme: Adventure

[1066] Feedback emotion: Fun

[1067] Story Current Scene: Taro meets a dragon and they go on a treasure hunt together to become friends.

[1068] Please suggest the following expansion:

[1069] In this way, a system can be implemented that provides users with a fun, emotionally driven story-creation experience.

[1070] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1071] Step 1:

[1072] The user inputs the title and theme of the story. The user's device receives these inputs and sends them to the server.

[1073] Input: User title and theme

[1074] Output: Title and theme sent to server

[1075] Specific operation: When a user enters characters into an input field on the user terminal and presses the send button, this data is sent to the server.

[1076] Step 2:

[1077] The server sends the title and theme to the generative AI model, which generates the initial plot and sends the results back to the server.

[1078] Input: Title and Theme

[1079] Output: Initial plot

[1080] What it does: The server sends the title and theme to the generative AI model and receives the generated initial plot.

[1081] Step 3:

[1082] The server sends the initial plot to the user terminal, which displays it.

[1083] Input: Initial plot

[1084] Output: Initial plot displayed on the user's terminal

[1085] Specific operation: The initial plot received from the server is displayed on the display screen of the user's terminal.

[1086] Step 4:

[1087] The user inputs the continuation of the story. The user's device receives this input and sends it to the server.

[1088] Input: User storyline

[1089] Output: Sending the storyline to the server

[1090] Specific behavior: When the user enters a storyline into the input field and presses the submit button, it is sent to the server.

[1091] Step 5:

[1092] The user device captures the user's facial expressions and voice in real time and sends them to the emotion engine for analysis.

[1093] Input: User's facial expressions and voice

[1094] Output: Emotion data

[1095] Specific operation: Uses the camera and microphone on the user's device to capture facial expression and voice data and sends it to the emotion engine.

[1096] Step 6:

[1097] The emotion engine analyzes the user's facial expressions and voice to generate emotion data, which is then sent to the server.

[1098] Input: facial expression data and voice data

[1099] Output: Emotion data

[1100] Specific operation: The emotion engine analyzes the received data, determines the user's emotions, generates emotion data, and sends it to the server.

[1101] Step 7:

[1102] The server sends the emotional data and the user's storyline to a generative AI model, which then generates a proposal for the next storyline.

[1103] Input: Emotion data and storyline

[1104] Output: Next storyline proposal

[1105] Specific operation: The server sends the story development and emotional data from the user to the generative AI model, which then generates a proposal for the next development and returns it to the server.

[1106] Step 8:

[1107] The server transmits the next story development proposal to the user terminal, which displays it.

[1108] Input: Next storyline suggestion

[1109] Output: Next expansion suggestion displayed on user terminal

[1110] Specific operation: The proposal received from the server is sent to the user terminal, and is displayed on the display screen of the user terminal.

[1111] Step 9:

[1112] The user completes the story and submits a binding request, which the user terminal sends to the server.

[1113] Input: Final story

[1114] Output: Binding request

[1115] Specific behavior: The user checks the final story and presses the binding request button, which sends the request to the server.

[1116] Step 10:

[1117] The server receives the binding request, works with the binding service to produce the picture book, and arranges for delivery.

[1118] Input: Binding request

[1119] Output: Physical storybook

[1120] Specific operation: After the server receives the binding request, it requests the binding service to produce and deliver the picture book.

[1121] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1122] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1123] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1124] [Third embodiment]

[1125] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1126] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1127] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1128] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1129] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1130] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1131] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1132] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1133] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1134] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1135] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1136] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1137] This paper describes a system that allows users to create original story picture books while interacting with a generative AI model. Specifically, it describes the configuration of the system, which includes a user terminal, a server, and a generative AI model, and the processing of the program.

[1138] System Overview

[1139] The system of the present invention receives input from the user's device, sends it to the generative AI model via the server, and then provides the output from the AI ​​back to the user. Through this process, users (especially children) are given the experience of creating their own story picture book. The system configuration and the functions of each element are described in detail below.

[1140] User terminal

[1141] The user device (e.g., PC, smartphone, tablet) provides an interface for the child to input the title and theme of the story.

[1142] 1. Input interface:

[1143] Provide a form for children to enter the title and theme of their story.

[1144] 2. Display interface:

[1145] Displays plot suggestions from AI and story developments entered by users.

[1146] 3. Editing features:

[1147] It provides an editor where users can edit, modify and save stories.

[1148] 4. Binding Request:

[1149] Provides an interface for submitting completed stories as binding requests.

[1150] server

[1151] The server is responsible for sending input data from users to the generative AI model and returning output data from the generative AI model to the user's device. It also stores data and manages binding requests. Its main functions are as follows:

[1152] 1. User authentication:

[1153] Verify your login information and authenticate.

[1154] 2. Data processing and analysis:

[1155] Receives input data from the user and passes it to the generative AI model.

[1156] 3. Data Retention:

[1157] Store user-created stories in a database.

[1158] 4. Generative AI model collaboration:

[1159] Send data to the AI ​​model and receive the results.

[1160] 5. Binding Request Management:

[1161] Receives binding requests and coordinates with binding services.

[1162] Generative AI Models

[1163] The generative AI model generates plot suggestions and storylines based on user input data.

[1164] 1. Initial plot generation:

[1165] Generates an initial plot based on the title and theme entered by the user.

[1166] 2. Storyline proposal:

[1167] Based on the story development entered by the user, new ideas and developments are suggested.

[1168] Specific examples

[1169] Imagine a child is creating a picture book with the theme of "adventure story."

[1170] Operation on the user device

[1171] The user (child) launches the application on the user's device and enters their login information. If login is successful, the home screen is displayed and the child clicks the "Create a new story" button. Next, they enter the story title "Adventure with a Dragon" and the theme "Adventure" in the form that appears, and press the submit button.

[1172] Processing on the server

[1173] The server stores the title and theme received from the user in a database and sends a request to the generative AI model to generate an initial plot. When the generative AI model receives the initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure..."), it sends the content to the user's device.

[1174] Viewing and editing on user devices

[1175] The user device displays the received initial plot, and the child can think up a new storyline. For example, they can input and submit a story in which a boy named Taro appears, becomes friends with a dragon, and goes on a treasure hunt together.

[1176] Collaboration between the server and the generative AI model

[1177] The server sends the new storyline input by the child to the generative AI model again and receives the next suggestion. When the server receives a new storyline from the AI ​​model (e.g., "Taro and Dragon find a treasure chest in a cave deep in the mountains, and inside is an old map..."), it sends it to the user's device.

[1178] Submit a binding request

[1179] After the story is completed, the user device displays a button for the child to send the completed story as a bookbinding request. When the child clicks the button, the request is sent to the server, which then works with the bookbinding service to arrange for the picture book to be produced and delivered.

[1180] This is the process by which children can create their own original story picture books using the system of the present invention, which not only allows children to exercise their creativity and learn while having fun, but also gives them a sense of accomplishment by having their own unique picture book.

[1181] The processing flow will be explained below.

[1182] Step 1:

[1183] The user starts the service application on the user device and the login screen is displayed. The child enters the user name and password and clicks the login button.

[1184] Step 2:

[1185] The server receives the login information and performs authentication. If authentication is successful, the server starts the user session and returns the home screen data to the device.

[1186] Step 3:

[1187] The device will display a home screen and offer a "Create a new story" button that the child can click to display a form where they can enter a title and theme.

[1188] Step 4:

[1189] The user enters a story title (e.g., "Adventure with a Dragon") and a theme (e.g., "Adventure") and clicks the submit button.

[1190] Step 5:

[1191] The server receives the input title and theme, stores them in a database, and then sends a request to the generative AI model to generate the initial plot.

[1192] Step 6:

[1193] The generative AI model generates an initial plot based on the title and theme, and sends the generated plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure...") back to the server.

[1194] Step 7:

[1195] The server sends the initial plot received from the generative AI model to the terminal.

[1196] Step 8:

[1197] The device displays the initial plot. The user (child) reads the plot and thinks of a new storyline to input (e.g., "Taro becomes friends with the dragon and they go on a treasure hunt together."). When they're done, they click the send button.

[1198] Step 9:

[1199] The server receives the new storyline input from the user and sends it back to the generative AI model, requesting new proposals.

[1200] Step 10:

[1201] The generative AI model generates the next proposal based on the new storyline and sends the result back to the server (e.g., "Tarro and Dragon find a treasure chest in a cave deep in the mountains, inside which is an old map").

[1202] Step 11:

[1203] The server sends the next proposal received from the generative AI model to the device.

[1204] Step 12:

[1205] The device will display a new suggestion, and the user can view the suggestion and edit the story. Once they've finished editing, they can click the save button again. When the final story is complete, they'll be given the option to request binding.

[1206] Step 13:

[1207] The user checks the completed story and clicks the binding request button.

[1208] Step 14:

[1209] The server receives the bookbinding request, stores it in a database, and sends the request to the bookbinding service. After receiving confirmation from the bookbinding service, the server arranges for the production and delivery of the picture book.

[1210] Step 15:

[1211] Users receive a bound picture book delivered to their home and enjoy their own original story.

[1212] Example 1

[1213] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1214] Conventional story creation systems require users to do a lot of manual work when creating a story, making the creation process complicated and time-consuming. They also lacked adequate support to fully bring out users' creativity, especially functions that allow children to enjoy creative activities. Furthermore, the process of binding completed stories into books was complicated, making it difficult for users to easily obtain the finished product.

[1215] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1216] In this invention, the server includes a means for transmitting data input from a user terminal to the server, a means for the server to authenticate the user and store the data in a database, and a means for the server to transmit a request to the generative AI model and receive the generated plot. This allows the user to intuitively input data for story creation, utilize the generative AI model to propose and edit the story development, and finally easily request bookbinding.

[1217] A "user terminal" is a device operated by a user, and examples include PCs, smartphones, tablets, etc.

[1218] "Input means" refers to an interface for a user to input data, and includes a keyboard, a touch screen, a microphone, etc.

[1219] A "generative AI model" is a system that uses artificial intelligence to generate data, specifically a model that generates a plot or storyline based on a prompt.

[1220] "Initial plot generator" refers to a process for generating an initial plot for a story based on a title and theme entered by a user.

[1221] "Display means" refers to an interface for visually displaying data on a user terminal, including a screen or monitor.

[1222] "Storage" refers to a method for long-term storage of data, including databases and file systems.

[1223] "Bookbinding request processing means" refers to a system that processes requests from users to bind their completed stories into actual books.

[1224] A "server" refers to a computer or a series of computer systems that connects to multiple user terminals via a network and processes and stores data.

[1225] "Authentication measures" refers to the process of verifying a user's identity and controlling access to a system, including verifying user IDs and passwords.

[1226] A "database" refers to a system for efficiently storing, retrieving, and managing structured data.

[1227] "Request sending means" refers to a method for sending a user's request to a generative AI model or other system component.

[1228] The present invention relates to a system in which a user creates an original story picture book while interacting with a generative AI model. The system includes a user terminal, a server, and a generative AI model, each of which plays a specific role.

[1229] User terminal

[1230] The user device provides an interface for users (especially children) to input the title and theme of the story. It also has an editor function that displays the plot and story development generated by the generative AI model and allows users to edit, modify, and save the story. User devices can be PCs, smartphones, tablets, or other devices.

[1231] As a concrete example, a user (child) starts an application on the device and enters login information. After successful login, the user clicks the "Create a new story" button, enters the title "Adventure with a Dragon" and the theme "Adventure" in the form, and presses the submit button.

[1232] server

[1233] The server receives input data from user devices and sends it to the generative AI model. It also authenticates users, stores data, sends plot generation requests, and manages binding requests. The server uses standard server hardware and database software (e.g., MySQL, PostgreSQL).

[1234] The server receives the title and theme entered by the user, stores them in a database, and sends a request to the generative AI model to generate an initial plot. For example, the server sends the following prompt to the generative AI model:

[1235] Title: Adventure with Dragons

[1236] Theme: Adventure

[1237] Prompt: One day, a brave boy meets a dragon who guards a hidden treasure...

[1238] Generative AI Models

[1239] The generative AI model generates a plot and storyline based on the input title and theme. For example, OpenAI's GPT-3 model is used. The generative AI model generates an initial plot based on prompts received from the server and sends the plot back to the server.

[1240] A generative AI model might generate a plot like this:

[1241] One day, a brave boy meets a dragon who is guarding a secret treasure. The boy and the dragon decide to join forces to search for the treasure...

[1242] A concrete example of the program's operation flow

[1243] When a user inputs the story title and theme, the user's device sends the data in JSON format to the server. The server stores the data in a database and sends a prompt to the generative AI model to generate an initial plot. The server receives the plot returned by the generative AI model and sends it to the user's device for display.

[1244] When a user begins to input a new storyline, that data is also sent via the server to the generative AI model, which then generates a new storyline. By repeating this process, the user and the generative AI model work together to create a unique story picture book. Once completed, the user submits a binding request, and the server connects the request to the binding service to produce and deliver the picture book.

[1245] Specific technical features

[1246] This system uses a generative AI model to automatically generate plots and story developments, and is equipped with the ability to efficiently process, store, and display data entered by users. The server acts as a central link between the user's device and the generative AI model, ensuring a smooth story creation process. It is also designed to automate the processing of binding requests, allowing users to easily obtain physical picture books.

[1247] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1248] Step 1:

[1249] The user uses the user terminal to input the title and theme of the story. Specifically, the user starts the application, enters the title "Adventure with a Dragon" and the theme "Adventure" in the story creation form, and presses the submit button. This operation generates the input data (title and theme) from the user terminal.

[1250] Step 2:

[1251] The user's device sends the entered title and theme data to the server. The entered data is converted to JSON format and sent as an HTTP POST request to the server's API endpoint. Authentication information such as the user ID must also be included in this process.

[1252] Step 3:

[1253] The server parses the received JSON data and authenticates the user. After successful authentication, it saves the entered title and theme in the database. The server then connects to the database and adds the data as a new entry to the corresponding table.

[1254] Step 4:

[1255] The server sends a request to the generative AI model to generate an initial plot based on the title and theme stored in the database, generates a prompt text, and sends a request including the prompt text to the API endpoint of the generative AI model.

[1256] Step 5:

[1257] The generative AI model generates an initial plot based on the prompt received from the server. For example, it might generate an initial plot such as, "One day, a brave boy meets a dragon. The dragon is guarding a secret treasure..." and send it back to the server.

[1258] Step 6:

[1259] The server receives the initial plot returned from the generative AI model, stores it in a database, and sends an HTTP response containing the initial plot to the user device.

[1260] Step 7:

[1261] The user terminal displays the initial plot received from the server on the user interface. The user checks the plot and inputs a new story development. For example, the user inputs a development such as "A boy named Taro appears, becomes friends with a dragon, and goes on a treasure hunt together."

[1262] Step 8:

[1263] The user device sends the newly entered storyline data to the server, which again converts the input data into JSON format and sends it to the server as an HTTP POST request.

[1264] Step 9:

[1265] The server receives the new storyline data and resends it to the generative AI model. It then generates a new prompt and sends a request to the generative AI model based on that.

[1266] Step 10:

[1267] The generative AI model generates new developments based on the new storylines sent in. For example, it generates a new plot like, "Tarro and Dragon find a treasure chest in a cave deep in the mountains, and inside is an old map..."

[1268] Step 11:

[1269] The server receives the newly generated deployment from the generative AI model, stores it in a database, and sends it to the user's device as an HTTP response.

[1270] Step 12:

[1271] The user's device displays the new developments received from the server on the user interface. The user continues to edit and revise the story until it is complete.

[1272] Step 13:

[1273] The user confirms the completion of the final story and sends a binding request, which is then sent to the server from the user terminal.

[1274] Step 14:

[1275] The server stores the received bookbinding request in a database and sends the request to the bookbinding service, which then processes the book for production and delivery.

[1276] Step 15:

[1277] The server receives a completion notification from the bookbinding service and notifies the user of the completion of binding and delivery information. The user receives the notification and receives the completed picture book.

[1278] (Application example 1)

[1279] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1280] Existing story generation systems lack a means for users to easily share the content they create with other users when creating original story picture books. Another issue is the lack of convenience for use as a content distribution service for parents and children to enjoy together.

[1281] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1282] In this invention, the server includes means for inputting a story title and theme from a user terminal, means for generating an initial plot using a generative AI model, means for displaying the plot received from the generative AI model on the user terminal, means for passing the story development input by the user to the generative AI model, means for displaying a new development proposed by the generative AI model on the user terminal, means for saving the final story edited by the user, means for processing a binding request based on the saved story, and means for delivering the generated plot or story development as a content delivery service, thereby enabling users to easily share the generated story with other users or use it as a content delivery service for parent-child enjoyment.

[1283] A "user terminal" is a terminal that a user uses to input the title and theme of a story and to display the generated plot and story development.

[1284] A "generative AI model" is an artificial intelligence model that automatically generates an initial plot and story development based on the title and theme entered by the user.

[1285] An "initial plot" is the basic structure of a story that the generative AI model initially generates based on the story title and theme provided by the user.

[1286] "Story development" is an item in which the user specifically describes the progress of the story or adds new scenes.

[1287] A "bookbinding request" is a request by a user to produce a completed story as a physical book.

[1288] A "content distribution service" is a service that distributes generated plots and storylines in a form that can be used by other users.

[1289] This invention relates to a system that allows users to create original story picture books while interacting with a generative AI model. This system includes a user terminal, a server, and a generative AI model, and describes the program processing.

[1290] System Overview

[1291] The system of the present invention receives input from the user's device, sends it to the generative AI model via the server, and provides the output from the AI ​​back to the user. Through this process, users (especially children) can experience creating their own story picture book.

[1292] User terminal

[1293] The user device (e.g., smartphone, tablet) provides an interface for inputting the story title and theme. It also has the function of displaying the generated plot and story development. The user device has the following functions:

[1294] 1. Input interface: Provides a form for users to enter the title and theme of the story.

[1295] 2. Display interface: Displays plot suggestions from the generative AI model and story developments entered by the user.

[1296] 3. Editing function: Provide an editor that allows users to edit, modify, and save stories.

[1297] 4. Binding Request: Provides an interface for submitting completed stories as binding requests.

[1298] server

[1299] The server is responsible for sending input data from the user to the generative AI model and returning output data from the generative AI model to the user's device. The hardware used is a cloud server (e.g., AWS, Google Cloud), and the software used is Flask (a lightweight web server framework built in Python). The server has the following functions:

[1300] 1. User authentication: Verify login information and perform authentication.

[1301] 2. Data processing and analysis: Receives input data from the user and passes it to the generative AI model.

[1302] 3. Data storage: Store the stories created by users in a database.

[1303] 4. Generative AI model integration: Send data to the generative AI model and receive the results.

[1304] 5. Binding Request Management: Receive binding requests and coordinate with the binding service.

[1305] 6. Content Distribution: Distributing the generated plot or storyline as a content distribution service.

[1306] Generative AI Models

[1307] Generative AI models (e.g., OpenAI GPT) generate plot suggestions and storylines based on user input data. Generative AI models have the following capabilities:

[1308] 1. Initial plot generation: Generate an initial plot based on the title and theme entered by the user.

[1309] 2. Story development suggestions: Based on the story development input by the user, new ideas and developments are suggested.

[1310] Specific examples

[1311] Let's say a child wants to create a picture book with the theme of "Adventure Story." The user (child) starts the application on the user's device and enters their login information. After successful login, the home screen is displayed and the child clicks the "Create a new story" button. Next, in the form that appears, the child enters the story title "Adventure with a Dragon" and the theme "Adventure," and presses the submit button.

[1312] The server stores the title and theme received from the user in a database and sends a request to the generative AI model to generate an initial plot. When the generative AI model receives the initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure..."), it sends the content to the user's device.

[1313] The user device displays the received initial plot, and the child can think up a new storyline. For example, they can input and submit a story in which a boy named Taro appears, becomes friends with a dragon, and goes on a treasure hunt together.

[1314] The server sends the new storyline input by the child to the generative AI model again and receives the next suggestion. When the server receives a new storyline from the AI ​​model (e.g., "Taro and Dragon find a treasure chest in a cave deep in the mountains, and inside is an old map..."), it sends it to the user's device.

[1315] After the story is completed, the user device displays a button for the child to send the completed story as a bookbinding request. When the child clicks the button, the request is sent to the server, which then works with the bookbinding service to arrange for the picture book to be produced and delivered.

[1316] An example of a prompt is:

[1317] "Title: The Brave Girl and the Magical Forest

[1318] Theme: Adventure

[1319] Generate the plot."

[1320] The format is as follows.

[1321] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1322] Step 1:

[1323] The user device launches the application and enters login information. The user device sends the entered login information to the server, which then authenticates the user. If authentication is successful, the server returns a successful authentication response to the user device. The user device then displays the home screen.

[1324] Step 2:

[1325] The user clicks the "Create a new story" button, which displays a form for entering the story title and theme. The user enters the title and theme in the form and presses the submit button. The user's device sends the input data to the server. The server stores the received title and theme in a database and sends a request to the generative AI model to generate an initial plot.

[1326] Step 3:

[1327] The generative AI model generates an initial plot based on the title and theme received from the user. The generative AI model returns the generated initial plot to the server. After receiving the plot, the server sends its contents to the user's device. The user's device displays the initial plot.

[1328] Step 4:

[1329] The user thinks of a new storyline and inputs the content they want to add. The user's device sends the user's input to the server. The server then sends the input data back to the generative AI model and requests a storyline proposal.

[1330] Step 5:

[1331] The generative AI model proposes a new storyline. The generative AI model generates a storyline and returns the result to the server. The server receives the result and sends the next storyline to the user's device. The user's device displays the generated storyline.

[1332] Step 6:

[1333] The user edits and saves the final story. The user terminal sends the edited story to the server, which stores it in the database. When saving is complete, the server sends a confirmation message to the user terminal.

[1334] Step 7:

[1335] The user clicks a button to request binding. The user terminal sends the request to the server, which receives the request. The server then works with the binding service to arrange for the production and delivery of the picture book. The server also processes the generated plot or story development to distribute to the content distribution service.

[1336] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1337] This invention combines an emotion engine with a system that allows users to create original story picture books while interacting with a generative AI model. Specifically, we explain the configuration of the system, which includes a user terminal, a server, a generative AI model, and an emotion engine, and how each element works together.

[1338] System Overview

[1339] The system of the present invention receives input from a user terminal, sends it to a generative AI model and an emotion engine via a server, and provides the output from the AI ​​and emotion engine back to the user terminal. This allows users (especially children) to experience creating their own story picture books. The following is a detailed description of the system's components and their functions.

[1340] User terminal

[1341] The user terminal is a device that provides an interface for creating stories. Its main functions are as follows:

[1342] 1. Input interface:

[1343] Provide a form for children to enter the title and theme of their story.

[1344] 2. Display interface:

[1345] It displays plot suggestions from generative AI models and storylines input by users.

[1346] 3. Editing features:

[1347] It provides an editor where users can edit, modify and save stories.

[1348] 4. Binding Request:

[1349] Provides an interface for submitting completed stories as binding requests.

[1350] 5. Emotion recognition:

[1351] It has the ability to analyze the user's facial expressions and voice and send the data to the emotion engine.

[1352] server

[1353] The server is responsible for sending input data from users to the generative AI model and emotion engine, and returning output data to the user device. Its main functions are as follows:

[1354] 1. User authentication:

[1355] Verify the user's login information and perform authentication.

[1356] 2. Data processing and analysis:

[1357] User input data is passed to the emotion engine and generative AI model.

[1358] 3. Data Retention:

[1359] Store user-created stories in a database.

[1360] 4. Collaboration with generative AI models and emotion engines:

[1361] The title, theme, and user emotion data are sent to the emotion engine and generative AI model, and the results are returned to the user's device.

[1362] 5. Binding Request Management:

[1363] Receives binding requests and coordinates with binding services.

[1364] Generative AI Models

[1365] The generative AI model generates plot suggestions and story developments based on user input data and emotional data sent from the emotion engine.

[1366] 1. Initial plot generation:

[1367] Generates an initial plot based on the title and theme entered by the user.

[1368] 2. Storyline proposal:

[1369] Based on the story development entered by the user, new ideas and developments are suggested.

[1370] Emotion Engine

[1371] The emotion engine analyzes emotions from the user's facial expressions and voice and passes that data to a generative AI model.

[1372] 1. Sentiment analysis:

[1373] Performs real-time emotion analysis of the user's facial expressions and voice.

[1374] 2. Emotion data generation:

[1375] Emotional data is generated based on the analysis results and passed to the generative AI model.

[1376] Specific examples

[1377] Imagine a child is creating a picture book with the theme of "adventure story."

[1378] Operation on the user device

[1379] The user (child) starts the application on the user's device and enters their login information. After successful login, the home screen is displayed and the child clicks the "Create a new story" button. In the form that is then displayed, the child enters the story title "Adventure with a Dragon" and the theme "Adventure" and presses the submit button.

[1380] Processing on the server

[1381] The server stores the title and theme received from the user in a database and sends the data to the generative AI model and emotion engine. The emotion engine analyzes the user's facial expressions and voice data and passes the generated emotion data to the generative AI model.

[1382] How generative AI models work

[1383] The generative AI model generates an initial plot based on the title, theme, and emotional data, and sends the generated initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure...") back to the server.

[1384] Collaboration between the server and the emotion engine

[1385] The server sends the initial plot received from the generative AI model to the terminal and makes user-level adjustments, taking into account feedback from the emotion engine.

[1386] Viewing and editing on user devices

[1387] The user's device displays the initial plot, and the child thinks up and inputs a new storyline (e.g., "Taro becomes friends with the dragon and they go on a treasure hunt together"). At this time, the emotion engine analyzes the user's emotions from facial expressions and voice and provides feedback in real time. Once input is complete, the child clicks the send button.

[1388] Reconnecting the server and the generative AI model

[1389] The server resends new developments input from the user or emotion engine to the generative AI model, receives new proposals, and the generative AI model generates the next proposal and sends the result back to the server.

[1390] Submit a binding request

[1391] After the story is completed, the user device displays a button to request the completed story to be bound. When the child clicks the button, the request is sent to the server, which then works with the binding service to arrange for the book to be produced and delivered.

[1392] This is the process by which children create their own original story picture books using the system of this invention. By combining it with an emotion engine, dynamic story development and feedback are provided according to the user's emotions, creating a richer, more interactive experience.

[1393] The processing flow will be explained below.

[1394] Step 1:

[1395] The user starts the service application on the user device and the login screen is displayed. The child enters the user name and password and clicks the login button.

[1396] Step 2:

[1397] The server receives the login information and performs user authentication. If authentication is successful, the server starts a user session and returns the home screen data to the device.

[1398] Step 3:

[1399] The device will display a home screen and offer a "Create a new story" button that the child can click to display a form where they can enter a title and theme.

[1400] Step 4:

[1401] The user enters a story title (e.g., "Adventure with a Dragon") and a theme (e.g., "Adventure") and clicks the submit button.

[1402] Step 5:

[1403] The server stores the input title and theme in a database and sends a request to generate an initial plot to the generative AI model. At the same time, the server sends the user's facial expressions and voice data from the user's device to the emotion engine.

[1404] Step 6:

[1405] The generative AI model generates an initial plot based on the title and theme and sends the results back to the server. The emotion engine also analyzes the user's facial expressions and voice in real time to generate emotion data.

[1406] Step 7:

[1407] The server integrates the initial plot received from the generative AI model with the emotion data from the emotion engine and sends it to the terminal.

[1408] Step 8:

[1409] The device displays an initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure..."), which the user reads. The user then inputs a new storyline (e.g., "Taro becomes friends with the dragon and they go on a treasure hunt together") and submits it. At the same time, the emotion engine continues to analyze the user's emotions from their facial expressions and voice.

[1410] Step 9:

[1411] The server receives the newly input storyline and emotion data from the user and sends it to the generative AI model, which then proposes the next storyline.

[1412] Step 10:

[1413] The generative AI model generates the next proposal based on the new storyline and sends the result back to the server (e.g., "Tarro and Dragon find a treasure chest in a cave deep in the mountains, inside which is an old map").

[1414] Step 11:

[1415] The server receives the next suggestion from the generative AI model and sends it to the user device and the emotion engine, which continues to monitor the user's reactions and analyze the emotion data.

[1416] Step 12:

[1417] The device displays new suggestions, and the user can view them and edit the story. Once edits are complete, the user clicks the save button. During this time, the emotion engine continues to analyze emotions.

[1418] Step 13:

[1419] The server receives the final edited story and emotion data from the user and stores it in a database.

[1420] Step 14:

[1421] The user checks the completed story and clicks the binding request button.

[1422] Step 15:

[1423] The server receives the bookbinding request, stores it in a database, and sends the request to the bookbinding service. After receiving confirmation from the bookbinding service, the server arranges for the production and delivery of the picture book.

[1424] Step 16:

[1425] Users receive a bound picture book delivered to their home and enjoy their own original story.

[1426] These are the specific processing steps for creating a story picture book using the system of the present invention. By combining it with an emotion engine, it is possible to provide an interactive story based on the user's emotions, providing a richer experience.

[1427] Example 2

[1428] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1429] In today's digital society, it is important to provide children with the opportunity to express their creativity and create original story picture books. However, existing systems are unable to reflect users' emotions in real time, limiting the interactive story creation experience. Furthermore, the process of binding the created stories is time-consuming, making it difficult for users to use.

[1430] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1431] In this invention, the server includes a means including an emotion engine for analyzing user emotions, a means for transmitting data to the generative AI model, and a means for returning the results to the user terminal. This allows users to receive real-time emotional feedback as they create stories, enabling a richer, more interactive story creation experience. Furthermore, requests for binding the generated stories are efficiently processed, improving user convenience.

[1432] A "user terminal" is a device that allows a user to input and edit the title and theme of a story.

[1433] A "generative AI model" is an artificial intelligence that generates an initial plot based on the title and theme entered by the user and suggests new story developments.

[1434] An "initial plot" is a story outline initially created by a generative AI model based on the title and theme entered by the user.

[1435] The "emotion engine" is a system element that analyzes the user's facial expressions and voice, generates emotional data, and passes it to the generative AI model.

[1436] A "plot" is a basic component that shows the development and progression of a story.

[1437] "Emotion data" is data that indicates an emotional state generated by an emotion engine from the user's facial expressions and voice.

[1438] A "binding request" is a request that a user submits to have a completed story bound into a physical book.

[1439] "Saving" means recording the final story edited by the user in a storage device such as a database.

[1440] "Real-time feedback" means that the emotion engine analyzes the user's emotional state in real time and immediately reflects the results to the user.

[1441] This invention combines an emotion engine with a system that allows users to create original story picture books while interacting with a generative AI model. Specifically, we explain the configuration of the system, which includes a user terminal, a server, a generative AI model, and an emotion engine, and how each element works together.

[1442] System Overview

[1443] The system of the present invention receives input from a user terminal, sends it to a generative AI model and emotion engine via a server, and provides the output from the AI ​​and emotion engine back to the user terminal, thereby providing users (especially children) with the experience of creating their own story picture books.

[1444] User terminal

[1445] The user terminal is a device that provides an interface for creating stories. Its main functions are as follows:

[1446] 1. Input interface:

[1447] Provide a form for the child to enter the title and theme of the story. For example, enter the title "Adventure with a Dragon" and the theme "Adventure."

[1448] 2. Display interface:

[1449] It displays plot suggestions from generative AI models and storylines input by users.

[1450] 3. Editing features:

[1451] It provides an editor where users can edit, modify and save stories. Users can add their own ideas or modify existing stories.

[1452] 4. Binding Request:

[1453] It provides a button to submit the completed story as a binding request. Clicking the button sends the request to the server.

[1454] 5. Emotion recognition:

[1455] It has the ability to analyze the user's facial expressions and voice and send that data to the emotion engine.

[1456] server

[1457] The server is responsible for sending input data from users to the generative AI model and emotion engine, and returning output data to the user device. Its main functions are as follows:

[1458] 1. User authentication:

[1459] Verify and authenticate the user's login information. For example, the user enters login information and the server verifies it.

[1460] 2. Data processing and analysis:

[1461] User input data is passed to the emotion engine and generative AI model.

[1462] 3. Data Retention:

[1463] Store user-created stories in a database.

[1464] 4. Collaboration with generative AI models and emotion engines:

[1465] The title, theme, and user emotion data are sent to the emotion engine and generative AI model, and the results are returned to the user's device.

[1466] 5. Binding Request Management:

[1467] Receives binding requests and coordinates with binding services.

[1468] Generative AI Models

[1469] The generative AI model generates plot suggestions and story developments based on user input data and emotional data sent from the emotion engine.

[1470] 1. Initial plot generation:

[1471] Generates an initial plot based on the title and theme entered by the user. For example, create a plot based on "Adventures with Dragons."

[1472] 2. Storyline proposal:

[1473] Based on the story development entered by the user, new ideas and developments are suggested.

[1474] Emotion Engine

[1475] The emotion engine analyzes emotions from the user's facial expressions and voice and passes that data to a generative AI model.

[1476] 1. Sentiment analysis:

[1477] Performs real-time emotion analysis of the user's facial expressions and voice.

[1478] 2. Emotion data generation:

[1479] Emotional data is generated based on the analysis results and passed to the generative AI model.

[1480] Specific examples

[1481] Operation on the user device

[1482] The user (child) launches the application on their device and enters their login information. After successfully logging in, the home screen appears and the child clicks the "Create a new story" button. In the form that appears next, the child enters "Adventure with a Dragon" and "Adventure" and presses the submit button.

[1483] Processing on the server

[1484] The server stores the title and theme received from the user in a database and sends the data to the generative AI model and emotion engine. The emotion engine analyzes the user's facial expressions and voice data and passes the generated emotion data to the generative AI model.

[1485] How generative AI models work

[1486] The generative AI model generates an initial plot based on the title, theme, and emotional data, for example, "One day, a brave boy meets a dragon who is guarding a secret treasure," and sends it back to the server.

[1487] Collaboration between the server and the emotion engine

[1488] The server sends the initial plot received from the generative AI model to the user's device and makes adjustments taking into account feedback from the emotion engine.

[1489] Viewing and editing on user devices

[1490] The user's device displays the initial plot, and the user can think of and input a new story development. For example, they can input "Taro becomes friends with a dragon and they go on a treasure hunt together." At this time, the emotion engine analyzes the user's emotions from facial expressions and voice and provides feedback in real time. Once input is complete, the user clicks the send button.

[1491] Reconnecting the server and the generative AI model

[1492] The server resubmits the new deployment data to the generative AI model, which then generates the next proposal and sends the result back to the server.

[1493] Submit a binding request

[1494] After the story is completed, the user's device displays a button for sending the completed story as a bookbinding request. When the user clicks the button, the request is sent to the server, which then works with the bookbinding service to produce the picture book and arrange for its delivery.

[1495] Examples of prompt statements

[1496] As an example of a prompt for a generative AI model, the following could be considered as input to generate the initial plot of a story themed around "adventures with dragons":

[1497] "The user has entered the title 'Adventure with the Dragon' and the theme 'Adventure'. Based on this, please generate an initial plot for the story, starting with the scene where the boy meets the dragon."

[1498] This concludes the detailed explanation of how to use the system of the present invention. This system allows users to interactively create original stories while receiving emotional feedback, and ultimately to complete the process of bookbinding.

[1499] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1500] Step 1:

[1501] Log in

[1502] The user accesses the login screen and enters their login information. The server receives the login information (input) sent by the user and authenticates it by comparing it with information in the database (data calculation). If authentication is successful, the home screen is displayed on the user's device (output).

[1503] Step 2:

[1504] Start creating your story

[1505] The user clicks the "Create a new story" button on the home screen. The user's device detects the click event (input) and displays a new story creation form (output).

[1506] Step 3:

[1507] Enter the title and theme

[1508] The user enters the story title and theme on a new screen. The user's device receives the title "Adventure with a Dragon" and theme "Adventure" entered by the user through an input form (input). The information entered into the form is temporarily stored in the device's memory.

[1509] Step 4:

[1510] Sending input data

[1511] The user presses the send button to send the data to the server. The user's device sends the title and theme data (input) to the server and displays a notification of successful transmission to the user (output).

[1512] Step 5:

[1513] User Authentication

[1514] The server checks the user's login information and performs authentication. If authentication is successful, the server saves the user data in the database (data calculation).

[1515] Step 6:

[1516] Data storage

[1517] The server stores the received title and theme in a database (output). The server records this data in a database in an appropriate format for further processing.

[1518] Step 7:

[1519] Sending data to a generative AI model

[1520] The server sends the title and theme data to the generative AI model. It sends the title "Adventure with Dragons" and the theme "Adventure" (input) to the generative AI model via API, and receives the generated initial plot (output).

[1521] Step 8:

[1522] Sending data to the emotion engine

[1523] The server sends the user's emotional data to the emotion engine. It also sends facial expression and voice data (input) acquired from the user's device to the emotion engine and receives the analysis results (output).

[1524] Step 9:

[1525] Receiving plots from generative AI models

[1526] The server receives the initial plot from the generative AI model. The initial plot data (output) is "One day, a brave boy meets a dragon. The dragon guards a secret treasure..." and is used for the next process.

[1527] Step 10:

[1528] Receiving feedback from the emotion engine

[1529] The server receives feedback from the emotion engine and understands the user's emotional state. Based on the feedback data (output), it modifies the output results of the generative AI model (data calculation).

[1530] Step 11:

[1531] Displaying the initial plot

[1532] The user terminal displays the initial plot to the user, displays the received initial plot data (input) on the screen, and asks the user for feedback (output).

[1533] Step 12:

[1534] User edits

[1535] The user thinks up and inputs a new story development based on the initial plot. The user's input data, "Taro becomes friends with the dragon and goes on a treasure hunt together" (input), is acquired by the user's device and temporarily saved as edited data.

[1536] Step 13:

[1537] Send edited data

[1538] The user resends the edited data to the server. The user terminal sends the edited new story development data (input) to the server and displays a notification that transmission is complete (output).

[1539] Step 14:

[1540] Resubmit to generative AI model

[1541] The server resubmits new story development data to the generative AI model. It sends new story development data (input) via API and receives new proposals from the generative AI model (output).

[1542] Step 15:

[1543] Receiving a resubmission

[1544] The server receives new proposals from the generative AI model. It receives new story development proposal data (output) from the generative AI model and uses it for further processing.

[1545] Step 16:

[1546] View the final story

[1547] The user terminal displays the final story development to the user. The generated final story data (input) is displayed on the screen and the user is prompted to confirm it (output).

[1548] Step 17:

[1549] Submit a binding request

[1550] The user submits the completed story as a binding request. The user terminal transmits the binding request data (input) to the server and displays a notification of completion of transmission (output).

[1551] Step 18:

[1552] Collaboration with bookbinding services

[1553] The server receives the binding request and works with the binding service to produce and arrange for delivery of the picture book. It sends the binding request data (input) to the binding service and records the binding progress in a database (output).

[1554] (Application example 2)

[1555] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1556] Conventional story creation systems lacked the ability to provide feedback based on the user's emotions or adjust the storyline, which prevented them from fully enriching the user's experience. Furthermore, they lacked the interactive elements that children needed to have fun creating stories, which led to a lack of interest.

[1557] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting the title and theme of the story from a user terminal, means for generating an initial plot using a generative AI model, means for displaying the plot received from the generative AI model on the user terminal, means for passing the story development input by the user to the generative AI model, means for displaying a new development proposed by the generative AI model on the user terminal, means for saving the final story edited by the user, means for processing a binding request based on the saved story, and means for adjusting the story feedback displayed on the user terminal based on the emotion, including an emotion engine that analyzes the user's facial expressions and voice to generate emotion data. This allows for dynamic story development and feedback in accordance with the user's emotions, enabling a richer and more interactive experience.

[1558] A "user terminal" is a device that provides an interface for creating a story and receives input from a user.

[1559] A "generative AI model" is an artificial intelligence that generates an initial plot or new story development based on user input data.

[1560] The "initial plot" is the initial story development that is generated based on the story title and theme entered by the user.

[1561] A "plot" indicates the story's outline and development pattern, and is generated by a generative AI model.

[1562] Theme refers to the overall theme or central idea of ​​the story.

[1563] The "emotion engine" is an engine that analyzes emotions from the user's facial expressions and voice, and adjusts the story development through the generated emotional data.

[1564] A "bookbinding request" is a procedure in which a user requests that a completed story be produced as a physical picture book.

[1565] "Emotion data" refers to emotional information analyzed from the user's facial expressions and voice, and is reflected in the story development.

[1566] "Interface" refers to the screen display and input methods on a user terminal that provide the means for a user to interact with a system.

[1567] "Feedback" refers to the information or suggestions that a system returns in response to a user's input or behavior.

[1568] To implement this invention, a user terminal on which a dedicated application is installed, a server located on the cloud, a generative AI model, and an emotion engine are used. Details of these are described below.

[1569] Overall system configuration

[1570] The system includes the following major components:

[1571] 1. User Device:

[1572] Input interface: Accepts title and theme input from the user.

[1573] Display interface: Shows the plot from the generative AI model and the final story.

[1574] Editing capabilities: Users can create, edit, and save stories.

[1575] Emotion Recognition: Uses a camera and microphone to capture the user's facial expressions and voice to generate emotional data.

[1576] Bookbinding Requests: Processing requests to have completed stories produced as physical picture books.

[1577] 2. Server:

[1578] User authentication: Verify the user's login information and perform authentication.

[1579] Data processing and analysis: Sending user input data to the emotion engine and generative AI model.

[1580] Data storage: Store user-created stories in a database.

[1581] Collaboration with generative AI models and emotion engines: Processes title, theme, and emotion data and returns the results to the user's device.

[1582] Binding Request Management: Send binding requests to binding services.

[1583] 3. Generative AI models (e.g., GPT-4):

[1584] Initial plot generation: Generates the basic plot of your story based on the title and theme.

[1585] Story development proposal: Based on new development input from the user, the next development proposal is made.

[1586] 4. Emotion engine (e.g. Affectiva SDK):

[1587] Sentiment analysis: Analyzes the user's facial expressions and voice to generate emotional data.

[1588] Emotional feedback: Adjust the storyline based on the generated emotional data.

[1589] Software and Hardware Configuration

[1590] User device: Smartphone application (e.g., using React Native)

[1591] Server: Node.js, Express framework, database (e.g. MongoDB)

[1592] Emotion recognition: Expo Camera, Expo Audio

[1593] Generative AI model: OpenAI GPT-4 API

[1594] Data processing flow

[1595] 1. Getting user input:

[1596] The user enters the story title and theme on their device and sends it to the server.

[1597] 2. Emotion capture and analysis:

[1598] It uses the camera and microphone on the user's device to capture facial expressions and voice.

[1599] The captured data is sent to the emotion engine via the server, where emotions are analyzed.

[1600] 3. Collaboration with generative AI models:

[1601] The server sends user input and emotion data to the generative AI model.

[1602] The generative AI model generates an initial plot and sends it to the user's device via the server.

[1603] 4. Providing User Feedback:

[1604] As users edit the storyline, the emotion engine provides real-time emotional feedback that is sent to the generative AI model.

[1605] The generative AI model generates new suggestions, which are displayed on the user's device.

[1606] 5. Processing binding requests:

[1607] Users submit completed stories as binding requests.

[1608] The server works with the bookbinding service to create the physical picture book and arrange for delivery.

[1609] Examples and prompts

[1610] Title: "The Adventures of Brave Taro"

[1611] Theme: Adventure

[1612] Scene: "Tarro meets a dragon and becomes friends with him."

[1613] Example prompts to input to a generative AI model:

[1614] Title: The Adventures of the Brave Taro

[1615] Theme: Adventure

[1616] Feedback emotion: Fun

[1617] Story Current Scene: Taro meets a dragon and they go on a treasure hunt together to become friends.

[1618] Please suggest the following expansion:

[1619] In this way, a system can be implemented that provides users with a fun, emotionally driven story-creation experience.

[1620] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1621] Step 1:

[1622] The user inputs the title and theme of the story. The user's device receives these inputs and sends them to the server.

[1623] Input: User title and theme

[1624] Output: Title and theme sent to server

[1625] Specific operation: When a user enters characters into an input field on the user terminal and presses the send button, this data is sent to the server.

[1626] Step 2:

[1627] The server sends the title and theme to the generative AI model, which generates the initial plot and sends the results back to the server.

[1628] Input: Title and Theme

[1629] Output: Initial plot

[1630] What it does: The server sends the title and theme to the generative AI model and receives the generated initial plot.

[1631] Step 3:

[1632] The server sends the initial plot to the user terminal, which displays it.

[1633] Input: Initial plot

[1634] Output: Initial plot displayed on the user's terminal

[1635] Specific operation: The initial plot received from the server is displayed on the display screen of the user's terminal.

[1636] Step 4:

[1637] The user inputs the continuation of the story. The user's device receives this input and sends it to the server.

[1638] Input: User storyline

[1639] Output: Sending the storyline to the server

[1640] Specific behavior: When the user enters a storyline into the input field and presses the submit button, it is sent to the server.

[1641] Step 5:

[1642] The user device captures the user's facial expressions and voice in real time and sends them to the emotion engine for analysis.

[1643] Input: User's facial expressions and voice

[1644] Output: Emotion data

[1645] Specific operation: Uses the camera and microphone on the user's device to capture facial expression and voice data and sends it to the emotion engine.

[1646] Step 6:

[1647] The emotion engine analyzes the user's facial expressions and voice to generate emotion data, which is then sent to the server.

[1648] Input: facial expression data and voice data

[1649] Output: Emotion data

[1650] Specific operation: The emotion engine analyzes the received data, determines the user's emotions, generates emotion data, and sends it to the server.

[1651] Step 7:

[1652] The server sends the emotional data and the user's storyline to a generative AI model, which then generates a proposal for the next storyline.

[1653] Input: Emotion data and storyline

[1654] Output: Next storyline proposal

[1655] Specific operation: The server sends the story development and emotional data from the user to the generative AI model, which then generates a proposal for the next development and returns it to the server.

[1656] Step 8:

[1657] The server transmits the next story development proposal to the user terminal, which displays it.

[1658] Input: Next storyline suggestion

[1659] Output: Next expansion suggestion displayed on user terminal

[1660] Specific operation: The proposal received from the server is sent to the user terminal, and is displayed on the display screen of the user terminal.

[1661] Step 9:

[1662] The user completes the story and submits a binding request, which the user terminal sends to the server.

[1663] Input: Final story

[1664] Output: Binding request

[1665] Specific behavior: The user checks the final story and presses the binding request button, which sends the request to the server.

[1666] Step 10:

[1667] The server receives the binding request, works with the binding service to produce the picture book, and arranges for delivery.

[1668] Input: Binding request

[1669] Output: Physical storybook

[1670] Specific operation: After the server receives the binding request, it requests the binding service to produce and deliver the picture book.

[1671] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1672] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1673] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1674] [Fourth embodiment]

[1675] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1676] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1677] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1678] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1679] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1680] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1681] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1682] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1683] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1684] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1685] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1686] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1687] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1688] This paper describes a system that allows users to create original story picture books while interacting with a generative AI model. Specifically, it describes the configuration of the system, which includes a user terminal, a server, and a generative AI model, and the processing of the program.

[1689] System Overview

[1690] The system of the present invention receives input from the user's device, sends it to the generative AI model via the server, and then provides the output from the AI ​​back to the user. Through this process, users (especially children) are given the experience of creating their own story picture book. The system configuration and the functions of each element are described in detail below.

[1691] User terminal

[1692] The user device (e.g., PC, smartphone, tablet) provides an interface for the child to input the title and theme of the story.

[1693] 1. Input interface:

[1694] Provide a form for children to enter the title and theme of their story.

[1695] 2. Display interface:

[1696] Displays plot suggestions from AI and story developments entered by users.

[1697] 3. Editing features:

[1698] It provides an editor where users can edit, modify and save stories.

[1699] 4. Binding Request:

[1700] Provides an interface for submitting completed stories as binding requests.

[1701] server

[1702] The server is responsible for sending input data from users to the generative AI model and returning output data from the generative AI model to the user's device. It also stores data and manages binding requests. Its main functions are as follows:

[1703] 1. User authentication:

[1704] Verify your login information and authenticate.

[1705] 2. Data processing and analysis:

[1706] Receives input data from the user and passes it to the generative AI model.

[1707] 3. Data Retention:

[1708] Store user-created stories in a database.

[1709] 4. Generative AI model collaboration:

[1710] Send data to the AI ​​model and receive the results.

[1711] 5. Binding Request Management:

[1712] Receives binding requests and coordinates with binding services.

[1713] Generative AI Models

[1714] The generative AI model generates plot suggestions and storylines based on user input data.

[1715] 1. Initial plot generation:

[1716] Generates an initial plot based on the title and theme entered by the user.

[1717] 2. Storyline proposal:

[1718] Based on the story development entered by the user, new ideas and developments are suggested.

[1719] Specific examples

[1720] Imagine a child is creating a picture book with the theme of "adventure story."

[1721] Operation on the user device

[1722] The user (child) launches the application on the user's device and enters their login information. If login is successful, the home screen is displayed and the child clicks the "Create a new story" button. Next, they enter the story title "Adventure with a Dragon" and the theme "Adventure" in the form that appears, and press the submit button.

[1723] Processing on the server

[1724] The server stores the title and theme received from the user in a database and sends a request to the generative AI model to generate an initial plot. When the generative AI model receives the initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure..."), it sends the content to the user's device.

[1725] Viewing and editing on user devices

[1726] The user device displays the received initial plot, and the child can think up a new storyline. For example, they can input and submit a story in which a boy named Taro appears, becomes friends with a dragon, and goes on a treasure hunt together.

[1727] Collaboration between the server and the generative AI model

[1728] The server sends the new storyline input by the child to the generative AI model again and receives the next suggestion. When the server receives a new storyline from the AI ​​model (e.g., "Taro and Dragon find a treasure chest in a cave deep in the mountains, and inside is an old map..."), it sends it to the user's device.

[1729] Submit a binding request

[1730] After the story is completed, the user device displays a button for the child to send the completed story as a bookbinding request. When the child clicks the button, the request is sent to the server, which then works with the bookbinding service to arrange for the picture book to be produced and delivered.

[1731] This is the process by which children can create their own original story picture books using the system of the present invention, which not only allows children to exercise their creativity and learn while having fun, but also gives them a sense of accomplishment by having their own unique picture book.

[1732] The processing flow will be explained below.

[1733] Step 1:

[1734] The user starts the service application on the user device and the login screen is displayed. The child enters the user name and password and clicks the login button.

[1735] Step 2:

[1736] The server receives the login information and performs authentication. If authentication is successful, the server starts the user session and returns the home screen data to the device.

[1737] Step 3:

[1738] The device will display a home screen and offer a "Create a new story" button that the child can click to display a form where they can enter a title and theme.

[1739] Step 4:

[1740] The user enters a story title (e.g., "Adventure with a Dragon") and a theme (e.g., "Adventure") and clicks the submit button.

[1741] Step 5:

[1742] The server receives the input title and theme, stores them in a database, and then sends a request to the generative AI model to generate the initial plot.

[1743] Step 6:

[1744] The generative AI model generates an initial plot based on the title and theme, and sends the generated plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure...") back to the server.

[1745] Step 7:

[1746] The server sends the initial plot received from the generative AI model to the terminal.

[1747] Step 8:

[1748] The device displays the initial plot. The user (child) reads the plot and thinks of a new storyline to input (e.g., "Taro becomes friends with the dragon and they go on a treasure hunt together."). When they're done, they click the send button.

[1749] Step 9:

[1750] The server receives the new storyline input from the user and sends it back to the generative AI model, requesting new proposals.

[1751] Step 10:

[1752] The generative AI model generates the next proposal based on the new storyline and sends the result back to the server (e.g., "Tarro and Dragon find a treasure chest in a cave deep in the mountains, inside which is an old map").

[1753] Step 11:

[1754] The server sends the next proposal received from the generative AI model to the device.

[1755] Step 12:

[1756] The device will display a new suggestion, and the user can view the suggestion and edit the story. Once they've finished editing, they can click the save button again. When the final story is complete, they'll be given the option to request binding.

[1757] Step 13:

[1758] The user checks the completed story and clicks the binding request button.

[1759] Step 14:

[1760] The server receives the bookbinding request, stores it in a database, and sends the request to the bookbinding service. After receiving confirmation from the bookbinding service, the server arranges for the production and delivery of the picture book.

[1761] Step 15:

[1762] Users receive a bound picture book delivered to their home and enjoy their own original story.

[1763] Example 1

[1764] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1765] Conventional story creation systems require users to do a lot of manual work when creating a story, making the creation process complicated and time-consuming. They also lacked adequate support to fully bring out users' creativity, especially functions that allow children to enjoy creative activities. Furthermore, the process of binding completed stories into books was complicated, making it difficult for users to easily obtain the finished product.

[1766] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1767] In this invention, the server includes a means for transmitting data input from a user terminal to the server, a means for the server to authenticate the user and store the data in a database, and a means for the server to transmit a request to the generative AI model and receive the generated plot. This allows the user to intuitively input data for story creation, utilize the generative AI model to propose and edit the story development, and finally easily request bookbinding.

[1768] A "user terminal" is a device operated by a user, and examples include PCs, smartphones, tablets, etc.

[1769] "Input means" refers to an interface for a user to input data, and includes a keyboard, a touch screen, a microphone, etc.

[1770] A "generative AI model" is a system that uses artificial intelligence to generate data, specifically a model that generates a plot or storyline based on a prompt.

[1771] "Initial plot generator" refers to a process for generating an initial plot for a story based on a title and theme entered by a user.

[1772] "Display means" refers to an interface for visually displaying data on a user terminal, including a screen or monitor.

[1773] "Storage" refers to a method for long-term storage of data, including databases and file systems.

[1774] "Bookbinding request processing means" refers to a system that processes requests from users to bind their completed stories into actual books.

[1775] A "server" refers to a computer or a series of computer systems that connects to multiple user terminals via a network and processes and stores data.

[1776] "Authentication measures" refers to the process of verifying a user's identity and controlling access to a system, including verifying user IDs and passwords.

[1777] A "database" refers to a system for efficiently storing, retrieving, and managing structured data.

[1778] "Request sending means" refers to a method for sending a user's request to a generative AI model or other system component.

[1779] The present invention relates to a system in which a user creates an original story picture book while interacting with a generative AI model. The system includes a user terminal, a server, and a generative AI model, each of which plays a specific role.

[1780] User terminal

[1781] The user device provides an interface for users (especially children) to input the title and theme of the story. It also has an editor function that displays the plot and story development generated by the generative AI model and allows users to edit, modify, and save the story. User devices can be PCs, smartphones, tablets, or other devices.

[1782] As a concrete example, a user (child) starts an application on the device and enters login information. After successful login, the user clicks the "Create a new story" button, enters the title "Adventure with a Dragon" and the theme "Adventure" in the form, and presses the submit button.

[1783] server

[1784] The server receives input data from user devices and sends it to the generative AI model. It also authenticates users, stores data, sends plot generation requests, and manages binding requests. The server uses standard server hardware and database software (e.g., MySQL, PostgreSQL).

[1785] The server receives the title and theme entered by the user, stores them in a database, and sends a request to the generative AI model to generate an initial plot. For example, the server sends the following prompt to the generative AI model:

[1786] Title: Adventure with Dragons

[1787] Theme: Adventure

[1788] Prompt: One day, a brave boy meets a dragon who guards a hidden treasure...

[1789] Generative AI Models

[1790] The generative AI model generates a plot and storyline based on the input title and theme. For example, OpenAI's GPT-3 model is used. The generative AI model generates an initial plot based on prompts received from the server and sends the plot back to the server.

[1791] A generative AI model might generate a plot like this:

[1792] One day, a brave boy meets a dragon who is guarding a secret treasure. The boy and the dragon decide to join forces to search for the treasure...

[1793] A concrete example of the program's operation flow

[1794] When a user inputs the story title and theme, the user's device sends the data in JSON format to the server. The server stores the data in a database and sends a prompt to the generative AI model to generate an initial plot. The server receives the plot returned by the generative AI model and sends it to the user's device for display.

[1795] When a user begins to input a new storyline, that data is also sent via the server to the generative AI model, which then generates a new storyline. By repeating this process, the user and the generative AI model work together to create a unique story picture book. Once completed, the user submits a binding request, and the server connects the request to the binding service to produce and deliver the picture book.

[1796] Specific technical features

[1797] This system uses a generative AI model to automatically generate plots and story developments, and is equipped with the ability to efficiently process, store, and display data entered by users. The server acts as a central link between the user's device and the generative AI model, ensuring a smooth story creation process. It is also designed to automate the processing of binding requests, allowing users to easily obtain physical picture books.

[1798] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1799] Step 1:

[1800] The user uses the user terminal to input the title and theme of the story. Specifically, the user starts the application, enters the title "Adventure with a Dragon" and the theme "Adventure" in the story creation form, and presses the submit button. This operation generates the input data (title and theme) from the user terminal.

[1801] Step 2:

[1802] The user's device sends the entered title and theme data to the server. The entered data is converted to JSON format and sent as an HTTP POST request to the server's API endpoint. Authentication information such as the user ID must also be included in this process.

[1803] Step 3:

[1804] The server parses the received JSON data and authenticates the user. After successful authentication, it saves the entered title and theme in the database. The server then connects to the database and adds the data as a new entry to the corresponding table.

[1805] Step 4:

[1806] The server sends a request to the generative AI model to generate an initial plot based on the title and theme stored in the database, generates a prompt text, and sends a request including the prompt text to the API endpoint of the generative AI model.

[1807] Step 5:

[1808] The generative AI model generates an initial plot based on the prompt received from the server. For example, it might generate an initial plot such as, "One day, a brave boy meets a dragon. The dragon is guarding a secret treasure..." and send it back to the server.

[1809] Step 6:

[1810] The server receives the initial plot returned from the generative AI model, stores it in a database, and sends an HTTP response containing the initial plot to the user device.

[1811] Step 7:

[1812] The user terminal displays the initial plot received from the server on the user interface. The user checks the plot and inputs a new story development. For example, the user inputs a development such as "A boy named Taro appears, becomes friends with a dragon, and goes on a treasure hunt together."

[1813] Step 8:

[1814] The user device sends the newly entered storyline data to the server, which again converts the input data into JSON format and sends it to the server as an HTTP POST request.

[1815] Step 9:

[1816] The server receives the new storyline data and resends it to the generative AI model. It then generates a new prompt and sends a request to the generative AI model based on that.

[1817] Step 10:

[1818] The generative AI model generates new developments based on the new storylines sent in. For example, it generates a new plot like, "Tarro and Dragon find a treasure chest in a cave deep in the mountains, and inside is an old map..."

[1819] Step 11:

[1820] The server receives the newly generated deployment from the generative AI model, stores it in a database, and sends it to the user's device as an HTTP response.

[1821] Step 12:

[1822] The user's device displays the new developments received from the server on the user interface. The user continues to edit and revise the story until it is complete.

[1823] Step 13:

[1824] The user confirms the completion of the final story and sends a binding request, which is then sent to the server from the user terminal.

[1825] Step 14:

[1826] The server stores the received bookbinding request in a database and sends the request to the bookbinding service, which then processes the book for production and delivery.

[1827] Step 15:

[1828] The server receives a completion notification from the bookbinding service and notifies the user of the completion of binding and delivery information. The user receives the notification and receives the completed picture book.

[1829] (Application example 1)

[1830] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1831] Existing story generation systems lack a means for users to easily share the content they create with other users when creating original story picture books. Another issue is the lack of convenience for use as a content distribution service for parents and children to enjoy together.

[1832] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1833] In this invention, the server includes means for inputting a story title and theme from a user terminal, means for generating an initial plot using a generative AI model, means for displaying the plot received from the generative AI model on the user terminal, means for passing the story development input by the user to the generative AI model, means for displaying a new development proposed by the generative AI model on the user terminal, means for saving the final story edited by the user, means for processing a binding request based on the saved story, and means for delivering the generated plot or story development as a content delivery service, thereby enabling users to easily share the generated story with other users or use it as a content delivery service for parent-child enjoyment.

[1834] A "user terminal" is a terminal that a user uses to input the title and theme of a story and to display the generated plot and story development.

[1835] A "generative AI model" is an artificial intelligence model that automatically generates an initial plot and story development based on the title and theme entered by the user.

[1836] An "initial plot" is the basic structure of a story that the generative AI model initially generates based on the story title and theme provided by the user.

[1837] "Story development" is an item in which the user specifically describes the progress of the story or adds new scenes.

[1838] A "bookbinding request" is a request by a user to produce a completed story as a physical book.

[1839] A "content distribution service" is a service that distributes generated plots and storylines in a form that can be used by other users.

[1840] This invention relates to a system that allows users to create original story picture books while interacting with a generative AI model. This system includes a user terminal, a server, and a generative AI model, and describes the program processing.

[1841] System Overview

[1842] The system of the present invention receives input from the user's device, sends it to the generative AI model via the server, and provides the output from the AI ​​back to the user. Through this process, users (especially children) can experience creating their own story picture book.

[1843] User terminal

[1844] The user device (e.g., smartphone, tablet) provides an interface for inputting the story title and theme. It also has the function of displaying the generated plot and story development. The user device has the following functions:

[1845] 1. Input interface: Provides a form for users to enter the title and theme of the story.

[1846] 2. Display interface: Displays plot suggestions from the generative AI model and story developments entered by the user.

[1847] 3. Editing function: Provide an editor that allows users to edit, modify, and save stories.

[1848] 4. Binding Request: Provides an interface for submitting completed stories as binding requests.

[1849] server

[1850] The server is responsible for sending input data from the user to the generative AI model and returning output data from the generative AI model to the user's device. The hardware used is a cloud server (e.g., AWS, Google Cloud), and the software used is Flask (a lightweight web server framework built in Python). The server has the following functions:

[1851] 1. User authentication: Verify login information and perform authentication.

[1852] 2. Data processing and analysis: Receives input data from the user and passes it to the generative AI model.

[1853] 3. Data storage: Store the stories created by users in a database.

[1854] 4. Generative AI model integration: Send data to the generative AI model and receive the results.

[1855] 5. Binding Request Management: Receive binding requests and coordinate with the binding service.

[1856] 6. Content Distribution: Distributing the generated plot or storyline as a content distribution service.

[1857] Generative AI Models

[1858] Generative AI models (e.g., OpenAI GPT) generate plot suggestions and storylines based on user input data. Generative AI models have the following capabilities:

[1859] 1. Initial plot generation: Generate an initial plot based on the title and theme entered by the user.

[1860] 2. Story development suggestions: Based on the story development input by the user, new ideas and developments are suggested.

[1861] Specific examples

[1862] Let's say a child wants to create a picture book with the theme of "Adventure Story." The user (child) starts the application on the user's device and enters their login information. After successful login, the home screen is displayed and the child clicks the "Create a new story" button. Next, in the form that appears, the child enters the story title "Adventure with a Dragon" and the theme "Adventure," and presses the submit button.

[1863] The server stores the title and theme received from the user in a database and sends a request to the generative AI model to generate an initial plot. When the generative AI model receives the initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure..."), it sends the content to the user's device.

[1864] The user device displays the received initial plot, and the child can think up a new storyline. For example, they can input and submit a story in which a boy named Taro appears, becomes friends with a dragon, and goes on a treasure hunt together.

[1865] The server sends the new storyline input by the child to the generative AI model again and receives the next suggestion. When the server receives a new storyline from the AI ​​model (e.g., "Taro and Dragon find a treasure chest in a cave deep in the mountains, and inside is an old map..."), it sends it to the user's device.

[1866] After the story is completed, the user device displays a button for the child to send the completed story as a bookbinding request. When the child clicks the button, the request is sent to the server, which then works with the bookbinding service to arrange for the picture book to be produced and delivered.

[1867] An example of a prompt is:

[1868] "Title: The Brave Girl and the Magical Forest

[1869] Theme: Adventure

[1870] Generate the plot."

[1871] The format is as follows.

[1872] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1873] Step 1:

[1874] The user device launches the application and enters login information. The user device sends the entered login information to the server, which then authenticates the user. If authentication is successful, the server returns a successful authentication response to the user device. The user device then displays the home screen.

[1875] Step 2:

[1876] The user clicks the "Create a new story" button, which displays a form for entering the story title and theme. The user enters the title and theme in the form and presses the submit button. The user's device sends the input data to the server. The server stores the received title and theme in a database and sends a request to the generative AI model to generate an initial plot.

[1877] Step 3:

[1878] The generative AI model generates an initial plot based on the title and theme received from the user. The generative AI model returns the generated initial plot to the server. After receiving the plot, the server sends its contents to the user's device. The user's device displays the initial plot.

[1879] Step 4:

[1880] The user thinks of a new storyline and inputs the content they want to add. The user's device sends the user's input to the server. The server then sends the input data back to the generative AI model and requests a storyline proposal.

[1881] Step 5:

[1882] The generative AI model proposes a new storyline. The generative AI model generates a storyline and returns the result to the server. The server receives the result and sends the next storyline to the user's device. The user's device displays the generated storyline.

[1883] Step 6:

[1884] The user edits and saves the final story. The user terminal sends the edited story to the server, which stores it in the database. When saving is complete, the server sends a confirmation message to the user terminal.

[1885] Step 7:

[1886] The user clicks a button to request binding. The user terminal sends the request to the server, which receives the request. The server then works with the binding service to arrange for the production and delivery of the picture book. The server also processes the generated plot or story development to distribute to the content distribution service.

[1887] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1888] This invention combines an emotion engine with a system that allows users to create original story picture books while interacting with a generative AI model. Specifically, we explain the configuration of the system, which includes a user terminal, a server, a generative AI model, and an emotion engine, and how each element works together.

[1889] System Overview

[1890] The system of the present invention receives input from a user terminal, sends it to a generative AI model and an emotion engine via a server, and provides the output from the AI ​​and emotion engine back to the user terminal. This allows users (especially children) to experience creating their own story picture books. The following is a detailed description of the system's components and their functions.

[1891] User terminal

[1892] The user terminal is a device that provides an interface for creating stories. Its main functions are as follows:

[1893] 1. Input interface:

[1894] Provide a form for children to enter the title and theme of their story.

[1895] 2. Display interface:

[1896] It displays plot suggestions from generative AI models and storylines input by users.

[1897] 3. Editing features:

[1898] It provides an editor where users can edit, modify and save stories.

[1899] 4. Binding Request:

[1900] Provides an interface for submitting completed stories as binding requests.

[1901] 5. Emotion recognition:

[1902] It has the ability to analyze the user's facial expressions and voice and send the data to the emotion engine.

[1903] server

[1904] The server is responsible for sending input data from users to the generative AI model and emotion engine, and returning output data to the user device. Its main functions are as follows:

[1905] 1. User authentication:

[1906] Verify the user's login information and perform authentication.

[1907] 2. Data processing and analysis:

[1908] User input data is passed to the emotion engine and generative AI model.

[1909] 3. Data Retention:

[1910] Store user-created stories in a database.

[1911] 4. Collaboration with generative AI models and emotion engines:

[1912] The title, theme, and user emotion data are sent to the emotion engine and generative AI model, and the results are returned to the user's device.

[1913] 5. Binding Request Management:

[1914] Receives binding requests and coordinates with binding services.

[1915] Generative AI Models

[1916] The generative AI model generates plot suggestions and story developments based on user input data and emotional data sent from the emotion engine.

[1917] 1. Initial plot generation:

[1918] Generates an initial plot based on the title and theme entered by the user.

[1919] 2. Storyline proposal:

[1920] Based on the story development entered by the user, new ideas and developments are suggested.

[1921] Emotion Engine

[1922] The emotion engine analyzes emotions from the user's facial expressions and voice and passes that data to a generative AI model.

[1923] 1. Sentiment analysis:

[1924] Performs real-time emotion analysis of the user's facial expressions and voice.

[1925] 2. Emotion data generation:

[1926] Emotional data is generated based on the analysis results and passed to the generative AI model.

[1927] Specific examples

[1928] Imagine a child is creating a picture book with the theme of "adventure story."

[1929] Operation on the user device

[1930] The user (child) starts the application on the user's device and enters their login information. After successful login, the home screen is displayed and the child clicks the "Create a new story" button. In the form that is then displayed, the child enters the story title "Adventure with a Dragon" and the theme "Adventure" and presses the submit button.

[1931] Processing on the server

[1932] The server stores the title and theme received from the user in a database and sends the data to the generative AI model and emotion engine. The emotion engine analyzes the user's facial expressions and voice data and passes the generated emotion data to the generative AI model.

[1933] How generative AI models work

[1934] The generative AI model generates an initial plot based on the title, theme, and emotional data, and sends the generated initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure...") back to the server.

[1935] Collaboration between the server and the emotion engine

[1936] The server sends the initial plot received from the generative AI model to the terminal and makes user-level adjustments, taking into account feedback from the emotion engine.

[1937] Viewing and editing on user devices

[1938] The user's device displays the initial plot, and the child thinks up and inputs a new storyline (e.g., "Taro becomes friends with the dragon and they go on a treasure hunt together"). At this time, the emotion engine analyzes the user's emotions from facial expressions and voice and provides feedback in real time. Once input is complete, the child clicks the send button.

[1939] Reconnecting the server and the generative AI model

[1940] The server resends new developments input from the user or emotion engine to the generative AI model, receives new proposals, and the generative AI model generates the next proposal and sends the result back to the server.

[1941] Submit a binding request

[1942] After the story is completed, the user device displays a button to request the completed story to be bound. When the child clicks the button, the request is sent to the server, which then works with the binding service to arrange for the book to be produced and delivered.

[1943] This is the process by which children create their own original story picture books using the system of this invention. By combining it with an emotion engine, dynamic story development and feedback are provided according to the user's emotions, creating a richer, more interactive experience.

[1944] The processing flow will be explained below.

[1945] Step 1:

[1946] The user starts the service application on the user device and the login screen is displayed. The child enters the user name and password and clicks the login button.

[1947] Step 2:

[1948] The server receives the login information and performs user authentication. If authentication is successful, the server starts a user session and returns the home screen data to the device.

[1949] Step 3:

[1950] The device will display a home screen and offer a "Create a new story" button that the child can click to display a form where they can enter a title and theme.

[1951] Step 4:

[1952] The user enters a story title (e.g., "Adventure with a Dragon") and a theme (e.g., "Adventure") and clicks the submit button.

[1953] Step 5:

[1954] The server stores the input title and theme in a database and sends a request to generate an initial plot to the generative AI model. At the same time, the server sends the user's facial expressions and voice data from the user's device to the emotion engine.

[1955] Step 6:

[1956] The generative AI model generates an initial plot based on the title and theme and sends the results back to the server. The emotion engine also analyzes the user's facial expressions and voice in real time to generate emotion data.

[1957] Step 7:

[1958] The server integrates the initial plot received from the generative AI model with the emotion data from the emotion engine and sends it to the terminal.

[1959] Step 8:

[1960] The device displays an initial plot (e.g., "One day, a brave boy meets a dragon. The dragon guards a secret treasure..."), which the user reads. The user then inputs a new storyline (e.g., "Taro becomes friends with the dragon and they go on a treasure hunt together") and submits it. At the same time, the emotion engine continues to analyze the user's emotions from their facial expressions and voice.

[1961] Step 9:

[1962] The server receives the newly input storyline and emotion data from the user and sends it to the generative AI model, which then proposes the next storyline.

[1963] Step 10:

[1964] The generative AI model generates the next proposal based on the new storyline and sends the result back to the server (e.g., "Tarro and Dragon find a treasure chest in a cave deep in the mountains, inside which is an old map").

[1965] Step 11:

[1966] The server receives the next suggestion from the generative AI model and sends it to the user device and the emotion engine, which continues to monitor the user's reactions and analyze the emotion data.

[1967] Step 12:

[1968] The device displays new suggestions, and the user can view them and edit the story. Once edits are complete, the user clicks the save button. During this time, the emotion engine continues to analyze emotions.

[1969] Step 13:

[1970] The server receives the final edited story and emotion data from the user and stores it in a database.

[1971] Step 14:

[1972] The user checks the completed story and clicks the binding request button.

[1973] Step 15:

[1974] The server receives the bookbinding request, stores it in a database, and sends the request to the bookbinding service. After receiving confirmation from the bookbinding service, the server arranges for the production and delivery of the picture book.

[1975] Step 16:

[1976] Users receive a bound picture book delivered to their home and enjoy their own original story.

[1977] These are the specific processing steps for creating a story picture book using the system of the present invention. By combining it with an emotion engine, it is possible to provide an interactive story based on the user's emotions, providing a richer experience.

[1978] Example 2

[1979] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1980] In today's digital society, it is important to provide children with the opportunity to express their creativity and create original story picture books. However, existing systems are unable to reflect users' emotions in real time, limiting the interactive story creation experience. Furthermore, the process of binding the created stories is time-consuming, making it difficult for users to use.

[1981] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1982] In this invention, the server includes a means including an emotion engine for analyzing user emotions, a means for transmitting data to the generative AI model, and a means for returning the results to the user terminal. This allows users to receive real-time emotional feedback as they create stories, enabling a richer, more interactive story creation experience. Furthermore, requests for binding the generated stories are efficiently processed, improving user convenience.

[1983] A "user terminal" is a device that allows a user to input and edit the title and theme of a story.

[1984] A "generative AI model" is an artificial intelligence that generates an initial plot based on the title and theme entered by the user and suggests new story developments.

[1985] An "initial plot" is a story outline initially created by a generative AI model based on the title and theme entered by the user.

[1986] The "emotion engine" is a system element that analyzes the user's facial expressions and voice, generates emotional data, and passes it to the generative AI model.

[1987] A "plot" is a basic component that shows the development and progression of a story.

[1988] "Emotion data" is data that indicates an emotional state generated by an emotion engine from the user's facial expressions and voice.

[1989] A "binding request" is a request that a user submits to have a completed story bound into a physical book.

[1990] "Saving" means recording the final story edited by the user in a storage device such as a database.

[1991] "Real-time feedback" means that the emotion engine analyzes the user's emotional state in real time and immediately reflects the results to the user.

[1992] This invention combines an emotion engine with a system that allows users to create original story picture books while interacting with a generative AI model. Specifically, we explain the configuration of the system, which includes a user terminal, a server, a generative AI model, and an emotion engine, and how each element works together.

[1993] System Overview

[1994] The system of the present invention receives input from a user terminal, sends it to a generative AI model and emotion engine via a server, and provides the output from the AI ​​and emotion engine back to the user terminal, thereby providing users (especially children) with the experience of creating their own story picture books.

[1995] User terminal

[1996] The user terminal is a device that provides an interface for creating stories. Its main functions are as follows:

[1997] 1. Input interface:

[1998] Provide a form for the child to enter the title and theme of the story. For example, enter the title "Adventure with a Dragon" and the theme "Adventure."

[1999] 2. Display interface:

[2000] It displays plot suggestions from generative AI models and storylines input by users.

[2001] 3. Editing features:

[2002] It provides an editor where users can edit, modify and save stories. Users can add their own ideas or modify existing stories.

[2003] 4. Binding Request:

[2004] It provides a button to submit the completed story as a binding request. Clicking the button sends the request to the server.

[2005] 5. Emotion recognition:

[2006] It has the ability to analyze the user's facial expressions and voice and send that data to the emotion engine.

[2007] server

[2008] The server is responsible for sending input data from users to the generative AI model and emotion engine, and returning output data to the user device. Its main functions are as follows:

[2009] 1. User authentication:

[2010] Verify and authenticate the user's login information. For example, the user enters login information and the server verifies it.

[2011] 2. Data processing and analysis:

[2012] User input data is passed to the emotion engine and generative AI model.

[2013] 3. Data Retention:

[2014] Store user-created stories in a database.

[2015] 4. Collaboration with generative AI models and emotion engines:

[2016] The title, theme, and user emotion data are sent to the emotion engine and generative AI model, and the results are returned to the user's device.

[2017] 5. Binding Request Management:

[2018] Receives binding requests and coordinates with binding services.

[2019] Generative AI Models

[2020] The generative AI model generates plot suggestions and story developments based on user input data and emotional data sent from the emotion engine.

[2021] 1. Initial plot generation:

[2022] Generates an initial plot based on the title and theme entered by the user. For example, create a plot based on "Adventures with Dragons."

[2023] 2. Storyline proposal:

[2024] Based on the story development entered by the user, new ideas and developments are suggested.

[2025] Emotion Engine

[2026] The emotion engine analyzes emotions from the user's facial expressions and voice and passes that data to a generative AI model.

[2027] 1. Sentiment analysis:

[2028] Performs real-time emotion analysis of the user's facial expressions and voice.

[2029] 2. Emotion data generation:

[2030] Emotional data is generated based on the analysis results and passed to the generative AI model.

[2031] Specific examples

[2032] Operation on the user device

[2033] The user (child) launches the application on their device and enters their login information. After successfully logging in, the home screen appears and the child clicks the "Create a new story" button. In the form that appears next, the child enters "Adventure with a Dragon" and "Adventure" and presses the submit button.

[2034] Processing on the server

[2035] The server stores the title and theme received from the user in a database and sends the data to the generative AI model and emotion engine. The emotion engine analyzes the user's facial expressions and voice data and passes the generated emotion data to the generative AI model.

[2036] How generative AI models work

[2037] The generative AI model generates an initial plot based on the title, theme, and emotional data, for example, "One day, a brave boy meets a dragon who is guarding a secret treasure," and sends it back to the server.

[2038] Collaboration between the server and the emotion engine

[2039] The server sends the initial plot received from the generative AI model to the user's device and makes adjustments taking into account feedback from the emotion engine.

[2040] Viewing and editing on user devices

[2041] The user's device displays the initial plot, and the user can think of and input a new story development. For example, they can input "Taro becomes friends with a dragon and they go on a treasure hunt together." At this time, the emotion engine analyzes the user's emotions from facial expressions and voice and provides feedback in real time. Once input is complete, the user clicks the send button.

[2042] Reconnecting the server and the generative AI model

[2043] The server resubmits the new deployment data to the generative AI model, which then generates the next proposal and sends the result back to the server.

[2044] Submit a binding request

[2045] After the story is completed, the user's device displays a button for sending the completed story as a bookbinding request. When the user clicks the button, the request is sent to the server, which then works with the bookbinding service to produce the picture book and arrange for its delivery.

[2046] Examples of prompt statements

[2047] As an example of a prompt for a generative AI model, the following could be considered as input to generate the initial plot of a story themed around "adventures with dragons":

[2048] "The user has entered the title 'Adventure with the Dragon' and the theme 'Adventure'. Based on this, please generate an initial plot for the story, starting with the scene where the boy meets the dragon."

[2049] This concludes the detailed explanation of how to use the system of the present invention. This system allows users to interactively create original stories while receiving emotional feedback, and ultimately to complete the process of bookbinding.

[2050] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2051] Step 1:

[2052] Log in

[2053] The user accesses the login screen and enters their login information. The server receives the login information (input) sent by the user and authenticates it by comparing it with information in the database (data calculation). If authentication is successful, the home screen is displayed on the user's device (output).

[2054] Step 2:

[2055] Start creating your story

[2056] The user clicks the "Create a new story" button on the home screen. The user's device detects the click event (input) and displays a new story creation form (output).

[2057] Step 3:

[2058] Enter the title and theme

[2059] The user enters the story title and theme on a new screen. The user's device receives the title "Adventure with a Dragon" and theme "Adventure" entered by the user through an input form (input). The information entered into the form is temporarily stored in the device's memory.

[2060] Step 4:

[2061] Sending input data

[2062] The user presses the send button to send the data to the server. The user's device sends the title and theme data (input) to the server and displays a notification of successful transmission to the user (output).

[2063] Step 5:

[2064] User Authentication

[2065] The server checks the user's login information and performs authentication. If authentication is successful, the server saves the user data in the database (data calculation).

[2066] Step 6:

[2067] Data storage

[2068] The server stores the received title and theme in a database (output). The server records this data in a database in an appropriate format for further processing.

[2069] Step 7:

[2070] Sending data to a generative AI model

[2071] The server sends the title and theme data to the generative AI model. It sends the title "Adventure with Dragons" and the theme "Adventure" (input) to the generative AI model via API, and receives the generated initial plot (output).

[2072] Step 8:

[2073] Sending data to the emotion engine

[2074] The server sends the user's emotional data to the emotion engine. It also sends facial expression and voice data (input) acquired from the user's device to the emotion engine and receives the analysis results (output).

[2075] Step 9:

[2076] Receiving plots from generative AI models

[2077] The server receives the initial plot from the generative AI model. The initial plot data (output) is "One day, a brave boy meets a dragon. The dragon guards a secret treasure..." and is used for the next process.

[2078] Step 10:

[2079] Receiving feedback from the emotion engine

[2080] The server receives feedback from the emotion engine and understands the user's emotional state. Based on the feedback data (output), it modifies the output results of the generative AI model (data calculation).

[2081] Step 11:

[2082] Displaying the initial plot

[2083] The user terminal displays the initial plot to the user, displays the received initial plot data (input) on the screen, and asks the user for feedback (output).

[2084] Step 12:

[2085] User edits

[2086] The user thinks up and inputs a new story development based on the initial plot. The user's input data, "Taro becomes friends with the dragon and goes on a treasure hunt together" (input), is acquired by the user's device and temporarily saved as edited data.

[2087] Step 13:

[2088] Send edited data

[2089] The user resends the edited data to the server. The user terminal sends the edited new story development data (input) to the server and displays a notification that transmission is complete (output).

[2090] Step 14:

[2091] Resubmit to generative AI model

[2092] The server resubmits new story development data to the generative AI model. It sends new story development data (input) via API and receives new proposals from the generative AI model (output).

[2093] Step 15:

[2094] Receiving a resubmission

[2095] The server receives new proposals from the generative AI model. It receives new story development proposal data (output) from the generative AI model and uses it for further processing.

[2096] Step 16:

[2097] View the final story

[2098] The user terminal displays the final story development to the user. The generated final story data (input) is displayed on the screen and the user is prompted to confirm it (output).

[2099] Step 17:

[2100] Submit a binding request

[2101] The user submits the completed story as a binding request. The user terminal transmits the binding request data (input) to the server and displays a notification of completion of transmission (output).

[2102] Step 18:

[2103] Collaboration with bookbinding services

[2104] The server receives the binding request and works with the binding service to produce and arrange for delivery of the picture book. It sends the binding request data (input) to the binding service and records the binding progress in a database (output).

[2105] (Application example 2)

[2106] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2107] Conventional story creation systems lacked the ability to provide feedback based on the user's emotions or adjust the storyline, which prevented them from fully enriching the user's experience. Furthermore, they lacked the interactive elements that children needed to have fun creating stories, which led to a lack of interest.

[2108] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting the title and theme of the story from a user terminal, means for generating an initial plot using a generative AI model, means for displaying the plot received from the generative AI model on the user terminal, means for passing the story development input by the user to the generative AI model, means for displaying a new development proposed by the generative AI model on the user terminal, means for saving the final story edited by the user, means for processing a binding request based on the saved story, and means for adjusting the story feedback displayed on the user terminal based on the emotion, including an emotion engine that analyzes the user's facial expressions and voice to generate emotion data. This allows for dynamic story development and feedback in accordance with the user's emotions, enabling a richer and more interactive experience.

[2109] A "user terminal" is a device that provides an interface for creating a story and receives input from a user.

[2110] A "generative AI model" is an artificial intelligence that generates an initial plot or new story development based on user input data.

[2111] The "initial plot" is the initial story development that is generated based on the story title and theme entered by the user.

[2112] A "plot" indicates the story's outline and development pattern, and is generated by a generative AI model.

[2113] Theme refers to the overall theme or central idea of ​​the story.

[2114] The "emotion engine" is an engine that analyzes emotions from the user's facial expressions and voice, and adjusts the story development through the generated emotional data.

[2115] A "bookbinding request" is a procedure in which a user requests that a completed story be produced as a physical picture book.

[2116] "Emotion data" refers to emotional information analyzed from the user's facial expressions and voice, and is reflected in the story development.

[2117] "Interface" refers to the screen display and input methods on a user terminal that provide the means for a user to interact with a system.

[2118] "Feedback" refers to the information or suggestions that a system returns in response to a user's input or behavior.

[2119] To implement this invention, a user terminal on which a dedicated application is installed, a server located on the cloud, a generative AI model, and an emotion engine are used. Details of these are described below.

[2120] Overall system configuration

[2121] The system includes the following major components:

[2122] 1. User Device:

[2123] Input interface: Accepts title and theme input from the user.

[2124] Display interface: Shows the plot from the generative AI model and the final story.

[2125] Editing capabilities: Users can create, edit, and save stories.

[2126] Emotion Recognition: Uses a camera and microphone to capture the user's facial expressions and voice to generate emotional data.

[2127] Bookbinding Requests: Processing requests to have completed stories produced as physical picture books.

[2128] 2. Server:

[2129] User authentication: Verify the user's login information and perform authentication.

[2130] Data processing and analysis: Sending user input data to the emotion engine and generative AI model.

[2131] Data storage: Store user-created stories in a database.

[2132] Collaboration with generative AI models and emotion engines: Processes title, theme, and emotion data and returns the results to the user's device.

[2133] Binding Request Management: Send binding requests to binding services.

[2134] 3. Generative AI models (e.g., GPT-4):

[2135] Initial plot generation: Generates the basic plot of your story based on the title and theme.

[2136] Story development proposal: Based on new development input from the user, the next development proposal is made.

[2137] 4. Emotion engine (e.g. Affectiva SDK):

[2138] Sentiment analysis: Analyzes the user's facial expressions and voice to generate emotional data.

[2139] Emotional feedback: Adjust the storyline based on the generated emotional data.

[2140] Software and Hardware Configuration

[2141] User device: Smartphone application (e.g., using React Native)

[2142] Server: Node.js, Express framework, database (e.g. MongoDB)

[2143] Emotion recognition: Expo Camera, Expo Audio

[2144] Generative AI model: OpenAI GPT-4 API

[2145] Data processing flow

[2146] 1. Getting user input:

[2147] The user enters the story title and theme on their device and sends it to the server.

[2148] 2. Emotion capture and analysis:

[2149] It uses the camera and microphone on the user's device to capture facial expressions and voice.

[2150] The captured data is sent to the emotion engine via the server, where emotions are analyzed.

[2151] 3. Collaboration with generative AI models:

[2152] The server sends user input and emotion data to the generative AI model.

[2153] The generative AI model generates an initial plot and sends it to the user's device via the server.

[2154] 4. Providing User Feedback:

[2155] As users edit the storyline, the emotion engine provides real-time emotional feedback that is sent to the generative AI model.

[2156] The generative AI model generates new suggestions, which are displayed on the user's device.

[2157] 5. Processing binding requests:

[2158] Users submit completed stories as binding requests.

[2159] The server works with the bookbinding service to create the physical picture book and arrange for delivery.

[2160] Examples and prompts

[2161] Title: "The Adventures of Brave Taro"

[2162] Theme: Adventure

[2163] Scene: "Tarro meets a dragon and becomes friends with him."

[2164] Example prompts to input to a generative AI model:

[2165] Title: The Adventures of the Brave Taro

[2166] Theme: Adventure

[2167] Feedback emotion: Fun

[2168] Story Current Scene: Taro meets a dragon and they go on a treasure hunt together to become friends.

[2169] Please suggest the following expansion:

[2170] In this way, a system can be implemented that provides users with a fun, emotionally driven story-creation experience.

[2171] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2172] Step 1:

[2173] The user inputs the title and theme of the story. The user's device receives these inputs and sends them to the server.

[2174] Input: User title and theme

[2175] Output: Title and theme sent to server

[2176] Specific operation: When a user enters characters into an input field on the user terminal and presses the send button, this data is sent to the server.

[2177] Step 2:

[2178] The server sends the title and theme to the generative AI model, which generates the initial plot and sends the results back to the server.

[2179] Input: Title and Theme

[2180] Output: Initial plot

[2181] What it does: The server sends the title and theme to the generative AI model and receives the generated initial plot.

[2182] Step 3:

[2183] The server sends the initial plot to the user terminal, which displays it.

[2184] Input: Initial plot

[2185] Output: Initial plot displayed on the user's terminal

[2186] Specific operation: The initial plot received from the server is displayed on the display screen of the user's terminal.

[2187] Step 4:

[2188] The user inputs the continuation of the story. The user's device receives this input and sends it to the server.

[2189] Input: User storyline

[2190] Output: Sending the storyline to the server

[2191] Specific behavior: When the user enters a storyline into the input field and presses the submit button, it is sent to the server.

[2192] Step 5:

[2193] The user device captures the user's facial expressions and voice in real time and sends them to the emotion engine for analysis.

[2194] Input: User's facial expressions and voice

[2195] Output: Emotion data

[2196] Specific operation: Uses the camera and microphone on the user's device to capture facial expression and voice data and sends it to the emotion engine.

[2197] Step 6:

[2198] The emotion engine analyzes the user's facial expressions and voice to generate emotion data, which is then sent to the server.

[2199] Input: facial expression data and voice data

[2200] Output: Emotion data

[2201] Specific operation: The emotion engine analyzes the received data, determines the user's emotions, generates emotion data, and sends it to the server.

[2202] Step 7:

[2203] The server sends the emotional data and the user's storyline to a generative AI model, which then generates a proposal for the next storyline.

[2204] Input: Emotion data and storyline

[2205] Output: Next storyline proposal

[2206] Specific operation: The server sends the story development and emotional data from the user to the generative AI model, which then generates a proposal for the next development and returns it to the server.

[2207] Step 8:

[2208] The server transmits the next story development proposal to the user terminal, which displays it.

[2209] Input: Next storyline suggestion

[2210] Output: Next expansion suggestion displayed on user terminal

[2211] Specific operation: The proposal received from the server is sent to the user terminal, and is displayed on the display screen of the user terminal.

[2212] Step 9:

[2213] The user completes the story and submits a binding request, which the user terminal sends to the server.

[2214] Input: Final story

[2215] Output: Binding request

[2216] Specific behavior: The user checks the final story and presses the binding request button, which sends the request to the server.

[2217] Step 10:

[2218] The server receives the binding request, works with the binding service to produce the picture book, and arranges for delivery.

[2219] Input: Binding request

[2220] Output: Physical storybook

[2221] Specific operation: After the server receives the binding request, it requests the binding service to produce and deliver the picture book.

[2222] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2223] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2224] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2225] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2226] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2227] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2228] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2229] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2230] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2231] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2232] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2233] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2234] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2235] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the sp...

Claims

1. a means for inputting the title and theme of the story from a user terminal; means for generating an initial plot using a generative AI model; a means for displaying the plot received from the generative AI model on a user device; A means of passing the user-input storyline to the generative AI model; A means for displaying new developments proposed by the generative AI model on a user device; a means for the user to save the final edited story; a means of processing binding requests based on the stored stories; A system including:

2. 10. The system of claim 1, including a text-to-speech function.

3. 10. The system of claim 1, including multilingual support.

Images