system

Abstract

Provide a system. [Solution] means for providing an interface for a user to input characters, themes, and key elements; means for collecting the input data and transmitting it to a server; a means for utilizing generative AI to generate a narrative based on the received input data; a means for storing the generated stories; means for displaying the generated story to a user; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional story creation is time-consuming and laborious, preventing many people from enjoying creative activities. In particular, there is a lack of a way to easily create stories and picture books tailored to individual needs. Furthermore, there are few additional features, such as saving the created stories, reading them aloud, or providing illustrations, making it difficult to meet the diverse needs of users. Therefore, there is a demand for a system that allows users to easily and quickly create original stories and picture books. [Means for solving the problem]

[0005] The system of the present invention includes a means for providing an interface for users to input characters, themes, and important elements, a means for collecting the input data and sending it to a server, a means for generating a story based on the received input data using a generation AI, a means for saving the generated story, and a means for displaying the generated story to the user. It can also include a means for providing the generated story by audio using a voice generation service, and a means for generating illustrations that match the generated story using an image generation AI and combining the illustrations with the story to provide the user. This allows users to easily and quickly create original stories and picture books, which can be saved and enjoyed in a variety of formats.

[0006] "User" refers to an individual or group that uses the system to create stories or picture books.

[0007] "Interface" refers to the software components that provide the screens and controls for users to input characters, themes, and key elements.

[0008] "Input Data" refers to character, theme, and key element information entered by a user through the interface.

[0009] "Server" refers to the computer system that receives input data and processes it to generate a story using generative AI.

[0010] "Generative AI" refers to artificial intelligence algorithms or models that generate stories based on the input data they receive.

[0011] "Story" refers to a text-based story generated by generative AI.

[0012] "Storage" refers to storing and maintaining the generated stories in a database or other storage means.

[0013] "Display" refers to outputting the generated story on an interface in a format that can be viewed by a user.

[0014] "Speech generation service" refers to a service that converts generated narrative text into speech and provides it.

[0015] "Image generation AI" refers to an artificial intelligence algorithm or model that generates illustrations that match the content of a generated story.

[0016] "Illustrations" refer to images generated by image generation AI to visually complement the content of the story.

[0017] "Combination" refers to integrating the generated story and illustrations into a single piece of content. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The system of the present invention is designed to enable users to easily create original stories and picture books. Specific embodiments will be described below.

[0040] Providing a user interface

[0041] The user inputs the story setting through a terminal interface. The interface provides fields for inputting characters, themes, key elements, etc. For example,

[0042] Character: "Taro"

[0043] Theme: "Adventure"

[0044] Key Element: "Magic Sword"

[0045] Enter items such as:

[0046] Sending data

[0047] The terminal collects the user's input data, converts it into an appropriate format such as JSON, and then sends it to the server.

[0048] Narrative Generation

[0049] The server uses a generation AI to generate a story based on the received data. The generation AI creates a story with natural sentences based on the input data. For example, if a user inputs the settings above, the generated story will look like this:

[0050] Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero.

[0051] Preserving the Genesis Story

[0052] The stories you create are stored in a database on the server. With the free plan, you can save up to three stories, after which you are restricted from saving new stories. With the paid plan, you can save four or more stories.

[0053] Showing the story

[0054] The server sends the generated story back to the terminal, which displays the story to the user, who can then view the generated story on the interface.

[0055] Voice Generation Options

[0056] As a paid option, the device can use a voice generation service to provide the generated story audibly. The device sends a voice generation request to the server, which uses TTS (Text to Speech) technology to convert the story into audio and send it back to the device. The user can then play the generated audio in the interface.

[0057] Illustration and binding options

[0058] Another paid option is for the device to provide a service that uses image generation AI to generate illustrations. The server receives a request from the device and generates illustrations that match the content of the story, integrates them with the story, and sends them back to the device. Furthermore, if the user requests it, the server can also bind the story and illustrations and provide them as a physical book.

[0059] This system allows users to quickly and effortlessly create original stories and picture books based on pre-defined elements, and enjoy them in a variety of formats. The stories created can be provided not only as text, but also in audio and illustration formats, making it possible to meet the diverse needs of users.

[0060] The processing flow will be explained below.

[0061] Step 1:

[0062] The user inputs the setting of the story. Using the terminal interface, the user inputs the characters, theme, and key elements. For example, the user inputs the character "Taro," the theme "adventure," and the key element "magic sword."

[0063] Step 2:

[0064] The device collects input data. It converts the data entered by the user into a format such as JSON. Example: {"character": "Taro", "theme": "Adventure", "keyElement": "Magic Sword"}

[0065] Step 3:

[0066] The device sends data to the server by using an HTTP POST request to send input data to a specified endpoint on the server.

[0067] Step 4:

[0068] The server receives the request. The request is received at the appropriate endpoint on the server and the input data is parsed.

[0069] Step 5:

[0070] The server gives instructions to the AI ​​generator. Based on the parsed data, it instructs the AI ​​generator to generate a story. For example, it provides the AI ​​generator with the following prompts: "Character: Taro," "Theme: Adventure," "Important Element: Magic Sword."

[0071] Step 6:

[0072] The generation AI generates a story. The generation AI creates a story based on the provided prompts. Example: The generated story is "Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero."

[0073] Step 7:

[0074] The server stores the generated stories. The generated stories are stored in a database. The free plan allows you to store up to three stories, and any more stories will be limited to one.

[0075] Step 8:

[0076] The server returns the generated story to the terminal as an HTTP response. Example: {"story": "Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero."}

[0077] Step 9:

[0078] The terminal displays the generated story. A user interface displays the generated story, allowing the user to view the story.

[0079] Step 10 (Optional):

[0080] The user selects the read aloud option: If the user wants to read aloud, the user selects this option in the interface.

[0081] Step 11 (Optional):

[0082] The device sends a request for speech generation to the server. Based on the input data and the generated story, a request for speech generation is sent to the server.

[0083] Step 12 (Optional):

[0084] The server converts the story into audio using a TTS service, converting the text of the generated story into audio and generating an audio file.

[0085] Step 13 (Optional):

[0086] The server returns the audio file to the device. The generated audio file is sent to the device as an HTTP response.

[0087] Step 14 (Optional):

[0088] The device plays the audio and displays an audio play button in the user interface, allowing the user to play the audio of the story.

[0089] Step 15 (Optional):

[0090] The user selects the Generate Illustration option: If the user wants to generate illustrations, the user selects this option in the interface.

[0091] Step 16 (Optional):

[0092] The terminal sends a request to the server to generate illustrations. Based on the input data and the generated story, a request to generate illustrations is sent to the server.

[0093] Step 17 (Optional):

[0094] The server generates illustrations using image generation AI, which creates corresponding illustrations based on the content of the story.

[0095] Step 18 (Optional):

[0096] The server returns the generated illustration to the terminal, which then sends the generated illustration to the terminal as an HTTP response.

[0097] Step 19 (Optional):

[0098] The terminal integrates the illustrations into the story and displays them. The user interface integrates the story and illustrations generated by the terminal so that the user can view them.

[0099] Step 20 (Optional):

[0100] The user selects the option to bind the story: If binding is desired, the user selects this option in the interface.

[0101] Step 21 (Optional):

[0102] The server arranges for a bookbinding service to bind the generated story and illustrations into a physical book and provide it to the user.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Conventional story creation systems generate stories based on specific templates, making it difficult for users to easily create their own stories. Furthermore, few systems allow users to enjoy stories in a richer format by adding audio and illustrations, rather than just displaying the generated stories in text format. Therefore, there is a need for a system that allows users to effortlessly create, save, and view their own stories in a variety of formats.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes means for generating a story based on received input data using a generative AI model, means for saving the generated story in a database, and means for returning the generated story to a user terminal and displaying it to the user. This allows users to easily create original stories, add audio and illustrations to them in addition to text, and save and view them.

[0108] "Interface" refers to the screens and input fields that allow users to perform operations and input data into the system.

[0109] "Input Data" refers to information about characters, themes, key elements, etc. that a user inputs into the system through an interface.

[0110] "Server" refers to the main computer system that receives and processes data sent from terminals over a network.

[0111] A "generative AI model" refers to an artificial intelligence algorithm that generates natural-looking text or images based on the data it receives.

[0112] "Narrative" refers to a story created by a generative AI model based on user input data.

[0113] "Database" refers to a system for structuring and storing generated narratives and other related data.

[0114] "User terminal" refers to a device such as a computer or smartphone that a user uses to access and operate the system.

[0115] "Text-to-speech technology" refers to technology that converts text data into voice data.

[0116] "Image generation AI" refers to an artificial intelligence algorithm that generates images and illustrations based on the content of a story.

[0117] A "prompt sentence" refers to a sentence that provides a generative AI model with the input data it needs to generate a story.

[0118] The system of the present invention is designed to enable users to easily create original stories and picture books. Specific embodiments will be described below.

[0119] The system provides an interface for users to input characters, themes, and key elements. Users use a terminal to input information into this interface. For example, they might input the following information:

[0120] Character: "Taro"

[0121] Theme: "Adventure"

[0122] Key Element: "Magic Sword"

[0123] The terminal collects the data entered by the user, converts it into JSON format, and sends it to the server via an HTTP POST request. The server receives the JSON data and analyzes its contents.

[0124] Next, the server uses a generative AI model (e.g., OpenAI's GPT-4) to generate a story based on the input data. Specifically, the server provides the generative AI model with a prompt like this:

[0125] Characters: Taro, Theme: Adventure, Key Elements: Magic Sword

[0126] Based on this prompt, the generative AI model generates the following story:

[0127] Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero.

[0128] The generated stories are stored in a database by the server. The free plan allows you to store up to three stories, after which you are restricted from saving new stories. Paid plans allow you to store more stories.

[0129] The server then sends the generated story back to the user terminal, which displays the received story on its interface so that the user can view the story.

[0130] As a paid option, devices can use a speech generation service. The device sends a speech generation request to a server, which converts the story into audio using text-to-speech (TTS) technology. The generated audio data is sent back to the device, allowing the user to play the audio in the interface.

[0131] There is also an option to use image generation AI (e.g., DALL-E) to generate illustrations that match the story. The device sends a request for illustration generation to the server, and the server generates the illustrations using the image generation AI. These illustrations are integrated with the story and sent back to the device. In this way, the user can enjoy the story and illustrations together. Furthermore, if the user requests it, there is also a service that will bind the story and illustrations and provide them as a physical book.

[0132] As described above, this system allows users to quickly and effortlessly create original stories and picture books based on pre-set elements, and enjoy them in a variety of formats. The stories created can be provided not only as text, but also in audio and illustration formats, making it a system that can meet the diverse needs of users.

[0133] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0134] Step 1:

[0135] Providing a user interface

[0136] The user uses a terminal to access an interface to input the elements needed to create a story. The user inputs characters, themes, and key elements. For example, "Taro", "adventure", and "magic sword". The input data is saved in form fields. Input: Data entered by the user (characters, themes, key elements). Output: Input data saved in form fields.

[0137] Step 2:

[0138] Sending data

[0139] The device collects data entered by the user and converts it into JSON format. The JSON data is then sent to the server using an HTTP POST request. The data includes characters, themes, key elements, etc. Input: The input data stored in the form fields. Output: The JSON data sent to the server.

[0140] Step 3:

[0141] Narrative Generation

[0142] The server receives JSON data sent from the device. After receiving it, it analyzes the data and calls a generative AI model (e.g., OpenAI GPT-4) to generate a story. The generative AI model is given a prompt. For example, the following prompt is input to the generative AI model: "Character: Taro, Theme: Adventure, Key Element: Magic Sword." The generative AI model generates a story based on the prompt. Input: JSON data received by the server. Output: Story generated by the generative AI model.

[0143] Step 4:

[0144] Preserving the Genesis Story

[0145] The server saves the generated stories in a database. The database has tables for managing multiple stories, into which the text data of the stories is inserted. The free plan allows for the saving of up to three stories, after which new stories cannot be saved. Paid plans allow for the saving of more stories. Input: Text data of the generated stories. Output: Stories saved in the database.

[0146] Step 5:

[0147] Showing the story

[0148] The server returns the generated story to the terminal. The terminal receives the returned story and displays it on the user interface. The user can view the displayed story. Input: Text data of the story stored in the database. Output: The story displayed on the user interface.

[0149] Step 6:

[0150] Voice Generation Options

[0151] If the user selects the paid option, the device sends a voice generation request to the server. The server uses text-to-speech (TTS) technology to convert the story into voice data. The generated voice data is sent back to the device, and the user can play the audio on the interface. Input: Story text data, voice generation request. Output: Generated voice data.

[0152] Step 7:

[0153] Illustration and binding options

[0154] If the user selects the illustration generation option, the device sends an image generation request to the server. The server uses image generation AI (e.g., DALL-E) to generate illustrations that match the content of the story. The generated illustrations are integrated with the story and sent back to the device. Furthermore, if the user requests it, the server also provides a service to bind the story and illustrations and provide them as a physical book. Input: Story text data, image generation request. Output: Generated illustrations and integrated story data.

[0155] (Application example 1)

[0156] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0157] Conventional story generation systems are convenient in that they allow users to create their own stories and picture books, but they have limitations on how the stories can be utilized, making it difficult to share them with other users or use them commercially. Furthermore, elements such as converting the generated stories into audio or adding illustrations must be handled individually, and are not provided as a unified system. This often requires a lot of effort and time for users.

[0158] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0159] In this invention, the server includes means for providing an interface for users to input characters, themes, and important elements, means for collecting the input data and sending it to the server, means for generating a story based on the received input data using a generation AI, means for saving the generated story, means for displaying the generated story to users, and means for sharing and purchasing the generated story with other users in a virtual store. This allows users to quickly and easily create original stories and picture books and share them with other users or use them commercially.

[0160] "Users" are individuals or groups who use the system to create stories and picture books.

[0161] A "character" is a character who plays a role in the story and moves the story forward.

[0162] "Theme" refers to the overall theme or central idea of ​​a story.

[0163] "Important elements" refer to keywords, items, and settings that are essential to the story.

[0164] "Interface" refers to the screen and operating means that users use to input data into the system.

[0165] "Input Data" refers to information about characters, themes, key elements, etc., entered by a user through an interface.

[0166] A "server" is a computer system that processes and stores data on a network.

[0167] "Generative AI" is an artificial intelligence model that automatically generates natural language narratives based on user-entered data.

[0168] A "story" is a collection of texts that form a series of events or episodes based on characters, themes, and key elements.

[0169] "Storage" means keeping the generated story in a storage device such as a database.

[0170] A "virtual store" is a virtual space where users can use products and services and share content over the Internet.

[0171] "Sharing" means making the created story available for other users to see.

[0172] "Purchase" is the act of another user paying a fee to acquire the created story.

[0173] The present invention is a system that allows users to easily create original stories and picture books, and further allows them to share and commercially use them with other users in a virtual store.

[0174] Providing a user interface

[0175] Users input the story setting through the interface of a device such as a smartphone or PC. The interface provides fields for inputting characters, themes, important elements, etc. For example, it is designed so that users can input items such as "Taro," "adventure," and "magic sword."

[0176] Submitting data and generating stories

[0177] The device collects the user's input data, converts it into an appropriate format such as JSON, and then sends it to a cloud server. The server uses a generative AI (for example, OpenAI's GPT-4) to generate a story based on the received data. The generative AI generates natural-sounding sentences based on the input data. For example, if a user inputs the above settings, the following story will be generated: "Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero."

[0178] Preserving and displaying generative stories

[0179] The generated story is stored in a database on a cloud server, and is then sent back to the user's device and displayed on the interface.

[0180] Sharing and purchasing within the virtual store

[0181] The stories created can be shared with other users in the virtual store, and can also be purchased by other users for a fee.

[0182] Voice Generation Options

[0183] The server also includes a speech generation service, and as a paid option, the device can send a speech generation request to the server. The server uses Text-to-Speech (TTS) technology to convert the story into speech and send it back to the device. The user can then play the generated speech on the interface and enjoy it.

[0184] Illustration and binding options

[0185] Image generation AI can also be used to generate illustrations that match the content of a story. In this case, the server receives a user request and generates illustrations that match the story, integrates them with the story, and sends them back to the device. Furthermore, if the user requests it, the server can also bind the generated story and illustrations and provide them as a physical book.

[0186] Specific examples

[0187] For example, if a user wanted to create a story about a winter night and snow spirits, the following prompts would be input into the generative AI model:

[0188] Character: Snow Spirit

[0189] Theme: Winter Night

[0190] Key Element: Magic Snowflake

[0191] Based on this, the server generates a story about a snow spirit who finds a magical snowflake on a winter's night and embarks on a heartwarming adventure.

[0192] This system allows users to quickly and effortlessly create original stories and picture books, and enjoy them in a variety of formats. The generated content can also be used commercially, meeting a variety of needs.

[0193] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0194] Step 1:

[0195] The user inputs characters, themes, key elements, etc. into the interface on their smartphone or PC. The interface provides input fields where the user can enter information such as "Taro," "adventure," and "magic sword." Once this data is entered, it is converted into a JSON-formatted object.

[0196] Step 2:

[0197] The device collects the input data, converts it to JSON format, and sends it to the cloud server using an HTTP POST request. The input data is sent to the server for further processing.

[0198] Step 3:

[0199] The server parses the received JSON-formatted input data. The parsed data is then input as a prompt to a generative AI (e.g., OpenAI's GPT-4). The generative AI then automatically generates a story based on the characters, theme, and key elements. For example, a sentence like "Taro obtains a magic sword and sets out on an adventure" is generated.

[0200] Step 4:

[0201] The server saves the generated stories in a database, where they can be accessed by users later. When saving, metadata such as the user ID and the story title are also saved.

[0202] Step 5:

[0203] The server sends the generated story back to the user's device. The HTTP response sends the generated story in JSON format to the device, where it is displayed on the user's interface.

[0204] Step 6:

[0205] Users can share stories created in the virtual store with other users. By sharing, other users can view or purchase the story. The data of purchased stories is managed on the server and kept as shared information between users.

[0206] Step 7:

[0207] As a paid option, users can also listen to the generated story. The device sends a request to generate the story to the server, which uses Text-to-Speech (TTS) technology to convert the story into audio. The generated audio file is sent back to the device and played on the interface.

[0208] Step 8:

[0209] Another paid option is to use image generation AI to generate illustrations. Upon receiving a user request, the server generates illustrations based on the content of the story. The generated illustrations are integrated into the story and sent back to the device. The user can then enjoy the story, including the generated illustrations, as an e-book.

[0210] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0211] The system of the present invention is designed to enable users to easily create original stories and picture books. Furthermore, it adds a function to recognize the user's emotions and adjust the content, tone, illustrations, and speech generation of the story based on those emotions. Specific embodiments are described below.

[0212] Providing a user interface

[0213] The user inputs the story setting through a terminal interface. The interface provides fields for inputting characters, themes, and key elements. For example,

[0214] Character: "Taro"

[0215] Theme: "Adventure"

[0216] Key Element: "Magic Sword"

[0217] Enter items such as:

[0218] Emotion recognition

[0219] The device is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice. The emotion engine recognizes the user's emotional state in real time through a camera and microphone. The recognized emotion data is quantified as, for example, "happiness," "sadness," or "excitement."

[0220] Sending data

[0221] The device collects input data and emotion data, converts it into an appropriate format such as JSON, and then sends it to the server.

[0222] Narrative Generation

[0223] The server uses a generation AI to generate a story based on the received data. The generation AI creates a natural-sounding story based on the input story setting data and emotion data. For example, if a user provides emotion data of "joy" with the theme of "adventure," the generated story will be written in a bright and cheerful tone.

[0224] Taro finds a magical sword and embarks on an exciting adventure, overcoming many amusing challenges and becoming a legendary hero all while having fun.

[0225] Preserving the Genesis Story

[0226] The stories that are created are stored in a database on the server. With the free plan, up to three stories can be saved, and saving new stories beyond that is restricted. With the paid plan, four or more stories can be saved.

[0227] Showing the story

[0228] The server sends the generated story back to the terminal, which displays the story to the user, who can then view the generated story on the interface.

[0229] Voice Generation Options

[0230] As a paid option, the device can use a voice generation service to deliver the generated story aloud. The tone and emotional expression of the voice are adjusted based on the user's emotional data. For example, a story generated with emotional data expressing joy will be read in a bright tone.

[0231] Illustration and binding options

[0232] Another paid option is for the device to provide a service that uses image generation AI to generate illustrations. The server receives a request from the device and uses the AI ​​to create corresponding illustrations based on the content of the story and the user's emotional data. For example, if a story contains a lot of emotional data about "surprise," an illustration that matches that emotion will be generated. Furthermore, if the user requests it, the service also provides a service to bind the story and illustrations and provide them as a physical book.

[0233] This system allows users to quickly and effortlessly create original stories and picture books based on pre-defined elements and their own emotions, and enjoy them in a variety of formats. The stories they create can be provided not only as text but also in audio and illustration formats, making it possible to meet the diverse needs of users. Adding emotion recognition functionality will provide a more personalized experience and increase user satisfaction.

[0234] The processing flow will be explained below.

[0235] Step 1:

[0236] The user inputs the setting of the story. Using the terminal interface, the user inputs the characters, theme, and key elements. For example, the user inputs the character "Taro," the theme "adventure," and the key element "magic sword."

[0237] Step 2:

[0238] The device collects input data. It converts the data entered by the user into a format such as JSON. Example: {"character": "Taro", "theme": "Adventure", "keyElement": "Magic Sword"}

[0239] Step 3:

[0240] The device recognizes the user's emotions. Using an emotion engine, it analyzes the user's facial expressions and tone of voice in real time via the camera and microphone to generate emotion data. Example: {"emotion": "joy"}

[0241] Step 4:

[0242] The device sends the collected input data and emotion data to the server using an HTTP POST request. Example: {"character": "Taro", "theme": "Adventure", "keyElement": "Magic Sword", "emotion": "Joy"}

[0243] Step 5:

[0244] The server receives the request. The request is received at the server endpoint and the input data and emotion data are parsed.

[0245] Step 6:

[0246] The server gives instructions to the AI ​​generator. Based on the parsed data, the AI ​​generator is instructed to generate a story. For example, the server provides the AI ​​with the following prompts: "Character: Taro," "Theme: Adventure," "Important Element: Magic Sword," and "Emotion: Joy."

[0247] Step 7:

[0248] The Generator AI generates a story. The Generator AI creates a story based on the provided prompts. Example: A generated story is "Taro obtains a magical sword and sets off on an exciting adventure. He overcomes many interesting challenges, has fun, and becomes a legendary hero."

[0249] Step 8:

[0250] The server stores the generated stories. The generated stories are stored in a database. The free plan allows you to store up to three stories, and any more stories will be limited to one.

[0251] Step 9:

[0252] The server sends the generated story back to the terminal. It sends back the generated story as an HTTP response. Example: {"story": "Taro gets his hands on a magical sword and sets off on an exciting adventure. He overcomes many interesting challenges, has fun, and becomes a legendary hero."}

[0253] Step 10:

[0254] The terminal displays the generated story. A user interface displays the generated story, allowing the user to view the story.

[0255] Step 11 (Optional):

[0256] The user selects the read aloud option: If the user wants to read aloud, the user selects this option in the interface.

[0257] Step 12 (Optional):

[0258] The device sends a request for speech generation to the server.,Based on the input data, the generated story, and,emotion data, a speech generation request is sent to the server.

[0259] Step 13 (Optional):

[0260] The server converts the story into audio using a TTS service. The generated story text is converted into audio and an audio file is generated. The tone and emotional expression of the audio are adjusted based on the emotional data.

[0261] Step 14 (Optional):

[0262] The server returns the audio file to the device. The generated audio file is sent to the device as an HTTP response.

[0263] Step 15 (Optional):

[0264] The device plays the audio and displays an audio play button in the user interface, allowing the user to play the audio of the story.

[0265] Step 16 (Optional):

[0266] The user selects the Generate Illustration option: If the user wants to generate illustrations, the user selects this option in the interface.

[0267] Step 17 (Optional):

[0268] The terminal sends a request to the server to generate an illustration. Based on the input data, the generated story, and the emotion data, the request to generate an illustration is sent to the server.

[0269] Step 18 (Optional):

[0270] The server uses image generation AI to generate illustrations. Based on the content of the story and emotional data, the AI ​​creates corresponding illustrations. For example, if a story contains a lot of emotional data for "surprise," an illustration that matches that emotion will be generated.

[0271] Step 19 (Optional):

[0272] The server returns the generated illustration to the terminal, which then sends the generated illustration to the terminal as an HTTP response.

[0273] Step 20 (Optional):

[0274] The terminal integrates the illustrations into the story and displays them. The user interface integrates the story and illustrations generated by the terminal so that the user can view them.

[0275] Step 21 (Optional):

[0276] The user selects the option to bind the story: If binding is desired, the user selects this option in the interface.

[0277] Step 22 (Optional):

[0278] The server arranges for a bookbinding service to bind the generated story and illustrations into a physical book and provide it to the user.

[0279] Example 2

[0280] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0281] Conventional story generation systems generate stories solely based on user input, making it difficult to provide a personalized experience. Furthermore, they are unable to adjust the content, tone, illustrations, and audio expression of the story according to the user's emotions and preferences. Therefore, a new system that takes the user's emotions into account during the story generation process is needed to improve user satisfaction.

[0282] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0283] In this invention, the server includes means for providing an interface for a user to input characters, themes, and important elements, means for collecting the input data, recognizing the user's emotional state using an emotion engine, and transmitting the input data to the server, means for generating a story based on the received input data and emotional data using a generative AI model, means for saving the generated story, and means for displaying the generated story to the user. This makes it possible to generate a personalized story according to the user's emotional state and provide content that is more satisfying.

[0284] An "interface" is the means by which a user inputs characters, themes, and key elements.

[0285] An "emotion engine" is a means having the function of recognizing the user's emotional state and collecting it as numerical data.

[0286] A "generative AI model" is a type of artificial intelligence used to generate stories based on input data and emotional data.

[0287] The "server" is a central computer system that receives input data and emotional data, and generates, stores, and displays stories.

[0288] A "voice generation service" is a means of providing a generated story in audio form and adjusting the tone and emotional expression of the audio based on the user's emotional data.

[0289] "Image generation AI" is a type of artificial intelligence used to generate illustrations that match a generated story.

[0290] "Storage means" refers to the methods and techniques used to store the generated stories in a database or storage device.

[0291] "Display means" refers to the methods and techniques for visually presenting the generated story on a user interface.

[0292] The present invention is a system that allows users to easily create original stories and picture books. This system has the function of recognizing the user's emotional state and adjusting the content, tone, illustrations, and audio expression of the story based on that emotion. An embodiment of the system is described in detail below.

[0293] The user inputs the story setting through a device interface. The interface provides fields for inputting characters, themes, and key elements. For example, the user can input "Taro," "adventure," and "magic sword." The interface is designed to be intuitive for the user, with text fields and drop-down menus.

[0294] The device is equipped with a camera and microphone, and the emotion engine uses these devices to analyze the user's facial expressions and tone of voice in real time. The emotion engine analyzes the collected data and quantifies emotions such as "happiness," "sadness," and "excitement." This emotional data is an important element required for story generation.

[0295] The device converts the story setting data and recognized emotion data entered by the user into JSON format and sends it to the server. The server analyzes the received data and generates a story using a generative AI model (e.g., OpenAI's GPT-3 (registered trademark)). The generative AI model creates a story with an appropriate tone and content based on the input elements and emotion data. For example, if the user provides emotion data for "adventure" and "joy," the generated story will have a bright and cheerful tone.

[0296] The stories that are created are stored in a database on the server. Users with the free plan can save up to three stories, but to save more, a paid plan is required. Saved stories can be reused at any time upon user request.

[0297] Once the story generation is complete, the server sends the generated story data back to the device. The device displays the received story on its user interface, allowing the user to check the content. In addition to viewing the generated story in text format, the user can also optionally use services that generate audio and illustrations.

[0298] When generating the voice, the device uses a voice generation service to provide the generated story aloud. The tone and emotional expression of the voice are adjusted based on the user's emotional data. For example, the story is read in a bright tone based on the emotional data of "joy."

[0299] Furthermore, when generating illustrations, the device uses image generation AI to generate them. The server uses a generative AI model to generate appropriate illustrations based on the story and emotion data. The generated illustrations are incorporated into the story and provided to the user. If the user wishes, a service is also available that binds the story and illustrations and provides them as a physical book.

[0300] Below are some example input prompts for the generative AI model:

[0301] Example prompt sentence:

[0302] Character: Taro

[0303] Theme: Adventure

[0304] Key Element: Magic Sword

[0305] Emotion data: Joy

[0306] This system allows users to easily create their own original stories and enjoy them in a variety of formats. By utilizing emotion data, it is possible to provide a more personalized experience and improve user satisfaction.

[0307] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0308] Step 1:

[0309] Users input story settings through the device interface, including characters, themes, and key elements, into text fields and drop-down menus.

[0310] Input: "Character: Taro" "Theme: Adventure" "Important element: Magic sword"

[0311] Output: Input data (Character: Taro, Theme: Adventure, Key element: Magic sword)

[0312] Step 2:

[0313] The device uses an emotion engine to recognize the user's emotional state in real time, analyzing the user's facial expressions and tone of voice using a camera and microphone, and converting emotional data such as "happiness," "sadness," and "excitement" into numerical values.

[0314] Input: User's facial expression images, voice audio data

[0315] Output: Emotion data (e.g., Happiness: 80%, Sadness: 10%, Excitement: 10%)

[0316] Step 3:

[0317] The device converts the story setting data and emotion data entered by the user into JSON format and sends it to the server, which converts the data into a specific format and sends an HTTP request.

[0318] Input: Story setting data (character: Taro, theme: adventure, key element: magic sword), emotion data (joy: 80%, sadness: 10%, excitement: 10%)

[0319] Output: JSON data (integration of story setting data and emotion data)

[0320] Step 4:

[0321] The server parses the received JSON data and generates a story using a generative AI model (e.g., OpenAI's GPT-3). It sends prompts to the generative AI model and generates a story based on the input elements and emotion data.

[0322] Input: JSON data (story setting data and emotion data)

[0323] Output: Generated story (e.g., "Taro finds a magical sword and embarks on an exciting adventure. He overcomes many interesting challenges, has fun, and becomes a legendary hero.")

[0324] Step 5:

[0325] The server saves the generated stories in a database. When saving, the user's plan information is taken into consideration, and free users can save up to three stories.

[0326] Input: Generated story, user information

[0327] Output: Narrative data stored in a database

[0328] Step 6:

[0329] The server sends the saved story data back to the device. The data transfer uses HTTP responses to send the data to the device.

[0330] Input: Narrative data in the database

[0331] Output: HTTP response with story data

[0332] Step 7:

[0333] The terminal displays the received story data on a user interface, allowing the user to view the generated story on the interface.

[0334] Input: Story data (HTTP response)

[0335] Output: A story displayed on a user interface

[0336] Step 8:

[0337] If the user selects the paid option, the device will use a voice generation service to provide the story audibly, with the tone and emotional expression of the voice adjusted based on the user's emotional data.

[0338] Input: Narrative data, emotion data

[0339] Output: Modified audio data

[0340] Step 9:

[0341] If the user requests further illustrations, the device sends a request to the image generation AI to generate illustrations that match the story. The generated illustrations are then incorporated into the story and displayed on the interface.

[0342] Input: Narrative data, emotion data

[0343] Output: Generated illustration data

[0344] (Application example 2)

[0345] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0346] Conventional story generation systems simply generate stories based on information entered by the user and are unable to reflect the user's emotions or experiences in the virtual environment. This has resulted in limited user experiences and makes it difficult to provide a personalized experience. The present invention aims to solve these problems and provide a more sophisticated, emotionally-sensitive personalized storybook within a virtual environment.

[0347] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the user's emotions and adjusting the tone and expression of the story based on the emotional data, means for displaying the story in a virtual environment, and means for providing the story generated by using a voice generation service by voice. This makes it possible to generate a personalized story tailored to the user's emotions and to experience the story in a immersive virtual environment.

[0348] A "user" is someone who interacts with the system to input story settings and enjoy a personalized experience.

[0349] "Characters" refers to people or characters that the user sets up in the story.

[0350] A "theme" is a central concept that a user specifies as the theme or scenario of a story.

[0351] "Important elements" refer to items and settings that play an important role in the progression of the story.

[0352] An "interface" is a screen or tool that allows a user to input characters, themes, and important elements.

[0353] "Means for collecting data" refers to the functionality for obtaining and storing information entered by the user into the interface.

[0354] "Server" refers to a remote computer system that stores and processes data.

[0355] "Generative AI" refers to artificial intelligence that automatically generates stories based on data entered by the user.

[0356] A "narrative" is a text-based story created by generative AI.

[0357] "Means for storage" refers to the function for storing the generated stories in a database or the like.

[0358] "Display means" is a function for displaying the generated story on the user's device.

[0359] "Means for analyzing emotions" refers to a function for acquiring emotional data by analyzing the user's facial expressions and tone of voice.

[0360] "Emotion data" is data that quantifies the analyzed emotional state of the user.

[0361] "Means for adjusting tone and expression" refers to a function for adjusting the atmosphere and content of a story's text based on emotional data.

[0362] "Virtual environment" refers to a system environment in which users can experience a story in virtual reality.

[0363] A "speech generation service" is a system that converts text into speech and allows the user to listen.

[0364] The present invention provides a system for generating a personalized story based on data input by a user and the user's emotions at the time, and displaying the story in a virtual environment. Specific embodiments are described below.

[0365] Providing a user interface

[0366] The user enters the setting of the story through a terminal interface. The interface provides fields for entering characters, themes, and key elements. For example,

[0367] Character: "Alice"

[0368] Theme: "Dream Adventure"

[0369] Key element: "Magic Flower"

[0370] Emotion recognition

[0371] The device is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice. The emotion engine recognizes the user's emotional state in real time through a camera and microphone. The recognized emotional data is quantified and expressed as excitement, joy, surprise, etc.

[0372] Sending data

[0373] The device collects story setting data and emotion data entered by the user, converts it into an appropriate format such as JSON, and then sends it to the server.

[0374] Narrative Generation

[0375] The server uses a generation AI to generate a story based on the received data. The generation AI creates a natural-sounding story based on the story setting data and emotion data entered by the user. For example, if a user provides emotion data for "joy" on the theme of "dream adventure," the generated story will be written in a bright and cheerful tone.

[0376] Display in a virtual environment

[0377] The generated story is sent back to the terminal from the server and displayed on the user's device. At this time, the user can experience the generated story in a highly immersive manner in a virtual space by using a smartphone or head-mounted display.

[0378] Voice generation

[0379] The server's voice generation service also provides the generated story in audio format. The story's voice is adjusted to match the user's emotional expression based on the user's emotional data. For example, a story generated with emotional data expressing joy will be read in a bright tone.

[0380] Illustration generation

[0381] Furthermore, image generation AI is used to generate illustrations that match the story. The style and content of the illustrations are also adjusted based on the user's emotional data. For example, if a story contains a lot of emotional data for "surprise," illustrations that match that emotion will be generated.

[0382] Hardware and software used

[0383] Emotion Engine: Camera and microphone for recognizing user emotions

[0384] Generative AI: An AI model for generating stories

[0385] Virtual environment display: head-mounted display and smartphone

[0386] Speech generation service: A system that provides generated stories in audio format.

[0387] Image generation AI: AI model for generating illustrations

[0388] Specific examples

[0389] Example prompt sentence:

[0390] Characters: Alice

[0391] Theme: Dream Adventure

[0392] Key Element: Magic Flower

[0393] Emotions: Joy and excitement

[0394] Generated story:

[0395] After finding the magic flower, Alice embarks on an exciting adventure, solving mysteries and having fun with new friends in the land of dreams.

[0396] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0397] Step 1:

[0398] The user inputs the characters, theme, and important elements through the terminal interface. The input information is collected as the character "Alice," the theme "Dream Adventure," and the important element "Magic Flower."

[0399] Step 2:

[0400] The device's built-in emotion engine recognizes the user's facial expressions and tone of voice in real time through a camera and microphone, and the recognized emotion data is converted into a numerical value such as "happiness" or "excitement."

[0401] Step 3:

[0402] Emotion data and story setting data entered by the user are converted into JSON format. For example, data is sent in the following format:

[0403] json

[0404] {

[0405] "characters": "Alice",

[0406] "theme": "Dream Adventure",

[0407] "elements": "Magic Flower",

[0408] "emotion_data": {

[0409] "joy": 0.8,

[0410] "excitement": 0.7

[0411] }

[0412] }

[0413] Step 4:

[0414] The device sends the collected data to the server, which receives it and checks the data for consistency and completeness.

[0415] Step 5:

[0416] The server runs the generative AI model based on the received data and generates a story. At this time, the generative AI model based on the prompt sentence operates as follows:

[0417] Prompt statement:

[0418] Characters: Alice

[0419] Theme: Dream Adventure

[0420] Key Element: Magic Flower

[0421] Emotions: Joy and excitement

[0422] Generated story:

[0423] After finding the magic flower, Alice embarks on an exciting adventure, solving mysteries and having fun with new friends in the land of dreams.

[0424] Step 6:

[0425] The server stores the generated story in a database, and the story is associated with the user's emotional data and input data.

[0426] Step 7:

[0427] The server sends the generated story back to the device, which then displays it to the user, who then experiences the story in a virtual environment using a smartphone or head-mounted display.

[0428] Step 8:

[0429] The server's voice generation service converts the generated story into audio, adjusting the tone and expression of the audio based on the user's emotional data.

[0430] Step 9:

[0431] Additionally, it uses image generation AI to generate illustrations that match the story, which are adjusted based on emotional data and provided to users along with the story.

[0432] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0433] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0434] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0435] [Second embodiment]

[0436] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0437] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0438] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0439] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0440] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0441] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0442] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0443] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0444] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0445] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0446] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0447] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0448] The system of the present invention is designed to enable users to easily create original stories and picture books. Specific embodiments will be described below.

[0449] Providing a user interface

[0450] The user inputs the story setting through a terminal interface. The interface provides fields for inputting characters, themes, key elements, etc. For example,

[0451] Character: "Taro"

[0452] Theme: "Adventure"

[0453] Key Element: "Magic Sword"

[0454] Enter items such as:

[0455] Sending data

[0456] The terminal collects the user's input data, converts it into an appropriate format such as JSON, and then sends it to the server.

[0457] Narrative Generation

[0458] The server uses a generation AI to generate a story based on the received data. The generation AI creates a story with natural sentences based on the input data. For example, if a user inputs the settings above, the generated story will look like this:

[0459] Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero.

[0460] Preserving the Genesis Story

[0461] The stories you create are stored in a database on the server. With the free plan, you can save up to three stories, after which you are restricted from saving new stories. With the paid plan, you can save four or more stories.

[0462] Showing the story

[0463] The server sends the generated story back to the terminal, which displays the story to the user, who can then view the generated story on the interface.

[0464] Voice Generation Options

[0465] As a paid option, the device can use a voice generation service to provide the generated story audibly. The device sends a voice generation request to the server, which uses TTS (Text to Speech) technology to convert the story into audio and send it back to the device. The user can then play the generated audio in the interface.

[0466] Illustration and binding options

[0467] Another paid option is for the device to provide a service that uses image generation AI to generate illustrations. The server receives a request from the device and generates illustrations that match the content of the story, integrates them with the story, and sends them back to the device. Furthermore, if the user requests it, the server can also bind the story and illustrations and provide them as a physical book.

[0468] This system allows users to quickly and effortlessly create original stories and picture books based on pre-defined elements, and enjoy them in a variety of formats. The stories created can be provided not only as text, but also in audio and illustration formats, making it possible to meet the diverse needs of users.

[0469] The processing flow will be explained below.

[0470] Step 1:

[0471] The user inputs the setting of the story. Using the terminal interface, the user inputs the characters, theme, and key elements. For example, the user inputs the character "Taro," the theme "adventure," and the key element "magic sword."

[0472] Step 2:

[0473] The device collects input data. It converts the data entered by the user into a format such as JSON. Example: {"character": "Taro", "theme": "Adventure", "keyElement": "Magic Sword"}

[0474] Step 3:

[0475] The device sends data to the server by using an HTTP POST request to send input data to a specified endpoint on the server.

[0476] Step 4:

[0477] The server receives the request. The request is received at the appropriate endpoint on the server and the input data is parsed.

[0478] Step 5:

[0479] The server gives instructions to the AI ​​generator. Based on the parsed data, it instructs the AI ​​generator to generate a story. For example, it provides the AI ​​generator with the following prompts: "Character: Taro," "Theme: Adventure," "Important Element: Magic Sword."

[0480] Step 6:

[0481] The generation AI generates a story. The generation AI creates a story based on the provided prompts. Example: The generated story is "Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero."

[0482] Step 7:

[0483] The server stores the generated stories. The generated stories are stored in a database. The free plan allows you to store up to three stories, and any more stories will be limited to one.

[0484] Step 8:

[0485] The server returns the generated story to the terminal as an HTTP response. Example: {"story": "Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero."}

[0486] Step 9:

[0487] The terminal displays the generated story. A user interface displays the generated story, allowing the user to view the story.

[0488] Step 10 (Optional):

[0489] The user selects the read aloud option: If the user wants to read aloud, the user selects this option in the interface.

[0490] Step 11 (Optional):

[0491] The device sends a request for speech generation to the server. Based on the input data and the generated story, a request for speech generation is sent to the server.

[0492] Step 12 (Optional):

[0493] The server converts the story into audio using a TTS service, converting the text of the generated story into audio and generating an audio file.

[0494] Step 13 (Optional):

[0495] The server returns the audio file to the device. The generated audio file is sent to the device as an HTTP response.

[0496] Step 14 (Optional):

[0497] The device plays the audio and displays an audio play button in the user interface, allowing the user to play the audio of the story.

[0498] Step 15 (Optional):

[0499] The user selects the Generate Illustration option: If the user wants to generate illustrations, the user selects this option in the interface.

[0500] Step 16 (Optional):

[0501] The terminal sends a request to the server to generate illustrations. Based on the input data and the generated story, a request to generate illustrations is sent to the server.

[0502] Step 17 (Optional):

[0503] The server generates illustrations using image generation AI, which creates corresponding illustrations based on the content of the story.

[0504] Step 18 (Optional):

[0505] The server returns the generated illustration to the terminal, which then sends the generated illustration to the terminal as an HTTP response.

[0506] Step 19 (Optional):

[0507] The terminal integrates the illustrations into the story and displays them. The user interface integrates the story and illustrations generated by the terminal so that the user can view them.

[0508] Step 20 (Optional):

[0509] The user selects the option to bind the story: If binding is desired, the user selects this option in the interface.

[0510] Step 21 (Optional):

[0511] The server arranges for a bookbinding service to bind the generated story and illustrations into a physical book and provide it to the user.

[0512] Example 1

[0513] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0514] Conventional story creation systems generate stories based on specific templates, making it difficult for users to easily create their own stories. Furthermore, few systems allow users to enjoy stories in a richer format by adding audio and illustrations, rather than just displaying the generated stories in text format. Therefore, there is a need for a system that allows users to effortlessly create, save, and view their own stories in a variety of formats.

[0515] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0516] In this invention, the server includes means for generating a story based on received input data using a generative AI model, means for saving the generated story in a database, and means for returning the generated story to a user terminal and displaying it to the user. This allows users to easily create original stories, add audio and illustrations to them in addition to text, and save and view them.

[0517] "Interface" refers to the screens and input fields that allow users to perform operations and input data into the system.

[0518] "Input Data" refers to information about characters, themes, key elements, etc. that a user inputs into the system through an interface.

[0519] "Server" refers to the main computer system that receives and processes data sent from terminals over a network.

[0520] A "generative AI model" refers to an artificial intelligence algorithm that generates natural-looking text or images based on the data it receives.

[0521] "Narrative" refers to a story created by a generative AI model based on user input data.

[0522] "Database" refers to a system for structuring and storing generated narratives and other related data.

[0523] "User terminal" refers to a device such as a computer or smartphone that a user uses to access and operate the system.

[0524] "Text-to-speech technology" refers to technology that converts text data into voice data.

[0525] "Image generation AI" refers to an artificial intelligence algorithm that generates images and illustrations based on the content of a story.

[0526] A "prompt sentence" refers to a sentence that provides a generative AI model with the input data it needs to generate a story.

[0527] The system of the present invention is designed to enable users to easily create original stories and picture books. Specific embodiments will be described below.

[0528] The system provides an interface for users to input characters, themes, and key elements. Users use a terminal to input information into this interface. For example, they might input the following information:

[0529] Character: "Taro"

[0530] Theme: "Adventure"

[0531] Key Element: "Magic Sword"

[0532] The terminal collects the data entered by the user, converts it into JSON format, and sends it to the server via an HTTP POST request. The server receives the JSON data and analyzes its contents.

[0533] Next, the server uses a generative AI model (e.g., OpenAI's GPT-4) to generate a story based on the input data. Specifically, it gives the generative AI model a prompt like this:

[0534] Characters: Taro, Theme: Adventure, Key Elements: Magic Sword

[0535] Based on this prompt, the generative AI model generates the following story:

[0536] Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero.

[0537] The generated stories are stored in a database by the server. The free plan allows you to store up to three stories, after which you are restricted from saving new stories. Paid plans allow you to store more stories.

[0538] The server then sends the generated story back to the user terminal, which displays the received story on its interface so that the user can view the story.

[0539] As a paid option, devices can use a speech generation service. The device sends a speech generation request to a server, which converts the story into audio using text-to-speech (TTS) technology. The generated audio data is sent back to the device, allowing the user to play the audio in the interface.

[0540] There is also an option to use image generation AI (e.g., DALL-E) to generate illustrations that match the story. The device sends a request for illustration generation to the server, and the server generates the illustrations using the image generation AI. These illustrations are integrated with the story and sent back to the device. In this way, the user can enjoy the story and illustrations together. Furthermore, if the user requests it, there is also a service that will bind the story and illustrations and provide them as a physical book.

[0541] As described above, this system allows users to quickly and effortlessly create original stories and picture books based on pre-set elements, and enjoy them in a variety of formats. The stories created can be provided not only as text, but also in audio and illustration formats, making it a system that can meet the diverse needs of users.

[0542] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0543] Step 1:

[0544] Providing a user interface

[0545] The user uses a terminal to access an interface to input the elements needed to create a story. The user inputs characters, themes, and key elements. For example, "Taro", "adventure", and "magic sword". The input data is saved in form fields. Input: Data entered by the user (characters, themes, key elements). Output: Input data saved in form fields.

[0546] Step 2:

[0547] Sending data

[0548] The device collects data entered by the user and converts it into JSON format. The JSON data is then sent to the server using an HTTP POST request. The data includes characters, themes, key elements, etc. Input: The input data stored in the form fields. Output: The JSON data sent to the server.

[0549] Step 3:

[0550] Narrative Generation

[0551] The server receives JSON data sent from the device. After receiving it, it analyzes the data and calls a generative AI model (e.g., OpenAI GPT-4) to generate a story. The generative AI model is given a prompt. For example, the following prompt is input to the generative AI model: "Character: Taro, Theme: Adventure, Key Element: Magic Sword." The generative AI model generates a story based on the prompt. Input: JSON data received by the server. Output: Story generated by the generative AI model.

[0552] Step 4:

[0553] Preserving the Genesis Story

[0554] The server saves the generated stories in a database. The database has tables for managing multiple stories, into which the text data of the stories is inserted. The free plan allows for the saving of up to three stories, after which new stories cannot be saved. Paid plans allow for the saving of more stories. Input: Text data of the generated stories. Output: Stories saved in the database.

[0555] Step 5:

[0556] Showing the story

[0557] The server returns the generated story to the terminal. The terminal receives the returned story and displays it on the user interface. The user can view the displayed story. Input: Text data of the story stored in the database. Output: The story displayed on the user interface.

[0558] Step 6:

[0559] Voice Generation Options

[0560] If the user selects the paid option, the device sends a voice generation request to the server. The server uses text-to-speech (TTS) technology to convert the story into voice data. The generated voice data is sent back to the device, and the user can play the audio on the interface. Input: Story text data, voice generation request. Output: Generated voice data.

[0561] Step 7:

[0562] Illustration and binding options

[0563] If the user selects the illustration generation option, the device sends an image generation request to the server. The server uses image generation AI (e.g., DALL-E) to generate illustrations that match the content of the story. The generated illustrations are integrated with the story and sent back to the device. Furthermore, if the user requests it, the server also provides a service to bind the story and illustrations and provide them as a physical book. Input: Story text data, image generation request. Output: Generated illustrations and integrated story data.

[0564] (Application example 1)

[0565] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0566] Conventional story generation systems are convenient in that they allow users to create their own stories and picture books, but they have limitations on how the stories can be utilized, making it difficult to share them with other users or use them commercially. Furthermore, elements such as converting the generated stories into audio or adding illustrations must be handled individually, and are not provided as a unified system. This often requires a lot of effort and time for users.

[0567] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0568] In this invention, the server includes means for providing an interface for users to input characters, themes, and important elements, means for collecting the input data and sending it to the server, means for generating a story based on the received input data using a generation AI, means for saving the generated story, means for displaying the generated story to users, and means for sharing and purchasing the generated story with other users in a virtual store. This allows users to quickly and easily create original stories and picture books and share them with other users or use them commercially.

[0569] "Users" are individuals or groups who use the system to create stories and picture books.

[0570] A "character" is a character who plays a role in the story and moves the story forward.

[0571] "Theme" refers to the overall theme or central idea of ​​a story.

[0572] "Important elements" refer to keywords, items, and settings that are essential to the story.

[0573] "Interface" refers to the screen and operating means that users use to input data into the system.

[0574] "Input Data" refers to information about characters, themes, key elements, etc., entered by a user through an interface.

[0575] A "server" is a computer system that processes and stores data on a network.

[0576] "Generative AI" is an artificial intelligence model that automatically generates natural language narratives based on user-entered data.

[0577] A "story" is a collection of texts that form a series of events or episodes based on characters, themes, and key elements.

[0578] "Storage" means keeping the generated story in a storage device such as a database.

[0579] A "virtual store" is a virtual space where users can use products and services and share content over the Internet.

[0580] "Sharing" means making the created story available for other users to see.

[0581] "Purchase" is the act of another user paying a fee to acquire the created story.

[0582] The present invention is a system that allows users to easily create original stories and picture books, and further allows them to share and commercially use them with other users in a virtual store.

[0583] Providing a user interface

[0584] Users input the story setting through the interface of a device such as a smartphone or PC. The interface provides fields for inputting characters, themes, important elements, etc. For example, it is designed so that users can input items such as "Taro," "adventure," and "magic sword."

[0585] Submitting data and generating stories

[0586] The device collects the user's input data, converts it into an appropriate format such as JSON, and then sends it to a cloud server. The server uses a generative AI (for example, OpenAI's GPT-4) to generate a story based on the received data. The generative AI generates natural-sounding sentences based on the input data. For example, if a user inputs the above settings, the following story will be generated: "Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero."

[0587] Preserving and displaying generative stories

[0588] The generated story is stored in a database on a cloud server, and is then sent back to the user's device and displayed on the interface.

[0589] Sharing and purchasing within the virtual store

[0590] The stories created can be shared with other users in the virtual store, and can also be purchased by other users for a fee.

[0591] Voice Generation Options

[0592] The server also includes a speech generation service, and as a paid option, the device can send a speech generation request to the server. The server uses Text-to-Speech (TTS) technology to convert the story into speech and send it back to the device. The user can then play the generated speech on the interface and enjoy it.

[0593] Illustration and binding options

[0594] Image generation AI can also be used to generate illustrations that match the content of a story. In this case, the server receives a user request and generates illustrations that match the story, integrates them with the story, and sends them back to the device. Furthermore, if the user requests it, the server can also bind the generated story and illustrations and provide them as a physical book.

[0595] Specific examples

[0596] For example, if a user wanted to create a story about a winter night and snow spirits, the following prompts would be input into the generative AI model:

[0597] Character: Snow Spirit

[0598] Theme: Winter Night

[0599] Key Element: Magic Snowflake

[0600] Based on this, the server generates a story about a snow spirit who finds a magical snowflake on a winter's night and embarks on a heartwarming adventure.

[0601] This system allows users to quickly and effortlessly create original stories and picture books, and enjoy them in a variety of formats. The generated content can also be used commercially, meeting a variety of needs.

[0602] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0603] Step 1:

[0604] The user inputs characters, themes, key elements, etc. into the interface on their smartphone or PC. The interface provides input fields where the user can enter information such as "Taro," "adventure," and "magic sword." Once this data is entered, it is converted into a JSON-formatted object.

[0605] Step 2:

[0606] The device collects the input data, converts it to JSON format, and sends it to the cloud server using an HTTP POST request. The input data is sent to the server for further processing.

[0607] Step 3:

[0608] The server parses the received JSON-formatted input data. The parsed data is then input as a prompt to a generative AI (e.g., OpenAI's GPT-4). The generative AI then automatically generates a story based on the characters, theme, and key elements. For example, a sentence like "Taro obtains a magic sword and sets out on an adventure" is generated.

[0609] Step 4:

[0610] The server saves the generated stories in a database, where they can be accessed by users later. When saving, metadata such as the user ID and the story title are also saved.

[0611] Step 5:

[0612] The server sends the generated story back to the user's device. The HTTP response sends the generated story in JSON format to the device, where it is displayed on the user's interface.

[0613] Step 6:

[0614] Users can share stories created in the virtual store with other users. By sharing, other users can view or purchase the story. The data of purchased stories is managed on the server and kept as shared information between users.

[0615] Step 7:

[0616] As a paid option, users can also listen to the generated story. The device sends a request to generate the story to the server, which uses Text-to-Speech (TTS) technology to convert the story into audio. The generated audio file is sent back to the device and played on the interface.

[0617] Step 8:

[0618] Another paid option is to use image generation AI to generate illustrations. Upon receiving a user request, the server generates illustrations based on the content of the story. The generated illustrations are integrated into the story and sent back to the device. The user can then enjoy the story, including the generated illustrations, as an e-book.

[0619] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0620] The system of the present invention is designed to enable users to easily create original stories and picture books. Furthermore, it adds a function to recognize the user's emotions and adjust the content, tone, illustrations, and speech generation of the story based on those emotions. Specific embodiments are described below.

[0621] Providing a user interface

[0622] The user inputs the story setting through a terminal interface. The interface provides fields for inputting characters, themes, and key elements. For example,

[0623] Character: "Taro"

[0624] Theme: "Adventure"

[0625] Key Element: "Magic Sword"

[0626] Enter items such as:

[0627] Emotion recognition

[0628] The device is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice. The emotion engine recognizes the user's emotional state in real time through a camera and microphone. The recognized emotion data is quantified as, for example, "happiness," "sadness," or "excitement."

[0629] Sending data

[0630] The device collects input data and emotion data, converts it into an appropriate format such as JSON, and then sends it to the server.

[0631] Narrative Generation

[0632] The server uses a generation AI to generate a story based on the received data. The generation AI creates a natural-sounding story based on the input story setting data and emotion data. For example, if a user provides emotion data of "joy" with the theme of "adventure," the generated story will be written in a bright and cheerful tone.

[0633] Taro finds a magical sword and embarks on an exciting adventure, overcoming many amusing challenges and becoming a legendary hero all while having fun.

[0634] Preserving the Genesis Story

[0635] The stories that are created are stored in a database on the server. With the free plan, up to three stories can be saved, and saving new stories beyond that is restricted. With the paid plan, four or more stories can be saved.

[0636] Showing the story

[0637] The server sends the generated story back to the terminal, which displays the story to the user, who can then view the generated story on the interface.

[0638] Voice Generation Options

[0639] As a paid option, the device can use a voice generation service to deliver the generated story aloud. The tone and emotional expression of the voice are adjusted based on the user's emotional data. For example, a story generated with emotional data expressing joy will be read in a bright tone.

[0640] Illustration and binding options

[0641] Another paid option is for the device to provide a service that uses image generation AI to generate illustrations. The server receives a request from the device and uses the AI ​​to create corresponding illustrations based on the content of the story and the user's emotional data. For example, if a story contains a lot of emotional data about "surprise," an illustration that matches that emotion will be generated. Furthermore, if the user requests it, the service also provides a service to bind the story and illustrations and provide them as a physical book.

[0642] This system allows users to quickly and effortlessly create original stories and picture books based on pre-defined elements and their own emotions, and enjoy them in a variety of formats. The stories they create can be provided not only as text but also in audio and illustration formats, making it possible to meet the diverse needs of users. Adding emotion recognition functionality will provide a more personalized experience and increase user satisfaction.

[0643] The processing flow will be explained below.

[0644] Step 1:

[0645] The user inputs the setting of the story. Using the terminal interface, the user inputs the characters, theme, and key elements. For example, the user inputs the character "Taro," the theme "adventure," and the key element "magic sword."

[0646] Step 2:

[0647] The device collects input data. It converts the data entered by the user into a format such as JSON. Example: {"character": "Taro", "theme": "Adventure", "keyElement": "Magic Sword"}

[0648] Step 3:

[0649] The device recognizes the user's emotions. Using an emotion engine, it analyzes the user's facial expressions and tone of voice in real time via the camera and microphone to generate emotion data. Example: {"emotion": "joy"}

[0650] Step 4:

[0651] The device sends the collected input data and emotion data to the server using an HTTP POST request. Example: {"character": "Taro", "theme": "Adventure", "keyElement": "Magic Sword", "emotion": "Joy"}

[0652] Step 5:

[0653] The server receives the request. The request is received at the server endpoint and the input data and emotion data are parsed.

[0654] Step 6:

[0655] The server gives instructions to the AI ​​generator. Based on the parsed data, the AI ​​generator is instructed to generate a story. For example, the server provides the AI ​​with the following prompts: "Character: Taro," "Theme: Adventure," "Important Element: Magic Sword," and "Emotion: Joy."

[0656] Step 7:

[0657] The Generator AI generates a story. The Generator AI creates a story based on the provided prompts. Example: A generated story is "Taro obtains a magical sword and sets off on an exciting adventure. He overcomes many interesting challenges, has fun, and becomes a legendary hero."

[0658] Step 8:

[0659] The server stores the generated stories. The generated stories are stored in a database. The free plan allows you to store up to three stories, and any more stories will be limited to one.

[0660] Step 9:

[0661] The server sends the generated story back to the terminal. It sends back the generated story as an HTTP response. Example: {"story": "Taro gets his hands on a magical sword and sets off on an exciting adventure. He overcomes many interesting challenges, has fun, and becomes a legendary hero."}

[0662] Step 10:

[0663] The terminal displays the generated story. A user interface displays the generated story, allowing the user to view the story.

[0664] Step 11 (Optional):

[0665] The user selects the read aloud option: If the user wants to read aloud, the user selects this option in the interface.

[0666] Step 12 (Optional):

[0667] The device sends a request for speech generation to the server.,Based on the input data, the generated story, and,emotion data, a speech generation request is sent to the server.

[0668] Step 13 (Optional):

[0669] The server converts the story into audio using a TTS service. The generated story text is converted into audio and an audio file is generated. The tone and emotional expression of the audio are adjusted based on the emotional data.

[0670] Step 14 (Optional):

[0671] The server returns the audio file to the device. The generated audio file is sent to the device as an HTTP response.

[0672] Step 15 (Optional):

[0673] The device plays the audio and displays an audio play button in the user interface, allowing the user to play the audio of the story.

[0674] Step 16 (Optional):

[0675] The user selects the Generate Illustration option: If the user wants to generate illustrations, the user selects this option in the interface.

[0676] Step 17 (Optional):

[0677] The terminal sends a request to the server to generate an illustration. Based on the input data, the generated story, and the emotion data, the request to generate an illustration is sent to the server.

[0678] Step 18 (Optional):

[0679] The server uses image generation AI to generate illustrations. Based on the content of the story and emotional data, the AI ​​creates corresponding illustrations. For example, if a story contains a lot of emotional data for "surprise," an illustration that matches that emotion will be generated.

[0680] Step 19 (Optional):

[0681] The server returns the generated illustration to the terminal, which then sends the generated illustration to the terminal as an HTTP response.

[0682] Step 20 (Optional):

[0683] The terminal integrates the illustrations into the story and displays them. The user interface integrates the story and illustrations generated by the terminal so that the user can view them.

[0684] Step 21 (Optional):

[0685] The user selects the option to bind the story: If binding is desired, the user selects this option in the interface.

[0686] Step 22 (Optional):

[0687] The server arranges for a bookbinding service to bind the generated story and illustrations into a physical book and provide it to the user.

[0688] Example 2

[0689] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0690] Conventional story generation systems generate stories solely based on user input, making it difficult to provide a personalized experience. Furthermore, they are unable to adjust the content, tone, illustrations, and audio expression of the story according to the user's emotions and preferences. Therefore, a new system that takes the user's emotions into account during the story generation process is needed to improve user satisfaction.

[0691] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0692] In this invention, the server includes means for providing an interface for a user to input characters, themes, and important elements, means for collecting the input data, recognizing the user's emotional state using an emotion engine, and transmitting the input data to the server, means for generating a story based on the received input data and emotional data using a generative AI model, means for saving the generated story, and means for displaying the generated story to the user. This makes it possible to generate a personalized story according to the user's emotional state and provide content that is more satisfying.

[0693] An "interface" is the means by which a user inputs characters, themes, and key elements.

[0694] An "emotion engine" is a means having the function of recognizing the user's emotional state and collecting it as numerical data.

[0695] A "generative AI model" is a type of artificial intelligence used to generate stories based on input data and emotional data.

[0696] The "server" is a central computer system that receives input data and emotional data, and generates, stores, and displays stories.

[0697] A "voice generation service" is a means of providing a generated story in audio form and adjusting the tone and emotional expression of the audio based on the user's emotional data.

[0698] "Image generation AI" is a type of artificial intelligence used to generate illustrations that match a generated story.

[0699] "Storage means" refers to the methods and techniques used to store the generated stories in a database or storage device.

[0700] "Display means" refers to the methods and techniques for visually presenting the generated story on a user interface.

[0701] The present invention is a system that allows users to easily create original stories and picture books. This system has the function of recognizing the user's emotional state and adjusting the content, tone, illustrations, and audio expression of the story based on that emotion. An embodiment of the system is described in detail below.

[0702] The user inputs the story setting through a device interface. The interface provides fields for inputting characters, themes, and key elements. For example, the user can input "Taro," "adventure," and "magic sword." The interface is designed to be intuitive for the user, with text fields and drop-down menus.

[0703] The device is equipped with a camera and microphone, and the emotion engine uses these devices to analyze the user's facial expressions and tone of voice in real time. The emotion engine analyzes the collected data and quantifies emotions such as "happiness," "sadness," and "excitement." This emotional data is an important element required for story generation.

[0704] The device converts the story setting data and recognized emotion data entered by the user into JSON format and sends it to the server. The server analyzes the received data and generates a story using a generative AI model (e.g., OpenAI's GPT-3). The generative AI model creates a story with an appropriate tone and content based on the input elements and emotion data. For example, if the user provides emotion data for "adventure" and "joy," the generated story will have a bright and cheerful tone.

[0705] The stories that are created are stored in a database on the server. Users with the free plan can save up to three stories, but to save more, a paid plan is required. Saved stories can be reused at any time upon user request.

[0706] Once the story generation is complete, the server sends the generated story data back to the device. The device displays the received story on its user interface, allowing the user to check the content. In addition to viewing the generated story in text format, the user can also optionally use services that generate audio and illustrations.

[0707] When generating the voice, the device uses a voice generation service to provide the generated story aloud. The tone and emotional expression of the voice are adjusted based on the user's emotional data. For example, the story is read in a bright tone based on the emotional data of "joy."

[0708] Furthermore, when generating illustrations, the device uses image generation AI to generate them. The server uses a generative AI model to generate appropriate illustrations based on the story and emotion data. The generated illustrations are incorporated into the story and provided to the user. If the user wishes, a service is also available that binds the story and illustrations and provides them as a physical book.

[0709] Below are some example input prompts for the generative AI model:

[0710] Example prompt sentence:

[0711] Character: Taro

[0712] Theme: Adventure

[0713] Key Element: Magic Sword

[0714] Emotion data: Joy

[0715] This system allows users to easily create their own original stories and enjoy them in a variety of formats. By utilizing emotion data, it is possible to provide a more personalized experience and improve user satisfaction.

[0716] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0717] Step 1:

[0718] Users input story settings through the device interface, including characters, themes, and key elements, into text fields and drop-down menus.

[0719] Input: "Character: Taro" "Theme: Adventure" "Important element: Magic sword"

[0720] Output: Input data (Character: Taro, Theme: Adventure, Key element: Magic sword)

[0721] Step 2:

[0722] The device uses an emotion engine to recognize the user's emotional state in real time, analyzing the user's facial expressions and tone of voice using a camera and microphone, and converting emotional data such as "happiness," "sadness," and "excitement" into numerical values.

[0723] Input: User's facial expression images, voice audio data

[0724] Output: Emotion data (e.g., Happiness: 80%, Sadness: 10%, Excitement: 10%)

[0725] Step 3:

[0726] The device converts the story setting data and emotion data entered by the user into JSON format and sends it to the server, which converts the data into a specific format and sends an HTTP request.

[0727] Input: Story setting data (character: Taro, theme: adventure, key element: magic sword), emotion data (joy: 80%, sadness: 10%, excitement: 10%)

[0728] Output: JSON data (integration of story setting data and emotion data)

[0729] Step 4:

[0730] The server parses the received JSON data and generates a story using a generative AI model (e.g., OpenAI's GPT-3). It sends prompts to the generative AI model and generates a story based on the input elements and emotion data.

[0731] Input: JSON data (story setting data and emotion data)

[0732] Output: Generated story (e.g., "Taro finds a magical sword and embarks on an exciting adventure. He overcomes many interesting challenges, has fun, and becomes a legendary hero.")

[0733] Step 5:

[0734] The server saves the generated stories in a database. When saving, the user's plan information is taken into consideration, and free users can save up to three stories.

[0735] Input: Generated story, user information

[0736] Output: Narrative data stored in a database

[0737] Step 6:

[0738] The server sends the saved story data back to the device. The data transfer uses HTTP responses to send the data to the device.

[0739] Input: Narrative data in the database

[0740] Output: HTTP response with story data

[0741] Step 7:

[0742] The terminal displays the received story data on a user interface, allowing the user to view the generated story on the interface.

[0743] Input: Story data (HTTP response)

[0744] Output: A story displayed on a user interface

[0745] Step 8:

[0746] If the user selects the paid option, the device will use a voice generation service to provide the story audibly, with the tone and emotional expression of the voice adjusted based on the user's emotional data.

[0747] Input: Narrative data, emotion data

[0748] Output: Modified audio data

[0749] Step 9:

[0750] If the user requests further illustrations, the device sends a request to the image generation AI to generate illustrations that match the story. The generated illustrations are then incorporated into the story and displayed on the interface.

[0751] Input: Narrative data, emotion data

[0752] Output: Generated illustration data

[0753] (Application example 2)

[0754] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0755] Conventional story generation systems simply generate stories based on information entered by the user and are unable to reflect the user's emotions or experiences in the virtual environment. This has resulted in limited user experiences and makes it difficult to provide a personalized experience. The present invention aims to solve these problems and provide a more sophisticated, emotionally-sensitive personalized storybook within a virtual environment.

[0756] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the user's emotions and adjusting the tone and expression of the story based on the emotional data, means for displaying the story in a virtual environment, and means for providing the story generated by using a voice generation service by voice. This makes it possible to generate a personalized story tailored to the user's emotions and to experience the story in a immersive virtual environment.

[0757] A "user" is someone who interacts with the system to input story settings and enjoy a personalized experience.

[0758] "Characters" refers to people or characters that the user sets up in the story.

[0759] A "theme" is a central concept that a user specifies as the theme or scenario of a story.

[0760] "Important elements" refer to items and settings that play an important role in the progression of the story.

[0761] An "interface" is a screen or tool that allows a user to input characters, themes, and important elements.

[0762] "Means for collecting data" refers to the functionality for obtaining and storing information entered by the user into the interface.

[0763] "Server" refers to a remote computer system that stores and processes data.

[0764] "Generative AI" refers to artificial intelligence that automatically generates stories based on data entered by the user.

[0765] A "narrative" is a text-based story created by generative AI.

[0766] "Means for storage" refers to the function for storing the generated stories in a database or the like.

[0767] "Display means" is a function for displaying the generated story on the user's device.

[0768] "Means for analyzing emotions" refers to a function for acquiring emotional data by analyzing the user's facial expressions and tone of voice.

[0769] "Emotion data" is data that quantifies the analyzed emotional state of the user.

[0770] "Means for adjusting tone and expression" refers to a function for adjusting the atmosphere and content of a story's text based on emotional data.

[0771] "Virtual environment" refers to a system environment in which users can experience a story in virtual reality.

[0772] A "speech generation service" is a system that converts text into speech and allows the user to listen.

[0773] The present invention provides a system for generating a personalized story based on data input by a user and the user's emotions at the time, and displaying the story in a virtual environment. Specific embodiments are described below.

[0774] Providing a user interface

[0775] The user enters the setting of the story through a terminal interface. The interface provides fields for entering characters, themes, and key elements. For example,

[0776] Character: "Alice"

[0777] Theme: "Dream Adventure"

[0778] Key element: "Magic Flower"

[0779] Emotion recognition

[0780] The device is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice. The emotion engine recognizes the user's emotional state in real time through a camera and microphone. The recognized emotional data is quantified and expressed as excitement, joy, surprise, etc.

[0781] Sending data

[0782] The device collects story setting data and emotion data entered by the user, converts it into an appropriate format such as JSON, and then sends it to the server.

[0783] Narrative Generation

[0784] The server uses a generation AI to generate a story based on the received data. The generation AI creates a natural-sounding story based on the story setting data and emotion data entered by the user. For example, if a user provides emotion data for "joy" on the theme of "dream adventure," the generated story will be written in a bright and cheerful tone.

[0785] Display in a virtual environment

[0786] The generated story is sent back to the terminal from the server and displayed on the user's device. At this time, the user can experience the generated story in a highly immersive manner in a virtual space by using a smartphone or head-mounted display.

[0787] Voice generation

[0788] The server's voice generation service also provides the generated story in audio format. The story's voice is adjusted to match the user's emotional expression based on the user's emotional data. For example, a story generated with emotional data expressing joy will be read in a bright tone.

[0789] Illustration generation

[0790] Furthermore, image generation AI is used to generate illustrations that match the story. The style and content of the illustrations are also adjusted based on the user's emotional data. For example, if a story contains a lot of emotional data for "surprise," illustrations that match that emotion will be generated.

[0791] Hardware and software used

[0792] Emotion Engine: Camera and microphone for recognizing user emotions

[0793] Generative AI: An AI model for generating stories

[0794] Virtual environment display: head-mounted display and smartphone

[0795] Speech generation service: A system that provides generated stories in audio format.

[0796] Image generation AI: AI model for generating illustrations

[0797] Specific examples

[0798] Example prompt sentence:

[0799] Characters: Alice

[0800] Theme: Dream Adventure

[0801] Key Element: Magic Flower

[0802] Emotions: Joy and excitement

[0803] Generated story:

[0804] After finding the magic flower, Alice embarks on an exciting adventure, solving mysteries and having fun with new friends in the land of dreams.

[0805] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0806] Step 1:

[0807] The user inputs the characters, theme, and important elements through the terminal interface. The input information is collected as the character "Alice," the theme "Dream Adventure," and the important element "Magic Flower."

[0808] Step 2:

[0809] The device's built-in emotion engine recognizes the user's facial expressions and tone of voice in real time via a camera and microphone, and the recognized emotion data is converted into a numerical value such as "happiness" or "excitement."

[0810] Step 3:

[0811] Emotion data and story setting data entered by the user are converted into JSON format. For example, data is sent in the following format:

[0812] json

[0813] {

[0814] "characters": "Alice",

[0815] "theme": "Dream Adventure",

[0816] "elements": "Magic Flower",

[0817] "emotion_data": {

[0818] "joy": 0.8,

[0819] "excitement": 0.7

[0820] }

[0821] }

[0822] Step 4:

[0823] The device sends the collected data to the server, which receives it and checks the data for consistency and completeness.

[0824] Step 5:

[0825] The server runs the generative AI model based on the received data and generates a story. At this time, the generative AI model based on the prompt sentence operates as follows:

[0826] Prompt statement:

[0827] Characters: Alice

[0828] Theme: Dream Adventure

[0829] Key Element: Magic Flower

[0830] Emotions: Joy and excitement

[0831] Generated story:

[0832] After finding the magic flower, Alice embarks on an exciting adventure, solving mysteries and having fun with new friends in the land of dreams.

[0833] Step 6:

[0834] The server stores the generated story in a database, and the story is associated with the user's emotional data and input data.

[0835] Step 7:

[0836] The server sends the generated story back to the device, which then displays it to the user, who then experiences the story in a virtual environment using a smartphone or head-mounted display.

[0837] Step 8:

[0838] The server's voice generation service converts the generated story into audio, adjusting the tone and expression of the audio based on the user's emotional data.

[0839] Step 9:

[0840] Additionally, it uses image generation AI to generate illustrations that match the story, which are adjusted based on emotional data and provided to users along with the story.

[0841] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0842] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0843] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0844] [Third embodiment]

[0845] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0846] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0847] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0848] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0849] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0850] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0851] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0852] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0853] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0854] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0855] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0856] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0857] The system of the present invention is designed to enable users to easily create original stories and picture books. Specific embodiments will be described below.

[0858] Providing a user interface

[0859] The user inputs the story setting through a terminal interface. The interface provides fields for inputting characters, themes, key elements, etc. For example,

[0860] Character: "Taro"

[0861] Theme: "Adventure"

[0862] Key Element: "Magic Sword"

[0863] Enter items such as:

[0864] Sending data

[0865] The terminal collects the user's input data, converts it into an appropriate format such as JSON, and then sends it to the server.

[0866] Narrative Generation

[0867] The server uses a generation AI to generate a story based on the received data. The generation AI creates a story with natural sentences based on the input data. For example, if a user inputs the settings above, the generated story will look like this:

[0868] Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero.

[0869] Preserving the Genesis Story

[0870] The stories you create are stored in a database on the server. With the free plan, you can save up to three stories, after which you are restricted from saving new stories. With the paid plan, you can save four or more stories.

[0871] Showing the story

[0872] The server sends the generated story back to the terminal, which displays the story to the user, who can then view the generated story on the interface.

[0873] Voice Generation Options

[0874] As a paid option, the device can use a voice generation service to provide the generated story audibly. The device sends a voice generation request to the server, which uses TTS (Text to Speech) technology to convert the story into audio and send it back to the device. The user can then play the generated audio in the interface.

[0875] Illustration and binding options

[0876] Another paid option is for the device to provide a service that uses image generation AI to generate illustrations. The server receives a request from the device and generates illustrations that match the content of the story, integrates them with the story, and sends them back to the device. Furthermore, if the user requests it, the server can also bind the story and illustrations and provide them as a physical book.

[0877] This system allows users to quickly and effortlessly create original stories and picture books based on pre-defined elements, and enjoy them in a variety of formats. The stories created can be provided not only as text, but also in audio and illustration formats, making it possible to meet the diverse needs of users.

[0878] The processing flow will be explained below.

[0879] Step 1:

[0880] The user inputs the setting of the story. Using the terminal interface, the user inputs the characters, theme, and key elements. For example, the user inputs the character "Taro," the theme "adventure," and the key element "magic sword."

[0881] Step 2:

[0882] The device collects input data. It converts the data entered by the user into a format such as JSON. Example: {"character": "Taro", "theme": "Adventure", "keyElement": "Magic Sword"}

[0883] Step 3:

[0884] The device sends data to the server by using an HTTP POST request to send input data to a specified endpoint on the server.

[0885] Step 4:

[0886] The server receives the request. The request is received at the appropriate endpoint on the server and the input data is parsed.

[0887] Step 5:

[0888] The server gives instructions to the AI ​​generator. Based on the parsed data, it instructs the AI ​​generator to generate a story. For example, it provides the AI ​​generator with the following prompts: "Character: Taro," "Theme: Adventure," "Important Element: Magic Sword."

[0889] Step 6:

[0890] The generation AI generates a story. The generation AI creates a story based on the provided prompts. Example: The generated story is "Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero."

[0891] Step 7:

[0892] The server stores the generated stories. The generated stories are stored in a database. The free plan allows you to store up to three stories, and any more stories will be limited to one.

[0893] Step 8:

[0894] The server returns the generated story to the terminal as an HTTP response. Example: {"story": "Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero."}

[0895] Step 9:

[0896] The terminal displays the generated story. A user interface displays the generated story, allowing the user to view the story.

[0897] Step 10 (Optional):

[0898] The user selects the read aloud option: If the user wants to read aloud, the user selects this option in the interface.

[0899] Step 11 (Optional):

[0900] The device sends a request for speech generation to the server. Based on the input data and the generated story, a request for speech generation is sent to the server.

[0901] Step 12 (Optional):

[0902] The server converts the story into audio using a TTS service, converting the text of the generated story into audio and generating an audio file.

[0903] Step 13 (Optional):

[0904] The server returns the audio file to the device. The generated audio file is sent to the device as an HTTP response.

[0905] Step 14 (Optional):

[0906] The device plays the audio and displays an audio play button in the user interface, allowing the user to play the audio of the story.

[0907] Step 15 (Optional):

[0908] The user selects the Generate Illustration option: If the user wants to generate illustrations, the user selects this option in the interface.

[0909] Step 16 (Optional):

[0910] The terminal sends a request to the server to generate illustrations. Based on the input data and the generated story, a request to generate illustrations is sent to the server.

[0911] Step 17 (Optional):

[0912] The server generates illustrations using image generation AI, which creates corresponding illustrations based on the content of the story.

[0913] Step 18 (Optional):

[0914] The server returns the generated illustration to the terminal, which then sends the generated illustration to the terminal as an HTTP response.

[0915] Step 19 (Optional):

[0916] The terminal integrates the illustrations into the story and displays them. The user interface integrates the story and illustrations generated by the terminal so that the user can view them.

[0917] Step 20 (Optional):

[0918] The user selects the option to bind the story: If binding is desired, the user selects this option in the interface.

[0919] Step 21 (Optional):

[0920] The server arranges for a bookbinding service to bind the generated story and illustrations into a physical book and provide it to the user.

[0921] Example 1

[0922] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0923] Conventional story creation systems generate stories based on specific templates, making it difficult for users to easily create their own stories. Furthermore, few systems allow users to enjoy stories in a richer format by adding audio and illustrations, rather than just displaying the generated stories in text format. Therefore, there is a need for a system that allows users to effortlessly create, save, and view their own stories in a variety of formats.

[0924] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0925] In this invention, the server includes means for generating a story based on received input data using a generative AI model, means for saving the generated story in a database, and means for returning the generated story to a user terminal and displaying it to the user. This allows users to easily create original stories, add audio and illustrations to them in addition to text, and save and view them.

[0926] "Interface" refers to the screens and input fields that allow users to perform operations and input data into the system.

[0927] "Input Data" refers to information about characters, themes, key elements, etc. that a user inputs into the system through an interface.

[0928] "Server" refers to the main computer system that receives and processes data sent from terminals over a network.

[0929] A "generative AI model" refers to an artificial intelligence algorithm that generates natural-looking text or images based on the data it receives.

[0930] "Narrative" refers to a story created by a generative AI model based on user input data.

[0931] "Database" refers to a system for structuring and storing generated narratives and other related data.

[0932] "User terminal" refers to a device such as a computer or smartphone that a user uses to access and operate the system.

[0933] "Text-to-speech technology" refers to technology that converts text data into voice data.

[0934] "Image generation AI" refers to an artificial intelligence algorithm that generates images and illustrations based on the content of a story.

[0935] A "prompt sentence" refers to a sentence that provides a generative AI model with the input data it needs to generate a story.

[0936] The system of the present invention is designed to enable users to easily create original stories and picture books. Specific embodiments will be described below.

[0937] The system provides an interface for users to input characters, themes, and key elements. Users use a terminal to input information into this interface. For example, they might input the following information:

[0938] Character: "Taro"

[0939] Theme: "Adventure"

[0940] Key Element: "Magic Sword"

[0941] The terminal collects the data entered by the user, converts it into JSON format, and sends it to the server via an HTTP POST request. The server receives the JSON data and analyzes its contents.

[0942] Next, the server uses a generative AI model (e.g., OpenAI's GPT-4) to generate a story based on the input data. Specifically, it gives the generative AI model a prompt like this:

[0943] Characters: Taro, Theme: Adventure, Key Elements: Magic Sword

[0944] Based on this prompt, the generative AI model generates the following story:

[0945] Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero.

[0946] The generated stories are stored in a database by the server. The free plan allows you to store up to three stories, after which you are restricted from saving new stories. Paid plans allow you to store more stories.

[0947] The server then sends the generated story back to the user terminal, which displays the received story on its interface so that the user can view the story.

[0948] As a paid option, devices can use a speech generation service. The device sends a speech generation request to a server, which converts the story into audio using text-to-speech (TTS) technology. The generated audio data is sent back to the device, allowing the user to play the audio in the interface.

[0949] There is also an option to use image generation AI (e.g., DALL-E) to generate illustrations that match the story. The device sends a request for illustration generation to the server, and the server generates the illustrations using the image generation AI. These illustrations are integrated with the story and sent back to the device. In this way, the user can enjoy the story and illustrations together. Furthermore, if the user requests it, there is also a service that will bind the story and illustrations and provide them as a physical book.

[0950] As described above, this system allows users to quickly and effortlessly create original stories and picture books based on pre-set elements, and enjoy them in a variety of formats. The stories created can be provided not only as text, but also in audio and illustration formats, making it a system that can meet the diverse needs of users.

[0951] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0952] Step 1:

[0953] Providing a user interface

[0954] The user uses a terminal to access an interface to input the elements needed to create a story. The user inputs characters, themes, and key elements. For example, "Taro", "adventure", and "magic sword". The input data is saved in form fields. Input: Data entered by the user (characters, themes, key elements). Output: Input data saved in form fields.

[0955] Step 2:

[0956] Sending data

[0957] The device collects data entered by the user and converts it into JSON format. The JSON data is then sent to the server using an HTTP POST request. The data includes characters, themes, key elements, etc. Input: The input data stored in the form fields. Output: The JSON data sent to the server.

[0958] Step 3:

[0959] Narrative Generation

[0960] The server receives JSON data sent from the device. After receiving it, it analyzes the data and calls a generative AI model (e.g., OpenAI GPT-4) to generate a story. The generative AI model is given a prompt. For example, the following prompt is input to the generative AI model: "Character: Taro, Theme: Adventure, Key Element: Magic Sword." The generative AI model generates a story based on the prompt. Input: JSON data received by the server. Output: Story generated by the generative AI model.

[0961] Step 4:

[0962] Preserving the Genesis Story

[0963] The server saves the generated stories in a database. The database has tables for managing multiple stories, into which the text data of the stories is inserted. The free plan allows for the saving of up to three stories, after which new stories cannot be saved. Paid plans allow for the saving of more stories. Input: Text data of the generated stories. Output: Stories saved in the database.

[0964] Step 5:

[0965] Showing the story

[0966] The server returns the generated story to the terminal. The terminal receives the returned story and displays it on the user interface. The user can view the displayed story. Input: Text data of the story stored in the database. Output: The story displayed on the user interface.

[0967] Step 6:

[0968] Voice Generation Options

[0969] If the user selects the paid option, the device sends a voice generation request to the server. The server uses text-to-speech (TTS) technology to convert the story into voice data. The generated voice data is sent back to the device, and the user can play the audio on the interface. Input: Story text data, voice generation request. Output: Generated voice data.

[0970] Step 7:

[0971] Illustration and binding options

[0972] If the user selects the illustration generation option, the device sends an image generation request to the server. The server uses image generation AI (e.g., DALL-E) to generate illustrations that match the content of the story. The generated illustrations are integrated with the story and sent back to the device. Furthermore, if the user requests it, the server also provides a service to bind the story and illustrations and provide them as a physical book. Input: Story text data, image generation request. Output: Generated illustrations and integrated story data.

[0973] (Application example 1)

[0974] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0975] Conventional story generation systems are convenient in that they allow users to create their own stories and picture books, but they have limitations on how the stories can be utilized, making it difficult to share them with other users or use them commercially. Furthermore, elements such as converting the generated stories into audio or adding illustrations must be handled individually, and are not provided as a unified system. This often requires a lot of effort and time for users.

[0976] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0977] In this invention, the server includes means for providing an interface for users to input characters, themes, and important elements, means for collecting the input data and sending it to the server, means for generating a story based on the received input data using a generation AI, means for saving the generated story, means for displaying the generated story to users, and means for sharing and purchasing the generated story with other users in a virtual store. This allows users to quickly and easily create original stories and picture books and share them with other users or use them commercially.

[0978] "Users" are individuals or groups who use the system to create stories and picture books.

[0979] A "character" is a character who plays a role in the story and moves the story forward.

[0980] "Theme" refers to the overall theme or central idea of ​​a story.

[0981] "Important elements" refer to keywords, items, and settings that are essential to the story.

[0982] "Interface" refers to the screen and operating means that users use to input data into the system.

[0983] "Input Data" refers to information about characters, themes, key elements, etc., entered by a user through an interface.

[0984] A "server" is a computer system that processes and stores data on a network.

[0985] "Generative AI" is an artificial intelligence model that automatically generates natural language narratives based on user-entered data.

[0986] A "story" is a collection of texts that form a series of events or episodes based on characters, themes, and key elements.

[0987] "Storage" means keeping the generated story in a storage device such as a database.

[0988] A "virtual store" is a virtual space where users can use products and services and share content over the Internet.

[0989] "Sharing" means making the created story available for other users to see.

[0990] "Purchase" is the act of another user paying a fee to acquire the created story.

[0991] The present invention is a system that allows users to easily create original stories and picture books, and further allows them to share and commercially use them with other users in a virtual store.

[0992] Providing a user interface

[0993] Users input the story setting through the interface of a device such as a smartphone or PC. The interface provides fields for inputting characters, themes, important elements, etc. For example, it is designed so that users can input items such as "Taro," "adventure," and "magic sword."

[0994] Submitting data and generating stories

[0995] The device collects the user's input data, converts it into an appropriate format such as JSON, and then sends it to a cloud server. The server uses a generative AI (for example, OpenAI's GPT-4) to generate a story based on the received data. The generative AI generates natural-sounding sentences based on the input data. For example, if a user inputs the above settings, the following story will be generated: "Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero."

[0996] Preserving and displaying generative stories

[0997] The generated story is stored in a database on a cloud server, and is then sent back to the user's device and displayed on the interface.

[0998] Sharing and purchasing within the virtual store

[0999] The stories created can be shared with other users in the virtual store, and can also be purchased by other users for a fee.

[1000] Voice Generation Options

[1001] The server also includes a speech generation service, and as a paid option, the device can send a speech generation request to the server. The server uses Text-to-Speech (TTS) technology to convert the story into speech and send it back to the device. The user can then play the generated speech on the interface and enjoy it.

[1002] Illustration and binding options

[1003] Image generation AI can also be used to generate illustrations that match the content of a story. In this case, the server receives a user request and generates illustrations that match the story, integrates them with the story, and sends them back to the device. Furthermore, if the user requests it, the server can also bind the generated story and illustrations and provide them as a physical book.

[1004] Specific examples

[1005] For example, if a user wanted to create a story about a winter night and snow spirits, the following prompts would be input into the generative AI model:

[1006] Character: Snow Spirit

[1007] Theme: Winter Night

[1008] Key Element: Magic Snowflake

[1009] Based on this, the server generates a story about a snow spirit who finds a magical snowflake on a winter's night and embarks on a heartwarming adventure.

[1010] This system allows users to quickly and effortlessly create original stories and picture books, and enjoy them in a variety of formats. The generated content can also be used commercially, meeting a variety of needs.

[1011] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1012] Step 1:

[1013] The user inputs characters, themes, key elements, etc. into the interface on their smartphone or PC. The interface provides input fields where the user can enter information such as "Taro," "adventure," and "magic sword." Once this data is entered, it is converted into a JSON-formatted object.

[1014] Step 2:

[1015] The device collects the input data, converts it to JSON format, and sends it to the cloud server using an HTTP POST request. The input data is sent to the server for further processing.

[1016] Step 3:

[1017] The server parses the received JSON-formatted input data. The parsed data is then input as a prompt to a generative AI (e.g., OpenAI's GPT-4). The generative AI then automatically generates a story based on the characters, theme, and key elements. For example, a sentence like "Taro obtains a magic sword and sets out on an adventure" is generated.

[1018] Step 4:

[1019] The server saves the generated stories in a database, where they can be accessed by users later. When saving, metadata such as the user ID and the story title are also saved.

[1020] Step 5:

[1021] The server sends the generated story back to the user's device. The HTTP response sends the generated story in JSON format to the device, where it is displayed on the user's interface.

[1022] Step 6:

[1023] Users can share stories created in the virtual store with other users. By sharing, other users can view or purchase the story. The data of purchased stories is managed on the server and kept as shared information between users.

[1024] Step 7:

[1025] As a paid option, users can also listen to the generated story. The device sends a request to generate the story to the server, which uses Text-to-Speech (TTS) technology to convert the story into audio. The generated audio file is sent back to the device and played on the interface.

[1026] Step 8:

[1027] Another paid option is to use image generation AI to generate illustrations. Upon receiving a user request, the server generates illustrations based on the content of the story. The generated illustrations are integrated into the story and sent back to the device. The user can then enjoy the story, including the generated illustrations, as an e-book.

[1028] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1029] The system of the present invention is designed to enable users to easily create original stories and picture books. Furthermore, it adds a function to recognize the user's emotions and adjust the content, tone, illustrations, and speech generation of the story based on those emotions. Specific embodiments are described below.

[1030] Providing a user interface

[1031] The user inputs the story setting through a terminal interface. The interface provides fields for inputting characters, themes, and key elements. For example,

[1032] Character: "Taro"

[1033] Theme: "Adventure"

[1034] Key Element: "Magic Sword"

[1035] Enter items such as:

[1036] Emotion recognition

[1037] The device is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice. The emotion engine recognizes the user's emotional state in real time through a camera and microphone. The recognized emotion data is quantified as, for example, "happiness," "sadness," or "excitement."

[1038] Sending data

[1039] The device collects input data and emotion data, converts it into an appropriate format such as JSON, and then sends it to the server.

[1040] Narrative Generation

[1041] The server uses a generation AI to generate a story based on the received data. The generation AI creates a natural-sounding story based on the input story setting data and emotion data. For example, if a user provides emotion data of "joy" with the theme of "adventure," the generated story will be written in a bright and cheerful tone.

[1042] Taro finds a magical sword and embarks on an exciting adventure, overcoming many amusing challenges and becoming a legendary hero all while having fun.

[1043] Preserving the Genesis Story

[1044] The stories that are created are stored in a database on the server. With the free plan, up to three stories can be saved, and saving new stories beyond that is restricted. With the paid plan, four or more stories can be saved.

[1045] Showing the story

[1046] The server sends the generated story back to the terminal, which displays the story to the user, who can then view the generated story on the interface.

[1047] Voice Generation Options

[1048] As a paid option, the device can use a voice generation service to deliver the generated story aloud. The tone and emotional expression of the voice are adjusted based on the user's emotional data. For example, a story generated with emotional data expressing joy will be read in a bright tone.

[1049] Illustration and binding options

[1050] Another paid option is for the device to provide a service that uses image generation AI to generate illustrations. The server receives a request from the device and uses the AI ​​to create corresponding illustrations based on the content of the story and the user's emotional data. For example, if a story contains a lot of emotional data about "surprise," an illustration that matches that emotion will be generated. Furthermore, if the user requests it, the service also provides a service to bind the story and illustrations and provide them as a physical book.

[1051] This system allows users to quickly and effortlessly create original stories and picture books based on pre-defined elements and their own emotions, and enjoy them in a variety of formats. The stories they create can be provided not only as text but also in audio and illustration formats, making it possible to meet the diverse needs of users. Adding emotion recognition functionality will provide a more personalized experience and increase user satisfaction.

[1052] The processing flow will be explained below.

[1053] Step 1:

[1054] The user inputs the setting of the story. Using the terminal interface, the user inputs the characters, theme, and key elements. For example, the user inputs the character "Taro," the theme "adventure," and the key element "magic sword."

[1055] Step 2:

[1056] The device collects input data. It converts the data entered by the user into a format such as JSON. Example: {"character": "Taro", "theme": "Adventure", "keyElement": "Magic Sword"}

[1057] Step 3:

[1058] The device recognizes the user's emotions. Using an emotion engine, it analyzes the user's facial expressions and tone of voice in real time via the camera and microphone to generate emotion data. Example: {"emotion": "joy"}

[1059] Step 4:

[1060] The device sends the collected input data and emotion data to the server using an HTTP POST request. Example: {"character": "Taro", "theme": "Adventure", "keyElement": "Magic Sword", "emotion": "Joy"}

[1061] Step 5:

[1062] The server receives the request. The request is received at the server endpoint and the input data and emotion data are parsed.

[1063] Step 6:

[1064] The server gives instructions to the AI ​​generator. Based on the parsed data, the AI ​​generator is instructed to generate a story. For example, the server provides the AI ​​with the following prompts: "Character: Taro," "Theme: Adventure," "Important Element: Magic Sword," and "Emotion: Joy."

[1065] Step 7:

[1066] The Generator AI generates a story. The Generator AI creates a story based on the provided prompts. Example: A generated story is "Taro obtains a magical sword and sets off on an exciting adventure. He overcomes many interesting challenges, has fun, and becomes a legendary hero."

[1067] Step 8:

[1068] The server stores the generated stories. The generated stories are stored in a database. The free plan allows you to store up to three stories, and any more stories will be limited to one.

[1069] Step 9:

[1070] The server sends the generated story back to the terminal. It sends back the generated story as an HTTP response. Example: {"story": "Taro gets his hands on a magical sword and sets off on an exciting adventure. He overcomes many interesting challenges, has fun, and becomes a legendary hero."}

[1071] Step 10:

[1072] The terminal displays the generated story. A user interface displays the generated story, allowing the user to view the story.

[1073] Step 11 (Optional):

[1074] The user selects the read aloud option: If the user wants to read aloud, the user selects this option in the interface.

[1075] Step 12 (Optional):

[1076] The device sends a request for speech generation to the server.,Based on the input data, the generated story, and,emotion data, a speech generation request is sent to the server.

[1077] Step 13 (Optional):

[1078] The server converts the story into audio using a TTS service. The generated story text is converted into audio and an audio file is generated. The tone and emotional expression of the audio are adjusted based on the emotional data.

[1079] Step 14 (Optional):

[1080] The server returns the audio file to the device. The generated audio file is sent to the device as an HTTP response.

[1081] Step 15 (Optional):

[1082] The device plays the audio and displays an audio play button in the user interface, allowing the user to play the audio of the story.

[1083] Step 16 (Optional):

[1084] The user selects the Generate Illustration option: If the user wants to generate illustrations, the user selects this option in the interface.

[1085] Step 17 (Optional):

[1086] The terminal sends a request to the server to generate an illustration. Based on the input data, the generated story, and the emotion data, the request to generate an illustration is sent to the server.

[1087] Step 18 (Optional):

[1088] The server uses image generation AI to generate illustrations. Based on the content of the story and emotional data, the AI ​​creates corresponding illustrations. For example, if a story contains a lot of emotional data for "surprise," an illustration that matches that emotion will be generated.

[1089] Step 19 (Optional):

[1090] The server returns the generated illustration to the terminal, which then sends the generated illustration to the terminal as an HTTP response.

[1091] Step 20 (Optional):

[1092] The terminal integrates the illustrations into the story and displays them. The user interface integrates the story and illustrations generated by the terminal so that the user can view them.

[1093] Step 21 (Optional):

[1094] The user selects the option to bind the story: If binding is desired, the user selects this option in the interface.

[1095] Step 22 (Optional):

[1096] The server arranges for a bookbinding service to bind the generated story and illustrations into a physical book and provide it to the user.

[1097] Example 2

[1098] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1099] Conventional story generation systems generate stories solely based on user input, making it difficult to provide a personalized experience. Furthermore, they are unable to adjust the content, tone, illustrations, and audio expression of the story according to the user's emotions and preferences. Therefore, a new system that takes the user's emotions into account during the story generation process is needed to improve user satisfaction.

[1100] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1101] In this invention, the server includes means for providing an interface for a user to input characters, themes, and important elements, means for collecting the input data, recognizing the user's emotional state using an emotion engine, and transmitting the input data to the server, means for generating a story based on the received input data and emotional data using a generative AI model, means for saving the generated story, and means for displaying the generated story to the user. This makes it possible to generate a personalized story according to the user's emotional state and provide content that is more satisfying.

[1102] An "interface" is the means by which a user inputs characters, themes, and key elements.

[1103] An "emotion engine" is a means having the function of recognizing the user's emotional state and collecting it as numerical data.

[1104] A "generative AI model" is a type of artificial intelligence used to generate stories based on input data and emotional data.

[1105] The "server" is a central computer system that receives input data and emotional data, and generates, stores, and displays stories.

[1106] A "voice generation service" is a means of providing a generated story in audio form and adjusting the tone and emotional expression of the audio based on the user's emotional data.

[1107] "Image generation AI" is a type of artificial intelligence used to generate illustrations that match a generated story.

[1108] "Storage means" refers to the methods and techniques used to store the generated stories in a database or storage device.

[1109] "Display means" refers to the methods and techniques for visually presenting the generated story on a user interface.

[1110] The present invention is a system that allows users to easily create original stories and picture books. This system has the function of recognizing the user's emotional state and adjusting the content, tone, illustrations, and audio expression of the story based on that emotion. An embodiment of the system is described in detail below.

[1111] The user inputs the story setting through a device interface. The interface provides fields for inputting characters, themes, and key elements. For example, the user can input "Taro," "adventure," and "magic sword." The interface is designed to be intuitive for the user, with text fields and drop-down menus.

[1112] The device is equipped with a camera and microphone, and the emotion engine uses these devices to analyze the user's facial expressions and tone of voice in real time. The emotion engine analyzes the collected data and quantifies emotions such as "happiness," "sadness," and "excitement." This emotional data is an important element required for story generation.

[1113] The device converts the story setting data and recognized emotion data entered by the user into JSON format and sends it to the server. The server analyzes the received data and generates a story using a generative AI model (e.g., OpenAI's GPT-3). The generative AI model creates a story with an appropriate tone and content based on the input elements and emotion data. For example, if the user provides emotion data for "adventure" and "joy," the generated story will have a bright and cheerful tone.

[1114] The stories that are created are stored in a database on the server. Users with the free plan can save up to three stories, but to save more, a paid plan is required. Saved stories can be reused at any time upon user request.

[1115] Once the story generation is complete, the server sends the generated story data back to the device. The device displays the received story on its user interface, allowing the user to check the content. In addition to viewing the generated story in text format, the user can also optionally use services that generate audio and illustrations.

[1116] When generating the voice, the device uses a voice generation service to provide the generated story aloud. The tone and emotional expression of the voice are adjusted based on the user's emotional data. For example, the story is read in a bright tone based on the emotional data of "joy."

[1117] Furthermore, when generating illustrations, the device uses image generation AI to generate them. The server uses a generative AI model to generate appropriate illustrations based on the story and emotion data. The generated illustrations are incorporated into the story and provided to the user. If the user wishes, a service is also available that binds the story and illustrations and provides them as a physical book.

[1118] Below are some example input prompts for the generative AI model:

[1119] Example prompt sentence:

[1120] Character: Taro

[1121] Theme: Adventure

[1122] Key Element: Magic Sword

[1123] Emotion data: Joy

[1124] This system allows users to easily create their own original stories and enjoy them in a variety of formats. By utilizing emotion data, it is possible to provide a more personalized experience and improve user satisfaction.

[1125] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1126] Step 1:

[1127] Users input story settings through the device interface, including characters, themes, and key elements, into text fields and drop-down menus.

[1128] Input: "Character: Taro" "Theme: Adventure" "Important element: Magic sword"

[1129] Output: Input data (Character: Taro, Theme: Adventure, Key element: Magic sword)

[1130] Step 2:

[1131] The device uses an emotion engine to recognize the user's emotional state in real time, analyzing the user's facial expressions and tone of voice using a camera and microphone, and converting emotional data such as "happiness," "sadness," and "excitement" into numerical values.

[1132] Input: User's facial expression images, voice audio data

[1133] Output: Emotion data (e.g., Happiness: 80%, Sadness: 10%, Excitement: 10%)

[1134] Step 3:

[1135] The device converts the story setting data and emotion data entered by the user into JSON format and sends it to the server, which converts the data into a specific format and sends an HTTP request.

[1136] Input: Story setting data (character: Taro, theme: adventure, key element: magic sword), emotion data (joy: 80%, sadness: 10%, excitement: 10%)

[1137] Output: JSON data (integration of story setting data and emotion data)

[1138] Step 4:

[1139] The server parses the received JSON data and generates a story using a generative AI model (e.g., OpenAI's GPT-3). It sends prompts to the generative AI model and generates a story based on the input elements and emotion data.

[1140] Input: JSON data (story setting data and emotion data)

[1141] Output: Generated story (e.g., "Taro finds a magical sword and embarks on an exciting adventure. He overcomes many interesting challenges, has fun, and becomes a legendary hero.")

[1142] Step 5:

[1143] The server saves the generated stories in a database. When saving, the user's plan information is taken into consideration, and free users can save up to three stories.

[1144] Input: Generated story, user information

[1145] Output: Narrative data stored in a database

[1146] Step 6:

[1147] The server sends the saved story data back to the device. The data transfer uses HTTP responses to send the data to the device.

[1148] Input: Narrative data in the database

[1149] Output: HTTP response with story data

[1150] Step 7:

[1151] The terminal displays the received story data on a user interface, allowing the user to view the generated story on the interface.

[1152] Input: Story data (HTTP response)

[1153] Output: A story displayed on a user interface

[1154] Step 8:

[1155] If the user selects the paid option, the device will use a voice generation service to provide the story audibly, with the tone and emotional expression of the voice adjusted based on the user's emotional data.

[1156] Input: Narrative data, emotion data

[1157] Output: Modified audio data

[1158] Step 9:

[1159] If the user requests further illustrations, the device sends a request to the image generation AI to generate illustrations that match the story. The generated illustrations are then incorporated into the story and displayed on the interface.

[1160] Input: Narrative data, emotion data

[1161] Output: Generated illustration data

[1162] (Application example 2)

[1163] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1164] Conventional story generation systems simply generate stories based on information entered by the user and are unable to reflect the user's emotions or experiences in the virtual environment. This has resulted in limited user experiences and makes it difficult to provide a personalized experience. The present invention aims to solve these problems and provide a more sophisticated, emotionally-sensitive personalized storybook within a virtual environment.

[1165] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the user's emotions and adjusting the tone and expression of the story based on the emotional data, means for displaying the story in a virtual environment, and means for providing the story generated by using a voice generation service by voice. This makes it possible to generate a personalized story tailored to the user's emotions and to experience the story in a immersive virtual environment.

[1166] A "user" is someone who interacts with the system to input story settings and enjoy a personalized experience.

[1167] "Characters" refers to people or characters that the user sets up in the story.

[1168] A "theme" is a central concept that a user specifies as the theme or scenario of a story.

[1169] "Important elements" refer to items and settings that play an important role in the progression of the story.

[1170] An "interface" is a screen or tool that allows a user to input characters, themes, and important elements.

[1171] "Means for collecting data" refers to the functionality for obtaining and storing information entered by the user into the interface.

[1172] "Server" refers to a remote computer system that stores and processes data.

[1173] "Generative AI" refers to artificial intelligence that automatically generates stories based on data entered by the user.

[1174] A "narrative" is a text-based story created by generative AI.

[1175] "Means for storage" refers to the function for storing the generated stories in a database or the like.

[1176] "Display means" is a function for displaying the generated story on the user's device.

[1177] "Means for analyzing emotions" refers to a function for acquiring emotional data by analyzing the user's facial expressions and tone of voice.

[1178] "Emotion data" is data that quantifies the analyzed emotional state of the user.

[1179] "Means for adjusting tone and expression" refers to a function for adjusting the atmosphere and content of a story's text based on emotional data.

[1180] "Virtual environment" refers to a system environment in which users can experience a story in virtual reality.

[1181] A "speech generation service" is a system that converts text into speech and allows the user to listen.

[1182] The present invention provides a system for generating a personalized story based on data input by a user and the user's emotions at the time, and displaying the story in a virtual environment. Specific embodiments are described below.

[1183] Providing a user interface

[1184] The user enters the setting of the story through a terminal interface. The interface provides fields for entering characters, themes, and key elements. For example,

[1185] Character: "Alice"

[1186] Theme: "Dream Adventure"

[1187] Key element: "Magic Flower"

[1188] Emotion recognition

[1189] The device is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice. The emotion engine recognizes the user's emotional state in real time through a camera and microphone. The recognized emotional data is quantified and expressed as excitement, joy, surprise, etc.

[1190] Sending data

[1191] The device collects story setting data and emotion data entered by the user, converts it into an appropriate format such as JSON, and then sends it to the server.

[1192] Narrative Generation

[1193] The server uses a generation AI to generate a story based on the received data. The generation AI creates a natural-sounding story based on the story setting data and emotion data entered by the user. For example, if a user provides emotion data for "joy" on the theme of "dream adventure," the generated story will be written in a bright and cheerful tone.

[1194] Display in a virtual environment

[1195] The generated story is sent back to the terminal from the server and displayed on the user's device. At this time, the user can experience the generated story in a highly immersive manner in a virtual space by using a smartphone or head-mounted display.

[1196] Voice generation

[1197] The server's voice generation service also provides the generated story in audio format. The story's voice is adjusted to match the user's emotional expression based on the user's emotional data. For example, a story generated with emotional data expressing joy will be read in a bright tone.

[1198] Illustration generation

[1199] Furthermore, image generation AI is used to generate illustrations that match the story. The style and content of the illustrations are also adjusted based on the user's emotional data. For example, if a story contains a lot of emotional data for "surprise," illustrations that match that emotion will be generated.

[1200] Hardware and software used

[1201] Emotion Engine: Camera and microphone for recognizing user emotions

[1202] Generative AI: An AI model for generating stories

[1203] Virtual environment display: head-mounted display and smartphone

[1204] Speech generation service: A system that provides generated stories in audio format.

[1205] Image generation AI: AI model for generating illustrations

[1206] Specific examples

[1207] Example prompt sentence:

[1208] Characters: Alice

[1209] Theme: Dream Adventure

[1210] Key Element: Magic Flower

[1211] Emotions: Joy and excitement

[1212] Generated story:

[1213] After finding the magic flower, Alice embarks on an exciting adventure, solving mysteries and having fun with new friends in the land of dreams.

[1214] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1215] Step 1:

[1216] The user inputs the characters, theme, and important elements through the terminal interface. The input information is collected as the character "Alice," the theme "Dream Adventure," and the important element "Magic Flower."

[1217] Step 2:

[1218] The device's built-in emotion engine recognizes the user's facial expressions and tone of voice in real time through a camera and microphone, and the recognized emotion data is converted into a numerical value such as "happiness" or "excitement."

[1219] Step 3:

[1220] Emotion data and story setting data entered by the user are converted into JSON format. For example, data is sent in the following format:

[1221] json

[1222] {

[1223] "characters": "Alice",

[1224] "theme": "Dream Adventure",

[1225] "elements": "Magic Flower",

[1226] "emotion_data": {

[1227] "joy": 0.8,

[1228] "excitement": 0.7

[1229] }

[1230] }

[1231] Step 4:

[1232] The device sends the collected data to the server, which receives it and checks the data for consistency and completeness.

[1233] Step 5:

[1234] The server runs the generative AI model based on the received data and generates a story. At this time, the generative AI model based on the prompt sentence operates as follows:

[1235] Prompt statement:

[1236] Characters: Alice

[1237] Theme: Dream Adventure

[1238] Key Element: Magic Flower

[1239] Emotions: Joy and excitement

[1240] Generated story:

[1241] After finding the magic flower, Alice embarks on an exciting adventure, solving mysteries and having fun with new friends in the land of dreams.

[1242] Step 6:

[1243] The server stores the generated story in a database, and the story is associated with the user's emotional data and input data.

[1244] Step 7:

[1245] The server sends the generated story back to the device, which then displays it to the user, who then experiences the story in a virtual environment using a smartphone or head-mounted display.

[1246] Step 8:

[1247] The server's voice generation service converts the generated story into audio, adjusting the tone and expression of the audio based on the user's emotional data.

[1248] Step 9:

[1249] Additionally, it uses image generation AI to generate illustrations that match the story, which are adjusted based on emotional data and provided to users along with the story.

[1250] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1251] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1252] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1253] [Fourth embodiment]

[1254] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1255] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1256] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1257] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1258] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1259] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1260] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1261] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1262] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1263] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1264] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1265] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1266] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1267] The system of the present invention is designed to enable users to easily create original stories and picture books. Specific embodiments will be described below.

[1268] Providing a user interface

[1269] The user inputs the story setting through a terminal interface. The interface provides fields for inputting characters, themes, key elements, etc. For example,

[1270] Character: "Taro"

[1271] Theme: "Adventure"

[1272] Key Element: "Magic Sword"

[1273] Enter items such as:

[1274] Sending data

[1275] The terminal collects the user's input data, converts it into an appropriate format such as JSON, and then sends it to the server.

[1276] Narrative Generation

[1277] The server uses a generation AI to generate a story based on the received data. The generation AI creates a story with natural sentences based on the input data. For example, if a user inputs the settings above, the generated story will look like this:

[1278] Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero.

[1279] Preserving the Genesis Story

[1280] The stories you create are stored in a database on the server. With the free plan, you can save up to three stories, after which you are restricted from saving new stories. With the paid plan, you can save four or more stories.

[1281] Showing the story

[1282] The server sends the generated story back to the terminal, which displays the story to the user, who can then view the generated story on the interface.

[1283] Voice Generation Options

[1284] As a paid option, the device can use a voice generation service to provide the generated story audibly. The device sends a voice generation request to the server, which uses TTS (Text to Speech) technology to convert the story into audio and send it back to the device. The user can then play the generated audio in the interface.

[1285] Illustration and binding options

[1286] Another paid option is for the device to provide a service that uses image generation AI to generate illustrations. The server receives a request from the device and generates illustrations that match the content of the story, integrates them with the story, and sends them back to the device. Furthermore, if the user requests it, the server can also bind the story and illustrations and provide them as a physical book.

[1287] This system allows users to quickly and effortlessly create original stories and picture books based on pre-defined elements, and enjoy them in a variety of formats. The stories created can be provided not only as text, but also in audio and illustration formats, making it possible to meet the diverse needs of users.

[1288] The processing flow will be explained below.

[1289] Step 1:

[1290] The user inputs the setting of the story. Using the terminal interface, the user inputs the characters, theme, and key elements. For example, the user inputs the character "Taro," the theme "adventure," and the key element "magic sword."

[1291] Step 2:

[1292] The device collects input data. It converts the data entered by the user into a format such as JSON. Example: {"character": "Taro", "theme": "Adventure", "keyElement": "Magic Sword"}

[1293] Step 3:

[1294] The device sends data to the server by using an HTTP POST request to send input data to a specified endpoint on the server.

[1295] Step 4:

[1296] The server receives the request. The request is received at the appropriate endpoint on the server and the input data is parsed.

[1297] Step 5:

[1298] The server gives instructions to the AI ​​generator. Based on the parsed data, it instructs the AI ​​generator to generate a story. For example, it provides the AI ​​generator with the following prompts: "Character: Taro," "Theme: Adventure," "Important Element: Magic Sword."

[1299] Step 6:

[1300] The generation AI generates a story. The generation AI creates a story based on the provided prompts. Example: The generated story is "Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero."

[1301] Step 7:

[1302] The server stores the generated stories. The generated stories are stored in a database. The free plan allows you to store up to three stories, and any more stories will be limited to one.

[1303] Step 8:

[1304] The server returns the generated story to the terminal as an HTTP response. Example: {"story": "Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero."}

[1305] Step 9:

[1306] The terminal displays the generated story. A user interface displays the generated story, allowing the user to view the story.

[1307] Step 10 (Optional):

[1308] The user selects the read aloud option: If the user wants to read aloud, the user selects this option in the interface.

[1309] Step 11 (Optional):

[1310] The device sends a request for speech generation to the server. Based on the input data and the generated story, a request for speech generation is sent to the server.

[1311] Step 12 (Optional):

[1312] The server converts the story into audio using a TTS service, converting the text of the generated story into audio and generating an audio file.

[1313] Step 13 (Optional):

[1314] The server returns the audio file to the device. The generated audio file is sent to the device as an HTTP response.

[1315] Step 14 (Optional):

[1316] The device plays the audio and displays an audio play button in the user interface, allowing the user to play the audio of the story.

[1317] Step 15 (Optional):

[1318] The user selects the Generate Illustration option: If the user wants to generate illustrations, the user selects this option in the interface.

[1319] Step 16 (Optional):

[1320] The terminal sends a request to the server to generate illustrations. Based on the input data and the generated story, a request to generate illustrations is sent to the server.

[1321] Step 17 (Optional):

[1322] The server generates illustrations using image generation AI, which creates corresponding illustrations based on the content of the story.

[1323] Step 18 (Optional):

[1324] The server returns the generated illustration to the terminal, which then sends the generated illustration to the terminal as an HTTP response.

[1325] Step 19 (Optional):

[1326] The terminal integrates the illustrations into the story and displays them. The user interface integrates the story and illustrations generated by the terminal so that the user can view them.

[1327] Step 20 (Optional):

[1328] The user selects the option to bind the story: If binding is desired, the user selects this option in the interface.

[1329] Step 21 (Optional):

[1330] The server arranges for a bookbinding service to bind the generated story and illustrations into a physical book and provide it to the user.

[1331] Example 1

[1332] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1333] Conventional story creation systems generate stories based on specific templates, making it difficult for users to easily create their own stories. Furthermore, few systems allow users to enjoy stories in a richer format by adding audio and illustrations, rather than just displaying the generated stories in text format. Therefore, there is a need for a system that allows users to effortlessly create, save, and view their own stories in a variety of formats.

[1334] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1335] In this invention, the server includes means for generating a story based on received input data using a generative AI model, means for saving the generated story in a database, and means for returning the generated story to a user terminal and displaying it to the user. This allows users to easily create original stories, add audio and illustrations to them in addition to text, and save and view them.

[1336] "Interface" refers to the screens and input fields that allow users to perform operations and input data into the system.

[1337] "Input Data" refers to information about characters, themes, key elements, etc. that a user inputs into the system through an interface.

[1338] "Server" refers to the main computer system that receives and processes data sent from terminals over a network.

[1339] A "generative AI model" refers to an artificial intelligence algorithm that generates natural-looking text or images based on the data it receives.

[1340] "Narrative" refers to a story created by a generative AI model based on user input data.

[1341] "Database" refers to a system for structuring and storing generated narratives and other related data.

[1342] "User terminal" refers to a device such as a computer or smartphone that a user uses to access and operate the system.

[1343] "Text-to-speech technology" refers to technology that converts text data into voice data.

[1344] "Image generation AI" refers to an artificial intelligence algorithm that generates images and illustrations based on the content of a story.

[1345] A "prompt sentence" refers to a sentence that provides a generative AI model with the input data it needs to generate a story.

[1346] The system of the present invention is designed to enable users to easily create original stories and picture books. Specific embodiments will be described below.

[1347] The system provides an interface for users to input characters, themes, and key elements. Users use a terminal to input information into this interface. For example, they might input the following information:

[1348] Character: "Taro"

[1349] Theme: "Adventure"

[1350] Key Element: "Magic Sword"

[1351] The terminal collects the data entered by the user, converts it into JSON format, and sends it to the server via an HTTP POST request. The server receives the JSON data and analyzes its contents.

[1352] Next, the server uses a generative AI model (e.g., OpenAI's GPT-4) to generate a story based on the input data. Specifically, it gives the generative AI model a prompt like this:

[1353] Characters: Taro, Theme: Adventure, Key Elements: Magic Sword

[1354] Based on this prompt, the generative AI model generates the following story:

[1355] Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero.

[1356] The generated stories are stored in a database by the server. The free plan allows you to store up to three stories, after which you are restricted from saving new stories. Paid plans allow you to store more stories.

[1357] The server then sends the generated story back to the user terminal, which displays the received story on its interface so that the user can view the story.

[1358] As a paid option, devices can use a speech generation service. The device sends a speech generation request to a server, which converts the story into audio using text-to-speech (TTS) technology. The generated audio data is sent back to the device, allowing the user to play the audio in the interface.

[1359] There is also an option to use image generation AI (e.g., DALL-E) to generate illustrations that match the story. The device sends a request for illustration generation to the server, and the server generates the illustrations using the image generation AI. These illustrations are integrated with the story and sent back to the device. In this way, the user can enjoy the story and illustrations together. Furthermore, if the user requests it, there is also a service that will bind the story and illustrations and provide them as a physical book.

[1360] As described above, this system allows users to quickly and effortlessly create original stories and picture books based on pre-set elements, and enjoy them in a variety of formats. The stories created can be provided not only as text, but also in audio and illustration formats, making it a system that can meet the diverse needs of users.

[1361] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1362] Step 1:

[1363] Providing a user interface

[1364] The user uses a terminal to access an interface to input the elements needed to create a story. The user inputs characters, themes, and key elements. For example, "Taro", "adventure", and "magic sword". The input data is saved in form fields. Input: Data entered by the user (characters, themes, key elements). Output: Input data saved in form fields.

[1365] Step 2:

[1366] Sending data

[1367] The device collects data entered by the user and converts it into JSON format. The JSON data is then sent to the server using an HTTP POST request. The data includes characters, themes, key elements, etc. Input: The input data stored in the form fields. Output: The JSON data sent to the server.

[1368] Step 3:

[1369] Narrative Generation

[1370] The server receives JSON data sent from the device. After receiving it, it analyzes the data and calls a generative AI model (e.g., OpenAI GPT-4) to generate a story. The generative AI model is given a prompt. For example, the following prompt is input to the generative AI model: "Character: Taro, Theme: Adventure, Key Element: Magic Sword." The generative AI model generates a story based on the prompt. Input: JSON data received by the server. Output: Story generated by the generative AI model.

[1371] Step 4:

[1372] Preserving the Genesis Story

[1373] The server saves the generated stories in a database. The database has tables for managing multiple stories, into which the text data of the stories is inserted. The free plan allows for the saving of up to three stories, after which new stories cannot be saved. Paid plans allow for the saving of more stories. Input: Text data of the generated stories. Output: Stories saved in the database.

[1374] Step 5:

[1375] Showing the story

[1376] The server returns the generated story to the terminal. The terminal receives the returned story and displays it on the user interface. The user can view the displayed story. Input: Text data of the story stored in the database. Output: The story displayed on the user interface.

[1377] Step 6:

[1378] Voice Generation Options

[1379] If the user selects the paid option, the device sends a voice generation request to the server. The server uses text-to-speech (TTS) technology to convert the story into voice data. The generated voice data is sent back to the device, and the user can play the audio on the interface. Input: Story text data, voice generation request. Output: Generated voice data.

[1380] Step 7:

[1381] Illustration and binding options

[1382] If the user selects the illustration generation option, the device sends an image generation request to the server. The server uses image generation AI (e.g., DALL-E) to generate illustrations that match the content of the story. The generated illustrations are integrated with the story and sent back to the device. Furthermore, if the user requests it, the server also provides a service to bind the story and illustrations and provide them as a physical book. Input: Story text data, image generation request. Output: Generated illustrations and integrated story data.

[1383] (Application example 1)

[1384] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1385] Conventional story generation systems are convenient in that they allow users to create their own stories and picture books, but they have limitations on how the stories can be utilized, making it difficult to share them with other users or use them commercially. Furthermore, elements such as converting the generated stories into audio or adding illustrations must be handled individually, and are not provided as a unified system. This often requires a lot of effort and time for users.

[1386] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1387] In this invention, the server includes means for providing an interface for users to input characters, themes, and important elements, means for collecting the input data and sending it to the server, means for generating a story based on the received input data using a generation AI, means for saving the generated story, means for displaying the generated story to users, and means for sharing and purchasing the generated story with other users in a virtual store. This allows users to quickly and easily create original stories and picture books and share them with other users or use them commercially.

[1388] "Users" are individuals or groups who use the system to create stories and picture books.

[1389] A "character" is a character who plays a role in the story and moves the story forward.

[1390] "Theme" refers to the overall theme or central idea of ​​a story.

[1391] "Important elements" refer to keywords, items, and settings that are essential to the story.

[1392] "Interface" refers to the screen and operating means that users use to input data into the system.

[1393] "Input Data" refers to information about characters, themes, key elements, etc., entered by a user through an interface.

[1394] A "server" is a computer system that processes and stores data on a network.

[1395] "Generative AI" is an artificial intelligence model that automatically generates natural language narratives based on user-entered data.

[1396] A "story" is a collection of texts that form a series of events or episodes based on characters, themes, and key elements.

[1397] "Storage" means keeping the generated story in a storage device such as a database.

[1398] A "virtual store" is a virtual space where users can use products and services and share content over the Internet.

[1399] "Sharing" means making the created story available for other users to see.

[1400] "Purchase" is the act of another user paying a fee to acquire the created story.

[1401] The present invention is a system that allows users to easily create original stories and picture books, and further allows them to share and commercially use them with other users in a virtual store.

[1402] Providing a user interface

[1403] Users input the story setting through the interface of a device such as a smartphone or PC. The interface provides fields for inputting characters, themes, important elements, etc. For example, it is designed so that users can input items such as "Taro," "adventure," and "magic sword."

[1404] Submitting data and generating stories

[1405] The device collects the user's input data, converts it into an appropriate format such as JSON, and then sends it to a cloud server. The server uses a generative AI (for example, OpenAI's GPT-4) to generate a story based on the received data. The generative AI generates natural-sounding sentences based on the input data. For example, if a user inputs the above settings, the following story will be generated: "Taro obtains a magical sword and sets out on an adventure. He overcomes many challenges and eventually becomes a legendary hero."

[1406] Preserving and displaying generative stories

[1407] The generated story is stored in a database on a cloud server, and is then sent back to the user's device and displayed on the interface.

[1408] Sharing and purchasing within the virtual store

[1409] The stories created can be shared with other users in the virtual store, and can also be purchased by other users for a fee.

[1410] Voice Generation Options

[1411] The server also includes a speech generation service, and as a paid option, the device can send a speech generation request to the server. The server uses Text-to-Speech (TTS) technology to convert the story into speech and send it back to the device. The user can then play the generated speech on the interface and enjoy it.

[1412] Illustration and binding options

[1413] Image generation AI can also be used to generate illustrations that match the content of a story. In this case, the server receives a user request and generates illustrations that match the story, integrates them with the story, and sends them back to the device. Furthermore, if the user requests it, the server can also bind the generated story and illustrations and provide them as a physical book.

[1414] Specific examples

[1415] For example, if a user wanted to create a story about a winter night and snow spirits, the following prompts would be input into the generative AI model:

[1416] Character: Snow Spirit

[1417] Theme: Winter Night

[1418] Key Element: Magic Snowflake

[1419] Based on this, the server generates a story about a snow spirit who finds a magical snowflake on a winter's night and embarks on a heartwarming adventure.

[1420] This system allows users to quickly and effortlessly create original stories and picture books, and enjoy them in a variety of formats. The generated content can also be used commercially, meeting a variety of needs.

[1421] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1422] Step 1:

[1423] The user inputs characters, themes, key elements, etc. into the interface on their smartphone or PC. The interface provides input fields where the user can enter information such as "Taro," "adventure," and "magic sword." Once this data is entered, it is converted into a JSON-formatted object.

[1424] Step 2:

[1425] The device collects the input data, converts it to JSON format, and sends it to the cloud server using an HTTP POST request. The input data is sent to the server for further processing.

[1426] Step 3:

[1427] The server parses the received JSON-formatted input data. The parsed data is then input as a prompt to a generative AI (e.g., OpenAI's GPT-4). The generative AI then automatically generates a story based on the characters, theme, and key elements. For example, a sentence like "Taro obtains a magic sword and sets out on an adventure" is generated.

[1428] Step 4:

[1429] The server saves the generated stories in a database, where they can be accessed by users later. When saving, metadata such as the user ID and the story title are also saved.

[1430] Step 5:

[1431] The server sends the generated story back to the user's device. The HTTP response sends the generated story in JSON format to the device, where it is displayed on the user's interface.

[1432] Step 6:

[1433] Users can share stories created in the virtual store with other users. By sharing, other users can view or purchase the story. The data of purchased stories is managed on the server and kept as shared information between users.

[1434] Step 7:

[1435] As a paid option, users can also listen to the generated story. The device sends a request to generate the story to the server, which uses Text-to-Speech (TTS) technology to convert the story into audio. The generated audio file is sent back to the device and played on the interface.

[1436] Step 8:

[1437] Another paid option is to use image generation AI to generate illustrations. Upon receiving a user request, the server generates illustrations based on the content of the story. The generated illustrations are integrated into the story and sent back to the device. The user can then enjoy the story, including the generated illustrations, as an e-book.

[1438] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1439] The system of the present invention is designed to enable users to easily create original stories and picture books. Furthermore, it adds a function to recognize the user's emotions and adjust the content, tone, illustrations, and speech generation of the story based on those emotions. Specific embodiments are described below.

[1440] Providing a user interface

[1441] The user inputs the story setting through a terminal interface. The interface provides fields for inputting characters, themes, and key elements. For example,

[1442] Character: "Taro"

[1443] Theme: "Adventure"

[1444] Key Element: "Magic Sword"

[1445] Enter items such as:

[1446] Emotion recognition

[1447] The device is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice. The emotion engine recognizes the user's emotional state in real time through a camera and microphone. The recognized emotion data is quantified as, for example, "happiness," "sadness," or "excitement."

[1448] Sending data

[1449] The device collects input data and emotion data, converts it into an appropriate format such as JSON, and then sends it to the server.

[1450] Narrative Generation

[1451] The server uses a generation AI to generate a story based on the received data. The generation AI creates a natural-sounding story based on the input story setting data and emotion data. For example, if a user provides emotion data of "joy" with the theme of "adventure," the generated story will be written in a bright and cheerful tone.

[1452] Taro finds a magical sword and embarks on an exciting adventure, overcoming many amusing challenges and becoming a legendary hero all while having fun.

[1453] Preserving the Genesis Story

[1454] The stories that are created are stored in a database on the server. With the free plan, up to three stories can be saved, and saving new stories beyond that is restricted. With the paid plan, four or more stories can be saved.

[1455] Showing the story

[1456] The server sends the generated story back to the terminal, which displays the story to the user, who can then view the generated story on the interface.

[1457] Voice Generation Options

[1458] As a paid option, the device can use a voice generation service to deliver the generated story aloud. The tone and emotional expression of the voice are adjusted based on the user's emotional data. For example, a story generated with emotional data expressing joy will be read in a bright tone.

[1459] Illustration and binding options

[1460] Another paid option is for the device to provide a service that uses image generation AI to generate illustrations. The server receives a request from the device and uses the AI ​​to create corresponding illustrations based on the content of the story and the user's emotional data. For example, if a story contains a lot of emotional data about "surprise," an illustration that matches that emotion will be generated. Furthermore, if the user requests it, the service also provides a service to bind the story and illustrations and provide them as a physical book.

[1461] This system allows users to quickly and effortlessly create original stories and picture books based on pre-defined elements and their own emotions, and enjoy them in a variety of formats. The stories they create can be provided not only as text but also in audio and illustration formats, making it possible to meet the diverse needs of users. Adding emotion recognition functionality will provide a more personalized experience and increase user satisfaction.

[1462] The processing flow will be explained below.

[1463] Step 1:

[1464] The user inputs the setting of the story. Using the terminal interface, the user inputs the characters, theme, and key elements. For example, the user inputs the character "Taro," the theme "adventure," and the key element "magic sword."

[1465] Step 2:

[1466] The device collects input data. It converts the data entered by the user into a format such as JSON. Example: {"character": "Taro", "theme": "Adventure", "keyElement": "Magic Sword"}

[1467] Step 3:

[1468] The device recognizes the user's emotions. Using an emotion engine, it analyzes the user's facial expressions and tone of voice in real time via the camera and microphone to generate emotion data. Example: {"emotion": "joy"}

[1469] Step 4:

[1470] The device sends the collected input data and emotion data to the server using an HTTP POST request. Example: {"character": "Taro", "theme": "Adventure", "keyElement": "Magic Sword", "emotion": "Joy"}

[1471] Step 5:

[1472] The server receives the request. The request is received at the server endpoint and the input data and emotion data are parsed.

[1473] Step 6:

[1474] The server gives instructions to the AI ​​generator. Based on the parsed data, the AI ​​generator is instructed to generate a story. For example, the server provides the AI ​​with the following prompts: "Character: Taro," "Theme: Adventure," "Important Element: Magic Sword," and "Emotion: Joy."

[1475] Step 7:

[1476] The Generator AI generates a story. The Generator AI creates a story based on the provided prompts. Example: A generated story is "Taro obtains a magical sword and sets off on an exciting adventure. He overcomes many interesting challenges, has fun, and becomes a legendary hero."

[1477] Step 8:

[1478] The server stores the generated stories. The generated stories are stored in a database. The free plan allows you to store up to three stories, and any more stories will be limited to one.

[1479] Step 9:

[1480] The server sends the generated story back to the terminal. It sends back the generated story as an HTTP response. Example: {"story": "Taro gets his hands on a magical sword and sets off on an exciting adventure. He overcomes many interesting challenges, has fun, and becomes a legendary hero."}

[1481] Step 10:

[1482] The terminal displays the generated story. A user interface displays the generated story, allowing the user to view the story.

[1483] Step 11 (Optional):

[1484] The user selects the read aloud option: If the user wants to read aloud, the user selects this option in the interface.

[1485] Step 12 (Optional):

[1486] The device sends a request for speech generation to the server.,Based on the input data, the generated story, and,emotion data, a speech generation request is sent to the server.

[1487] Step 13 (Optional):

[1488] The server converts the story into audio using a TTS service. The generated story text is converted into audio and an audio file is generated. The tone and emotional expression of the audio are adjusted based on the emotional data.

[1489] Step 14 (Optional):

[1490] The server returns the audio file to the device. The generated audio file is sent to the device as an HTTP response.

[1491] Step 15 (Optional):

[1492] The device plays the audio and displays an audio play button in the user interface, allowing the user to play the audio of the story.

[1493] Step 16 (Optional):

[1494] The user selects the Generate Illustration option: If the user wants to generate illustrations, the user selects this option in the interface.

[1495] Step 17 (Optional):

[1496] The terminal sends a request to the server to generate an illustration. Based on the input data, the generated story, and the emotion data, the request to generate an illustration is sent to the server.

[1497] Step 18 (Optional):

[1498] The server uses image generation AI to generate illustrations. Based on the content of the story and emotional data, the AI ​​creates corresponding illustrations. For example, if a story contains a lot of emotional data for "surprise," an illustration that matches that emotion will be generated.

[1499] Step 19 (Optional):

[1500] The server returns the generated illustration to the terminal, which then sends the generated illustration to the terminal as an HTTP response.

[1501] Step 20 (Optional):

[1502] The terminal integrates the illustrations into the story and displays them. The user interface integrates the story and illustrations generated by the terminal so that the user can view them.

[1503] Step 21 (Optional):

[1504] The user selects the option to bind the story: If binding is desired, the user selects this option in the interface.

[1505] Step 22 (Optional):

[1506] The server arranges for a bookbinding service to bind the generated story and illustrations into a physical book and provide it to the user.

[1507] Example 2

[1508] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1509] Conventional story generation systems generate stories solely based on user input, making it difficult to provide a personalized experience. Furthermore, they are unable to adjust the content, tone, illustrations, and audio expression of the story according to the user's emotions and preferences. Therefore, a new system that takes the user's emotions into account during the story generation process is needed to improve user satisfaction.

[1510] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1511] In this invention, the server includes means for providing an interface for a user to input characters, themes, and important elements, means for collecting the input data, recognizing the user's emotional state using an emotion engine, and transmitting the input data to the server, means for generating a story based on the received input data and emotional data using a generative AI model, means for saving the generated story, and means for displaying the generated story to the user. This makes it possible to generate a personalized story according to the user's emotional state and provide content that is more satisfying.

[1512] An "interface" is the means by which a user inputs characters, themes, and key elements.

[1513] An "emotion engine" is a means having the function of recognizing the user's emotional state and collecting it as numerical data.

[1514] A "generative AI model" is a type of artificial intelligence used to generate stories based on input data and emotional data.

[1515] The "server" is a central computer system that receives input data and emotional data, and generates, stores, and displays stories.

[1516] A "voice generation service" is a means of providing a generated story in audio form and adjusting the tone and emotional expression of the audio based on the user's emotional data.

[1517] "Image generation AI" is a type of artificial intelligence used to generate illustrations that match a generated story.

[1518] "Storage means" refers to the methods and techniques used to store the generated stories in a database or storage device.

[1519] "Display means" refers to the methods and techniques for visually presenting the generated story on a user interface.

[1520] The present invention is a system that allows users to easily create original stories and picture books. This system has the function of recognizing the user's emotional state and adjusting the content, tone, illustrations, and audio expression of the story based on that emotion. An embodiment of the system is described in detail below.

[1521] The user inputs the story setting through a device interface. The interface provides fields for inputting characters, themes, and key elements. For example, the user can input "Taro," "adventure," and "magic sword." The interface is designed to be intuitive for the user, with text fields and drop-down menus.

[1522] The device is equipped with a camera and microphone, and the emotion engine uses these devices to analyze the user's facial expressions and tone of voice in real time. The emotion engine analyzes the collected data and quantifies emotions such as "happiness," "sadness," and "excitement." This emotional data is an important element required for story generation.

[1523] The device converts the story setting data and recognized emotion data entered by the user into JSON format and sends it to the server. The server analyzes the received data and generates a story using a generative AI model (e.g., OpenAI's GPT-3). The generative AI model creates a story with an appropriate tone and content based on the input elements and emotion data. For example, if the user provides emotion data for "adventure" and "joy," the generated story will have a bright and cheerful tone.

[1524] The stories that are created are stored in a database on the server. Users with the free plan can save up to three stories, but to save more, a paid plan is required. Saved stories can be reused at any time upon user request.

[1525] Once the story generation is complete, the server sends the generated story data back to the device. The device displays the received story on its user interface, allowing the user to check the content. In addition to viewing the generated story in text format, the user can also optionally use services that generate audio and illustrations.

[1526] When generating the voice, the device uses a voice generation service to provide the generated story aloud. The tone and emotional expression of the voice are adjusted based on the user's emotional data. For example, the story is read in a bright tone based on the emotional data of "joy."

[1527] Furthermore, when generating illustrations, the device uses image generation AI to generate them. The server uses a generative AI model to generate appropriate illustrations based on the story and emotion data. The generated illustrations are incorporated into the story and provided to the user. If the user wishes, a service is also available that binds the story and illustrations and provides them as a physical book.

[1528] Below are some example input prompts for the generative AI model:

[1529] Example prompt sentence:

[1530] Character: Taro

[1531] Theme: Adventure

[1532] Key Element: Magic Sword

[1533] Emotion data: Joy

[1534] This system allows users to easily create their own original stories and enjoy them in a variety of formats. By utilizing emotion data, it is possible to provide a more personalized experience and improve user satisfaction.

[1535] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1536] Step 1:

[1537] Users input story settings through the device interface, including characters, themes, and key elements, into text fields and drop-down menus.

[1538] Input: "Character: Taro" "Theme: Adventure" "Important element: Magic sword"

[1539] Output: Input data (Character: Taro, Theme: Adventure, Key element: Magic sword)

[1540] Step 2:

[1541] The device uses an emotion engine to recognize the user's emotional state in real time, analyzing the user's facial expressions and tone of voice using a camera and microphone, and converting emotional data such as "happiness," "sadness," and "excitement" into numerical values.

[1542] Input: User's facial expression images, voice audio data

[1543] Output: Emotion data (e.g., Happiness: 80%, Sadness: 10%, Excitement: 10%)

[1544] Step 3:

[1545] The device converts the story setting data and emotion data entered by the user into JSON format and sends it to the server, which converts the data into a specific format and sends an HTTP request.

[1546] Input: Story setting data (character: Taro, theme: adventure, key element: magic sword), emotion data (joy: 80%, sadness: 10%, excitement: 10%)

[1547] Output: JSON data (integration of story setting data and emotion data)

[1548] Step 4:

[1549] The server parses the received JSON data and generates a story using a generative AI model (e.g., OpenAI's GPT-3). It sends prompts to the generative AI model and generates a story based on the input elements and emotion data.

[1550] Input: JSON data (story setting data and emotion data)

[1551] Output: Generated story (e.g., "Taro finds a magical sword and embarks on an exciting adventure. He overcomes many interesting challenges, has fun, and becomes a legendary hero.")

[1552] Step 5:

[1553] The server saves the generated stories in a database. When saving, the user's plan information is taken into consideration, and free users can save up to three stories.

[1554] Input: Generated story, user information

[1555] Output: Narrative data stored in a database

[1556] Step 6:

[1557] The server sends the saved story data back to the device. The data transfer uses HTTP responses to send the data to the device.

[1558] Input: Narrative data in the database

[1559] Output: HTTP response with story data

[1560] Step 7:

[1561] The terminal displays the received story data on a user interface, allowing the user to view the generated story on the interface.

[1562] Input: Story data (HTTP response)

[1563] Output: A story displayed on a user interface

[1564] Step 8:

[1565] If the user selects the paid option, the device will use a voice generation service to provide the story audibly, with the tone and emotional expression of the voice adjusted based on the user's emotional data.

[1566] Input: Narrative data, emotion data

[1567] Output: Modified audio data

[1568] Step 9:

[1569] If the user requests further illustrations, the device sends a request to the image generation AI to generate illustrations that match the story. The generated illustrations are then incorporated into the story and displayed on the interface.

[1570] Input: Narrative data, emotion data

[1571] Output: Generated illustration data

[1572] (Application example 2)

[1573] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1574] Conventional story generation systems simply generate stories based on information entered by the user and are unable to reflect the user's emotions or experiences in the virtual environment. This has resulted in limited user experiences and makes it difficult to provide a personalized experience. The present invention aims to solve these problems and provide a more sophisticated, emotionally-sensitive personalized storybook within a virtual environment.

[1575] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing the user's emotions and adjusting the tone and expression of the story based on the emotional data, means for displaying the story in a virtual environment, and means for providing the story generated by using a voice generation service by voice. This makes it possible to generate a personalized story tailored to the user's emotions and to experience the story in a immersive virtual environment.

[1576] A "user" is someone who interacts with the system to input story settings and enjoy a personalized experience.

[1577] "Characters" refers to people or characters that the user sets up in the story.

[1578] A "theme" is a central concept that a user specifies as the theme or scenario of a story.

[1579] "Important elements" refer to items and settings that play an important role in the progression of the story.

[1580] An "interface" is a screen or tool that allows a user to input characters, themes, and important elements.

[1581] "Means for collecting data" refers to the functionality for obtaining and storing information entered by the user into the interface.

[1582] "Server" refers to a remote computer system that stores and processes data.

[1583] "Generative AI" refers to artificial intelligence that automatically generates stories based on data entered by the user.

[1584] A "narrative" is a text-based story created by generative AI.

[1585] "Means for storage" refers to the function for storing the generated stories in a database or the like.

[1586] "Display means" is a function for displaying the generated story on the user's device.

[1587] "Means for analyzing emotions" refers to a function for acquiring emotional data by analyzing the user's facial expressions and tone of voice.

[1588] "Emotion data" is data that quantifies the analyzed emotional state of the user.

[1589] "Means for adjusting tone and expression" refers to a function for adjusting the atmosphere and content of a story's text based on emotional data.

[1590] "Virtual environment" refers to a system environment in which users can experience a story in virtual reality.

[1591] A "speech generation service" is a system that converts text into speech and allows the user to listen.

[1592] The present invention provides a system for generating a personalized story based on data input by a user and the user's emotions at the time, and displaying the story in a virtual environment. Specific embodiments are described below.

[1593] Providing a user interface

[1594] The user enters the setting of the story through a terminal interface. The interface provides fields for entering characters, themes, and key elements. For example,

[1595] Character: "Alice"

[1596] Theme: "Dream Adventure"

[1597] Key element: "Magic Flower"

[1598] Emotion recognition

[1599] The device is equipped with an emotion engine that analyzes the user's facial expressions and tone of voice. The emotion engine recognizes the user's emotional state in real time through a camera and microphone. The recognized emotional data is quantified and expressed as excitement, joy, surprise, etc.

[1600] Sending data

[1601] The device collects story setting data and emotion data entered by the user, converts it into an appropriate format such as JSON, and then sends it to the server.

[1602] Narrative Generation

[1603] The server uses a generation AI to generate a story based on the received data. The generation AI creates a natural-sounding story based on the story setting data and emotion data entered by the user. For example, if a user provides emotion data for "joy" on the theme of "dream adventure," the generated story will be written in a bright and cheerful tone.

[1604] Display in a virtual environment

[1605] The generated story is sent back to the terminal from the server and displayed on the user's device. At this time, the user can experience the generated story in a highly immersive manner in a virtual space by using a smartphone or head-mounted display.

[1606] Voice generation

[1607] The server's voice generation service also provides the generated story in audio format. The story's voice is adjusted to match the user's emotional expression based on the user's emotional data. For example, a story generated with emotional data expressing joy will be read in a bright tone.

[1608] Illustration generation

[1609] Furthermore, image generation AI is used to generate illustrations that match the story. The style and content of the illustrations are also adjusted based on the user's emotional data. For example, if a story contains a lot of emotional data for "surprise," illustrations that match that emotion will be generated.

[1610] Hardware and software used

[1611] Emotion Engine: Camera and microphone for recognizing user emotions

[1612] Generative AI: An AI model for generating stories

[1613] Virtual environment display: head-mounted display and smartphone

[1614] Speech generation service: A system that provides generated stories in audio format.

[1615] Image generation AI: AI model for generating illustrations

[1616] Specific examples

[1617] Example prompt sentence:

[1618] Characters: Alice

[1619] Theme: Dream Adventure

[1620] Key Element: Magic Flower

[1621] Emotions: Joy and excitement

[1622] Generated story:

[1623] After finding the magic flower, Alice embarks on an exciting adventure, solving mysteries and having fun with new friends in the land of dreams.

[1624] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1625] Step 1:

[1626] The user inputs the characters, theme, and important elements through the terminal interface. The input information is collected as the character "Alice," the theme "Dream Adventure," and the important element "Magic Flower."

[1627] Step 2:

[1628] The device's built-in emotion engine recognizes the user's facial expressions and tone of voice in real time via a camera and microphone, and the recognized emotion data is converted into a numerical value such as "happiness" or "excitement."

[1629] Step 3:

[1630] Emotion data and story setting data entered by the user are converted into JSON format. For example, data is sent in the following format:

[1631] json

[1632] {

[1633] "characters": "Alice",

[1634] "theme": "Dream Adventure",

[1635] "elements": "Magic Flower",

[1636] "emotion_data": {

[1637] "joy": 0.8,

[1638] "excitement": 0.7

[1639] }

[1640] }

[1641] Step 4:

[1642] The device sends the collected data to the server, which receives it and checks the data for consistency and completeness.

[1643] Step 5:

[1644] The server runs the generative AI model based on the received data and generates a story. At this time, the generative AI model based on the prompt sentence operates as follows:

[1645] Prompt statement:

[1646] Characters: Alice

[1647] Theme: Dream Adventure

[1648] Key Element: Magic Flower

[1649] Emotions: Joy and excitement

[1650] Generated story:

[1651] After finding the magic flower, Alice embarks on an exciting adventure, solving mysteries and having fun with new friends in the land of dreams.

[1652] Step 6:

[1653] The server stores the generated story in a database, and the story is associated with the user's emotional data and input data.

[1654] Step 7:

[1655] The server sends the generated story back to the device, which then displays it to the user, who then experiences the story in a virtual environment using a smartphone or head-mounted display.

[1656] Step 8:

[1657] The server's voice generation service converts the generated story into audio, adjusting the tone and expression of the audio based on the user's emotional data.

[1658] Step 9:

[1659] Additionally, it uses image generation AI to generate illustrations that match the story, which are adjusted based on emotional data and provided to users along with the story.

[1660] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1661] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1662] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1663] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1664] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1665] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1666] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1667] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1668] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1669] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1670] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1671] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1672] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1673] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1674] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1675] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1676] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1677] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1678] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1679] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1680] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1681] The following is further disclosed regarding the above embodiment.

[1682] (Claim 1)

[1683] means for providing an interface for a user to input characters, themes, and key elements;

[1684] means for collecting the input data and transmitting it to a server;

[1685] a means for utilizing generative AI to generate a narrative based on the received input data;

[1686] a means for storing the generated stories;

[1687] means for displaying the generated story to a user;

[1688] A system including:

[1689] (Claim 2)

[1690] 10. The system of claim 1, further comprising means for utilizing a voice generation service to provide the generated narrative audibly.

[1691] (Claim 3)

[1692] The system of claim 1 further comprising means for using an image generation AI to generate illustrations that match the generated story and combining the illustrations with the story to provide to the user.

[1693] "Example 1"

[1694] (Claim 1)

[1695] means for providing an interface for a user to input characters, themes, and key elements;

[1696] means for collecting the input data and transmitting it to a server;

[1697] a means for utilizing a generative AI model to generate a narrative based on the received input data;

[1698] a means for storing the generated stories in a database;

[1699] means for returning the generated story to the user terminal and displaying it to the user;

[1700] A system including:

[1701] (Claim 2)

[1702] 10. The system of claim 1, further comprising means for providing the generated narrative audibly using text-to-speech technology.

[1703] (Claim 3)

[1704] The system of claim 1 further comprising means for using an image generation AI to generate illustrations that match the generated story and combining the illustrations with the story to provide to the user.

[1705] "Application Example 1"

[1706] (Claim 1)

[1707] means for providing an interface for a user to input characters, themes, and key elements;

[1708] means for collecting the input data and transmitting it to a server;

[1709] a means for utilizing generative AI to generate a narrative based on the received input data;

[1710] a means for storing the generated stories;

[1711] means for displaying the generated story to a user;

[1712] means for sharing and purchasing the generated stories with other users in a virtual store;

[1713] A system including:

[1714] (Claim 2)

[1715] 10. The system of claim 1, further comprising means for utilizing a voice generation service to provide the generated narrative audibly.

[1716] (Claim 3)

[1717] The system of claim 1 further comprising means for using an image generation AI to generate illustrations that match the generated story and combining the illustrations with the story to provide to the user.

[1718] "Example 2: Combining Emotion Engines"

[1719] (Claim 1)

[1720] means for providing an interface for a user to input characters, themes, and key elements;

[1721] means for collecting the input data, recognizing the user's emotional state using an emotion engine, and transmitting the data to a server;

[1722] a means for utilizing a generative AI model to generate a narrative based on the received input data and emotion data;

[1723] a means for storing the generated stories;

[1724] means for displaying the generated story to a user;

[1725] A system including:

[1726] (Claim 2)

[1727] 10. The system of claim 1, further comprising means for utilizing a voice generation service to provide the generated narrative audibly, and for adjusting tone and emotional expression of the voice based on the user's emotional data.

[1728] (Claim 3)

[1729] The system of claim 1, further comprising means for utilizing image generation AI to generate illustrations that match the generated story, and providing the generated illustrations and story in combination to the user.

[1730] "Application example 2 when combining emotion engines"

[1731] (Claim 1)

[1732] means for providing an interface for a user to input characters, themes, and key elements;

[1733] means for collecting the input data and transmitting it to a server;

[1734] a means for utilizing generative AI to generate a narrative based on the received input data;

[1735] a means for storing the generated stories;

[1736] means for displaying the generated story to a user;

[1737] A means for analyzing a user's emotions and adjusting the tone and expression of the story based on the emotional data;

[1738] a means of displaying stories in a virtual environment;

[1739] A system including:

[1740] (Claim 2)

[1741] 10. The system of claim 1, further comprising means for utilizing a voice generation service to provide the generated story audibly.

[1742] (Claim 3)

[1743] The system of claim 1 further comprises means for using an image generation AI to generate illustrations that match the generated story, and combining the illustrations with the story to provide to the user. [Explanation of symbols]

[1744] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for providing an interface for a user to input characters, themes, and key elements; means for collecting the input data and transmitting it to a server; a means for utilizing generative AI to generate a narrative based on the received input data; a means for storing the generated stories; means for displaying the generated story to a user; A system including:

2. The system of claim 1 , further comprising means for utilizing a voice generation service to provide the generated story audibly.

3. The system of claim 1 further comprising means for using an image generation AI to generate illustrations that match the generated story, and combining the illustrations with the story to provide to the user.

Images